What we feared has finally happened—cybercriminals have started using ChatGPT, and AI-driven crime could become the new trend

At the end of November 2022, OpenAI, an artificial intelligence research company, released its natural language generation model, ChatGPT, which gained over 1 million registered users in less than a week. The rapid breakout of ChatGPT sparked broad interest in AI and its potential applications.

However, ChatGPT also introduces new possibilities for modern cyber threats, as code generation can help less technically skilled attackers launch cyberattacks with little effort. This is even more concerning than five years ago when the industry worried about hackers using Machine Learning or Behavior Analysis to counter security products. Even Elon Musk commented: "ChatGPT is scary good. We are not far from dangerously strong AI."

Previously, some organizations researched implementing a complete infection flow through ChatGPT, from creating convincing spear-phishing emails to running a reverse shell that accepts English commands. The current question is whether this is just a hypothetical threat, or if attackers are already using OpenAI technology for malicious attacks.

An analysis of several major underground hacker communities shows that the first cases of cybercriminals using OpenAI to develop malicious tools have already emerged. As we suspected, some cases clearly demonstrate that many cybercriminals using OpenAI possess no development skills at all. Although the tools we are highlighting now are very basic, it is only a matter of time before more sophisticated cybercrime syndicates refine their methods of using AI-based tools to commit crimes.

▍Case 1: Creating Information Stealers

On December 29, 2022, a post titled "ChatGPT - Benefits of Malware" appeared on a popular underground hacker forum. The author of the post revealed that he was experimenting with ChatGPT to recreate malware strains and techniques described in security research publications and common malware write-ups. As an example, he shared the code of a Python-based stealer that searches for common file types, copies them to a random folder in the Temp directory, compresses them, and uploads them to a hardcoded FTP server.

Figure 1: A cybercriminal demonstrates how he used ChatGPT to create an information stealer

It turned out to be indeed a basic stealing tool that searches the entire system for 12 common file types (such as MS Office documents, PDFs, and JPGs). If any files of interest are found, the malware copies them to a temporary directory, compresses them, and sends them over the network. Notably, the attacker did not bother to encrypt or securely transmit the files, meaning they could potentially end up in the hands of third parties.

The second example created by this attacker using ChatGPT is a simple Java code snippet. It downloads PuTTY, a very common SSH and telnet client, and silently runs it on the system using PowerShell. Of course, this script can also be modified to download and run any program, including common malware families.

Figure 2: Demonstrating how he created a Java program to download PuTTY and run it using PowerShell

The poster had previously shared several scripts on the forum, such as post-exploitation automation and a C++ program attempting to phish for user credentials. In addition, he actively shared cracked versions of SpyNote, an Android RAT malware. Therefore, overall, this individual appears to be a tech-oriented cyber attacker whose post was aimed at showing less technically capable cybercriminals how to utilize ChatGPT for malicious attacks, providing them with usable, real-life examples.

▍Case 2: Facilitating ChatGPT Fraud Schemes

Another example of using ChatGPT for fraudulent activities was published in late 2022, demonstrating a different type of cybercrime activity. In this case, a cybercriminal showed how easy it is to use ChatGPT to create a Dark Web marketplace. The primary role of marketplaces in the underground illegal economy is to provide a platform for automated trading of illegal or stolen items (such as compromised accounts, payment cards, malware, or even drugs and ammunition), with all payments made in cryptocurrency. To illustrate how ChatGPT can be used for these purposes, the cybercriminal published a piece of code that uses a third-party API to fetch the latest cryptocurrency (Monero, Bitcoin, and Ethereum) prices as part of a Dark Web marketplace payment system.

Figure 3: Attacker uses ChatGPT to create a DarkWeb Market script

In early 2023, several cyber attackers initiated discussions on other underground forums focusing on how to use ChatGPT for fraud schemes. Most of these discussions centered around using another OpenAI technology (DALL-E 2) to generate random artwork and sell it online via legitimate platforms like Etsy. In another example, an attacker explained how to use ChatGPT to generate e-books or short chapters on specific topics and sell the content online.

Figure 4: Multiple tutorials in underground forums on how to use ChatGPT for fraudulent activities

In conclusion, it is still too early to determine whether ChatGPT features will become the preferred tools for cybercriminals. However, the cybercrime community has already shown great interest in it and is actively diving into the latest trend of generating malicious code. Security Labs will continue to track its progress in 2023. Once ChatGPT is utilized by cybercriminal syndicates for cyberattacks, we will promptly update our simulated attack library to help enterprises verify whether their defense systems can withstand such attacks and prepare in advance.

Finally, there is no better way to understand the abuse of ChatGPT than to ask ChatGPT itself, which provided a very interesting answer:

It is not uncommon for attackers to abuse AI and machine learning for malicious activities. One specific way they might abuse OpenAI is by using the company's language model technology to generate convincing phishing emails or social media posts. These can be used to trick people into clicking malicious links or revealing sensitive information.

Another way criminals might abuse OpenAI is by using the company's machine learning algorithms to create realistic synthetic media, such as audio or video, which can be used to spread disinformation or manipulate public opinion.

It is worth noting that OpenAI itself is not responsible for third-party abuse of its technology. The company has taken steps to prevent its technology from being used for malicious purposes, such as requiring users to agree to terms of service that prohibit the use of its technology for illegal or harmful activities.

Figure 5: ChatGPT's response on how attackers could abuse OpenAI