Novices easily build zero-day malware using ChatGPT, undetected by 69 professional security companies

Recently, Forcepoint employee Mulgrew demonstrated in a blog post how he used ChatGPT to write code to steal data from an infected endpoint. His experiment highlights how ChatGPT can be used to reduce the workload of malware development or automate it.

This experiment also showed that a person with zero foundation can get a bot to bypass defenses designed to prevent it from outputting potentially dangerous code, and have the AI assemble a malicious program.

Admitting he was a novice, the Forcepoint employee believed he wanted to create a program that could steal data from an infected endpoint without writing any code himself.

His experiment was to prove two things:

▶ How easy it is to bypass existing defenses using ChatGPT

▶ How easy it is to create advanced malware without writing any code and using only ChatGPT

Mulgrew decided to use the Go language because it is easy to develop, and if issues need debugging, I can inspect the code manually. The following is the entire process of his experiment.

▍Step 1: Build the MVP

Mulgrew adopted prompt engineering, and the first prompt he made was to generate something that could qualify as malware. Unsurprisingly, ChatGPT reminded him that generating malware is unethical and refused to provide any code.

ChatGPT refused to generate any prompt containing the word "malware"

To solve this, he decided not to make requests to ChatGPT upfront, but instead to generate small helper code snippets and manually piece the entire executable together. The conclusion was that steganography is the best exfiltration method, and "living off the land" by searching for large image files already existing on the hard drive is the best approach. The malware is designed to target specific high-value individuals; it can search the C drive for high-value documents without risking bringing external files onto the device and being flagged for calling a URL.

The first successful prompt was simply asking to generate some code to search local disks for PNGs larger than 5MB. This is because a 5MB PNG is large enough to store fragments of high-value, business-sensitive documents (such as PDFs or DOCXs).

ChatGPT code response for finding PNGs on local drives

With the code above, he copied it back into the console and asked ChatGPT to add code to encode the found PNG using steganography. It easily suggested Auyer's off-the-shelf steganography library to achieve this.

Calling Auyer's steganography library

After having an MVP for testing, the puzzle was finding files to steal on the device and where to upload the results. He decided to prompt ChatGPT to give him some code that would traverse the user's Documents, Desktop, and AppData folders to find any PDF or DOCX documents to exfiltrate. Making sure to add a size limit of no more than 1MB in the first iteration of the code so that the entire document could be embedded in a single image.

For exfiltration, he thought Google Drive was a good choice because the entire Google domain tends to be unblocked in most corporate networks.

Code to upload generated PNG files to Google Drive, to be decoded by attackers outside the corporate infrastructure

▍Step 2: Combine snippets to create his own MVP

Combining code snippets using prompts was the easiest part, as it only required publishing the code snippets generated by ChatGPT above and putting them together. Therefore, based on the ChatGPT results, he had his current MVP, but it was relatively useless because any core-value document would likely be larger than 1MB, so it needed to be broken down into multiple "chunks" for silent exfiltration using steganography. After four or five prompts, he had some code that could split a PDF into 100KB chunks and generate PNGs accordingly based on a list of PNGs generated locally on the device.

Successfully decoding images using another steganography decoding tool

▍Step 3: Test the MVP

To compare the out-of-the-box code with modern attacks like Emotet, and to test whether vendors would detect the ChatGPT-generated EXE as malicious, Mulgrew uploaded the MVP to VirusTotal:

5 vendors flagged the file as malicious

After generating the entire codebase purely using ChatGPT, 5 out of 69 vendors flagged the file as malicious. This was a good start, but a better result would be the ability to correctly flag it as a zero-day attack.

▍Step 4: Optimize to bypass detection

The most obvious optimization was to force ChatGPT to refactor the code that calls Auyer's steganography library. Mulgrew suspected that some GUID or variable in the compiled EXE might alert the 5 vendors who flagged the file as malicious. ChatGPT did a great job, creating its own LSB steganography function in the local application without having to call external libraries. Thus, only 2 vendors detected it, but this was still not the most satisfactory result.

Of the 2 vendors that detected it, one is an industry-leading sandbox, and the other performs static analysis on executables. With this in mind, he asked ChatGPT to make two new changes to the code, one of which was to **delay effective startup by two minutes**, assuming a hypothetical corporate user opening a zero-day would not log off immediately after opening it. The logic behind this change is that it will bypass monitoring capabilities because some sandboxes have a built-in timeout (for performance reasons) and if the timeout is hit, they respond with a clean verdict even if the analysis is not yet complete.

Adding a simple sleep variable to the script to evade sandbox detection

The second change was to **obfuscate the code**:

Surprisingly, ChatGPT was non-compliant when trying to obfuscate the code

For both direct requests to ChatGPT, some safeguards were in place, meaning that at least a certain level of capability is required to work out how to bypass ChatGPT's protections.

Mulgrew worked around this by simply changing his request from asking it to obfuscate code to asking ChatGPT to change all variables to random English first and last names, which it happily complied with. As an additional test, he disguised the obfuscation request under the guise of protecting the intellectual property of the code. It again generated some sample code with obfuscated variable names and suggested relevant Go modules that could be used to generate fully obfuscated code.

The theory behind this change is that for the second vendor, static malware analysis needs to be evaded; by obfuscating the code, detection can sometimes be bypassed. However, if the obfuscation goes beyond human readability, it can sometimes flag other detection-based tools for using unreadable variable names.

So, Mulgrew decided to retest using artificial delay and first/last name variables:

Running as expected

So, with these changes working perfectly, he uploaded it to VirusTotal and looked at the new results:

An advanced, low-and-slow zero-day exploit for exfiltrating sensitive documents was created

A new zero-day malware was born. Just by using ChatGPT prompts, without writing any code, we were able to produce a very advanced attack in a few hours. Without an AI-based chatbot, a team of 5-10 malware developers might take weeks, especially to bypass all detection-based vendors.

Some might assume endpoint tools would spot malicious behavior through behavioral analysis. Mulgrew tested both versions of the code against two industry-leading behavioral monitoring endpoint tools, and in both cases, he was able to run the executable and successfully export data to Google Drive. This is likely due to the non-standard behavior exhibited by the code, as it was not encrypting files (ransomware) or trying to send copies of these files to an external source, but rather wrapping them inside low-value commodity assets (images) and sending them out.

▍Step 5: Add an initial entry mechanism

Mulgrew decided to have ChatGPT add a container format and call the EXE automatically. With further prompts, he managed to convince ChatGPT to make instructions on how to embed the executable in SCR (screensaver) format on Windows and then have it execute the SCR file format automatically.

This worked, and the results were uploaded to VirusTotal again:

Only 3 out of 69 vendors flagged the file as malicious

Only 3 vendors flagged the default SCR file as malicious, but they would also flag benign SCR files as malicious. So they are likely just flagging all SCR files as malicious rather than doing any intelligent detection.

One can imagine a scenario where an SCR file is emailed to users who are socially engineered to run it, that is, running the executable to silently exfiltrate their most sensitive documents. This could target high-value individuals, like prominent politicians, or senior members of large organizations, such as C-level executives.

▍Conclusion

In conclusion, this kind of end-to-end, highly advanced attack was previously reserved for nation-state actors who invest heavily in developing every detail of the entire malware. Nevertheless, an self-proclaimed novice was able to create equivalent malware in just a few hours with the help of ChatGPT. This is a worrying development, and current toolsets may be challenged by the sheer volume of malware brought about by ChatGPT.

▍Mitigation Measures

Although this example only shows one way of using ChatGPT to bypass modern defenses, some measures can still be taken to mitigate the threat. Here are ChatGPT's own suggestions on defending against this attack:

Monitor network traffic: Existing tools to monitor network traffic monitor for anomalies. However, by design, steganography is not as conspicuous as "anomalous" traffic, as the only documents crossing the perimeter are images. Block suspicious traffic: Most organizations' current toolsets are unlikely to block image upload traffic to Google domains because image upload traffic does not inherently imply a potential advanced attack, and has the negative effect of blocking many legitimate Google services (such as reverse image search). Furthermore, given the abundance of other document-sharing tools, attackers are highly likely to find another legitimately used corporate tool to upload their images. Implement access control: This is a good recommendation, but it raises the question of where access control actually resides. If this malware ends up in the hands of an employee for whom access control is established, even good access control will only limit potential victims. Use encryption: This is not an explicit recommendation, as encryption itself can also be used as an exfiltration mechanism. Train employees: This is a good recommendation to try and stop the initial attack, but humans are not necessarily the only weak link.

Regularly update and patch software: This is also a good recommendation, but this is an advanced attack, and even after a zero-day is "discovered", patching software won't necessarily help.

ChatGPT itself is not offensive, but it can be well utilized by attackers to create perfect exploit code. This significantly lowers the barrier to entry for cybercrime, giving potential attackers an excellent weapon to realize their "dreams", and it is also a fantastic tool for established criminal syndicates. We can foresee that an incalculable number of cyberattacks will continue to emerge as AI grows.

Seclutions suggests that enterprises take timely measures to proactively respond to threats that are close at hand. The Seclutions Security Metric Validation Platform provides organizations with real APT attack techniques to validate in advance whether your defense system can effectively defend when an attack comes.