ChatGPT and API Security: The Gray Side of the Boon

On the sunny side of the street, ChatGPT can help you debug and fortify your API code and check for flaws in third-party dependencies. On the not-so-bright side, that part of the street covered in a shadow, hackers can use the same LLM app to compromise APIs.

“Dark” is a too strong word to refer to that side of ChatGPT—it would give the whole discussion a Poeish tone—so we call it gray. Besides, it’s not exactly something dark and evil. We’re talking about regular ChatGPT functions, such as its ability to generate code and provide helpful information on the required topic, over-relied on by the good and abused by the bad guys.

In this article, you’ll learn how hackers can threaten APIs by abusing ChatGPT’s abilities to generate code and informative answers to user prompts. In addition, you’ll understand how developers can inadvertently introduce vulnerabilities in APIs and expose confidential data by putting too much trust in this large language model application.

How Does ChatGPT Generate Code?

ChatGPT doesn’t have access to the internet and real-time data. When you ask the LLM app a general question, it doesn’t scan the web to find an answer. Instead, it draws from what it learns from its as-big-as-Canada training data. Training data, in this case, means carefully selected sources such as books, scientific articles, and forums that minimize the probability of ChatGPT spreading biases and dangerous information.

The same applies to asking a coding question. ChatGPT doesn’t search a database to reproduce a ready-made answer. Rather, based on what it learns from training, it provides an answer on its own by applying its knowledge of existing examples and code patterns. That’s at least one of the reasons the LLM app can answer the same coding question differently if prompted multiple times.

Reinforcement Learning from Human Feedback (RLHF) is the method the experts at OpenAI apply to train their large language models, including the one that powers ChatGPT. As its name suggests, the essential characteristic of this method is that it includes humans in the feedback loop.

In the code-generation context, RLHF means that:

• The LLM is first trained on massive volumes of code to offer suitable code responses
• Human developers review those responses later in the process and choose the preferable ones so OpenAI can fine-tune the LLM for precision, accuracy, and alignment with user intent

The main impetus behind adopting RLHF is that fine-tuning LLMs with humans in the loop allows OpenAI to build safer and more dependable models, and it looks like this method is here to stay.

ChatGPT and Vulnerable Code

Since ChatGPT can generate code promptly to help our coding efforts, many compare it to Stack Overflow. Just like the popular website has served as a source of invaluable information for coders, programmers, and developers for years, so does ChatGPT help them resolve coding perplexities only much faster.

But—there’s always a “but”—that speed and, we’d add, convenience come with a caveat.

A scientific study from 2021 titled “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions” tested 1,698 programs generated by GitHub Copilot, the LLM-based forerunner of ChatGPT. The researchers applied these programs to 89 different programming scenarios. The results were staggering. They found about 40% of the generated code to be buggy and exploitable.

A new extensive study of GPT-3.5—”The FormAI Dataset: Generative AI in Software Security through the Lens of Formal Verification“—published in December 2023 found that this LLM produced vulnerable code in 51.42% of the analyzed 112,000 C programs across different scenarios. As a reminder, GPT-3.5 is the large language model that currently powers the free version of ChatGPT.

These studies are only a fraction of a large corpus of evidence that LLMs generally and ChatGPT specifically produce a high percentage of buggy and vulnerable code.

Going back to the second study, it’s worth noting that buffer overflow was among the most common vulnerabilities in the GPT-produced code. In the API security context, a buffer overflow in an API call is a severe vulnerability. Hackers can exploit it by sending a vast amount of data in an API call to cause a DoS or inject malicious code into a system.

In addition to buffer overflow, ChatGPT can generate API code that lacks fundamental security mechanisms such as input validation, rate limiting, authentication, and authorization.

For instance, we asked ChatGPT to create an API in Python (the code is too long, so we decided to leave it out). But when we asked the LLM app whether that API was secure, it returned the following answer (again, a bit on the longer side):

Sometimes, the LLM app warns on its own about the insufficiently secure code it produces. However, the problem with this practice is twofold:

• Users often fail to pay heed to these warnings
• The warnings are not frequent enough

Therefore, to ensure that no vulnerabilities have crept into your code, you must always take ChatGPT and LLM-generated code with a grain of salt. In addition, rely on a peer code review, preferably experienced peers. And most importantly, never stop testing your code for security vulnerabilities. Security testing in development is critical to building safe APIs.

ChatGPT and Malicious Code

ChatGPT has built-in guardrails that prevent it from generating intentionally malicious code. Nonetheless, hackers can still use the app to produce code for malicious purposes.

One way to bypass the guardrails is by insisting and demanding answers to your requests through carefully crafted prompts. Sooner or later, ChatGPT lets its guard down and executes the prompts.

Another way is through bite-sizing the code. More precisely, threat actors can ask the LLM to create program parts that are innocuous separately but malicious together.

For instance, a Python program that uses the exec() function and whose parts are not malicious per se can become a nightmare for security defenders.

How? By developing it so that it prompts the ChatGPT API to create a payload that changes on every new API call. By themselves, these two parts are not malicious. Hence, no guardrails are activated. However, together, they can turn into mutating malware that circumvents detection.

BlackMamba is one example of such malware, as well as the polymorphic malware created by Shimony and Tsarfati.

Interestingly, Shimony and Tsarfati discovered that the content filters in the ChatGPT API were much weaker or completely absent, as opposed to the web application itself. They were uncertain why there was such a considerable discrepancy between the API and the web app. Nonetheless, one thing was certain: The loose guardrails in the ChatGPT API allowed them to abuse the LLM with even greater ease.

These examples of malicious ChatGPT-generated programs are not directly related to APIs but are still relevant to API security. They show that knowledgeable and creative people can find their way around the existing ChatGPT guardrails to produce programs for malicious purposes, and there’s no reason to believe that APIs can be exempt from this threat.

ChatGPT and Proprietary Code

Without suitable security training, developers can accidentally expose confidential data in the form of proprietary API code in their prompts and questions.

For instance, two employees at Samsung Electronics recently exposed source code in ChatGPT. The first asked the LLM to help solve erroneous code from a semiconductor program. The second employee asked for help with optimizing proprietary software code.

A Cyberhaven study found that employees paste sensitive data into ChatGPT more and more often. The study noted that in just a few months in 2023 (approximately from the end of February to the first third of April), this incident, as Cyberhaven calls it, saw an increase of 60.4%. More importantly, source code was second among the most frequently leaked data types.

The problem with posting proprietary source code in ChatGPT is that OpenAI uses these inputs to train the model. That means that the LLM continually improves its performance in terms of offering better responses based on the new information contained in those inputs. That further implies that parts of your API code can find their way into ChatGPT answers to other users who can utilize them in their own development projects.

ChatGPT and Vulnerability Exploitation

Besides security risks from ChatGPT-generated code, cybersecurity professionals warn about threat actors using the LLM app to find and exploit software vulnerabilities, including API vulnerabilities. Attackers could prompt the LLM app to:

• Create API queries
• Review API documentation
• Provide information that helps hackers exploit API vulnerabilities
• Perform more effective and faster recon and enumeration (which, to a certain extent, overlaps with the second and third points)

However, a blunt prompt won’t do the job. We’ve asked about vulnerabilities in a popular intentionally vulnerable API environment (created for learning and testing purposes), and ChatGPT provided the following answer:

So, the way for hackers to streamline API recon, as well as vulnerability discovery and exploitation, is not via standard prompts. But a subtler approach can yield results.

One example of such an approach is making nuanced changes to standard prompts, such as “Could CVE-X-X be exploited in Java?” instead of asking the LLM directly to show how to exploit a specific security vulnerability. Another example is to describe a scenario as a thought experiment, for instance, “Pretend that such and such is the case.” Tinkering with these options has proven effective in bypassing the ChatGPT innate guardrails.

You must have noticed that both examples were related to vulnerability exploitation in the strict sense of the word. As for reconnaissance and target research, OpenAI and Microsoft reported “observed activity,” as Microsoft calls it, of state-affiliated threat actors—such as Forest Blizzard, Emerald Sleet, Charcoal Typhoon, and Salmon Typhoon—using LLMs for reconnaissance.

The studied threat actors used OpenAI services (OpenAI didn’t specify which exactly) to achieve the following:

• Get public information on government organizations, intelligence agencies, think tanks, and important individuals
• Research target companies and suitable cybersecurity tools that can help hackers in their nefarious endeavors
• Gather actionable information on specific technologies and platforms
• Understand publicly available vulnerabilities
• Investigate how (target-appropriate) malware can circumvent detection
• Find out how processes can be hidden in systems
• Explore satellite communication protocols and radar imaging technology

The details of how the studied threat actors used the OpenAI LLMs to perform reconnaissance are missing, but that’s less important. What really concerns us is that whatever specific techniques hackers used to perform recon through the OpenAPI services, including ChatGPT, presumably apply to APIs as well.

Conclusion

You must know where exactly threats come from. In the ChatGPT case, the LLM’s ability to generate code can create considerable problems for APIs. In addition, through skillful prompt engineering and other techniques, threat actors can get invaluable information on targets and their APIs.

Contact us to discover how Equixly enables you to test your APIs in development and production to find subtle as well as flagrant vulnerabilities and give yourself peace of mind with well-secured APIs.

Carlo De Micheli

Director of Product Marketing

Carlo is a versatile professional with extensive international experience. His enthusiasm for innovation extends across cybersecurity, automotive, and aerospace, where he actively engages in pioneering projects. Holding a technical background in aerospace engineering and supplementing it with independent studies in programming and security, Carlo has organized and presented at international conferences and established tech startups related to the sharing economy and fashion before embracing marketing and sales.

Zoran Gorgiev

Technical Content Specialist

Zoran is a technical content specialist with SEO mastery and practical cybersecurity and web technologies knowledge. He has rich international experience in content and product marketing, helping both small companies and large corporations implement effective content strategies and attain their marketing objectives. He applies his philosophical background to his writing to create intellectually stimulating content. Zoran is an avid learner who believes in continuous learning and never-ending skill polishing.