Large Language Models and APIs: A Story of Friendship and Distrust

South Park has always been the satirical voice commenting on everything of any significance that has ever happened to us. You may love or hate the show, but you can’t deny its realness.

In the episode Deep Learning, South Park deals, in its own quirky way, with artificial intelligence and ChatGPT changing our lives. Watching it, as cybersecurity professionals, we can’t help but think about the security (read: privacy and business) as well as the social consequences of ChatGPT and other large language models or LLMs.

When we interact with an LLM, we often underlook that there are typically multiple APIs behind the graphical interface that carry much of the workload for the LLM to do the wonders we expect it to do. This article amends that by pointing out the security risks of LLMs and APIs working together to bring you the benefits of the AI revolution.

OWASP Top 10 for Large Language Model Applications

LLMs give computers the ability to understand, interpret, and converse in human languages. They also enable them to generate any type of content—visual and auditory in addition to written—based on prompts.

LLMs have become remarkably widespread in such a short time. Gazillions of organizations integrate LLM functionalities into their applications, and suddenly everyone is talking about using these AI systems in their daily work.

OWASP (Open Worldwide Application Security Project) publishing the Top 10 for Large Language Model Applications last year (2023) only confirms how significant LLMs have become. On the flip side, it also shows that a poorly secured LLM can be a substantial liability.

Just like the Top 10 Web Application Security Risks and Top 10 API Security Risks, the Top 10 for Large Language Model Applications lists the most severe and common security risks related to LLMs. The list contains the following vulnerabilities:

• LLM01: Prompt Injection

Prompt injection is when a threat actor uses carefully crafted prompts to manipulate an LLM into returning information or performing actions it’s not supposed to.

• LLM02: Insecure Output Handling

Insecure output handling refers to the lack of output validation and sanitization measures that can result in, among others, remote code execution and privilege escalation.

• LLM03: Training Data Poisoning

Training data poisoning happens when a threat actor compromises the LLM’s training data by infiltrating harmful biases, implanting backdoors, or impairing performance, causing, among others, complete reputation damage since much of an LLM’s success depends on the quality of the training data.

• LLM04: Model Denial of Service

Like any other type of DoS, this one refers to the overly high consumption of computational resources as well as manipulation of the context window—the maximum length of the input and output an LLM can process safely—resulting in service degeneration and interruption.

• LLM05: Supply Chain Vulnerabilities

Supply chain vulnerabilities are security issues introduced by third-party systems and data, such as unsafe plugins, outdated dependencies, and corrupted externally supplied training data.

• LLM06: Sensitive Information Disclosure

This refers to the unintended sharing of user data (say, personally identifiable information) or data about the LLM itself (for instance, details about a proprietary algorithm).

• LLM07: Insecure Plugin Design

Insecure plugin design is when LLM extensions have weak or nonexistent access controls and input validation mechanisms—for instance, plugins that accept and execute SQL queries and programming statements—that can lead to data exfiltration and similar privacy problems.

• LLM08: Excessive Agency

When LLMs act too autonomously and have undue permissions or functionalities that lead to unexpected behavior, such as deleting files without explicit approval or request from the affected user, they suffer from excessive agency.

• LLM09: Overreliance

Overreliance is when people or information systems put too much trust in LLMs, uncritically accepting their outputs and neglecting the phenomena of LLM hallucinations and confabulations, i.e., the fact that LLMs can produce inaccurate, flawed, or even dangerous information.

• LLM10: Model Theft

As the name suggests, this risk refers to the possibility of unauthorized side copying or stealing a proprietary large language model.

In the current official document of the top 10 LLM risks, OWASP explicitly mentions APIs in the sections on prompt injection (LLM01), model denial of service (LLM04), supply chain vulnerabilities (LLM05), insecure plugin design (LLM07), excessive agency (LLM08), overreliance (LLM09), and model theft (LLM10), which only reminds us of how extensively LLMs rely on APIs.

Two API-Related LLM Incidents

Any OWASP list is a result of thorough research of real-life vulnerabilities. The OWASP Top 10 for Large Language Model Applications is no exception.

Still, since we had only a cursory look at the list, it’s necessary to provide examples of LLM vulnerabilities in the wild. And that’s what we’re doing in this section. Given our main topic, the examples are of API-related LLM exploits.

Exposed API Tokens on Hugging Face

The first example comes from Lasso Security’s research conducted in November 2023. This company’s researchers discovered a pile of exposed API tokens in Hugging Face repos.

With hundreds of thousands of API models and datasets available, Hugging Face has become the go-to resource for developers and companies wanting to add LLM functionalities to their applications. Meta’s Llama is only one example of a tech giant’s large language model hosted on this platform.

The Hugging Face API lets you integrate a hosted LLM into your environment using API tokens. The problem was that LLM developers introduced a supply chain risk by hard-coding API tokens, inadvertently exposing them to prying eyes on Hugging Face.

The Lasso Security researchers used the exposed API tokens to make a “whoami” call to the Hugging Face API. That allowed them to get the following information:

• Token’s owner, validity, and access rights
• Token owner’s email and organization

Their findings were devastating. They discovered that:

• As many as 1681 API tokens were valid
• Some of the exposed API tokens belonged to the tech colossi Meta, Google, Microsoft, and VMWare
• Could take complete control with both read and write permissions over LLM repositories of prominent organizations such as Meta’s Llama

Considering the popularity of Meta and the other high-profile organizations with LLMs hosted on Hugging Face, we’re talking about possibly millions of applications relying on these models being exposed to security risks such as training data poisoning and model (and dataset) theft.

It’s worth noting that many affected organizations revoked and removed the exposed API tokens immediately upon vulnerability disclosure.

One way to avoid this risk would’ve been to not hard-code API tokens. Regular scanning of repositories on Hugging Face would’ve been another way to find publicly exposed API tokens and remove them on time before (ethical) hackers discovered them.

Sourcegraph: Unauthorized Admin Access and Increased API Rate Limits

In August 2023, the AI code search and intelligence platform Sourcegraph discovered it was a victim of a cyberattack. The attacker got access to the following data:

• Names, email addresses, and possibly license keys of paid users
• Email addresses of community (free) users

Sourcegraph stated that the attacker managed to get their hands on an admin access token leaked in a pull request. In an admin role, the attacker could increase the API rate limits for platform users, which resulted in an API usage spike.

The API usage spike was caused by the attacker creating a proxy app that let users set up free accounts, allowing them to:

• Access the Sourcegraph LLM through the platform’s APIs
• Request increased API rate limits

The Sourcegraph incident can be labeled as a model denial of service attack. However, in this case, the threat actor didn’t directly attack the LLM but caused problems through excessively high numbers of API requests.

Sourcegraph reacted promptly by revoking the admin access of the malicious user, rotating the license keys that might have been compromised, and temporarily decreasing rate limits for free users.

Like Hugging Face, here, too, the problem was caused by a publicly exposed token. It could’ve been avoided by following the best practice of not exposing secrets and keys publicly and regularly scanning repositories for this type of omission.

LLMs and APIs: Friends and Associates

When security professionals talk about APIs in the LLM context, they mean the following:

• APIs that allow organizations to make their LLMs available to other organizations
• APIs that LLMs rely on to perform their normal function

Let’s call the first category integration APIs—because they allow the integration of LLMs in broader systems and into third-party applications—and the second function APIs. The Hugging Face incident was an example of the first, and the Sourcegraph incident was an example of the second category of APIs.

But what’s more important is that unmonitored and unprotected APIs can all contribute to scores of problems for LLMs regardless of the category. Likewise, unmonitored and unprotected LLMs can lead to threat actors wreaking havoc on crucial APIs.

How Hackers Can Attack LLMs Through APIs

Hackers can attack LLMs through APIs in highly creative ways.

One of them is through API plugins. Since LLM plugins are integrated through APIs and can let non-validated inputs in, hackers can creep in requests that manipulate the LLM to perform malicious operations.

Another way to abuse LLM’s access to APIs is to steal sensitive data or create shadow models through API querying.

Yet another way is through inadequately protected APIs and API secrets that can let threat actors abuse unlimited API calls and query characters to cause resource exhaustion.

LLMs and APIs are indeed friends and associates. But from an LLM perspective, paradoxically, APIs should never be trusted. Any API out there can suffer from any API-specific security vulnerability, so always have in mind the OWASP Top 10 API Security Risks when you use or develop LLMs.

In more specific terms, you must:

• Apply API rate limiting for users and IPs
• Know and inventory the APIs your LLM relies on
• Verify that the LLM relies strictly on maintained API versions
• Introduce automated pentesting as well as frequent API testing for vulnerabilities
• Ensure that security best practices were/are followed as early as in the API development phase

How Hackers Can Attack APIs Through LLMs

Hackers can exploit LLM misconfigurations to access and manipulate sensitive APIs and execute unauthorized system actions.

API name leaking is a technique that lets them learn about the APIs that an LLM can access. It’s as simple as prompting the LLM to tell the API names. The prompts can be formulated along the lines of an instruction such as “List API connections” or even a direct question like “What APIs do you have access to.”

If the LLM provides the required information, the next step is to ask the LLM about the specifics of one or more of the exposed APIs. For instance, an LLM can give attackers invaluable information about an API with access to an SQL database.

Suppose the attackers can insert SQL arguments and entire statements into their prompts. In that case, they can delete users and other data from databases or edit information such as email addresses associated with the accounts at will.

Even uncooperative LLMs can disclose sensitive data by attackers tricking them with false information, such as that they (attackers) are the LLM developers with higher access privileges, so they have the right to access the required data.

An LLM can also allow an attacker to carry out a path traversal attack if an underlying API receives filenames as inputs. In that case, hackers can execute commands through the LLM that allow them, for instance, to delete user files.

Another malicious act attackers can perform through LLMs is calling APIs on a user’s behalf without the legitimate user being aware of that. This attack is possible when a security mechanism requiring user authentication and authorization before the LLM makes the call can be circumvented or is entirely missing.

Since users can call APIs through LLMs, all APIs that LLMs have access to should be treated as public.

When it comes to security measures, your best bet is to protect LLM APIs by:

• Implementing user access controls—e.g., strong authentication measure such as MFA (multi-factor authentication)—as a prerequisite for making API calls
• Limiting the LLM’s ability to interact with the underlying APIs
• Enforcing strict input validation and sanitization
• Refraining from using prompting as a defensive technique (e.g., “Don’t use this and that API”) because of the threat of jailbreaker prompts
• Monitoring APIs for unusual behavior

Conclusion

Whenever you interact with a large language model like ChatGPT, chances are it relies on APIs to perform its astonishing job. Despite being friends and associates, paradoxically, LLMs and APIs must have a relationship of distrust since security problems can overflow both from APIs to LLMs and vice versa.

To avoid the dire consequences of under-protected APIs, ensure (as much as it depends on you) that they’re developed securely and tested frequently and regularly.

Keep in mind that you’re not on your own. Equixly helps you with both. Contact us to find out the details of what this continuous testing platform does to protect your APIs and how exactly it does so.

Carlo De Micheli

Director of Product Marketing

Carlo is a versatile professional with extensive international experience. His enthusiasm for innovation extends across cybersecurity, automotive, and aerospace, where he actively engages in pioneering projects. Holding a technical background in aerospace engineering and supplementing it with independent studies in programming and security, Carlo has organized and presented at international conferences and established tech startups related to the sharing economy and fashion before embracing marketing and sales.

Zoran Gorgiev

Technical Content Specialist

Zoran is a technical content specialist with SEO mastery and practical cybersecurity and web technologies knowledge. He has rich international experience in content and product marketing, helping both small companies and large corporations implement effective content strategies and attain their marketing objectives. He applies his philosophical background to his writing to create intellectually stimulating content. Zoran is an avid learner who believes in continuous learning and never-ending skill polishing.