Hackers Are Stealing GenAI Credentials, So What Sensitive Company Data Are They Getting their Hands On?

On the Underground Hacker Markets, a criminal can purchase any number of illegal goods. They can get everything from online banking credentials, credit card numbers, guns, Fullz (full identity packets used for identity theft), drugs, PayPal credentials, passports, and now they can even purchase your Generative AI (GenAI) account credentials. As seen in (Figure 1), these include credentials for ChatGPT, Quillbot, Notion, Huggingface, Replit and the list goes on, as discovered by eSentire’s cybersecurity research team, the Threat Response Unit (TRU).

Security research organization, Sysdig, describes LLM Jacking– whereby threat actors take control of a large pool of LLMs. Sysdig demonstrated evidence of a campaign leveraging a reverse proxy – a useful component for covertly reselling LLM access. They lacked any evidence of how LLMs are used by cybercriminals in the wild.

In the present report, eSentire’s Threat Response Unit leverages insights from the cybercriminal underground to showcase how threat actors acquire, resell, and abuse access to LLMs.

GenAI Credentials on the Hacker Underground—Plenty to Go Around

It should come as no surprise that cybercriminals are selling GenAI credentials because threat actors always discover ways to monetize a piece of data. TRU found that threat actors are selling the usernames and passwords for approximately 400 individual GenAI accounts a day (Figure 1).

Cybercriminals are advertising the credentials on popular Russian Underground Markets, which specialize in everything from malware to infostealers to crypters. Many of the GenAI credentials are stolen from corporate end-users' computers when they get infected with an infostealer.

An infostealer is a piece of malware which retrieves everything entered by an end-user into their computer’s Internet Browser. This can include a user’s log-in credentials for their company’s IT network, their online bank account, their Amazon account, PayPal account, and healthcare provider portal.

If the end-user is a subscriber to a GenAI platform, then their credentials will be captured as well. The culmination of all the information retrieved, via an infostealer, is called a Stealer Log. Currently on the underground markets, it costs $10 for each stealer log.

Welcome to LLM Paradise—an Underground Market for Stolen GPT-4 and Claude API Keys

One of the underground services, which was selling stolen GenAI account credentials earlier this year, was called LLM Paradise. The threat actor running this market had a knack for marketing jargon, naming their store LLM Paradise and touting stolen GPT-4 and Claude API keys with ads reading: “The Only Place to get GPT-4 APIKEYS for unbeatable prices.”

The threat actor advertised GPT-4/Claude API keys starting at only $15 each, while typical prices for various OpenAI models run between $5 and $30 per million tokens utilized.

For whatever reason, it appears the proprietor of LLM Paradise couldn’t keep the doors open and the Underground Market was shuttered, although the threat actor even went as far as advertising the stolen GPT-4 API keys on TikTok, and the ad is still live (Figure 2).

The Untold Value of Access to Stolen GenAI Credentials and GenAI Data

eSentire discovered that cybercriminals are finding many ways to monetize stolen GenAI account credentials. Threat actors are using popular AI platforms to create convincing phishing campaigns, develop sophisticated malware, and produce chatbots for their underground forums.

Additionally, by accessing an organization’s corporate GenAI accounts, cybercriminals have the potential to get their hands on valuable and sensitive corporate data, such as customer information, financial data, valuable proprietary intellectual property and employee Personal Identifiable Information (PII).

Companies using GenAI platforms are not the only organizations at risk. Hackers are also attacking GenAI platform providers, which can pave the way for access to valuable data belonging to their corporate subscribers.

As of April 2024, there are three basic ways the Cybersecurity and Infrastructure Security Agency (CISA) defines GenAI threats:

• Attacks On GenAI Platforms
• Attacks Using AI
• Mishandling of AI

TRU conducted an integrated analysis of Open-Source Intelligence (OSINT) and cyber threat intelligence, which demonstrates the presence of all three GenAI threats in the past year. In addition, TRU researchers observed news cycles and cybercriminal discussions exploring these threats. The primary GenAI threats observed by TRU are:

• LLM Jacking – Threat actors are stealing and selling GenAI services to other threat actors on an ongoing basis.
• Abuse of stolen GenAI accounts for cybercrime – Threat actors are stealing or buying stolen GenAI credentials to create social engineering schemes, including convincing phishing campaigns; malware writing; and development of malicious chatbots. A chatbot is a virtual online assistant of which threat actors can ask questions. These can include queries about malware development, social engineering methods and operational security (Figures 3 and 4).
• Prompt Injection attacks to bypass guardrails – Prompt Injection is when an LLM user attempts to provoke the LLM to do something it’s not supposed to, such as leaking sensitive data or valuable intellectual property by using particular instructions or formatting. Guardrails are various techniques that LLM platform providers (like OpenAI) implement to enforce appropriate usage of their product (Figure 5).
• Cyberattacks launched against GenAI companies – Threat actors search for valuable data that GenAI companies could be storing. Once valuable or sensitive data is submitted to a cloud-based LLM platform it inherits the risks of those platforms, including traditional data exfiltration attacks, misconfiguration leaks, and credential theft.

One stark example of a GenAI company suffering a breach is OpenAI. In July 2024, the New York Times broke a story confirming that OpenAI, the developers of ChatGPT, did suffer a breach in 2023. The Times reported that the attacker “did not access the systems housing and building the AI but did steal discussions from an employee forum.”

Additionally, the Times revealed that OpenAI “did not publicly disclose the incident nor inform the FBI because, it claims, no information about customers nor partners was stolen, and the breach was not considered a threat to national security. The firm decided that the attack was down to a single person with no known association to any foreign government.” TRU has also observed threat actors discussing new research showing how one can gain access to different GenAI platforms (Figure 6).

TRU has also observed threat actors discussing new research showing how one can gain access to different GenAI platforms (Figure 6).

• Aggressive Data Collection – This is when GenAI companies ignore copyright and privacy ethics for better training of their model. This makes sensitive data available to any user of that platform and creates the potential for criminals and unsavory types to extract training data through crafted prompts.
• Supply Chain Risks – As GenAI becomes integrated with the data supply chain*. A data supply chain is where multiple companies may exchange, store or process data, providing opportunities to disrupt data supply chains, which are created by the rapid expansion of the GenAI attack surface.

*Traditionally, supply chain attacks often pertain to the software development layer – for example, poisoned updates as in the SolarWinds attack. Data supply chains consist of data going through several steps of storage, processing, analysis and dissemination – often this workload is distributed across several entities.

For example, healthcare documents are passed between several providers, government entities and access systems for the patients. If any point of this chain becomes compromised, the threat actors gain access to all data flowing through that point, giving them an opportunity to either exfiltrate – or poison – that data.

The most likely scenario is exfiltration of data for monetary incentives; poisoning is a more likely outcome of attackers motivated by political influence. One of the more well-known examples of a data supply chain attack is the MoveIT breach.

OpenAI—the Most Stolen GenAI Credentials

Of all the GenAI credentials being offered on the Underground, OpenAI usernames and passwords are the most prevalent. OpenAI has become one of the leading players in AI. ChatGPT has gained more than 100 million users since its public launch in November 2022.

TRU observes an average of over 200 OpenAI credentials posted for sale per day (Figure 7). These usernames and passwords are typically stolen from computer users’ internet browsers by cybercriminals using infostealers.

In the case of OpenAI, access to a subscriber’s account grants access to:

• Recent prompt history
• Any files uploaded to the OpenAI chat assistant endpoint
• The subscriber’s organization and account details

If a company is using GenAI solutions such as OpenAI in projects involving sensitive, valuable corporate data then the company should be implementing information security measures, starting with a general awareness of usage to help shape the development of policies around GenAI use. They must have a clear and thorough understanding of the potential risks and the protections implemented by the organization to defend against those risks.

The Future of the LLM Threat Landscape

The uncertain attack surface of LLMs lies mostly in the shadows of on-premises usage, and in cases where companies use LLMs as part of a data supply chain. As quickly as the corporate world has adopted GenAI, so have cybercriminals. Cybercriminals take interest in researching how to exploit artificial intelligence, and they share the articles on underground forums.

As LLMs become integrated with data supply chains, they create unexplored and fractal attack surfaces. And cybercriminals that have experience with hybrid attacks on LLMs will combine traditional attacks with prompt injection to explore these attack surfaces for exploitation.

Recommendations on How to Protect Against GenAI Attacks

• Monitor your employees’ usage of common cloud-based GenAI offerings (e.g. OpenAI, Claude, Perplexity) to eliminate blind spots. Make sure the GenAI usage monitoring extends beyond applications to include prompts used, files shared, trends, and more for comprehensive visibility.
• Urge your GenAI vendors to implement WebAuthn in their portals, allowing users to select FIDO2 security keys/passkeys as the sole multi-factor authentication (MFA) method. It is the best method for securing cloud accounts against Adversary-in-the-Middle (AitM) phishing attacks.
• If passkey security is not available from the GenAI platform provider, then follow password best practices. For example, never reuse passwords, and ensure that you are using uniquely generated passwords. Use a password keeper to avoid having to memorize simple passwords. Many password keepers come with a random password generator.
• Leverage dark web monitoring services to identify stolen credentials when they’re advertised on the underground hacker markets.

If you are not currently engaged with a Managed Detection and Response (MDR) provider, we highly recommend you partner with us for security services to disrupt threats before they impact your business. Want to learn more? Connect with an eSentire Security Specialist.