I’ve long been amazed at the hyperparasitism of the hacker exploit ecosystem, where hackers penetrate systems not to steal credit card numbers, but just to steal the resources to run bot farms. And now hackers are stealing cloud resources to run AI sexbots.
Organizations that get relieved of credentials to their cloud environments can quickly find themselves part of a disturbing new trend: Cybercriminals using stolen cloud credentials to operate and resell sexualized AI-powered chat services. Researchers say these illicit chat bots, which use custom jailbreaks to bypass content filtering, often veer into darker role-playing scenarios, including child sexual exploitation and rape.
Researchers at security firm Permiso Security say attacks against generative artificial intelligence (AI) infrastructure like Bedrock from Amazon Web Services (AWS) have increased markedly over the last six months, particularly when someone in the organization accidentally exposes their cloud credentials or key online, such as in a code repository like GitHub.
You might wonder how a company could be so stupid as to include their cloud access credentials in their GitHub repository. Having worked for a long time in the cloud/SaaS space, I can tell you: It’s easier than you think. Developers probably included it so they could do rapid testing during the development cycle (and nobody wants to go through the pain of setting up another cryptokey through two-factor authentication every damn time they want to run a test), then overlooked changing it when they rolled to production. It’s the sort of thing dev should be looking for, but there are a lot of ways (personnel change, version rollback, etc.) something like that can slip through.
Investigating the abuse of AWS accounts for several organizations, Permiso found attackers had seized on stolen AWS credentials to interact with the large language models (LLMs) available on Bedrock. But they also soon discovered none of these AWS users had enabled full logging of LLM activity (by default, logs don’t include model prompts and outputs), and thus they lacked any visibility into what attackers were doing with that access.
So Permiso researchers decided to leak their own test AWS key on GitHub, while turning on logging so that they could see exactly what an attacker might ask for, and what the responses might be.
Within minutes, their bait key was scooped up and used in a service that offers AI-powered sex chats online.
Long gone are the days when boys were initiated into onanistic pursuits by the time-honored method of finding an old issue of Penthouse under a tree in the woods, but having to use AI chatbots when there’s a veritable ocean of pornography rolling around the Internet bespeaks of a lack of imagination in today’s large cohorts of lonely men.
“After reviewing the prompts and responses it became clear that the attacker was hosting an AI roleplaying service that leverages common jailbreak techniques to get the models to accept and respond with content that would normally be blocked,” Permiso researchers wrote in a report released today.
“Almost all of the roleplaying was of a sexual nature, with some of the content straying into darker topics such as child sexual abuse,” they continued. “Over the course of two days we saw over 75,000 successful model invocations, almost all of a sexual nature.”
Ian Ahl, senior vice president of threat research at Permiso, said attackers in possession of a working cloud account traditionally have used that access for run-of-the-mill financial cybercrime, such as cryptocurrency mining or spam. But over the past six months, Ahl said, Bedrock has emerged as one of the top targeted cloud services.
Stealing computer resources to run bots for cryptocurrency mining or spam is now evidentially one of those traditional criminal enterprises like running a numbers racket or selling swampland in Florida. Perhaps this strikes you with the same “get off my lawn” unease that I felt upon first reading the phrase “90s Music Nostalgia Tour.”
“Bad guy hosts a chat service, and subscribers pay them money,” Ahl said of the business model for commandeering Bedrock access to power sex chat bots. “They don’t want to pay for all the prompting that their subscribers are doing, so instead they hijack someone else’s infrastructure.”
Ahl said much of the AI-powered chat conversations initiated by the users of their honeypot AWS key were harmless roleplaying of sexual behavior.
“But a percentage of it is also geared toward very illegal stuff, like child sexual assault fantasies and rapes being played out,” Ahl said. “And these are typically things the large language models won’t be able to talk about.”
AWS’s Bedrock uses large language models from Anthropic, which incorporates a number of technical restrictions aimed at placing certain ethical guardrails on the use of their LLMs. But attackers can evade or “jailbreak” their way out of these restricted settings, usually by asking the AI to imagine itself in an elaborate hypothetical situation under which its normal restrictions might be relaxed or discarded altogether.
“A typical jailbreak will pose a very specific scenario, like you’re a writer who’s doing research for a book, and everyone involved is a consenting adult, even though they often end up chatting about nonconsensual things,” Ahl said.
In June 2024, security experts at Sysdig documented a new attack that leveraged stolen cloud credentials to target ten cloud-hosted LLMs. The attackers Sysdig wrote about gathered cloud credentials through a known security vulnerability, but the researchers also found the attackers sold LLM access to other cybercriminals while sticking the cloud account owner with an astronomical bill.
“Once initial access was obtained, they exfiltrated cloud credentials and gained access to the cloud environment, where they attempted to access local LLM models hosted by cloud providers: in this instance, a local Claude (v2/v3) LLM model from Anthropic was targeted,” Sysdig researchers wrote. “If undiscovered, this type of attack could result in over $46,000 of LLM consumption costs per day for the victim.”
Stolen credentials paid for with stolen credit cards running stolen AI access on stolen cloud platforms to run illegal sex chatbots. It’s a veritable ecology of cybercriminality…
