Most people know to be cautious of suspicious emails and unexpected links. But as AI tools like ChatGPT become more popular, cybercriminals are finding new ways to take advantage of the trust people place in them.
A recent attack shows exactly how that can happen and it’s one we recently encountered within our own client base. Fortunately, KaufmanIT’s security monitoring tools detected the activity, our team quickly quarantined the affected device and the threat was contained before it could spread elsewhere in the environment.
What Happened?
Researchers discovered a campaign in which attackers created a malicious custom version of ChatGPT and promoted it through Google ads. Users searching for AI tools clicked the sponsored result and landed on what appeared to be a legitimate ChatGPT experience.
The attackers weren’t hacking ChatGPT itself. Instead, they were abusing a legitimate feature offered by OpenAI.
What Is a Custom GPT?
OpenAI allows users to create specialized versions of ChatGPT called custom GPTs. These can be tailored for specific purposes such as marketing, coding, legal research or travel planning. Creators can publish these GPTs for others to use, allowing anyone to interact with them through ChatGPT. The criminals in this case simply created their own public GPT and used it as a lure for victims.
How the Attack Worked
The attack chain was straightforward:
- The attacker created a public custom GPT.
- They promoted it through Google Ads.
- Users clicked the ad and interacted with the GPT.
- The GPT directed users to another website.
- The website instructed users to run a command on their computer.
- Malware was installed.
What made the scam effective is that everything appeared legitimate. The victim saw ChatGPT, Google Ads and a Google-hosted website, making the instructions seem trustworthy.
The Biggest Red Flag
The website instructed users to open PowerShell and paste in a command.
For most users, that should be an immediate warning sign.
As a rule of thumb, never run commands on your computer because a website, chatbot or AI tool tells you to. Leveraging a familiar tool to lure a user into feeling safe about a download is a tactic hackers increasingly use, as we’ve written about in the past.
Legitimate services rarely ask everyday users to do this.
What Was Installed?
The malware was a Remote Access Trojan (RAT), which can give attackers remote access to a victim’s computer. According to researchers, it could search files, gather information, access devices and install additional malware.
The Takeaway
This attack wasn’t successful because ChatGPT was compromised. It was successful because attackers found a new way to exploit human trust.
The good news is that OpenAI has announced plans to retire custom GPTs on December 11, which should eliminate this specific attack method. However, the broader lesson remains the same: cybercriminals will continue looking for ways to abuse trusted platforms and emerging technologies.
Our recent experience reinforces that point. The threat wasn’t a software vulnerability. It was a convincing social engineering attack that persuaded a user to trust instructions that appeared to come from legitimate platforms. Fortunately, layered security tools and rapid response helped prevent any broader impact.
As AI becomes part of everyday business life, the same skepticism that protects us from phishing emails should also apply to chatbots, websites and online instructions. If an AI tool asks you to download software or run commands on your computer, stop and verify before proceeding.
