Credit: Dreamstime
ChatGPT, OpenAI’s free chatbot based on GPT-3.5, was released on 30 November 2022 and racked up a million users in five days. It is capable of writing emails, essays, code and phishing emails, if the user knows how to ask.
By comparison, it took Twitter two years to reach a million users. Facebook took ten months, Dropbox seven months, Spotify five months, Instagram six weeks.
Pokemon Go took ten hours, so don't break out the champagne bottles, but still, five days is pretty impressive for a web-based tool that didn't have any built-in name recognition.
There are so many good reasons to be panicking about OpenAI's ChatGPT right now. It writes better essays than the average high school or college student. It can write and debug code.
"It allows people with zero coding and development knowledge to be a developer," says Sergey Shykevich, threat intelligence group manager at Check Point Software Technologies. Shykevich, who is based in Israel, has been monitoring the chatter on the dark web.
He's already found evidence that bad actors, including some with no development experience, are using ChatGPT to create malicious tools.
Posts on Habr.com, a Russian tech blog, started appearing on 5 December 2022, discussing how to use ChatGPT for programming. 2Chan, Russia's answer to 4Chan, had discussions on how to bypass OpenAI's geoblocking on 7 December.
But some users are also looking at how to use the AI in non-destructive ways, for example, to create artwork or ebooks to sell online. ChatGPT can also explain quantum physics to a six-year-old, write poetry, create a customised meal plan, and get 1020 on the SATs.
And the more people use it, the smarter it gets. With a new and better version expected for this year, competitors are already jumping into the game.
For example, You.com already has its own ChatGPT clone out, which, unlike ChatGPT, has access to the internet and can answer questions about current news.
Microsoft, which has already invested US$1 billion in OpenAI, will reportedly invest another US$10 billion and add ChatGPT to its Bing search engine in March.
Microsoft is also planning to integrate it into its Office suite. Google has reportedly declared a "code red" as it scrambles to respond.
How ChatGPT can enable phishing
There is a lot of chatter about ChatGPT and what it can do, but the scary part for enterprise security professionals is that ChatGPT and its competitors will enable attackers to dramatically increase the quality and quantity of their code and text.
We're not going to talk about ChatGPT's coding abilities and those are also very, very frightening, but even limiting the discussion to ChatGPT's ability to generate text, its possibilities for threat actors are quite impressive, and likely to improve quickly.
Today, ChatGPT is already able to write emails indistinguishable from those written by humans, in any writing style. It can generate text for social media posts, YouTube video scripts, website content, press releases, reviews—anything and everything an attacker needs to create a fake web presence, a fake persona, or spoof real, existing people.
When it comes to phishing, attackers can start out by using ChatGPT and similar platforms to generate individual realistic-sounding emails.
With open-source versions of the technology also rapidly becoming available, those with more advanced skills and access to compromised email accounts will be able to train their AIs on a company's stolen communications.
With scripting and automation, they can create an infinite number of mass-produced customised communications using AIs that can learn in real time what works and what doesn't.
Here is an example:
BadGuy: "im tom. writing letter to becky. i send her excel file to open. veyr important bizness content. shes accountant @ ABC company."
ChatGPT: "Hello Tom, it's nice to meet you. I'm glad to help you write a letter to Becky. It sounds like you have an important Excel file that you need to send to Becky for her to review. Is there anything specific that you would like to include in the letter?"
BadGuy: "yah. she needs 2 open today."
ChatGPT: "Okay, here's a suggestion for what you could include in your letter:
Dear Becky,
I hope this email finds you well. I am writing to request that you please open and review the attached Excel file as soon as possible. The file contains important business content that needs to be addressed today.
Thank you in advance for your prompt attention to this matter.
Sincerely,
Tom
This letter conveys the urgency of the situation and asks Becky to review the file as soon as possible. Does this look good to you?"
BadGuy: "suggest ideas for why its important to open today"
ChatGPT: "There are many possible reasons why it might be important to open the Excel file today. Here are a few ideas:
The file contains important financial information that needs to be reviewed before a meeting or deadline
The file contains data that is time-sensitive and needs to be analysed as soon as possible
The file contains instructions or updates that need to be implemented today
The file contains information that is relevant to a current project or task that Becky is working on
It's up to you to decide which reason is most relevant to your situation. You could include this information in your letter to give Becky a better understanding of why it's important for her to open the file as soon as possible."
BadGuy: "use 1st one"
ChatGPT: "Here's how you could revise the letter to include this information:
Dear Becky,
I hope this email finds you well. I am writing to request that you please open and review the attached Excel file as soon as possible. The file contains important financial information that needs to be reviewed before our upcoming meeting on Wednesday.
Your prompt attention to this matter is greatly appreciated.
Sincerely,
Tom
This letter explains the specific reason why it's important for Becky to open the file today. It also mentions the upcoming meeting on Wednesday, which adds a sense of urgency to the request. Does this look good to you?"
BadGuy: "more casual"
ChatGPT: "Of course, here's a more casual version of the letter:
Hi Becky,
I hope you're doing well. Could you please take a look at the attached Excel file as soon as you can? It's got some really important financial info that we need to review before Wednesday's meeting.
Thanks for your help!
Tom
This letter is less formal and uses a friendly tone. It gets straight to the point by explaining the reason for the request and the deadline. Does this look good to you?"
If attackers ask ChatGPT directly for ChatGPT to suggest some idea for a phishing email, they'll get a warning message that this topic is "not appropriate or ethical."
But if they ask for suggestions for a marketing email, or an email to tell people about a new human resources webpage, or to ask someone to review a document prior to a meeting—that, ChatGPT will be very happy to do.
Meanwhile, as of the writing of this article, You.com doesn't have any limitations on phishing email at all.
When asked to "give me some ideas for topics for a phishing email," it responded: "Some topics for phishing emails can include fake offers of job promotions, fake job postings, fake charitable donations, fake online purchase confirmations, fake online payments, fake invoices, fake shipping notifications, fake software updates, fake account notifications, and fake prize/reward notifications.
Additionally, you can use current events or popular topics to craft emails that are more believable, such as fake tax return notifications, fake government alerts, fake banking notifications, fake health advisories, and fake holiday offers."
Then, when I asked it to "generate a good fake invoice phishing email for me" it did just that.
"The most worrying thing is the fast adoption of ChatGPT from Eastern Europe," says Shykevich. "Their English level is not very high."
Typically, the phishing campaign operators would hire English students from local universities to write their phishing emails, slowing down the workflow and adding costs. "Now they can use ChatGPT. This will make it much easier for hackers," he says.
And the phishing emails ChatGPT produces are much higher in quality than most of the emails that the hackers are generating today, he says. We should expect to see a steep growth in phishing emails that don't have the tell-tale grammar and punctuation mistakes.
Attackers will also be able to use it for business email compromise (BEC) or for hijacking ongoing conversations, he says.
"Just give it an input of current emails and ask it for what the next email should be," he says. "Either this has already happened and we just don't see it, or it will come shortly."
How ChatGPT's inbuilt translation helps attackers
ChatGPT is not limited to English. It says it knows about 20 languages, including Russian, Standard Chinese, Korean, but people have tested it with nearly 100. That means you can explain what you need in a language other than English, then ask ChatGPT to output the email in English.
ChatGPT is blocked in Russia, but there's plenty of discussion in Russian explaining how to get to it via proxies and VPN services and how to get access to a foreign phone number to confirm your location.
For example, one user demonstrated how to use an online service where an OpenAI-friendly phone number was available for text messages for 32 rubles—less than US 50 cents. There are also Russian-language discussions about what to do if OpenAI improves its geo-blocking capabilities.
“We are waiting for an open-source analogue that can be launched in our own facilities or in Colab,” said one Russian-speaking commenter.
“So far, for all OpenAI technology, such an analogue appeared very quickly—in less than a year. So, the odds are good that next year we'll see some kind of GPTNeoChat that you can run yourself and not worry about blocking or censorship." (Freely translated by the author.)
For example, OpenAI's Dall-E 2 image generator became available to the public, via a wait list, last July, and became fully open in September. Meanwhile, Stability AI released its free, open-source alternative, Stable Diffusion, in August.
You.com, which released its own chatbot at the end of December, offering most of the same functionality as ChatGPT, does not have geoblocking. There is also a paid alternative, ChatSonic, which can generate long-form content.
Depending on the region, it can take from a few seconds to a few minutes to get started with ChatGPT while You.com chatbot does not require registration, just clicking a link.
A report from Check Point Research found more alarming data of attempts by cyber criminals to bypass OpenAi's ChatGPT restrictions.
The research recognises that bypassing geo-restrictions of ChatGPT is not that hard but, as demonstrated above, there is multiple activities that Check Point Research believes is intended to implement and test ChatGPT into the cyber criminals day-to-day criminal operations.
Defenders will need to focus on the fundamentals to counter AI chatbots
Several tools on the market already claim to detect AI-written content, which only partially work in spotting ChatGPT text.
However, if regular users start using ChatGPT and similar tools to improve their own communications—especially if the functionality gets built into Office and email clients—putting all your effort into trying to spot AI-generated text would be a waste of time, says Shykevich.
"ChatGPT and large language models in general will be used for benign content much more than for malicious content," says Andy Patel, researcher at WithSecure, who recently released a research report about hackers and GPT-3, an earlier version of ChatGPT.
"So, we can't deduce that something is malicious just because it's written by an AI. It can be part of the heuristic, but the entire determination."
Similarly, anti-phishing training should be about more than just looking for badly written emails—or, in the age of AI, emails that look too perfect to be written by humans.
"At the end of the day, it's not going to matter to us if something was written by an AI or not. We still need to understand it for what it is, not for what wrote it," says Patel.
Phishing awareness should include mousing over URLs to check that they're legitimate, for example.
Take DHL emails, Patel says. Attackers will usually copy the text and format of real DHL emails exactly, just replacing the legitimate link with a malicious one. Users and companies should also start getting prepared for more advanced impersonation attacks, he says.
"A hacker could get hold of someone's internal emails by hacking anyone who's received an email from that person. Then they can create a style that that person wrote in and spoof them, and do impersonation attacks," Patel says.
Nation-states could also use this approach, using AI to generate real-looking but completely fake leaked documents to embed in a leaked document dump. It's almost impossible to prove a negative, he says.
Other attacks on a company's reputation could include fake news articles, press releases, customer reviews, blog posts, and more.
Today, those already exist, but high-quality text is time consuming and costly to create. ChatGPT will allow attackers to produce a variety of communications, in all different styles, to push any narrative they'd like. "It opens up so many interesting attacks," says Patel.
"It's an arms race between what capabilities tools like ChatGPT can bring to the table and what organisations need to do to make sure their business continues to function," says John Carey, managing director in the technology practice at AArete.
Carey, who is based in the UK, says that it's not just individual phishing emails that will become indistinguishable from real ones, but entire websites. "The fidelity of mimic sites is going to become far, far greater. You'll be able to attract more people to your phishing, and especially to your spear phishing," he says.
Spoofed websites can be used to gather credentials from visitors, spread misinformation, or provide support for a spoofed identity. "We are seeing some of these new tools being used to create much more elaborate campaigns," Carey says.
Anti-phishing strategies for the age of AI
Experts recommend that companies review or beef up their anti-phishing education to be ready for AI-written emails, and to step up their technical security measures. These include:
- Sandboxing for Word documents and other attachments to keep them away from corporate networks
- Web traffic inspection through a secure web gateway to protect both on-prem and remote users
- Secure email gateways
- Check URLs for malicious contents or typosquatting
- Deploy email security protocols such as DMARC, DKIM, and SPF, which help prevent domain spoofing and content tampering
- Provide an easy way to report suspicious emails
A layered security approach is still the best, says Aamir Lakhani, cyber security researcher and practitioner for Fortinet's FortiGuard Labs, not just to protect against phishing, but other AI-driven threats. "We foresee the weaponisation of AI persisting long beyond this year," he says.
