(Photo by Dilara Irem Sancar/Anadolu via Getty Images)
Don't miss out on our latest stories. Add PCMag as a preferred source on Google.
A North Korean hacking group appears to have used the technology behind OpenAI’s ChatGPT to help them develop a phishing attack.
The scheme was traced to the Kimsuky group, which has been known to engage in cyber espionage for the North Korean government. On Monday, security vendor Genians published a report about phishing emails, which involved those impersonating “a South Korean defense-related institution” charged with issuing IDs for military-affiliated officials.
Sent back in July, the emails used a domain meant to “mimic the official domain of a South Korean military institution" and included a .zip file as an attachment, Genians said. The .zip file contained the recipient’s real name, but partially masked, which could lend the email some legitimacy and entice the target to open the attachment.
(Credit: Genians)The .zip file contained a malicious shortcut meant to fool the recipient into manually running a PowerShell command on their PC. This command would secretly connect their computer to the hacker’s server and download malware capable of acting as a backdoor. The same process also fetched a fake government military ID image to mask the intrusion and convince victims that nothing unusual had happened.
(Credit: Genians)Genians looked closely at one of the fake military ID images and uncovered evidence that it was sourced to OpenAI’s older GPT-4o model. The image's metadata referenced the words “GPT-4o*OpenAI API” along with the name ChatGPT.
Genians also ran the fake military ID image through a DeepFake detector and found a 98% probability that it was AI-generated.
(Credit: Genians)The report notes that OpenAI’s systems already have built-in safeguards that prevent the company’s chatbots from generating AI images for government IDs. Still, Genians suspects the Kimsuky group was able to develop a workaround, also known as jailbreak, to trick ChatGPT into creating the IDs.
“For example, it may respond to requests framed as creating a mock-up or sample design for legitimate purposes rather than reproducing an actual military ID,” the vendor wrote.
OpenAI didn’t immediately respond to a request for comment. But the company has detected and prevented state-sponsored hackers from using its technology in the past.
Still, the report from Genians suggests that North Korean hackers are continuing to find ways to access ChatGPT and leverage it for nefarious purposes. This comes as North Korea has increasingly been found using other AI technologies, including real-time deepfakes, to help them spread malware or even land remote jobs at US companies.
Disclosure: Ziff Davis, PCMag's parent company, filed a lawsuit against OpenAI in April 2025, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.