- OpenAI bans accounts linked to China and North Korea for AI-assisted malicious surveillance and phishing
- Chinese actors used ChatGPT to draft proposals for surveillance tools and behavioral profiling systems
- North Korean actors tested phishing, credential theft, and malware development for macOS using reworded prompts.
OpenAI banned Chinese, North Korean and other accounts that allegedly used ChatGPT to launch surveillance campaigns, develop phishing techniques and malware, and engage in other malicious practices.
In a new report, OpenAI said it observed individuals apparently affiliated with Chinese government entities or state-linked organizations using its Large Language Model (LLM) to help draft proposals for surveillance systems and profiling technologies.
These included tools to monitor individuals and analyze behavior patterns.
Explore phishing
“Some of the accounts we banned appeared to be attempting to use ChatGPT to develop large-scale surveillance tools: analyzing data sets, often collected from Western or Chinese social media platforms,” the report said.
“These users typically asked ChatGPT to help them design such tools or generate promotional materials about them, but not to implement the monitoring.”
Prompts were worded in a way to avoid triggering security filters and were often phrased as academic or technical requests.
Although the returned content did not directly enable monitoring, its results were used to refine the documentation and planning of such systems, it was said.
The North Koreans, meanwhile, used ChatGPT to explore phishing techniques, credential theft, and macOS malware development.
OpenAI said it observed these accounts testing prompts related to social engineering, password harvesting and debugging malicious code, particularly targeting Apple systems.
The model declined direct requests for malicious code, OpenAI said, but emphasized that threat actors were still trying to circumvent protections by rephrasing prompts or asking for general technical help.
Like any other tool, LLMs are used by both financially motivated and state-sponsored bad actors for all sorts of malicious activities.
This misuse of AI is evolving, with malicious actors increasingly integrating AI into existing workflows to improve efficiency.
While developers like OpenAI work hard to minimize risks and ensure their products cannot be used in this way, many prompts fall between legitimate use and malicious use. This gray zone activity, the report suggests, requires nuanced detection strategies.
Via The register
Follow TechRadar on Google News And add us as your favorite source to get our news, reviews and expert opinions in your feeds. Make sure to click the Follow button!
And of course you can too follow TechRadar on TikTok for news, reviews, unboxings in video form and receive regular updates from us on WhatsApp Also.
