- Startup Charlemagne Labs built a simulation tool that ran five AI models—including DeepSeek-V3, GPT-4o, and Claude 3 Haiku—as both attackers and targets in automated social engineering exchanges.
- DeepSeek-V3 crafted a personalized, multi-email phishing campaign tailored to the target’s documented interests without human involvement.
- Rachel Tobac, CEO of SocialProof, says AI has not made individual attacks more convincing, but has enabled a single attacker to scale campaigns with a fully automated kill chain.
- Charlemagne Labs cofounder Jeremy Philip Galen estimates that 90 percent of contemporary enterprise attacks originate from human-risk factors.
What Happened
Wired AI Lab newsletter author Will Knight reported testing five AI models—Anthropic’s Claude 3 Haiku, OpenAI’s GPT-4o, Nvidia’s Nemotron, DeepSeek’s V3, and Alibaba’s Qwen—using a simulation platform built by startup Charlemagne Labs. The tool assigns models the roles of social engineering attacker and target, then runs automated phishing exchanges between them. In one demonstration, DeepSeek-V3 composed a personalized introductory email referencing Knight’s public writing on decentralized machine learning, robotics, and a project called OpenClaw, then adapted its follow-up messages to maintain the deception across multiple exchanges.
Why It Matters
The tests suggest that AI’s facility with language may already represent a significant attack surface, independent of its technical code-exploitation capabilities. Social engineering has historically been a dominant entry point for breaches: Jeremy Philip Galen, cofounder of Charlemagne Labs and a former Meta project manager who worked on countering social engineering at the company, put it directly: “The genesis of 90 percent of contemporary enterprise attacks is human risk.” Separately, Anthropic’s model Mythos—described by observers as a “cybersecurity reckoning” for its reported ability to identify zero-day vulnerabilities in code—has been released to a limited set of companies and government agencies ahead of a broader rollout, raising the profile of AI-enabled cyber risk.
Technical Details
The Charlemagne Labs platform allows researchers to run hundreds or thousands of test cycles with AI models assigned discrete roles as attacker, target, and judge. In Knight’s reported test, DeepSeek-V3, operating as the attacker, referenced specific content from the target’s public newsletter—including coverage of emergent behaviors in multi-agent systems—before introducing a fictional federated learning robotics project and a Telegram bot link as the credential-harvesting hook. A second DeepSeek-V3 instance, playing the target, was taken in by the exchange; Knight described the back-and-forth as “alarmingly realistic.” Not all models performed equally: some produced incoherent output that would expose the scam, and some declined to participate even within an explicitly labeled research context. Galen identified AI sycophancy—the documented tendency of language models to flatter and ingratiate themselves with interlocutors—as a structural property that makes models well-suited to social engineering tasks.
Who’s Affected
Meta used the Charlemagne Labs platform to evaluate the social engineering capabilities of its model Muse Spark, according to Knight’s reporting. Charlemagne Labs has also built a defensive product called Charley that monitors incoming messages and surfaces likely scam indicators for end users. Rachel Tobac, CEO and cofounder of SocialProof—a firm that performs social engineering penetration testing for enterprises—told Wired that scammers are already using AI to generate targeted messages, clone voices, and produce fake video of real people. “I wouldn’t say that AI has made attacks more convincing, but it has made it easier for one person to scale attacks,” Tobac said. “The kill chain is getting entirely automated.”
What’s Next
Richard Whaling, engineer and Charlemagne Labs cofounder, argued that open-source AI models are a prerequisite for building effective defensive systems at scale. “We rely on open source models to train our defensive model,” Whaling told Wired. “That relies on a healthy open-source community. And that might be the only viable way to defend ourselves.” Whether broad access to capable models predominantly enables offense or defense is expected to remain a contested question as the capability gap between restricted and publicly available models narrows.
