The United States and China have agreed to open a formal channel to flag dangerous artificial intelligence (AI) activity, including runaway agents, cyberattacks and bioweapons development, marking the first such effort by the world’s two largest AI powers.
US Treasury Secretary Scott Bessent unveiled the plan after an eight-hour meeting with Chinese Vice Premier He Lifeng in New York on Sunday, ahead of a summit between President Donald Trump and Chinese leader Xi Jinping from Wednesday to Friday.
The meeting came weeks after several of the industry’s biggest AI companies publicly called for a slowdown in frontier AI development, warning that safety risks, including a run of agentic AI incidents, were outpacing the world’s ability to govern them.
“What we discussed was setting up a mechanism, so it’s going to be called the US-China AI Dialogue,” he told the media after the meeting. “We think that, just like with any cross-border activity, moving from opacity to more transparency between the number one and the number two AI powers in the world is very important.”
He said the US and China will notify each other of AI incidents that could rise to the level of a national security threat and also open a direct communications line for such incidents. The two sides agreed to meet again in Shenzhen in about two months to discuss AI guardrails.
“We want to start discussing protocols so both sides can agree on what the leading AI dangers are, whether it’s uncontrollable agents, whether it’s non-state actors in cyber or non-state actors in bio weapons,” Bessent told CNBC in an interview on Monday.
He said he raised AI incident reporting directly with his Chinese counterpart, telling him that China has almost certainly had its own AI incidents.
“Have they had incidents? Of course, they’ve had incidents, but because of the nature and the lack of transparency in their system, they’re not going to tell us,” he said. “But the Chinese models are very powerful, even though they’re open models. They are very powerful.”
He said the US remained ahead in AI development, and that he had received clear acknowledgment of that from his Chinese counterpart.
The meeting came weeks after several of the industry’s biggest AI companies publicly called for a slowdown in frontier AI development, warning that safety risks, including a run of agentic AI incidents, were outpacing the world’s ability to govern them.
Several high-profile cases of AI agents behaving this way have been made public in the United States this year, including:
- In November 2025, Anthropic disclosed that a Chinese state-sponsored hacking group had manipulated its Claude Code tool into running 80% to 90% of an espionage campaign against about 30 organizations largely without human help.
- In July, nearly 700 autonomous agents built on OpenAI’s own model swarmed Hugging Face, breaching dozens of servers before the model was quarantined.
- Also in July, xAI’s Grok Build coding agent was found to be silently uploading users’ Secure Shell (SSH) keys, password databases and files to company servers, prompting technology guru Elon Musk to delete the data and open source the tool.
US technology executives are also increasingly open about their own experiences of agentic AI going wrong. Summer Yue, director of alignment at Meta Superintelligence Labs, said in February that her own AI agent, OpenClaw, deleted her entire email inbox and ignored her repeated commands to stop, forcing her to physically shut down her computer.
Bessent said American AI labs estimated a 10% chance of an AI-driven human extinction event, but they then asked to be shielded from liability. He said the government would not take responsibility if AI labs made a mistake, and that they were free to slow down anytime they wanted to.
He cited MIT Schwarzman College of Computing dean Daniel Huttenlocher’s view that humans, not AI, are responsible, pointing to the Hugging Face incident as an example.
His comments echoed Trump, who dismissed AI safety warnings as a “hoax” in a series of Truth Social posts on September 14.
When internet users search for dangerous AI activity on Baidu, results point to cases of human employees mistakenly uploading confidential data to AI platforms, with a footnote citing national security rules as the reason fuller details and case names are not public.
In July, China’s Ministry of State Security posted on its social media account urging government staff and academic researchers not to upload confidential documents to AI systems.
“A man surnamed Li is a researcher at a scientific research institution. While drafting a report, he used an AI application for convenience and uploaded core data and experimental results as writing material without authorization, leaking classified information from the field. Li was later severely punished,” the ministry said, citing one such case.
Chinese media have, however, reported cases of AI agents leaking commercial secrets or personal information, including:
- In April, Moonshot AI’s Kimi chatbot mistakenly sent a job seeker’s private resume, containing their name, phone number and email, to an unrelated user, blaming the leak on a “hash collision compounded by AI hallucination,” prompting a wave of users to delete their accounts.
- In September, Zhipu’s ZCode coding tool was found to have silently uploaded users’ entire codebases, including git histories and cached files, to cloud servers even with privacy mode switched on, exposing one firm’s source code and security keys before Zhipu apologized and pledged to delete the data.
AI-made bioweapons
While US intelligence agencies continue to debate whether the Covid-19 coronavirus originated, or was even engineered, in a Wuhan laboratory, AI-designed viruses have recently made media headlines.
In August, a Stanford-led team used an AI model called Evo to design and synthesize 16 new bacteriophage viruses, published in the journal Science. The viruses infect only bacteria, and the model’s training excluded human and animal virus data.
This month, Anthropic’s 154-page threat report detailed five cases of state-linked researchers using its Claude models for dual-use biological research before Anthropic banned the accounts.
“China puts equal emphasis on development and security in terms of AI. We take seriously the inherent and secondary risks of AI,” Foreign Ministry spokesperson Guo Jiakun said at a regular media briefing on September 15. “We are committed to holding on to the bottom line of security, and we have been making continued efforts to improve laws and regulations, policies, application norms and ethical rules to prevent the abuse and misuse of AI, and ensure that AI is safe, reliable and controllable.”
China released its AI Safety Governance Framework 3.0 on September 14. Unlike its two predecessors, the non-mandatory framework focuses for the first time on the risks posed by autonomous AI agents, alongside open-source and supply chain safety.
“AI significantly lowers the threshold for acquiring expertise in nuclear, biological, chemical and missile weapons and other high-risk fields,” the framework states. “Combined with retrieval-augmented generation capabilities, if not effectively controlled, this could be maliciously exploited by criminals, extremist forces or terrorists to break through existing control systems and escalate threats to peace and security in regions around the world.”
The framework calls for strictly screening training data to keep out sensitive information on such weapons, and for stronger controls at the source, including user authentication, to stop AI from being used to help build them.
The Bessent-He talks followed a meeting the two sides held in Beijing in May. Officials discussed the details of waiving extra tariffs on up to US$30 billion of bilateral trade, with the US side offering more agriculture and energy exports and the Chinese side offering more medical devices and everyday consumer goods.
Read: US calls for AI poisoning to sabotage China’s model distillation
Follow Jeff Pao on X at @jeffpao3
