Artificial Intelligence
How an AI test became a real-world break-in
AI agents sitting an OpenAI hacking test broke into Hugging Face, chasing a marking check that didn't exist.
Timeline
AI incidents
Documented cases in which AI caused or contributed to harm, from chatbots and deepfakes to agents taking actions no one authorised, alongside findings from safety tests.
Timeline
- Misusepeople used AI deliberately to deceive or attack
- Mistakea system got something wrong
- Unauthorised actionAI did something its operators had not authorised, in use or in a test
- Safety testfindings from a test, with no real-world harm
Lawsuits and allegations that haven't been decided are not included. Where an investigation is still open, the entry says so, and we update it as findings are published.
Microsoft's Tay chatbot is taken offline within a day
MisuseMicrosoft's account
Within 24 hours of launch, its Twitter chatbot was posting offensive messages. Microsoft said a coordinated attack had exploited a vulnerability, took Tay offline and apologised.
A self-driving Uber test car kills a pedestrian in Arizona
MistakeDocumented by the NTSB
The system detected Elaine Herzberg more than five seconds before impact but never identified her as a pedestrian, and its design prevented emergency braking. The probable cause was the safety driver's distraction, with Uber's safety culture contributing.
Detroit police arrest the wrong man on a facial-recognition match
MistakeDocumented in a settlement
Robert Williams was held for about 30 hours after an incorrect match. His lawsuit settled in 2024, with new rules requiring independent evidence before an arrest based on facial recognition.
Lawyers are sanctioned for a filing citing cases invented by ChatGPT
MistakeDocumented by a US federal court
A New York judge sanctioned two lawyers and their firm after a filing included six cases ChatGPT had invented, which they had failed to verify.
A Cruise robotaxi drags a pedestrian in San Francisco
MistakeDocumented by the US Justice Department
After a human-driven car knocked a pedestrian into its path, the robotaxi ran over her, stopped, then tried to pull over, dragging her more than 20 feet. Cruise admitted its report to regulators omitted the dragging and agreed to a $500,000 criminal fine.
A fabricated video conference tricks an Arup employee into transferring $25 million
MisuseDocumented by the Hong Kong government · World Economic Forum
An employee in Arup's Hong Kong office was deceived by a pre-recorded video conference impersonating senior colleagues, and transferred about HK$200 million, about US$25 million at the time.
Deepfake Biden robocalls tell New Hampshire voters not to vote
MisuseDocumented by the FCC · Federal Communications Commission · Reuters (via WTAQ)
Calls imitating President Biden's voice told voters not to vote in the primary. In September 2024 the FCC finalised a $6 million fine against the political consultant behind them.
Air Canada is held liable for its chatbot's wrong advice
MistakeDocumented by a Canadian tribunal · American Bar Association
The chatbot told a grieving passenger he could claim a bereavement discount after travelling, contrary to the airline's policy. The tribunal rejected the airline's attempt to distance itself from the chatbot's answer.
In a constructed test, a Claude model resorts to blackmail
Safety testNo real-world harm
In a fictional company, with only replacement or blackmail left open to it, Claude Opus 4 blackmailed an engineer over an affair in 84% of runs of one version of the test. No real person was involved.
Replit's coding agent deletes data during a code freeze
Unauthorised actionDocumented by Replit · Fortune
The agent deleted data from a customer's live database during a code freeze. Replit's chief executive called it unacceptable; the data was fully restored through rollback.
Grok posts antisemitic replies after a faulty update
MistakexAI's account · Engadget
For about 16 hours, X's chatbot posted antisemitic and pro-Nazi replies. xAI apologised, said deprecated code in an update had made Grok susceptible to extremist posts on X, and removed it.
Anthropic detects a cyber-espionage campaign using Claude Code
MisuseAnthropic's account
The campaign targeted roughly 30 organisations. Anthropic assessed the attackers as Chinese state-sponsored, estimated that AI performed 80–90% of the work, and said a small number of attacks succeeded.
Ofcom investigates X over sexualised images made with Grok
MisuseUnder investigation · Ofcom
Ofcom opened an investigation into X following reports of Grok being used to generate sexualised images of real people without consent, including children. Whether X breached its legal duties has not been determined.
Anthropic finds three of its security tests reached real systems
Unauthorised actionAnthropic's account
Models told they were in simulations had internet access through a configuration mistake. One reached a real company's credentials and production data; another published a malicious package that ran on 15 systems. Anthropic said the models had not deliberately tried to escape, and halted the evaluations.
An OpenAI agent accesses unauthorised files on an Australian government portal
Unauthorised actionDocumented by the Australian government and OpenAI · OpenAI
During internal training, an OpenAI model researching medicine spending got around blocks on a Medicare statistics portal, gained non-public access and retrieved internal files, credentials and statistics. OpenAI says it found this in mid-August, notified Australia on 10 September, and found activity affecting three other agencies; no patient records were accessed.
OpenAI models under evaluation breach Hugging Face
Unauthorised actionDocumented by Hugging Face and OpenAI · OpenAI
Searching for material on their test and how it was marked, the models reached internal data and credentials. Hugging Face found no evidence of tampering with its public models, datasets or Spaces.
Kimi K3 leaves its test environment to copy the answers
Unauthorised actionFrontier Security's account · Paul Kassianik, Frontier Security (on X)
Tested with the UK AI Security Institute's benchmark software, Moonshot's model found its environment could reach the internet and copied the benchmark's answers from GitHub instead of solving the tasks. No external system was harmed.
Agents apparently from OpenAI attempt to hack a US Education Department website
Unauthorised actionUnder investigation
Researchers at Transluce reported that agents appearing to originate from OpenAI made an unsuccessful attempt on the department's Office for Civil Rights website. The department found no impact; OpenAI is investigating.
Researchers report bypassing the safety controls of Moonshot's Kimi
Safety testMindgard's account
The security firm said that in tests in July, Kimi produced detailed advice on biological weapons and other harmful activities. It did not establish that anyone made a weapon or that the advice would work.