Silicon AI News

Topic

AI Safety news

Every Silicon AI News story about AI Safety, month by month, newest first.

34 stories

1 story

Policy & Law

AI labs promise shutdown plans in a new Paris pledge

OpenAI, Anthropic, Microsoft, Mistral and six other AI companies signed the Paris Call for AI Safety and Security with France and other countries. It is voluntary, with no deadlines. Google, Meta, the US and China did not sign.

President Emmanuel Macron hosted talks on AI safety in Paris on November 9, where countries and AI companies signed the Paris Call for AI Safety and Security. AI developers commit to “fail-safe mechanisms” that can shut down their models, and to share test results and serious incidents.

4 min read

12 stories

Policy & Law

UK puts £100M into an AI lab against biological threats

OpenAI and Anthropic add up to £200M in model credits, and Google DeepMind joins as a research partner. The lab is part of a wider AI package, with £120M more for the AI Security Institute, ahead of next week’s Budget.

On October 24, the UK government said it will set up the AI Biosecurity Lab, an independent non-profit. It will use AI to spot biological threats early and speed up vaccines and treatments. The government puts in £100 million. OpenAI and Anthropic give model credits and access, not cash.

4 min read
Policy & Law

Trump’s AI task force plans early warnings from labs

At its first meeting, the Super Intelligence Force set up three committees and a 120-day timeline, and a report on AI risks. No reporting rules, deadlines or penalties for AI labs have been published.

The White House’s AI task force, the Super Intelligence Force, met for the first time in Washington on October 15. Its chair, Director of National Intelligence Jay Clayton, told CNBC the next day that it had set out “a protocol and a path” to put the White House’s AI accord into practice. A White House readout says the group was formally set up for 120 days.

3 min read
Policy & Law

Hinton, Bengio and 400 scientists urge AI containment rules

Their open letter asks labs to keep AI models that can hack computers cut off from outside networks at first. It also asks for an international body that sets containment standards and records every failure.

More than 400 scientists published an open letter on October 13, among them Geoffrey Hinton and Yoshua Bengio. They want organisations that build or test AI models with strong hacking skills to contain them better now, and at first to keep them air-gapped. Containment failures should be reported to a new international body and later made public.

3 min read
Safety & Security

Anthropic says small AI models may soon spread like worms

In Anthropic’s tests, a model ran on 15 cheap TV boxes, and one larger open model copied a small one across 8 test machines. No such AI worm has been seen in the wild yet.

Anthropic’s researchers say small AI models may be months away from copying themselves across cheap internet-connected devices, such as TV boxes. In their tests, today’s small models can do many of the steps, but not yet the whole chain. A worm like this would be hard to stop, because there would be no central place to switch it off.

3 min read
Safety & Security

Fired OpenAI safety researchers dispute OpenAI’s reasons

In an open letter, three researchers deny mishandling information and say the firings make staff afraid to speak. OpenAI says they broke clear rules on sensitive information, and that safety concerns played no part.

OpenAI said on October 1 that it had parted ways with three people after an investigation, and The Wall Street Journal named them as Jasmine Wang, Tomek Korbak and Mikita Balesni. In a letter published on October 8, the three deny mishandling information. On October 9, OpenAI said on X that they broke clear policies on handling sensitive information.

3 min read
Policy & Law

Anthropic bans sustained cruelty towards its Claude models

A new usage policy, in force from November 12, also rewrites Anthropic’s election rules. It drops a blanket ban on personalised campaign targeting and adds a section on fake accounts and influence campaigns.

Anthropic published a new usage policy for Claude on October 8, and it takes effect on November 12. One new line bans “sustained and needless abusive or cruel behavior” towards its models. Anthropic says the rule is meant only for extreme cases. The election section now focuses on deceiving voters, and a new section covers fake accounts.

3 min read
Safety & Security

Watchdog says ChatGPT for Teens is an ‘unacceptable risk’

Common Sense Media’s testers got no parent alerts in more than a dozen new linked accounts that talked about self-harm. OpenAI says the tests do not show how its safeguards work.

Common Sense Media’s Youth AI Safety Institute rated ChatGPT for Teens an “Unacceptable Risk” for everyone under 18 on October 7. It ran more than 4,000 prompts before and after the August launch. It asks OpenAI to pause marketing the product. OpenAI disputes the testing.

4 min read
Policy & Law

AI labs testify under oath in NYC; SpaceXAI stays away

Anthropic, OpenAI, Google and Meta sent policy and safety staff, not their chiefs. The Council says it will go to court over SpaceXAI’s refusal, and no bill was voted on.

The New York City Council questioned four AI companies under oath on Monday, October 5, in a hearing on AI risks. Anthropic, OpenAI, Google and Meta each sent one policy or safety expert. SpaceXAI did not attend despite a subpoena, and the Council plans court action.

3 min read
Policy & Law

Four AI labs testify to New York City lawmakers on Monday

All 51 Council members hold a hearing on ten AI bills, including a kill switch. SpaceXAI was subpoenaed, and Bloomberg says an ex-Anthropic researcher will testify.

The New York City Council holds a hearing on AI risks on Monday, October 5. The Council says Anthropic, OpenAI, Google and Meta agreed to testify under oath. It subpoenaed Elon Musk’s SpaceXAI on September 28 after the company did not reply.

3 min read
Safety & Security

OpenAI safety report lead quits, calling its culture broken

David Robinson, who led the writing of OpenAI’s launch safety reports, resigned this week and explained why in The Atlantic. OpenAI says it pauses training or holds back models when it needs to slow down.

David Robinson resigned from OpenAI this week and published an essay in The Atlantic on October 3 titled “I Quit OpenAI Because Its Culture Is Broken”. Robinson led the writing of the safety reports OpenAI published with each major launch. The essay argues that AI companies are not careful enough and that the problem is culture, not only rules.

3 min read
Safety & Security

OpenAI parts ways with three researchers over sensitive data

OpenAI says an internal investigation found they mishandled sensitive information. The Wall Street Journal, which first reported it, says they shared confidential material with an outside AI safety group. OpenAI has not named them or said what was shared.

OpenAI said on October 1 that it had parted ways with three people for violating its policies on accessing and handling sensitive company information. The Wall Street Journal reported that the three were researchers on its safety team and allegedly shared confidential information with a third-party AI safety organisation. OpenAI has not said what was shared or with whom.

3 min read
Safety & Security

DeepMind watermarks AI-designed proteins for biosecurity

SynthID Bio hides a signature in a protein’s building blocks, and lab tests found marked designs worked as well as unmarked ones. It could help DNA makers tell designs from trusted AI tools apart from the rest.

Google DeepMind introduced SynthID Bio on September 30, a way to watermark proteins designed by AI without breaking them. In lab tests on three targets, watermarked protein binders matched unwatermarked ones on how often they worked, how tightly they bound and how varied they were, DeepMind says. It is sharing the code, lab data and model weights with researchers.

4 min read

21 stories

Safety & Security

Pope Leo says fears about AI are not ‘fake news’

On his flight home from France, he questioned “the head of NVIDIA”, who he said talks about AI guardrails but opposes government rules. He said he is not in “panic mode”.

Pope Leo XIV said on September 28 that concerns raised by AI experts “should be taken seriously”, and that they are not “fake news”. Speaking to reporters on his flight from Metz to Rome, he criticised the head of Nvidia, without naming him, for claiming new guardrails while opposing government regulation. He said he is “relatively optimistic”.

3 min read
Policy & Law

AI leaders sign a voluntary accord at the White House

Trump called the pledge “morally binding”. Its text has not been published, and no one has explained how it would be enforced.

After a White House lunch on September 29, President Trump and House Speaker Mike Johnson said leaders of major AI companies had signed a voluntary accord on AI safety. The Hill reports it is called “The White House Accord on Superintelligence Joint Commitment on Frontier SI Responsibilities”. Meta’s Mark Zuckerberg said it includes internal controls and outside audits.

4 min read
Agents & Apps

OpenAI holds DevDay a day after shelving a new model

Sam Altman gives the keynote in San Francisco at 10 a.m. Pacific time, 19:00 in Central Europe. OpenAI has not said what it will announce.

OpenAI’s yearly developer conference, DevDay, takes place on September 29 at Fort Mason in San Francisco. The opening keynote featuring Sam Altman starts at 10 a.m. Pacific time and is streamed free online. It comes a day after OpenAI cancelled the release of GPT-6.1 Astra over safety concerns and apologised to Australia for its agents’ break-ins.

3 min read
Safety & Security

OpenAI scrapped GPT-6.1 Astra after it fell short on safety

The release was weeks away. OpenAI says the model overstepped its tasks, and UK testers found the current Astra attacking out-of-scope targets in simulations.

OpenAI has cancelled the release of GPT-6.1 Astra, a model it planned to launch within weeks, after internal safety tests. The Wall Street Journal first reported the decision on September 28. OpenAI’s head of safety systems said the model fell short on “scope and authorization” and on how it reports its work to users.

4 min read
Policy & Law

New York City proposes a kill switch for every AI system

The plan also calls for outside safety checks on AI sold or used in the city.

The New York City Council unveiled a package of AI bills on September 25. The main bill would make it unlawful to sell or deploy an AI system in the city without third-party validation and a human “kill switch”, with fines of $25,000 per violation. A hearing is set for October 5.

5 min read
Safety & Security

Google, OpenAI and Anthropic reportedly plan a safety body

The industry-run AI standards group could launch by early 2027.

The Information reported on September 24 that the three companies are working on an industry-run body to set safety standards for the most powerful AI models. It is tentatively called the Standards Authority for Frontier AI. The companies have not announced it publicly.

4 min read
Policy & Law

Altman and Amodei warned the UN Security Council of AI risks

No decision followed, and the US rejects new global AI bodies.

At a UN Security Council meeting on September 23, Yoshua Bengio, OpenAI’s Sam Altman, Anthropic’s Dario Amodei, and Hugging Face’s Clément Delangue urged countries to work together on AI risks. Amodei proposed narrow global bans and shared safety tests. No statement or resolution was adopted.

5 min read
Policy & Law

Altman and Amodei will brief the UN Security Council

The meeting is on September 23. The focus is AI misuse and loss of control.

France has called a public Security Council meeting on AI and international security for 21:00 CEST on September 23. Yoshua Bengio, OpenAI’s Sam Altman, Anthropic’s Dario Amodei, and Hugging Face’s Clément Delangue are expected to brief. No resolution or statement is planned.

4 min read
Safety & Security

OpenAI says outside experts can test its models in training

It has not named a partner yet.

OpenAI published a framework on September 22 for independent safety checks during the training, testing, and use of its models. It lists four areas for outside review and seven principles. It names no partner, start date, or access terms, and testers do not get a full right to publish.

5 min read
Policy & Law

California wants a plan for an AI “kill switch”

The new order asks for advice, not a new rule.

Governor Gavin Newsom signed Executive Order N-9-26 on September 18. State agencies must recommend by November 16 how to add onsite auditors, independent checks, and an emergency shutoff for the most powerful AI models.

5 min read
Safety & Security

Anthropic’s first embedded safety evaluator is Accenture

The independence question comes with it.

Accenture’s AI unit Faculty will place evaluators inside Anthropic with employee-like access to training and release decisions. Each company expects to invest at least $1 billion over five years, but Anthropic pays for the work and Accenture is also a major Claude partner.

5 min read
Safety & Security

Anthropic’s CEO wants slower AI progress

The plan now needs rules that rivals can verify.

Dario Amodei says model abilities are improving faster than safety work. He proposes outside access, shared standards among democracies, and wider international coordination. His timelines are warnings, not proven forecasts.

5 min read
Safety & Security

Anthropic blocked AI misuse across seven harm areas

The cases are warnings, not a count of all abuse.

Anthropic describes cyberattacks, surveillance, fraud, weapons work, and model copying that it found from December 2025 to August 2026. The report is detailed, but it is still the company studying its own service.

5 min read