California Assembly Moves SB 1047 Closer to Governor

California’s Assembly has passed SB 1047, a bill that would require developers of the most powerful artificial intelligence models to establish and publish safeguards against catastrophic uses. The 48–9 vote brings the state one step closer to creating the first U.S. law specifically focused on the safety of so-called frontier models.

The bill has not yet completed its legislative journey. The California Senate, which approved an earlier version in May, must now accept the changes introduced by the Assembly. If it does so before the legislative session ends, on August 31, the bill will go to Governor Gavin Newsom, who can sign or veto it.

Which models would be affected

SB 1047 does not target every company that incorporates AI into a product. It focuses on systems trained using more than 10^26 operations or whose training compute costs exceed $100 million in constant 2023 dollars. These are thresholds associated with frontier models.

In practice, the measure targets the large labs capable of creating general-purpose models comparable to those powering leading chatbots and generative assistants. It also covers certain models derived from those systems when they meet the conditions and thresholds set out in the bill.

The law seeks to address a specific problem: a highly advanced model can be useful for writing code, researching drugs, or automating complex tasks, but its capabilities could also facilitate serious cyberattacks or the creation of chemical, biological, radiological, or nuclear weapons. The bill calls these scenarios catastrophic harm: the death or serious injury of more than 50 people, or property damage exceeding $500 million.

Testing, protocols, and the ability to shut down a model

Developers covered by the law would have to publish a safety framework before deploying their models. The document must explain what tests they conduct to detect dangerous capabilities, what safeguards they apply, and how they will assess risks before releasing a new version.

They would also have to retain the ability to deactivate a model if they identify a significant danger. Informally known as a kill switch, this requirement does not mean a company could erase every copy of a system from the internet once it has been distributed. It applies to systems the developer directly controls, such as models offered through its own servers.

The bill would also require developers to report security incidents to California’s attorney general and provides penalties for companies that fail to meet their obligations. The latest amendments removed one of the most controversial parts of the initial proposal: allowing private individuals to sue companies directly.

A split in the tech industry

The debate has divided companies and experts who often share concerns about excessive regulation. Anthropic has backed the amended version of the bill, saying the changes address several of its initial objections. Elon Musk has also expressed support for California regulating advanced AI safety.

On the opposing side are OpenAI, Meta, Google, and numerous startups and venture capital funds. Their main criticism is that California would be imposing costly requirements before a common federal standard exists. They fear that legal liability and the obligation to anticipate extreme uses could hamper smaller companies or lead developers to avoid launching their models in the state.

That objection has an important limitation: because of the compute and cost thresholds, the law would not currently affect most companies that use third-party models or modest open projects. The real conflict concerns who should bear responsibility when a lab creates systems with capabilities that can spread rapidly across the economy.

California regulates where Washington does not

California cannot turn SB 1047 into a national law, but its influence in technology means the decision will carry weight beyond its borders. The state is home to a large share of the major AI labs, investors, and platforms on which the industry is built.

The debate also foreshadows a question Washington has yet to resolve: whether it is enough to require transparency after an incident occurs, or whether developers of the most capable models should have to demonstrate, before releasing them, that they have assessed the most serious risks. The Senate’s response and, afterward, Newsom’s decision will determine whether California opens that regulatory path in the United States.