How do you know when AI is powerful enough to be dangerous? Regulators try to do the math

How do you know if an artificial intelligence system is so powerful that it poses a security danger and shouldn’t be unleashed without careful oversight?

For regulators trying to put guardrails on AI, it’s mostly about the arithmetic. Specifically, an AI model trained on 10 to the 26th floating-point operations per second must now be reported to the U.S. government and could soon trigger even stricter requirements in California.

Say what? Well, if you’re counting the zeroes, that’s 100,000,000,000,000,000,000,000,000, or 100 septillion, calculations each second, using a measure known as flops.

What it signals to some lawmakers and AI safety advocates is a level of computing power that might enable rapidly advancing AI technology to create or proliferate weapons of mass destruction, or conduct catastrophic cyberattacks.

Those who’ve crafted such regulations acknowledge they are an imperfect starting point to distinguish today’s highest-performing generative AI systems — largely made by California-based companies like Anthropic, Google, Meta Platforms and ChatGPT-maker OpenAI — from the next generation that could be even more powerful.

Critics have pounced on the thresholds as arbitrary — an attempt by governments to regulate math.

“Ten to the 26th flops,” said venture capitalist Ben Horowitz on a podcast this summer. “Well, what if that’s the size of the model you need to, like, cure cancer?”

An executive order signed by President Joe Biden last year relies on that threshold. So does California’s newly passed AI safety legislation — which Gov. Gavin Newsom has until Sept. 30 to sign into law or veto. California adds a second metric to the equation: regulated AI models must also cost at least $100 million to build.

Following Biden’s footsteps, the European Union’s sweeping AI Act also measures floating-point operations per second, or flops, but sets the bar 10 times lower at 10 to the 25th power. That covers some AI systems already in operation. China’s government has also looked at measuring computing power to determine which AI systems need safeguards.

No publicly available models meet the higher California threshold, though it’s likely that some companies have already started to build them. If so, they’re supposed to be sharing certain details and safety precautions with the U.S. government. Biden employed a Korean War-era law to compel tech companies to alert the U.S. Commerce Department if they’re building such AI models.

AI researchers are still debating how best to evaluate the capabilities of the latest generative AI technology and how it compares to human intelligence. There are tests that judge AI on solving puzzles, logical reasoning or how swiftly and accurately it predicts what text will answer a person’s chatbot query. Those measurements help assess an AI tool’s usefulness for a given task, but there’s no easy way of knowing which one is so widely capable that it poses a danger to humanity.

“This computation, this flop number, by general consensus is sort of the best thing we have along those lines,” said physicist Anthony Aguirre, executive director of the Future of Life Institute, which has advocated for the passage of California’s Senate Bill 1047 and other AI safety rules around the world.