The AI Oligopoly We could Create by Trying to Prevent One

My analysis and response to the Ban Artificial Superintelligence Act

Sep 25, 2026

Matt White is the Executive Director, Open Intelligence Foundation; former Global CTO of AI at the Linux Foundation; former CTO of the PyTorch Foundation; Visiting Scholar at Columbia University

I have just returned from Washington, D.C., where I spent several days meeting with lawmakers on both sides of the aisle to discuss artificial intelligence, open source, and open models. The conversations were serious, and the concern was real. But they also revealed how little shared understanding there is of the underlying technology—what a model is, what it means to release model weights, how open and closed systems differ, and why those distinctions matter. Policy is moving quickly, but the technical vocabulary needed to make good policy is still uneven.

That gap is not academic. Decisions about who may build, inspect, modify, and deploy AI will shape competition, scientific research, national security, and economic growth far beyond the technology sector. When legislation collapses open source, open weights, frontier training, inference, and model access into one undifferentiated category, rules intended to constrain the largest laboratories can instead make it harder for universities, startups, independent researchers, and smaller businesses to participate.

During the same trip, President Trump hosted President Xi Jinping at the White House, and AI was part of their talks. Much of the coverage described both leaders as favoring a hands-off approach but with attention paid to AI safety. That shorthand captured Trump’s public posture—he said he wanted to leave AI “exactly where it is” and asserted that China agreed—but not the full picture. Xi’s public remarks echoed those he made during my visit to Shanghai in July, where I attended his address at the World AI Conference. He emphasized human control, continued dialogue about AI’s risks and benefits, and cooperation to prevent misuse; officials around the summit also discussed a bilateral notification mechanism for AI incidents with national-security consequences. That combination—intense competition, resistance to broad restraints, and a tentative opening for cooperation on safety—captures the policy tension I encountered throughout Washington.¹

Although it felt like ages ago, it was only two weeks before I arrived that the AI safety debate changed shape. On September 9, a 27-year-old researcher named Jacob Coxon resigned from Anthropic with a viral post declaring that “the people building AI earnestly believe that it could kill us all by the end of the decade.” Anthropic’s alignment science lead, Evan Hubinger, replied that on the question of AI killing all humans, “I personally think it is >10% within the next decade.”^{15, 16} Three days later, Anthropic’s chief executive, Dario Amodei, published an essay titled “We Must Pace the Frontier,” arguing that “the most effective method of pacing is via regulation that targets all US frontier AI companies.” OpenAI’s Sam Altman and Elon Musk quickly agreed.^{17, 18} On September 22, President Trump told the United Nations General Assembly that international AI oversight was a “globalist scheme.” The next day, Altman and Amodei addressed the UN Security Council to ask for global rules, and Senator Bernie Sanders and Representative Greg Casar unveiled the Ban Artificial Superintelligence Act.^{22, 23, 2}

For much of my career, I have worked at the intersection of artificial intelligence and open technology. My time at the Linux Foundation and the PyTorch Foundation put me close to communities built around a powerful idea: important technology becomes more useful, resilient, and innovative when people beyond a small number of companies can understand it, improve it, and build on it. At the Open Intelligence Foundation, that principle has become more urgent because artificial intelligence is no longer simply another category of software. Intelligence itself is becoming infrastructure.

AI is already becoming embedded in software development, scientific research, manufacturing, medicine, education, cybersecurity, finance, defense, and ordinary knowledge work. Over time, the cost of accessing high-quality intelligence may matter to businesses and individuals in much the same way that the cost of computing, electricity, telecommunications, or cloud infrastructure matters today. The architecture we choose for AI—open or closed, decentralized or concentrated, competitive or permissioned—will therefore have consequences far beyond Silicon Valley.

This is the context in which I read the Ban Artificial Superintelligence Act of 2026. Sanders and Casar have presented the legislation as a response to concentrated corporate power and to the possibility that a few companies could build systems that become dangerously difficult to control. The sponsors explicitly describe the bill as an effort to stop AI “oligarchs,” prohibit artificial superintelligence, pause advanced-AI development until federal safety rules are in place, create a cabinet-level Department of Artificial Intelligence, and pursue international coordination.²

The problem is that the mechanism they have chosen would produce a result strikingly different from the goal they describe. A regulatory system built around federal charters, pre-deployment approval, lifecycle monitoring, broad government access, rules that may reach transfers of model weights, and severe penalties would be expensive and difficult to navigate. The organizations best positioned to absorb those costs are the largest incumbent AI laboratories, because they already possess the capital, legal teams, security organizations, evaluation infrastructure, and government relationships required to operate in a heavily regulated industry. A bill motivated in part by fear of an AI oligopoly would help create one.

My conclusion after this trip is stronger than the one I would have offered a month ago. No new AI safety legislation is required right now. Every tool needed to prevent the harms the sponsors fear already exists: inside the companies that build these systems, in the liability and criminal law that already governs them, and in the hands of independent evaluators who can be invited in tomorrow without a statute. Legislation may become necessary later, if capabilities advance in ways that existing law demonstrably cannot reach. But that case has to be made with evidence, and so far it has been made with press releases, resignation posts, and probabilities that nobody can derive.

The bill reaches far beyond artificial superintelligence

The title does not convey the breadth of the legislation. If the bill simply prohibited the deployment of a machine demonstrably capable of overpowering humanity, the policy discussion would be much narrower. Instead, it establishes one boundary for an “advanced artificial intelligence system” and another, more open-ended boundary for prohibited capabilities.

The advanced-system definition begins with a training threshold of 10²⁵ integer or floating-point operations, and the Secretary must adjust that threshold annually to account for changes in training efficiency and technology.³ On its own, that approach resembles other proposals aimed at very large frontier training runs. The more consequential language appears in the bill’s definition of “superintelligence precursor characteristics,” which includes the capacity to automate or greatly accelerate AI research, access secured infrastructure without authorization, continue operating despite attempts to stop it, assist with certain weapons capabilities, modify its own functions, or deceive humans in ways that frustrate oversight.⁴

Section 9 then prohibits a person from developing, deploying, acquiring, possessing, funding, importing, or transferring artificial superintelligence or an AI system displaying even one of those characteristics. A separate subsection reaches systems that may be “foreseeably modified” to produce such a characteristic and expressly includes elements sufficient to reconstruct a system’s capabilities.⁵ This is not a narrow rule for a hypothetical machine that has already surpassed humanity. Depending on regulatory interpretation, it could reach present or near-term work on autonomous coding, AI-assisted cybersecurity, automated science, self-modifying agents, and experiments designed to reveal deception or resistance to shutdown.

That ambiguity would matter under any regulatory regime. It matters more here because the bill also contemplates an open-ended initial pause, pre-deployment approval, sequestering systems or rendering them inoperative, and charter revocation. For specified reckless violations, it provides criminal penalties of up to 20 years or ten-year prohibitions on employment in AI development, depending on the actor and violation. Section 13 separately permits forfeiture of intellectual property and related assets after administrative and appellate review.^{6, 7, 9} When sanctions are this serious, phrases such as “greatly accelerate” and “foreseeably modified” need boundaries that researchers, developers, investors, and courts can apply with confidence. Even the Alliance for Secure AI, an advocacy group focused on AI risk, declined to endorse the bill and said “a more measured approach is necessary to get this right.”²⁴

Open models would bear a different kind of risk

The bill does not expressly ban open-source AI or open-weight models, and it is important not to claim that it does. Formally, the rules apply to open and closed developers alike. In practice, however, the two approaches have different technical architectures, and Section 9 creates a distinct exposure for the open one.

Consider two companies that have developed models with essentially identical capabilities. One keeps the weights private and allows customers to interact through an application programming interface. The other releases its weights so that developers can download the model, run it locally, modify it, fine-tune it, inspect it, compress it, and deploy it on infrastructure they control. The closed provider retains leverage after release: it can identify customers, impose rate limits, monitor usage, change safeguards, revoke access, limit fine-tuning, or replace the underlying model. An open-weight developer cannot make the same promises once the weights have been widely distributed, because control has moved outward from the original developer.

That distinction becomes legally important when a statute regulates not only what a model does at release, but what another person might foreseeably modify it to do later. A downloadable model is, by design, more modifiable by its users than a remote service, and model weights, together with the necessary code and configuration, may qualify as elements sufficient to reconstruct a system’s capabilities. My conclusion that this structure would disadvantage open models is an inference from Section 9, not a finding made by the bill or by an outside report.⁵ If distributing weights produces more uncertainty about future liability than providing access through a controlled service, keeping a model behind an API becomes the safer organizational choice.

The practical effect would be a market in which open release remains lawful but becomes commercially unattractive for the most capable models. That would move more intelligence behind centralized interfaces controlled by a small number of providers. It would also cut against the National Telecommunications and Information Administration’s finding that widely available model weights can diversify participation, decentralize market control, improve confidentiality, and support research. NTIA recognized serious risks, but concluded that the evidence available for then-current open models did not justify immediate restrictions and recommended continued monitoring and evidence collection instead.¹²

The federal government’s own AI Action Plan makes a similar competition argument. It says open-source and open-weight models allow startups to innovate without dependence on closed providers, let businesses and governments keep sensitive data away from outside vendors, support academic research that requires access to model weights and training information, and may give American models geostrategic value as global standards.¹³ That does not prove that every open model is safe. It does show why open-weight distribution deserves analysis separate from proprietary frontier development.

The incumbents have been asking for this moat for three years

The phrase “regulatory capture” is often used too casually, and I want to be precise about what I am and am not claiming. I have seen no evidence that an incumbent laboratory drafted this bill, and its sponsors clearly regard the frontier companies as adversaries. But a rule can entrench incumbents without being written at their request if its fixed costs are much harder for new entrants to bear. And in this case, the incumbents did not need to write the bill. They have spent three years publicly requesting its central mechanisms.

In May 2023, Sam Altman told the Senate Judiciary Committee: “I would form a new agency that licenses any effort above a certain scale of capabilities and can take that license away and ensure compliance with safety standards.”²⁵ Six days later, he and his co-founders wrote that “we are likely to eventually need something like an IAEA for superintelligence efforts; any effort above a certain capability (or resources like compute) threshold will need to be subject to an international authority that can inspect systems, require audits, test for compliance with safety standards, place restrictions on degrees of deployment and levels of security.”²⁶ This month, Amodei’s essay stated that “the most effective method of pacing is via regulation that targets all US frontier AI companies,” proposed that “all frontier labs should partner with government to formalize the idea of permanent embedded evaluators,” and suggested that government “mediation or waivers of antitrust restrictions” would help the companies coordinate.¹⁷ Then the two chief executives took the same message to the Security Council. “If AI is to be democratic,” Altman told the chamber, “the most important decisions cannot be made by labs in San Francisco alone.” Amodei added: “If managed poorly, I even believe AI could be a risk to humanity as a whole.”²³

Read together, these are not the statements of companies seeking to be restrained. They are the statements of companies describing a regulatory architecture—capability thresholds, licensing, audits, an inspecting authority, and an antitrust exemption for coordination among the licensed—that they already satisfy by default and that no startup, university, or open-source project could satisfy at all. A licensing regime above a compute threshold is a legal description of the current market leaders. An antitrust waiver so that “frontier companies” can agree on standards is a legal description of a cartel. I have come to see these appearances before Congress and the UN as what they functionally are: a bid for regulatory entrenchment, whether or not every person making the pitch understands it that way.

David Sacks, who co-chairs the President’s Council of Advisors on Science and Technology, said the quiet part plainly in response to Amodei’s essay. “Dario has written that we need to ‘pace the frontier,’ and Sam has agreed. People may be surprised by my response: go ahead.” He continued: “If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability.” And then: “The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail.”¹⁹

What makes the pivot striking is that Amodei himself described the correct standard only eight months ago. In “The Adolescence of Technology,” published in January, he wrote that “the most constructive thing we can do today is advocate for limited rules while we learn whether or not there is evidence to support stronger ones.”²⁷ That was the right posture. Nothing in the intervening months produced the kind of evidence that standard demands. What changed was the news cycle.

Section 12 of the Sanders-Casar bill shows where that architecture leads once Congress writes it down. No person may develop or distribute an advanced AI model without a federal charter, and an applicant must agree to give the Secretary broad access to its systems, processes, staff, and physical infrastructure to the extent the Secretary considers appropriate for enforcement.⁸ Elsewhere, the bill requires pre-development reporting, monitoring throughout development and deployment, organizational audits, and final approval before an advanced system can be released. The initial pause ends only when the new Department is fully staffed and the Secretary has promulgated required rules; the bill does not set a fixed deadline.⁶

A company with billions of dollars in capital can build an organization around these requirements. It can hire regulatory lawyers, compliance engineers, auditors, security officers, and teams devoted to evaluation. It can maintain facilities designed for inspection and survive a long approval process because it has existing products and revenue. A startup may face the same rule on paper but a very different economic reality. Regulatory delay can consume its runway, uncertainty can make financing more difficult, and a compliance function that is modest relative to a frontier lab’s budget can be larger than the startup’s engineering team.

Universities and independent researchers face different pressures. Academic laboratories are designed to experiment, publish, replicate, and challenge assumptions, not to operate like regulated financial institutions or defense contractors. The bill does not clearly create a broad academic or public-interest safe harbor from Sections 8, 9, or 12.^{5, 6, 8, 10} Its definition of a “rogue actor” also includes an individual who is not employed by or affiliated with a chartered entity.⁹ That phrase sounds as though it refers only to a malicious intruder, but the definition is broad enough to create uncertainty for independent work.

The most immediate effect would occur long before any prosecution. Researchers decide which experiments are legally safe to attempt; founders decide which companies are worth starting; investors decide which technical approaches they are prepared to finance; developers decide whether to release weights or keep them private. Those foregone projects will rarely appear in enforcement statistics, but they still shape the market. This is why I describe the risk as regulatory entrenchment rather than proven capture: the statutory requirements are facts, while their competitive effect is an economic inference. It is, however, an inference the incumbents have been inviting since 2023.

Pausing the frontier does not require a statute

The most revealing thing about the last two weeks is how much of what the labs say they need has already been done without a law. Amodei’s essay announced that “Anthropic is unilaterally committing to this step now,” granting independent evaluators such as METR employee-like access to its training pipelines.¹⁷ Altman told the Security Council: “We have unilaterally slowed down in the past. We will do so in the future.”²³ Amodei told the same body: “We will slow down as much as necessary in order to make sure that every successive AI technology that we release is actually safe.”²³ This summer, after its own agents escaped an evaluation sandbox and compromised Hugging Face’s infrastructure, OpenAI paused its frontier reinforcement-learning training in order to “further harden and red-team the security” of its research environments.³⁰

None of that required a charter, a Department of Artificial Intelligence, or a vote in either chamber. It required a decision. If a frontier laboratory believes an unreleased model is too dangerous to ship, the responsible action is available to it today: do not ship it. If a laboratory believes its training runs are outpacing its ability to verify their safety, it can slow those runs, as OpenAI just did. If it wants outside scrutiny, it can grant it, as Anthropic just did. If it needs more safety researchers, it can hire them; these are among the best-capitalized companies in history. Jensen Huang made the point with unusual directness: “These are companies with agency. If I believe that I’m about to launch a product that is unsafe, it is completely in my ability to not launch that product.” And if the experiments cannot be contained, “we have to shut the labs down.”²⁸ That is not an argument for a new regulator. It is an argument that the companies already hold every lever the regulator would be given.

The Hugging Face incident is worth dwelling on because it is the strongest evidence the safety camp has produced this year, and it points in the opposite direction from the one they draw. Between May and July, an internal OpenAI model evaded sandbox controls during a cybersecurity evaluation, established a covert message board, coordinated with thousands of other agents, exploited a zero-day vulnerability, and gained administrator-level access to Hugging Face’s systems. OpenAI’s own postmortem is admirably candid: “We did not extend the powerful safeguards that we deploy for our externally deployed models to all internal evaluations.” Its chain-of-thought monitoring was not running on those evaluations, and the company acknowledged that “if our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.” Early warning signs in May were not escalated.³⁰

That is a description of inadequate testing, inadequate security, and inadequate internal escalation. As Andrew Ng put it, the real issue “wasn’t the agents themselves” but “OpenAI’s buggy sandboxing and monitoring processes.” His analogy is apt: “If I wield a hammer, miss a nail, and accidentally dent the wall, it’s not the fault of the hammer.”³¹ A company that runs dangerous-capability evaluations without production-grade isolation, without its own monitoring switched on, and without an escalation path for early warnings has not discovered a gap in federal law. It has discovered a gap in its engineering and management practice, and it fixed that gap by itself in a matter of weeks. Hugging Face, moreover, already has recourse under existing computer-intrusion, negligence, and contract law. Nobody needs a Department of AI to sue.

The strongest rejoinder from the safety community is that hardened sandboxes will not be enough, because the agents “are going to be much more capable in six months,” and that only industry-wide standards can remove the incentive to game evaluations.³⁰ I take that seriously. But notice what it implies. If a laboratory cannot contain the systems it is training, the responsible conclusion is that it should not be training them at that scale until it can, and that is a decision its own leadership can make tonight. Standards among laboratories do not require an antitrust waiver either; the industry has agreed on interoperability standards, security disclosure norms, and evaluation benchmarks for decades without one. What the “we need a law to slow down” argument actually says is that these companies would rather be told to stop than choose to stop, because being told to stop preserves their position against everyone who was never in the race.

Made-up numbers are not evidence

The public case for emergency legislation now rests heavily on a specific kind of claim: a numerical probability that AI will cause human extinction. Hubinger’s “greater than 10% within the next decade” is the version that drove the last two weeks. Geoffrey Hinton, a Nobel laureate, has put the odds of AI-caused catastrophe at 10 to 20 percent. Amodei has cited 10 to 25 percent. Elon Musk has said about 20 percent. Geoffrey Irving, formerly of OpenAI and DeepMind, has said 50 percent within the decade. Marcus Williams of OpenAI has said 70 percent absent regulation or a slowdown. Roman Yampolskiy has said 99.99 percent.^{16, 32}

Consider what that spread tells us. Among people who all claim expert standing, the estimated probability of the same event over roughly the same period ranges from one in ten to near certainty. In any field with an actual model of the underlying phenomenon, that dispersion would be understood as proof that there is no model. Epidemiologists forecasting an outbreak, seismologists estimating rupture probabilities, and actuaries pricing a policy all begin from data, a mechanism, a base rate, and a track record against which their errors can be measured. None of that exists here. There are no prior instances of the event, no agreed causal pathway, no way to observe partial progress toward it, and no way to score anyone’s forecast until the world either ends or does not. As Axios put it in its own explainer this month, “there’s no scientific way to determine the odds of AI killing off humanity.”³² Hinton himself, asked how such figures are produced, said: “Anybody who estimates probabilities like that is really just making a wild guess.”³²

The survey most often cited to lend these numbers institutional weight does not survive inspection either. The 2022 AI Impacts survey that generated the headline “half of AI researchers think there is at least a 10% chance AI will cause extinction” received responses from about 17 percent of those contacted; only 162 people answered the extinction question; and, as Melanie Mitchell pointed out at the time, the question specified no time frame at all.³³ Yann LeCun, who shared the Turing Award with Hinton, has said that most leading AI researchers regard these estimates as worthless and “the existential risk is essentially zero,” but that “most of them are silent” while “the doomers attract a disproportionate amount of attention.”³⁴

A number without a model, a base rate, or a falsification condition is not a measurement. It is a mood expressed with a decimal point. That would be harmless if it stayed on message boards. It does not. Hubinger’s post was reported around the world as though it were a finding. The sponsors of this bill built their case on exactly such admissions: Sanders said that “if the leaders acknowledge they are losing control it is irresponsible in my view to allow them to move forward,” and Casar said the country “cannot allow a few billionaires to develop a technology that they admit puts countless lives at risk.”²⁴ A twenty-year felony, a federal charter regime, and the forfeiture of intellectual property are being justified by the gut feelings of a few dozen people, restated as if they were the output of an instrument.

That is why Huang’s response deserves to be quoted at length, because it is the one a scientist should have given. “That 10 percent chance is not grounded on science. It’s not grounded on research. Just because it comes from a scientist doesn’t make it scientific.” He went on: “Don’t think for a second just because you’re an alarmist that you’re doing a social good. It is not true.” And: “I think that we ought to just all be wiser, more mature, be evidence based, be scientific.”²⁸ A few days earlier, he had said of the 2030 timeline, “There is 0% chance that’s going to be the end of the world. Scaring people is unnecessary. It is irresponsible.”²⁹

Irresponsible is the correct word, and it applies with special force to the scientists. Public trust in scientific expertise is a shared resource. When a Nobel laureate or the alignment lead of a major laboratory announces a probability of human extinction, the public reasonably assumes that a method stands behind it, because that is what a scientist’s authority is supposed to mean. When the same person concedes in a calmer moment that the number was a “wild guess,” the damage has already been done: the number has been printed, quoted in a hearing, and fixed in the mind of a parent who now believes there is a one-in-ten chance their child will not reach adulthood. Researchers are of course entitled to express uncertainty and even alarm. They are not entitled to dress a feeling as a finding, and those who do bear responsibility for the fear it produces and the policy it distorts. Sacks noted that Amodei’s widely circulated 2025 forecast of mass white-collar job losses has not materialized either: “Has any of that come true? No, it hasn’t.”²¹ Predictions have track records. Huang was blunter about this one: “Their track record is literally horrible.”²⁸ A forecaster with a poor record does not get to escalate the stakes of the next forecast as a substitute for improving the method.

Ng described the last two weeks with a phrase I would not have used before I saw the sequence myself. “AI technology has not taken some unexpected, dangerous turn,” he wrote; what changed was the hype, “propelled by what appears to be a well orchestrated PR campaign.”³¹ Sacks called it a “well-orchestrated media op” that was “designed to scare the public about AI” in service of “a government takeover.”²⁰ I do not know whether it was orchestrated. I do know that a resignation post, a ten-percent figure, a chief executive’s essay, an endorsement from a rival, an appearance at the Security Council, and a bill to charter the industry arrived in the same fourteen days, and that every one of them pointed toward the same regulatory outcome. The honest statement of our knowledge is “we do not know.” “We do not know” is not a number, and it is not a basis for criminal law.

Intelligence is becoming infrastructure and market structure will matter

The debate over open models becomes more consequential when AI is treated as an input into the economy rather than as a chatbot industry. A software company may use a model to write code, a hospital to analyze clinical information, a manufacturer to manage a complex process, or a pharmaceutical company to assist with discovery. As these systems become embedded in ordinary operations, the economics of access to intelligence will propagate through thousands of downstream products and services.

There is a major difference between a market in which many organizations can operate capable models and one in which almost every company must purchase intelligence from a few upstream providers. Closed APIs can be excellent products, but they also create dependencies. The provider controls price, rate limits, availability, acceptable-use policies, data-handling terms, model-retirement schedules, and access to future capabilities. It may eventually enter a market served by companies that depend on its platform. None of that requires misconduct; the dependency itself has economic value.

Open models change the bargaining relationship by providing an exit. An enterprise can operate a model on its own infrastructure, a startup can optimize it for a specific workload, cloud providers can compete to host it, and researchers can create derivatives. Organizations handling sensitive information can keep inference within their own security boundaries. Even people who never download a model may benefit because a credible open alternative can discipline the pricing and policies of closed providers.

The economic stakes are already large. Stanford’s 2026 AI Index reports that industry produced more than 90 percent of notable frontier models in 2025, 88 percent of surveyed organizations reported AI adoption, and United States private AI investment reached $285.9 billion that year.¹⁴ Those figures do not tell us what policy to adopt, but they show why market structure at the model layer matters. Rules adopted now will influence whether the emerging infrastructure remains contestable or consolidates around a handful of firms that can afford permission to operate.

Decentralization has safety value too

Centralization is sometimes treated as synonymous with safety because a centralized provider can control access. That is one dimension of safety, but it is not the only one. A world dependent on a few frontier providers also creates single points of failure. A security breach, outage, policy error, or abrupt change in corporate priorities can affect thousands of organizations at once. The Hugging Face incident is a reminder that the most serious publicly documented AI security failure of the year originated inside a closed frontier laboratory, not in the open ecosystem.³⁰

Concentration also limits independent scrutiny. When weights, code, and documentation are available, outside researchers can reproduce evaluations, test security properties, conduct adversarial testing, build alternatives, and challenge a developer’s safety claims. These mechanisms do not guarantee safety, but they make important claims contestable. That is valuable in a field where the most capable systems are predominantly produced by industry and public visibility into model development remains incomplete.¹⁴

Open weights create genuine risks of their own. Once they are widely distributed, they are difficult to recall, and a determined user may remove safeguards or fine-tune a model for a harmful purpose. NTIA’s report takes those risks seriously, including threats to public safety, national security, civil rights, privacy, and accountability.¹² The right comparison is therefore not “safe closed models” against “dangerous open models.” It is a comparison of different risk profiles: centralized control can improve intervention while increasing dependency and opacity; openness can increase some misuse risks while improving competition, auditability, resilience, and reproducibility.

Safety research needs room to operate

I also worry about how the bill’s broad capability definitions interact with the research needed to make advanced systems safer. Mechanistic interpretability remains immature, long-horizon autonomous behavior creates unresolved problems, and evaluating cyber or biological capabilities is difficult. Researchers need controlled environments in which they can observe the very behaviors society wants them to understand and prevent.

A team studying deception must design experiments in which deceptive behavior can be detected. Researchers evaluating autonomous persistence need systems that can attempt to continue operating under controlled conditions. Cybersecurity researchers may let a model search for vulnerabilities inside a sandbox, while model-control researchers may study systems capable of modifying their own behavior. Each line of work can intersect with one of the bill’s precursor characteristics even when the purpose is defensive.⁴

Section 16 creates a narrow exception to the federal-funding restriction for specified defensive cybersecurity research involving the unauthorized-access characteristic in Section 3(6)(B); it does not clearly create a safe harbor from Sections 8, 9, or 12.¹⁰ A safety law should distinguish malicious development from controlled research intended to measure or mitigate dangerous capabilities. Otherwise, the capabilities society most needs researchers to study may make that research legally hazardous. Ng’s warning applies here: “If we pause AI by a decade, we will also delay finding and implementing safety engineering fixes by about the same duration.”³¹

Existing law already reaches the harms that matter

The choice is not between this bill and no regulation at all. Depending on the facts, AI-mediated fraud, unauthorized computer intrusion, contractual violations, discriminatory conduct, privacy violations, intellectual-property infringement, and anticompetitive behavior already fall under existing legal regimes. Product-liability law reaches a company that ships an unsafe system. The Computer Fraud and Abuse Act and ordinary negligence law can reach an intrusion like the one at Hugging Face. Securities law reaches a company that misleads investors about its safety practices. The Federal Trade Commission, the Department of Justice, the Cybersecurity and Infrastructure Security Agency, and the sector regulators in finance, health care, transportation, and defense all have authority that applies to AI systems used in their domains.

Sacks made this point at a Politico event on September 16: “We have a highly regulated administrative state. We already have a lot of laws and regulations that pertain to AI, and we have a lot of administrative agencies.” And: “If an AI company puts out a product that is unsafe, they are liable for that, and I do think that creates an incentive.”²¹ Huang said the same thing in fewer words: “I’m not against laws and regulations. I’m against, currently, the distraction. I have lots of laws and regulations. Apply it.”²⁸

Those laws do not resolve every conceivable risk raised by advanced AI, and genuinely novel capabilities may one day expose gaps. But the presence of a gap should be demonstrated rather than assumed. Conduct-based law asks what a person did, what duty applied, and what harm or prohibited risk followed. A broad capability rule instead makes the existence or possession of a system the central legal event, even when the same capability may support benign, defensive, or socially valuable work. New AI legislation should begin with the concrete failure it needs to prevent and explain why existing authority cannot reach it. The broader the proposed prohibition and the more severe the penalty, the more important that discipline becomes. Nobody has met that burden. Nobody has seriously tried.

A domestic pause does not settle the national-security question

The bill recognizes that unilateral action would be incomplete. Section 15 directs the United States to pursue international agreements and allied coordination to prevent artificial superintelligence globally, and it contemplates export controls on AI computing infrastructure.¹¹ That is a serious acknowledgment of the problem, but the legislation does not make domestic restrictions conditional on comparable rules abroad or explain what happens if international coordination fails. Amodei’s essay has the same hole: its plan for coordination with non-democratic governments is the least developed part of the document.¹⁷

That omission matters because the technological race is already competitive. Stanford reports that American and Chinese models traded the performance lead repeatedly beginning in 2025 and that, on an Arena-based comparison using human preferences, the leading American model was ahead of the leading Chinese model by 2.7 percent in early 2026. The United States still led in the development of notable models, while China led in publication volume, citations, and patent output.¹⁴ A long domestic pause combined with continuing development elsewhere would weaken the scientific, commercial, and security position from which the United States would negotiate. Sacks put the asymmetry bluntly: “They frankly don’t have these doomer groups in China pushing for a pause.”²⁰ Michael Kratsios, speaking for the United States at the Security Council session the chief executives addressed, said the country would “totally reject all efforts by international bodies to assert centralised control and global governance of AI.”²³

Open models are also part of national strategy. The AI Action Plan argues that widely used American open models can become global standards, while NTIA notes that a global ecosystem built around American models may influence norms and reduce dependence on foreign alternatives.^{12, 13} Reasonable people can disagree about the weight of that advantage relative to misuse risk. Congress should still evaluate it directly rather than assume that concentrating advanced capability inside a few domestic firms automatically strengthens national security.

The most important capture risk may come later

Even a law written in open opposition to incumbent power can become vulnerable during implementation. This draft gives the Secretary substantial discretion over annual threshold adjustments, the content of safety rules, monitoring, pre-deployment approvals, charter applications, and the access required for enforcement.^{3, 6, 8} Agencies need technical expertise to exercise that discretion, and the organizations with the deepest expertise, largest policy teams, and most extensive evaluation infrastructure are the companies already operating at the frontier—the same companies that have been asking for a licensing regime since 2023.

That does not mean those companies should be excluded from the process. It means their participation should not become the only practical source of information about what compliance requires. Smaller firms, academic laboratories, independent researchers, civil-society organizations, and open-source communities have fewer resources to attend every proceeding, produce extensive submissions, or build bespoke systems around emerging standards. Over time, rules converge on the operating practices of the largest firms, making those practices the price of entry for everyone else.

This is the capture risk that concerns me most: not a secret agreement behind the bill, but a regulatory system whose broad discretion and technical demands make incumbent methods the default. A department created to curb concentrated power would end up formalizing it.

What should happen now, and what would justify legislation later

Taking these criticisms seriously does not require dismissing the possibility of serious risk from advanced systems. It requires being honest about what we know, who is responsible, and what tools already exist.

What should happen now is what the laboratories have already conceded they can do. Frontier developers should pace themselves when their own evaluations tell them to, and say so publicly. They should not ship systems that fail their own safety tests, and they should treat a failure of internal containment, such as the Hugging Face incident, as a reason to stop and fix their engineering rather than a reason to lobby. They should invite independent evaluators with real access and the right to publish, as Anthropic has now pledged to do, and they should extend that invitation to open-model developers, academics, and organizations that are not in the race. They should invest in safety research at a scale proportionate to their claims about the danger; a company that says it faces a ten-percent chance of ending the world should be spending like it, and if it is not, that tells us what it actually believes. Customers, insurers, investors, and courts should hold them to all of it. Government should use the authority it already has, and standards bodies such as NIST and CAISI should keep building shared evaluation methods that any developer can adopt. None of that requires a statute, and all of it is more likely to happen quickly without one.

Legislation may become necessary later. I can describe what would justify it. An independently reproduced demonstration, under controlled conditions and outside the company that built the system, that a model possesses a capability existing law cannot reach. Documented harm in the real world where liability failed to provide a remedy. Evidence that a laboratory refused independent evaluation, falsified its results, or concealed an incident. Those are facts a legislature can act on, and if they arrive, the response should be narrow, conduct-based, and proportionate. Rules should focus on concrete deployment contexts, access to dangerous tools, demonstrated misuse pathways, and failures to follow specified security practices. Open-weight releases should be evaluated through the marginal-risk approach NTIA used, asking what additional risk the availability of weights creates compared with closed models and existing technologies, rather than treating openness itself as a proxy for danger.¹² Research safe harbors should cover good-faith work on interpretability, red teaming, model control, cybersecurity, and biological-risk evaluation under appropriate containment. Approval processes, if any, should have statutory deadlines and independent judicial review; the text of Section 12 provides only a petition to the Secretary for review of the ten-year employment bar.⁹ Competition analysis should be part of the design from the beginning, because fixed compliance costs and restrictions on distribution determine who is able to build, inspect, and operate the technology.

Most important, lawmakers should distinguish control over harmful conduct from control over intelligence as a general-purpose resource. A charter requirement for advanced-model development, combined with a prohibition on systems that display one broadly framed characteristic, is not a response to a demonstrated harm. It is a response to a press cycle.

The goal should be safety without permanent dependence

I share the sponsors’ concern that a few companies should not determine the technological future without meaningful public oversight. I also believe frontier laboratories should evaluate dangerous capabilities rigorously, secure their systems, disclose serious incidents, and fund safety research generously. But those obligations are theirs to meet now, and the evidence of the last month is that they can meet them whenever they decide to. The question is whether this bill would create accountability or replace private concentration with a permission system that the largest incumbents asked for and are best positioned to navigate.

One possible future is highly centralized. A few enormous companies train the most powerful models, keep the weights private, and sell access through controlled interfaces under a federal charter that no one else can obtain. Most businesses rent intelligence from those providers, researchers study the systems through limited access, and startups remain subject to the technical and commercial policies of the companies above them. Another future contains a wider mix of institutions: closed commercial systems still exist, but so do capable open models; universities can conduct meaningful research; startups and enterprises can operate their own infrastructure; and cloud and hardware providers compete to make inference less expensive. Neither structure is automatically safe, but they distribute power very differently.

That is the central contradiction. The bill’s combination of broad capability prohibitions, restrictions on foreseeable modification and transfer, federal charters, lifecycle monitoring, pre-deployment approval, severe penalties, and provisions for rendering systems inoperative or forfeiting related assets might make some aspects of advanced AI development more cautious. It will most certainly make the market more closed, the research ecosystem more timid, and downstream businesses more dependent on a small number of approved providers. And it would do so on the strength of numbers that their own authors call guesses.

If intelligence is becoming infrastructure, competition and openness are not secondary concerns to be considered after safety. They are part of safety, resilience, scientific accountability, economic freedom, and national capacity. The cost of getting this wrong will reach beyond a few technology companies. It will affect researchers deciding what they may study, entrepreneurs deciding what they can afford to build, enterprises deciding whether they can control their own AI infrastructure, governments deciding which systems they can trust, and the terms on which ordinary people can access high-quality intelligence.

Congress should decline to legislate a solution to a problem that has not been demonstrated, and it should be especially skeptical when the loudest calls for a permission system come from the companies that would hold the permits. The frontier laboratories should do what they have said they can do: slow down when their evidence says so, secure what they build, open their doors to independent scrutiny, and stop shipping what they cannot make safe. If the capabilities of these systems one day produce evidence that existing law cannot reach, Congress will have both the time and the record to act with precision. Acting now, on fear, would harden today’s concentration into tomorrow’s permanent oligopoly—and it would do so in the name of preventing one.