When I cloned Mario with an LLM, I said it was more powerful than I thought it would be. What I took seriously was the accuracy and the speed. The jump physics felt right far sooner than I expected, and I never had to explain what coyote time is, because it already knew that’s what makes a jump feel forgiving. A system that is that accurate that quickly on a game can be that accurate that quickly on anything made of text and code. This post is where that thought leads.
The people who built it are worried
I’m not the first to notice, and the loudest warnings come from inside.
Geoffrey Hinton left Google in 2023 and said why: “I want to talk about AI safety issues without having to worry about how it interacts with Google’s business.” About the systems themselves he said, “Right now, they’re not more intelligent than us, as far as I can tell. But I think they soon may be.” (AP)
Yoshua Bengio started a nonprofit, LawZero, because “current frontier systems are already showing signs of self-preservation and deceptive behaviours, and this will only accelerate as their capabilities and degree of agency increase.” He describes it as insulated from “market and government pressures.” (LawZero)
The most recent voice is Jacob Coxon, a former pretraining researcher at both Anthropic and OpenAI. He resigned from Anthropic on September 8. On October 6 he told Jon Stewart on The Daily Show that the frontier labs are “racing straight to self-improving superintelligence and gambling with our lives,” and that same day told the NYC Council “we do not know how to control any AI system yet.” He also argued that a real slowdown would need international coordination, including China. (AI Weekly · IBTimes)
Put those together. The people who built these systems say they can’t reliably control them, and the companies building them are still racing.
Who decides when it is too dangerous
Paul Triolo makes a point I keep coming back to in the Cairo Review. Once a lab says a model is too dangerous for general release, it has conceded that some capabilities need to be controlled by someone other than the customer. Then the question is who decides to release, delay or restrict a frontier model. His line is that “the battleground is no longer ownership but authority.” (Cairo Review)
Right now that authority belongs to a private company. A board decides, with investors and competitors watching. The public gets whatever answer the company reaches.
The promise and the plumbing
Coxon described the future the builders have in mind: no jobs, with the benefits of automated production shared out. He said it as a description of their plan, and he said it with a warning attached.
Zack Exley’s series explains why the sharing part can’t be assumed. His argument starts from how AI differs from earlier automation. Earlier tools needed people to run them, and the new industries they created needed people too. Exley argues AI can staff the new industries itself, so there is nowhere for displaced workers to go. Competition forces every firm to adopt it, because an executive who refuses gets undercut by a competitor who doesn’t. (part 1)
Then comes the mechanism that matters most. In his words, “Capitalism distributes goods through paychecks. AI eliminates the paychecks.” A universal basic income doesn’t fix it under private ownership, he argues, because the producers take a cut every cycle and the money slowly drains out. His answer is public ownership of productive capacity, and he admits the hard question that follows is distribution: “Politics has to answer it.” (part 4)
Read the two side by side. If the shared-benefits future arrives, someone has to do the sharing. Whoever owns the most capable systems owns what they produce. A promise to distribute the gains is only as good as the owner’s willingness to keep it, and nothing in how these systems are being built obliges them to.
It was built from what we all wrote
Language models learn from text, and the text came from us. Meta’s published description of its LLaMA training data is a concrete example: most of it is web pages scraped through Common Crawl, with Wikipedia in 20 languages, public code on GitHub, two collections of books, scientific papers from arXiv and Stack Exchange questions and answers. (LLaMA paper) The leading commercial labs don’t publish this level of detail, and the RedPajama authors note that “many of the top-performing models lack transparency in their dataset curation.” (RedPajama)
Some of it was taken in ways even the courts objected to. Anthropic agreed to pay $1.5 billion to settle a class action by authors after downloading more than 7 million books it “knew had been pirated,” according to the judge. The same judge ruled that training on copyrighted books wasn’t inherently illegal, only the way those copies were acquired. (AP via Inquirer)
That ruling matters for my argument. It means the law, at least in that court, doesn’t give the public a claim to what was learned from the books. If there is a claim, it has to be made politically. Mine is that the written record of humanity was gathered over thousands of years by people from everywhere, and a handful of companies turned it into a product. The record belongs to everyone who wrote it and everyone who didn’t. Not only Americans. All people.
Rob May points out that the public already helps pay for these systems through research grants, power infrastructure and defense contracts, and asks: “why should 100% of that equity appreciation flow only to private investors?” (Investing in AI)
How you get from here to public ownership
Nobody serious expects a sudden takeover, and I’d rather lay out the steps than pretend otherwise.
Cheng and Katzke call the first phase soft nationalization: the government gains control step by step through oversight, compute restrictions and personnel rules, with total takeover as a last resort. (LessWrong) Triolo says this is already happening, in three stages. Visibility comes first, with mandatory reporting, model evaluations and compute disclosure. Then conditional permission, with gated deployments and limits on releasing model weights. Then operational integration, with classified work and defense procurement. He expects formal nationalization only if those stages fail or a crisis forces it. (Cairo Review)
The objections are real, and they come from people thinking about the same problem:
- Government stakes. May’s version trades land, power and clearances for equity in a public wealth fund. His own warning is that “give the government a financial stake in the success of a company, and you have fundamentally compromised its ability to regulate that company.” (Investing in AI)
- Coercion backfires. John Allard argues that “the frontier is a living process, not an asset.” Much of the capability is tacit knowledge in researchers’ heads, researchers can leave, and pressure can start a slowdown that leaves the government controlling a lab that has fallen behind. His conclusion is that control will come through “patience rather than force,” as labs automate their own research and depend less on specific people. (Allard)
I take both seriously, and I still land where I do. Allard’s endpoint is that control becomes possible once the labs no longer need their people. That’s also the point where who owns them matters most.
I’d go as far as full nationalization, and it has to mean public ownership for everyone, not ownership by one government against another. Triolo notes that each side reads the other’s soft-nationalization measures as a step toward militarization. Nationalizing frontier AI inside a single country feeds that. He endorses an IAEA-style international body, which would be the nearest existing model to a system owned by everyone. (Cairo Review)
Read more
- Cheng and Katzke, Soft Nationalization: How the US Government Will Control AI Labs
- Paul Triolo, Will Washington and Beijing Nationalize AI Labs? (Cairo Review)
- John Allard, Can You Nationalize a Frontier AI Lab?
- Rob May, Should the Government Take Stakes in Frontier Labs?
- Zack Exley, Why Capitalism Can’t Survive AI: part 1 · part 2 · part 3 · part 4