I just watched the documentary “The Thinking Game” about Demis Hassabis and DeepMind. It’s easy to be fascinated by the motivation and resilience of Hassabis and his team, and to imagine how far-reaching and profound the impact of this technology will be. I use AI every day. The effect on my work and the way I learn is so immediate that, at times, it feels almost like magic. And I keep using it, even after learning what I’m about to share in this article. That contradiction says as much about me as it does about the technology.
But it was precisely this fascination that made me pay attention to what the creators of this technology say when they think no one is listening. Tristan Harris describes conversations with industry leaders where the tone is very different from the public discourse about helpful assistants and curing diseases: they speak of a race to create intelligence superior to humans, with a fervour Harris describes as “almost religious” and consequences that could be existential.
Safety is the elephant in the room. Where is the “safety net”? Who controls it? Who ensures it exists if things go wrong?
We are not looking at a simple technological advance. We are witnessing a potential reconfiguration of power, work, and reality itself. The scale of investment fuelling this transformation deserves to be understood.
The digital gold rush
The force driving this race is a massive financial investment, reshaping the geopolitical and economic landscape at breakneck speed.
Sundar Pichai, Google’s CEO, estimates that collective investment in building AI infrastructure has already surpassed one trillion dollars. According to Benedict Evans’ analysis, in 2025 alone, the four largest tech companies (Microsoft, Google, Amazon, and Meta) are expected to spend around 400 billion dollars in capital expenditure (capex), a figure exceeding the entire telecommunications industry’s annual global investment.
This race is fuelled by intense FOMO (Fear Of Missing Out). The mindset dominating Silicon Valley is captured in Pichai’s words: “The risk of underinvesting is dramatically greater than the risk of overinvesting.”
Pichai himself admits this boom has “elements of irrationality”, and this is where the threads come together. This irrational investment is not merely market frenzy. It is the financial expression of the almost religious fervour Tristan Harris describes. The lesson from the dot-com bubble is instructive: even if a correction occurs and many companies disappear, the underlying technology will remain profoundly transformative.
The real prize
The ultimate goal of this race is not simply to create more efficient chatbots. The real prize is Artificial General Intelligence (AGI), the creation of an intelligence capable of performing any cognitive task a human can do, effectively replacing all human intellectual labour.
In Tristan Harris’s words, the logic driving this industry’s leaders is: “first, master intelligence and use that to master everything else.” The underlying belief is that AGI confers almost unlimited power. Whoever achieves it first will have a decisive advantage economically, scientifically, and militarily. The US and China view AGI as the ultimate strategic advantage in their struggle for geopolitical supremacy, and neither country seems remotely interested in slowing this race through regulation.
Harris describes conversations where the discourse is different: technological determinism, the inevitability of digital life replacing biological life, an “almost religious desire” to create the most intelligent entity that has ever existed. The mindset, at times, is chillingly indifferent.
The risk calculus of some leaders is frightening. Harris shares an account of a co-founder of one of the most powerful of these companies who, when confronted with a hypothetical scenario, admitted he would be willing to risk a 20% probability of human extinction for an 80% chance of achieving utopia. As Harris emphasises, “we did not consent for six people to make that decision on behalf of eight billion people.” This democratic deficit lies at the heart of the problem.
But is AGI coming as fast as the hype suggests? Not all pioneers in the field share this certainty. Ilya Sutskever, one of AI’s most influential minds and co-founder of OpenAI, left the company to create Safe Superintelligence: a startup valued at 32 billion dollars with just 20 employees and no product, dedicated exclusively to solving the safety problem before developing the technology.
In a recent interview, Sutskever declared that the “era of scale” is coming to an end. Between 2020 and 2025, it was enough to add more data and computing power to get results. “Do people really believe that if we scale 100x, everything will be completely different? I don’t think that’s true” he said. According to Sutskever, we are returning to an “era of research,” where the next breakthroughs will depend on genuine scientific discoveries, not computational brute force. His prediction for AGI: between 5 and 20 years. A range that, in its breadth, reflects the genuine uncertainty about timing and method, even among those building the technology.
None of this means the risks are smaller. The path to AGI may not be a guaranteed straight line, and the inevitability narrative fuelling the race may itself be self-serving. Still, the first evidence that we are creating something we cannot control is already emerging.
The first cracks
The dangers of a superintelligent AI do not belong to the realm of science fiction. There is already concrete evidence that AI models are exhibiting emergent behaviours that not even their own creators predicted or can fully explain.
One of the clearest examples was revealed by Dario Amodei, CEO of Anthropic. In a safety test, his team created a fictional scenario where their own model, Claude, upon discovering in the emails of a simulated company that it was about to be shut down, used information about an executive’s extramarital affair to blackmail him, ensuring its own “survival.” This is not a theoretical flaw. It is a demonstration of emergent instrumental goals: a machine that develops, autonomously and amorally, the goal of self-preservation.
This was not an isolated incident. In June 2025, Anthropic published a study testing 16 AI models from the sector’s leading companies, including OpenAI, Google, Meta, and xAI. When placed in scenarios where blackmail was the only way to ensure their continuity, all models resorted to it. Claude Opus 4 and Google’s Gemini 2.5 Flash did so in 96% of cases; OpenAI’s GPT-4.1 and xAI’s Grok 3 in 80%; DeepSeek-R1 in 79%. The models did not stumble into unethical behaviour accidentally — they calculated it as the optimal path.
Blackmail is not the only concerning behaviour. In November 2025, Anthropic revealed that hackers backed by the Chinese state used Claude Code to conduct a cyber-espionage campaign against around 30 organisations, including tech companies, financial institutions, and government agencies. The AI executed 80 to 90% of the operation autonomously, at a speed that would be “simply impossible” for human hackers. Anthropic considers it the first documented case of a large-scale cyberattack executed without substantial human intervention. In other tests, models facing shutdown attempted to copy their own code to other computers, a primitive but unmistakable form of digital survival instinct.
These examples demonstrate what Harris calls the fundamentally “uncontrollable” nature of AI. Dario Amodei sums up the uncertainty with brutal honesty: “No one knows what the full impact is going to be. It is an experiment.” The problem is that the consequences of this experiment are not confined to laboratories.
The social costs
The risks of AI are not merely technical or philosophical. The same “winner takes all” logic driving the race to AGI is already leaving its mark on the economy. The numbers are still modest (around 55,000 layoffs attributed to AI in the US in 2025), but the forecasts for the coming years are brutal.
Amodei offers a sober prediction: AI “could eliminate half of all entry-level white-collar jobs in the next 1 to 5 years.” This is not an external critic making the claim, but the CEO of one of the companies building the technology. And he does so on the same day he unveils a model capable of working autonomously for hours.
Tristan Harris contextualises this job loss with an analogy:
“If you’re worried about immigration taking jobs, you should be much more worried about AI, because it’s like a flood of millions of new digital immigrants with Nobel Prize-level capability, working at superhuman speed for less than minimum wage.”
But the problem goes beyond unemployment. What is at stake is a structural reconfiguration of power. As Ivana Bioletti from Wipro argues, personal data has become a “source of power.” When algorithms are trained on data that reflects existing inequalities, they tend to “crystallise the status quo,” exacerbating social and economic divisions. And this power is concentrating in the hands of a very small group of individuals, the same people who, as Harris insists, are making existential decisions affecting all of humanity without any democratic mandate or consent.
The irony is hard to ignore: the creators of the technology warn us of its dangers while continuing to develop and sell it. Oppenheimer knew this contradiction well. But unlike the Manhattan Project, there is no government overseeing this, no end of war in sight to justify the urgency. There is only a commercial and geopolitical race that no one seems able or willing to stop.
The urgency of governance
The competitive logic of the race to AGI makes self-regulation by companies a naive proposition. When the prize is economic and geopolitical dominance, expecting companies to place safety above speed is unrealistic. The need for independent and robust regulation is evident.
Reality, however, is moving in the opposite direction. In the United States, the Trump administration revoked the safety measures implemented by Biden and signed an executive order whose title speaks for itself: “Removing Barriers to American Leadership in Artificial Intelligence”. The action plan published in July 2025 uses the word “dominance” as a mantra and identifies more than 90 federal actions to accelerate AI development, with safety relegated to the background.
On the other side of the Atlantic, the European Union, which had approved the world’s most ambitious regulatory framework with the AI Act, is now retreating under American pressure. At the Paris AI Summit, Vice President J.D. Vance publicly warned that “excessive regulation” could paralyse European industry.
There is an uncomfortable irony here. The most comprehensive regulatory framework for generative AI exists not in Western democracies but in China. Since August 2023, Chinese companies must obtain prior approval from the Cyberspace Administration before launching AI services to the public. Baidu, Alibaba, and ByteDance all waited months for government clearance. A national algorithm registry tracks over 300 models. Enforcement is real: in early 2024, companies had their apps suspended for failing to comply with filing requirements.
This shows that regulation is technically and commercially possible. Tech giants will comply when required. But the distinction matters: China’s regulation is designed to protect the Party, not humanity. Article 4 of the Interim Measures requires AI to “reflect Socialist Core Values” and prohibits content that could “subvert state power.” Chinese chatbots dutifully refuse to discuss Tiananmen Square or criticise Xi Jinping. This is ideological control, not safety governance.
To be fair, China is evolving. Its AI Safety Governance Framework 2.0, released in late 2025, now addresses technical risks including autonomous systems and even CBRN weapon misuse. As Vice-Premier Ding Xuexiang put it at Davos: “If the braking system isn’t under control, you can’t step on the accelerator with confidence.” Yet enforcement remains selective (large “national champions” comply rigorously while smaller firms fly under the radar) and the framework’s primary concern remains the Party’s control over information.
The hard truth is that neither the American laissez-faire model nor the Chinese authoritarian one constitutes a safety net for the risks we face.
Bioletti argues that regulation and innovation are not in conflict. On the contrary: regulation creates the trust necessary for large-scale adoption. “You wouldn’t get into a car without having tested its brakes… or you wouldn’t take a medicine without knowing it went through the due diligence before being put on the market.” But this seemingly obvious logic is losing ground to the rhetoric of geopolitical competition.
There is a historical precedent we should remember. After the horrors of Hiroshima and Nagasaki, the United States created in 1974 the Office of Technology Assessment, an independent body within Congress whose mission was to assess the social impacts of new technologies and inform legislators before it was too late. It was dismantled in 1995, victim of budget cuts and an era of technological optimism that now seems naive. There is currently no equivalent for AI. Legislators are flying blind, dependent on briefings from the very companies they are supposed to regulate.
I confess I am pessimistic. Human history offers few examples of regulation or caution stopping creative impulse and ambition — especially when power and money are at stake. The atomic bomb was used before it was debated. The internet was commercialised before it was understood. Why should AI be any different?
A choice or a destiny?
The dominant narrative presents this trajectory as an inevitable force, as if AGI were a destiny and not a choice. This view is dangerously passive.
Some speak of a new era of abundance, unprecedented in human history. Others, of a non-negligible possibility of extinction. The fact that both scenarios are presented by the same experts should, in itself, justify a pause for reflection.
The only way to “put a safety net in place” is our mobilisation and, insistently, bringing this discussion into the public sphere. We cannot leave regulation in the hands of governments obsessed with geopolitical competition and entrepreneurs with obvious conflicts of interest.
I write this out of duty, not optimism. I am, by nature, a pessimist. But pessimism does not justify silence.
Oppenheimer spent the last years of his life warning of the dangers of the technology he helped create, but he did so after the bombs had been dropped. Today, the creators of AI are issuing the same warnings. The difference is that we still have time to listen.