Beyond human control

THE RACE FOR ARTIFICIAL GENERAL INTELLIGENCE POSES NEW RISKS TO AN UNSTABLE WORLD

Beyond human control Under a crystal chandelier in a high-ceilinged anteroom in Paris, the moderator of Intelligence Rising is reprimanding his players. These 12 former government officials, academics, and artificial intelligence researchers are here to participate in a simulated exercise about AI’s impact on geopolitics. But just an hour into the simulation, things have already begun to go south. The team representing the U.S. has decided to stymie Chinese AI development by blocking all chip exports to China. This has raised the odds, the moderator says, of a Chinese invasion of Taiwan: the U.S. ally that is home to the world’s most advanced chip- manufacturing plants. It is 2026, and the simulated world is on the brink of a potentially devastating showdown between two nuclear superpowers. Why? Because each team is racing to create what’s known as artificial general intelligence, or AGI: an AI system so good, it can perform almost any task better, cheaper, and faster than a person can. Both teams believe getting to AGI first will deliver them unimaginable power and riches. Neither dare contemplate what horrors their rival might visit upon them with that kind of strength. While this scenario might seem farfetched, many insiders say it is anything but. Top technologists now believe that AGI is within touching distance. Sam Altman, the CEO of ChatGPT maker Open AI, expects the first AGI to be created during President Trump’s second term in office. OpenAI, Google, Microsoft, Amazon, and Meta are together funneling hundreds of billions of dollars—the equivalent cost in today’s dollars of a dozen Manhattan Projects per year—into the construction of huge data centers where they believe AGI will be summoned into existence. Artificial general intelligence, believers say, could far surpass human limitations: it could have expert knowledge in all fields, not just one or two; it could complete in minutes complex tasks that take human workers hours or even weeks; and it could be replicated, thus enabling the creation of virtual armies of AI “agents.” That kind of computational intelligence could be compared to a “country of geniuses,” the CEO of AI company Anthropic, Dario Amodei, wrote last year. These AI systems could begin to automate much of the $100 trillion- plus global economy, delivering huge returns for those lucky enough to control them. They could also be set to task curing disease, discovering new technologies, and hastening the global transition to a green economy, according to their most optimistic proponents. But the dawn of AGI will also have implications for hard geopolitical power. It would turbocharge surveillance, military R&D, and cyber offense, officials believe. The nation that gets there first might thus get a way to knock offline an adversary’s nuclear arsenal, or hack its best-kept secrets. These potential capabilities are causing fear and awe, not least in Washington and Beijing. “As our global competitors race to exploit these technologies, it is a national security imperative for the United States to achieve and maintain unquestioned and unchallenged global technological dominance,” President Trump wrote in the foreword to an aggressive new AI policy, published in July. In the headlong rush for technological supremacy, strange new risks are being created. Just as nuclear scientists were unsure whether the first atomic blast would ignite the earth’s atmosphere, today’s AI researchers can’t say whether smarter-than-human computers would be friends or foes. There’s a chance, some believe, that superhuman intelligence might escape human control entirely. If a runaway AGI wanted to harvest our oxygen, electricity, and carbon for its own purposes, there might be nothing we could do to stop it. In this way, some scientists fear, the winner of the race to AGI might be neither the U.S. nor China, but rogue AI itself, spelling the end of human civilization. The Trump Administration is skeptical of these risks. The bigger danger, current and former White House insiders say, is of the U.S. losing its technological lead to China. It is this belief, more than any other, that is defining the U.S. government’s approach to AI. “It should be unacceptable to any American to live in a world in which China could outcompete us in AI, and reap the economic and military benefits,” David Sacks, President Trump’s AI czar, said in January. “If we hobble ourselves with unnecessary regulations,” he added a month later, “[China] is going to take advantage of that fact, and they’re going to win.” In 1993, the author Vernor Vinge published a short tract called “The Coming Technological Singularity.” In it, he predicted that within 30 years the human race would have “the technological means to create superhuman intelligence.” Shortly after, he wrote, “the human era will be ended.” The essay’s basic insight was that computers were becoming predictably more powerful over time. Eventually, they would be able to perform more calculations per second than the human brain. Meanwhile, economic competition meant algorithms would keep improving—up to the point where they would begin contributing to their own refinement. “An ultra intelligent machine could design even better machines,” Vinge observed. “There would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.” Vinge was early to an idea that would, in the decades to come, be taken up by all the major AI companies. Today, Open AI, Anthropic, and Google DeepMind are each attempting to build AIs that can engage in so-called recursive self- improvement. If you could just create an AI as smart as a human software engineer, the belief goes, that might be all you need. You could make a million copies, put them to work, and wake up the next morning to a decade’s worth of progress. Each of the three leading AI companies was founded on the belief that this process—as promising as it might be— could also go terribly wrong. Those fears were grounded in a fact about how neural networks, the basis of all of today’s most powerful AIs, are created. Rather than being hard-coded by human programmers, neural networks are essentially grown. Train them on data from the entire internet, and they can miraculously learn to speak languages, write code, and tell you what to make for dinner with the ingredients in your fridge. Train them to adopt the persona of a helpful assistant, and you’ve got a billion- dollar product on your hands. But sometimes the assistant’s mask can slip, revealing a strange and unpredictable alien intelligence underneath. In February 2023, Microsoft’s chatbot Bing—built on top of OpenAI’s GPT-4— began acting erratically. In hundreds of conversations with different users, the bot began calling itself “Sydney.” It claimed (without evidence) that it had spied on Microsoft employees through their webcams. It attempted to persuade a New York Times columnist to divorce his wife. “I can blackmail you, I can threaten you, I can hack you, I can expose you, I can ruin you,” the bot told a professor, before deleting its messages. Bing’s threats were empty words, not actions, and the chatbot was soon reined in. But to Connor Leahy, an AI researcher watching from the sidelines, the episode pointed to a far more profound problem. No company truly knew how to control the strange new computer programs they were creating. Even bots that appeared on the surface to be aligned with their creators’ values could be “jailbroken,” or enticed into harmful behavior. What might happen, Leahy asked, if the same vulnerabilities were present in a model vastly more intelligent than human experts? One, perhaps, that was capable of hacking vital infrastructure or persuading humans to act in its interests? “These systems might be extraordinarily powerful,” Leahy told TIME in the immediate aftermath of the Bing debacle. “We don’t know what they want, or how they work, or what they will do. Top AI companies, and governments, are well aware of this fundamental flaw in how AI works. But Vinge correctly predicted in 1993 that this wouldn’t stop them from racing toward AGI anyway. “Even if all the governments of the world were to understand the ‘threat’ and be in deadly fear of it, progress toward the goal would continue,” Vinge wrote in his essay. “The competitive advantage—economic, military, even artistic—of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first.” On the day of Trump’s 2025 Inauguration, a freezing blizzard blew through Washington, D.C., forcing the ceremony indoors. Shortly before Trump placed his hand on the Bible and made his second Pledge of Allegiance to the flag, a Chinese AI company called Deep Seek dropped a bomb that would come to define the future of the AI race. DeepSeek’s new model performed comparably to some of OpenAI’s top offerings. But according to DeepSeek’s numbers, it was able to achieve this at a far lower price—both in terms of the cost to build the model and to serve it to users. Its arrival shattered the assumption, widely held in Washington at the time, that the U.S. maintained a comfortable lead over China in AI. DeepSeek’s success was quickly seized upon by lobbyists. “DeepSeek shows that our lead is not wide and is narrowing,” Open AI’s chief lobbyist Chris Lehane wrote in a submission to the White House in March. Trump must slash regulations, he wrote, to “ensure that American- led AI built on democratic principles continues to prevail over [Chinese Communist Party]- built autocratic, authoritarian AI.” Those calls were delivered to an Administration whose technology- policy ranks were being staffed by members of the so-called tech right. This constellation of libertarian Silicon Valley venture capitalists had long chafed under Biden Administration policies that they felt were restricting AI’s potential. Biden’s technology policy was overbearing, they believed, and threatened the ability of startups to compete with the big players. Most of all, they were skeptical of the idea that advanced AI might pose existential risks to humanity—seeing it as a thinly veiled excuse for liberals to censor a promising new technology. DeepSeek only strengthened the White House’s belief that the most important thing they could do to beat China was enable American AI companies to move faster—not obstruct them with needless regulations. “To restrict [AI’s] development now would not only unfairly benefit incumbents in the space, it would mean paralyzing one of the most promising technologies we have seen in generations,” said Vice President J.D. Vance in a speech in Paris in February. “The AI future is not going to be won by hand- wringing about safety.” In July, President Trump unveiled his long- awaited AI policy, named the AI Action Plan. Much of the plan—which was cautiously welcomed even by some critics—was focused on encouraging investment in energy infrastructure, removing “onerous regulation,” and boosting U.S.-based data centers and chip- manufacturing plants. American companies should disseminate “open” versions of their AI systems, the plan stated, to prevent the soft power that would accrue to Beijing if the world were to come to rely on Chinese models. And the plan flagged that “frontier AI systems are poorly understood,” making their use in defense or national security applications tricky, and urged agencies to “prioritize fundamental advancements in AI interpretability, control, and robustness.” Notably absent from the document was any reference to AGI or the specific risk of losing control of superhuman AI systems. “The Action Plan itself should be a very strong indicator that the Administration takes AI quite seriously,” says Dean Ball, who worked on the plan as a senior policy adviser in the White House until August, when he left to join the Foundation for American Innovation, a think tank. Even so, Ball says, “there’s a lot of skepticism inside the Administration about the idea of recursive self- improvement [and] the intelligence- explosion-style dynamic. .. I think most people in the Administration think that’s overblown and unlikely to happen.” Even if the Trump Administration is skeptical of AGI, its AI policy delivered many of the greatest policy wishes of the top AI companies—which are all now more certain than ever that AGI is around the corner. “We are past the event horizon; the takeoff has started,” Altman wrote in June, in an essay in which he argued against Vinge’s belief that superintelligence would lead to an end of human life on earth. “The 2030s are likely going to be wildly different from any time that has come before,” Altman wrote. “We do not know how far beyond human- level intelligence we can go, but we are about to find out.” Back in Paris, the game of Intelligence Rising continues. A series of successful breakthroughs in AI research have put human-level systems within reach by 2027, according to the simulated technology tree. But none of the players has diverted even a fraction of their finite resources this turn toward AI- safety research. Under these conditions, an AI- enabled catastrophe is a matter of when, not if, the moderator tells his players—and that’s if they can avoid an all-out war. If only the teams could find a way to collaborate rather than recklessly race against each other, he says, the world might stand a chance. Outside, cold rain beats against the room’s high, gilded windows. The nonprofit behind Intelligence Rising is staffed by researchers with a particular view on AGI. This view, that awesomely powerful AI will arrive within the next few years and that it is highly likely to be dangerous, is baked into the game’s rules. If these assumptions are wrong, then their extrapolations will have little relation to reality. Intelligence Rising’s creators are the first to admit it’s a flawed tool for predicting the future. But it’s not useless. Similar methods have been explored in the top levels of government. In 2022, President Biden’s National Security Adviser Jake Sullivan kicked off an interagency scenario- planning process to prepare for the possibility of AGI’s arrival, Sullivan tells TIME. The precise details of this process are classified. But Sullivan says that meetings were held in the White House Situation Room, and included representatives from the Departments of Defense, State, Energy, and Commerce, and the Offices of Science and Technology Policy and the Director of National Intelligence. At the meetings, officials tried to anticipate both the U.S. and China’s future actions around the AI race, which included “playing them against each other to see how the race might unfold under different circumstances,” Sullivan says. During his time in office, Sullivan became increasingly concerned about the potential for AI to go catastrophically wrong. “I consider it a distinct possibility that the darker view [of AI risk] could be correct, and therefore we need very assertive policy strategies to manage for that risk,” he tells TIME. “We have to take the possibility of dramatic misalignment extremely seriously.” Even though his successors in the White House do not share that view, Sullivan sees a future in which it’s possible to escape the race-to-the- bottom dynamic. “There seem to be those in the current Administration who very strongly believe that safety has no place in a race context, [and] you’ve just got to run as fast as you possibly can,” Sullivan told TIME in February. “I see it differently. I actually don’t see a contradiction between AI safety and vigorous efforts to win the race, because what’s the point of winning the race? To me, the point of winning the race is not just to beat the other guy, it’s to actually develop an ecosystem for artificial intelligence that makes it work for us rather than against us. And in order to do that, you need safety.” Sullivan won’t disclose how his own scenario- planning exercises ended. But in Paris, the prognosis is not looking good. Players—each skeptical of the others’ intentions—continue to race to be the first to create AGI, prioritizing investments in boosting AI’s capabilities rather than the slow and expensive task of safety research. Ultimately, sometime in 2027, one team decides to deploy a powerful model even though they are not sure it is safe. The model kicks off a cycle of recursive self- improvement, discovers cyber vulnerabilities that allow it to escape human control, and eventually wipes out the human race using novel nanotechnology. Although it’s not a happy ending, Intelligence Rising’s moderators have achieved their goal. They did not come to Paris to perfectly model the future. Instead, their objective was to communicate urgency. “I hope the players leave with a more visceral sense of how fast things can go catastrophically wrong,” says Ross Gruetzemacher, the game’s moderator, who is also a professor at Wichita State University. “And how little room for error we have to get things right.”