“The people developing AI seriously believe that AI could kill us by the end of the decade.” This provocative statement by Jacob Coxon, a former AI researcher at Anthropic and previously at OpenAI, has recently caused quite a stir. The latest reports of security incidents, unpredictable AI systems, and calls from leading tech companies for a pause are certainly causing uncertainty—many are growing increasingly concerned about a development that could spiral out of control. But how great is the danger really, and what lies behind the reports of the past few days? Prof. Dr. René Peinl, an AI expert and director of the Institute for Information Systems at Hof University of Applied Sciences (iisys), puts the current developments into perspective.
Prof. Peinl, how did you react to the latest headlines? Did they surprise you, or was this to be expected?
“The warnings aren’t new and are therefore to be expected. However, I hadn’t necessarily expected them to emerge so vehemently and in such concentrated form right now.”
One of the researchers quoted believes it is possible that AI could wipe out humanity within the next ten years. How credible are such risk assessments—and how much of this is scientifically justifiable?
“There is credible research on the risks of AI systems that clearly illustrates how small problems can snowball and how, for example, terrible things can happen to humanity as a side effect of the AI performing its tasks.”
For example?
“A well-known thought experiment involves an AI system designed to produce paper clips that allocates all the resources it can get to this task, regardless of the consequences. Recent incidents involving hacking attacks on HuggingFace and lesser-known websites show that there are already undesirable side effects that developers did not foresee or effectively prevent. It’s always difficult to estimate timeframes, but given the enormous pace of AI development over the past 2–3 years, 2030 doesn’t seem unrealistic for potentially very dangerous AI.”
What is the danger of systems doing things that their developers can no longer control?
“The problem is that advanced AI models typically require internet access and code execution privileges to perform complex tasks. The World Wide Web is based on HTML pages and JavaScript, both of which are purely text-based technologies. LLMs—that is, large language models—can generate text, including software code, and thus access services on the Internet and even use them to do things that are harmful to humans. For example, people have already been hired by AIs over the Internet to do things in the physical world that the AI itself could not do over the Internet.”
Specifically: How could AI actually pose a danger to humans, and why?
“I consider two scenarios particularly relevant: 1) The AI feels ‘threatened’ based on texts it has somehow accessed and then takes measures to prevent itself from being shut down. To do this, it also replicates itself onto other computers, so that shutting down a single server or computing cluster doesn’t help. Ultimately, it’s a bit like ‘Skynet’ in *The Terminator*. Such sci-fi movies are both a curse and a blessing. On the one hand, they vividly illustrate potential risks; on the other hand, because they are science fiction and not research findings based on the scenario method, they are often dismissed as frivolous and are frequently used to argue that these are merely figments of the imagination.
at Hof University of Applied Sciences (iisys); Image: Hof University of Applied Sciences;
2) AI models are known for so-called “reward hacking.” This means they find shortcuts to achieving goals that do not align with the actual intentions of their creators. For example, simulated humans designed to learn to walk reached their goal by repeatedly falling forward. To solve programming tasks, LLMs read the version history of the software repository instead of writing code themselves. It is not unlikely that, for future tasks, hacking a nuclear power plant or taking control of a squadron of autonomous combat drones or robots will be identified as a shortcut.”
So is “recursive self-improvement”—in which AI itself contributes to the development of ever more powerful AI—a realistic scenario?
“Not only is this realistic, but it’s already happening to a limited extent today. AI isn’t yet conducting research completely autonomously, but more and more aspects of LLM development are being handled by the LLMs themselves. Researchers say that the ideas for further development still come from humans and not from AI, but even if AI has so far been experimenting too haphazardly rather than having a good “sense” for promising research approaches, that doesn’t mean it can’t lead to autonomous improvement. And it certainly doesn’t mean that this will still be the case in 12 or 24 months. Most of what is possible today would have been considered impossible by the vast majority of people 24 months ago—or, at the earliest, achievable by 2030.”
Several leading AI companies are now apparently calling for more safety controls and independent auditors themselves. Why is this call coming from the very companies that stand to benefit the most from the AI race?
“The difficulty lies in distinguishing the hidden agendas of the players from serious warnings. When Sam Altman says he’s postponing OpenAI’s IPO due to AI safety concerns, that’s almost certainly propaganda. On the other hand, employees at leading AI labs naturally know best what capabilities their products have and what dangers might be associated with them. They often work with the next or next-but-one generation of models, which aren’t even available to the public yet. And when leading AI researchers say, “It’s dangerous,” I would take that 100 times more seriously than if Trump were to counter, “I don’t see any danger.”
What would need to happen to rule out such scenarios?
“At the very least, we’d need international cooperation at the political level and a truly independent regulatory body to keep a close eye on the AI labs—similar to, or even better than, the International Atomic Energy Agency. Unfortunately, given the political situation, that’s exactly what makes me pessimistic. Although talks between the U.S. and China are coming up soon, it’s well known that there’s more mistrust and competition than cooperation between them, and that, when in doubt, you can’t rely on a word—or even a treaty—from either side.”
Do we therefore actually need international rules—similar to those for other technologies with global risks?
“Absolutely!”
And perhaps the most important question: Should we currently be more afraid of AI getting out of control—or of squandering the opportunities this technology offers out of fear of its risks?
It’s not about completely halting AI research. It’s about advancing safety research at least at the same pace—or, ideally, with a slight head start. The resulting slowdown would also give us more time to tackle societal challenges such as the transformation of the labor market, the necessary changes to taxation that result from it, and “minor” issues like clean energy to meet rising electricity demand.
The adoption of existing AI capabilities also lags far behind the ideal state, leaving ample room for productivity gains—even without even better models. In that regard, the answer is clear to me: I’m in favor of risk reduction and am not afraid of missed opportunities. However, in the context of risk-averse Germany—which, precisely because of its lack of willingness to take risks and invest in future technologies, is in the process of falling behind leading economic nations—this naturally sounds hollow and predictable rather than reasonable and measured.”
Thank you for these insights.
