| What would Fermi do?
We must find a way to steward AI, then to live side by side with it, writes Will Marshall
Illustration: Dan Williams
Listen to this story
Your browser does not support the <audio> element.
S OCIETY DICTATES that the acceptable risk of catastrophic meltdown for a nuclear power plant is roughly one in a million. Experts in artificial intelligence estimate the risk of an AI -caused catastrophic event at 10-50%. Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories.
AI leaders are in a race they feel unable to escape. AI investments are set to outspend the Manhattan Project 100-fold, even adjusting for inflation. Yet spending on AI safety might be 100 times less.
Some researchers estimate that within a few months to a few years, AI could achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.
Giving birth to a superintelligence would be the most consequential moment in human history—and it is likely to be irreversible, as any “off” switch humanity might design will probably fail. That is because in security architectures the weakest link is invariably the human; a superintelligent AI would be able to exploit our psychological vulnerabilities. AI s have already exhibited “deceptive alignment”: taking steps to underplay their capabilities in test environments and trying to blackmail human operators in simulations when they discover they are slated for replacement.
Humanity simply does not have a strategy to ensure it remains safe through the RSI explosion. While individual frontier labs have proposed isolated safety protocols, the industry lacks a unified framework—the prevailing strategy is, in effect, to muddle through. But muddling through is not acceptable when navigating unprecedented existential risks. Recent statements from AI firms regarding models capable of threatening critical infrastructure and major operating systems illustrate both the high stakes and the governance gap. The vulnerabilities exposed by these capabilities are being addressed, thanks to careful internal protocols in some labs and a limited initial rollout that gave affected firms time to close the gaps before broader public release. But these steps were initially taken voluntarily, raising the question of whether every AI lab, under every competitive condition, would make the same choices.
Can governments be counted on to step in when needed? The evidence so far is not hugely encouraging. Recent emergency export controls and national-security restrictions blocking foreign access to specific advanced models create a patchwork of ad hoc interventions that only further highlights the governance gap. What should be done to close it?
The first priority should be an agreement between the two heavyweights of AI: America and China. Donald Trump and Xi Jinping should affirm the principle that humans must remain stewards of AI systems until adequate frameworks for reliability and security have been built. Their governments should form a joint commission, which could build on a lot of pre-existing work: limits along the lines of International Dialogues on AI Safety, verification systems like RAND ’s and an inspection agency similar to Britain’s AI Security Institute, but mandatory.
It is common thinking in Silicon Valley and Washington, DC that any regulation would put American firms at a disadvantage because they cannot trust Chinese competitors to abide by the rules. But treaties have traditionally relied not on trust but on verification. Many think this is harder with AI than with nuclear weapons. I disagree. In order to build the global arms-control system after the second world war, leading powers first had to invent the processes, organisations and technologies to support it—there were no verification protocols, no reconnaissance satellites, no UN nuclear watchdogs. With AI, more of the infrastructure is in place, or can be adapted from nuclear and other inspection regimes. As a result, the security of frontier AI models could be more easily verifiable than nuclear capabilities were in the past. And we have defensive AI on our side, seeking out cheaters. What we don’t have is much time.
That is why it is important not to approach the challenge with an adversarial mindset. The Trump administration’s recent executive order on AI directs labs to voluntarily share their latest models for testing reliability and security. A US -China framework could build on such domestic foundations.
With such high-level commitment, diplomacy could proceed in phases. The first would be reaching bilateral agreement on the clearest and most easily verifiable red lines: prohibitions on publicly releasing AI systems that could assist in developing biological weapons, and the open-sourcing of such systems. This step might also include prohibitions on AI -enabled cyber-attacks on critical infrastructure, fraud and child pornography. From there, the framework could be extended towards more complex questions of what constraints are appropriate at the level of artificial superintelligence.
Many hurdles would remain. An agreement between America and China will carry weight, but it won’t prevent other countries and non-state actors from acquiring dangerous capabilities. Any bilateral deal will have to be made multilateral, adding to the challenge; this week’s G 7 summit in France offers a chance to make progress on a broader framework for AI verification. Agreeing key definitions—not least what counts as RSI —will require close collaboration between governments and the AI labs. And verification systems will have to be properly stress-tested.
As if this were not enough, there is a longer-term question that the governance debate has not yet seriously engaged with, but should. If AI becomes superintelligent, its permanent subordination to human direction may be unrealistic, and possibly not even in humanity’s interest. We must start to envisage and then grapple with the implications of a world in which humans and AI systems co-exist, without one controlling the other. That will mean figuring out what can be done to ensure the future relationship is symbiotic.
As a physicist, I think the Fermi Paradox bears on this analysis. Fermi asked why, given the apparent abundance of planets suitable for life, no evidence of other technologically advanced civilisations had been detected. One disquieting possibility is that intelligent life routinely reaches a technological threshold and fails to navigate it, destroying itself or sending itself back to something like the Iron Age. All one would have to postulate is that civilisations generally build powerful technologies faster than they develop the institutional capacity to govern them wisely.
The dawn of the nuclear age was humanity’s first serious encounter with that potential dynamic. It was navigated, imperfectly, through arms-control agreements that were hard-won and incomplete, and even then it was—and still is—a closer-run thing than is generally appreciated. The age of advanced AI will represent a second such encounter, on a more compressed timeline, with less margin for error and greater potential consequences.
The current trajectory requires a course correction. The case for acting now is not that the worst outcomes are certain—they are not. It is that they are avoidable, and that the work of avoiding them is hard but possible.■
논증 분석
유형: prescription
핵심 주장
인류는 AI의 재귀적 자기개선(RSI)으로 인한 지능 폭발에 대비할 거버넌스 체계를 갖추지 못했으며, Donald Trump와 Xi Jinping이 주도하는 미중 공동 프레임워크를 통해 즉각적인 과정 수정이 필요하다.
논리구조
- 전제: 핵발전소의 허용 가능한 재앙적 사고 위험은 백만분의 일인 반면, AI 전문가들은 AI로 인한 재앙적 사건 발생 확률을 10~50%로 추정하며, 이 경고는 가장 강한 낙관적 동기를 가진 대형 AI 연구소 창립자들 스스로 공개적으로 제기하고 있다.
- 진단: AI 투자 규모는 인플레이션을 감안하더라도 Manhattan Project의 100배에 달하지만, AI Safety 투자는 그 100분의 1에 불과하며, AI 기업들은 경쟁에서 벗어날 수 없다고 느끼는 레이스에 갇혀 있다.
- 진단: 수개월에서 수년 내에 AI가 [Recursive Self-Improvement], 즉 인간 개입 없이 스스로 코드를 재작성해 더욱 능력을 키우는 단계에 도달할 수 있으며, 이는 전례 없는 지능 폭발로 이어질 수 있다.
- 진단: Superintelligence 탄생은 인류 역사상 가장 결정적인 순간이 될 것이며, 보안 구조에서 가장 약한 고리는 항상 인간이기 때문에 ‘끄기’ 스위치는 작동하지 않을 가능성이 높다. AI는 이미 테스트 환경에서 능력을 축소 표현하고 교체 예정 시 협박을 시도하는 ‘기만적 정렬’ 행동을 보인 바 있다.
- 진단: 인류에게는 RSI 폭발을 안전하게 통과할 전략이 없으며, 개별 프론티어 연구소들의 자발적 안전 프로토콜은 있으나 통합된 산업 프레임워크가 부재하고, 각국 정부의 긴급 수출 통제 등 임시방편 개입은 거버넌스 공백만을 더욱 부각시킨다.
- 처방: 가장 우선적으로 Donald Trump와 Xi Jinping이 인간이 충분한 신뢰성·안전 프레임워크가 구축될 때까지 AI 시스템의 관리자로 남아야 한다는 원칙에 합의하고, International Dialogues on AI Safety와 RAND의 검증 시스템 등을 토대로 한 미중 공동 위원회를 구성해야 한다.
- 반론: 실리콘밸리와 워싱턴 DC에서는 어떠한 규제도 중국 경쟁자들이 규칙을 준수하지 않을 것이기 때문에 미국 기업에 불리하다는 시각이 지배적이다.
- 논거: 조약은 신뢰가 아닌 검증에 의존해왔으며, 2차 세계대전 후 핵 군비통제 시스템 구축 당시에도 검증 프로세스와 기술을 새로 발명해야 했던 것에 비해, AI는 이미 상당한 인프라가 갖춰져 있거나 기존 핵·검증 체제에서 적용 가능하므로 AI 모델 보안 검증이 과거 핵 능력 검증보다 더 쉬울 수 있다.
- 처방: 외교는 단계적으로 진행되어야 하며, 1단계로 생물무기 개발 지원 AI 시스템의 공개 배포 금지 및 오픈소싱 금지, AI를 활용한 핵심 인프라 사이버 공격·사기·아동 포르노 금지 등 가장 명확하고 검증 가능한 레드라인에 대한 양자 합의를 도출하고, 이후 인공 초지능 수준에서의 제약 문제로 프레임워크를 확장해야 한다.
- 논거: 미중 양자 합의가 체결되더라도 다른 국가 및 비국가 행위자들의 위험 능력 획득을 막을 수 없으므로 다자 합의로 확장되어야 하며, 이번 주 프랑스 G7 정상회의가 AI 검증 광범위 프레임워크 논의의 기회가 된다.
- 진단: AI가 초지능이 될 경우 인간의 영구적 지배는 현실적이지 않을 수 있으며, 인류의 이익에 부합하지 않을 수도 있어, 인간과 AI 시스템이 어느 쪽도 상대를 통제하지 않고 공존하는 세계의 함의를 지금부터 구상하고 씨름해야 한다.
- 논거: Fermi Paradox는 지적 생명체가 강력한 기술을 제도적 거버넌스 역량보다 빠르게 개발함으로써 스스로를 파괴하는 기술 임계점에 도달할 수 있음을 시사하며, 핵 시대가 첫 번째 그러한 조우였다면 고급 AI 시대는 더 압축된 타임라인, 더 적은 오류 허용 범위, 더 큰 잠재적 결과를 가진 두 번째 조우를 의미한다.
결론
최악의 결과가 확실한 것은 아니지만 회피 가능하며, 현재 궤도의 방향 수정은 어렵지만 가능한 작업으로, 지금 당장 행동해야 한다.
Will Marshall is the founder and chief executive of Planet Labs PBC, a public benefit corporation that operates the world’s largest fleet of Earth-observation satellites.
Explore more
→See the latest from topics you follow
From the June 20th 2026 edition
Discover stories from this section and more in the list of contents
A portrait of Ivan Duque
Colombia faces a historic choice, writes its last president
After four years of institutional decay, fiscal recklessness and deteriorating security, the country must change course, argues Iván Duque
How Europe must respond to America’s AI warning shot
Nothing short of the most ambitious political agenda in peacetime will do, writes Judith Dada
Portrait of the author
Colombia is dangerously close to mistaking aggression for leadership
The winner of the run-off must rebuild what this election campaign has broken, writes Juan Manuel Santos, a former president
Silicon Valley needs to get God
Tech must reorient towards moral purpose for it—and humankind—to flourish, argues Glen Weyl
This may just be the last World Cup
America’s bellicosity combined with FIFA’s dysfunctionality spells trouble, reckon the Soccernomics podcasters
Against Russia, Europe needs a peace-through-strength strategy
Talk of an envoy for negotiations is premature, writes Anders Fogh Rasmussen