핵심 요약
효과적 이타주의의 실증적 장점과 AI 위험 경고를 인정하되, 도덕적 최적화와 권력 집중의 위험을 비판한다.
논증 분석
논리구조
-
자선 효율화에서 시작된 운동이 AI 기업과 자산가를 통해 제도적 영향력을 얻었다는 흐름을 제시한다.
-
이성과 경험적 검증의 가치, AI 위험에 대한 일부 선행 경고를 먼저 인정해 비판의 대상을 운동 전체의 선의와 구분한다.
-
미래 인구·기술·할인율의 불확실성이 큰데도 멸종 위험의 거대한 손실값이 현재의 빈곤·권리 문제를 압도할 수 있다고 지적한다.
-
단일 목표를 극단적으로 최적화하는 AI에 대한 비판을, AI 통제권을 독점하려는 소수 인간 집단에도 적용한다.
결론
역사적 혁명과의 비유는 현재 운동이 같은 결과를 낳는다는 증거가 아니다. 일부 지지자의 극단적 입장을 전체 운동으로 일반화하지 않는다. 미래의 수조 명은 관측 인구가 아닌 철학적 가정이므로 통계로 추출하지 않았다. 본문의 모델 출시 관련 서술은 원문 주장으로만 보존하며 별도 확인하지 않았다.
토론 질문
미래의 큰 위험을 고려하면서도 현재 개인의 권리와 복지를 침해하지 않는 의사결정 규칙은 무엇일까? AI 안전의 결정권은 누가 감시해야 할까?
원문
The movement is admirable, but dangerous
Illustration: Vasya Kolotusha
THOUGHT PRECEDES deed as lightning precedes thunder, wrote the poet Heinrich Heine. So it has proved with effective altruism (EA), an ethical doctrine cooked up by philosophers in Oxford that has gone from the ivory tower to the corridors of power with remarkable speed. Effective altruists fill Silicon Valley, staff the world’s leading artificial-intelligence companies—and alarm the Trump administration. “The United States will NEVER be an effective-altruist country,” pledges the Pentagon. The secretary of war brands the philosophy “defective altruism”; others in the administration call it “dystopian”. Take those attacks as a sign of its influence.
Some complain that the 21st century has seen little fresh thinking. Into the void have stepped MAGA and Co, making nostalgic populism the century’s big political project. But the century’s big idea is effective altruism. As we report, what began in the 2000s as a geeky way to make charitable giving more cost-effective has changed into a movement that subjects the world to complex, even bizarre, moral calculations. For many of its adherents, the urgent ethical requirement for people who want to do good is to stop artificial intelligence from wiping out humanity.
EA is sociologically important, too. EA s are not numerous, but the idea has swept up a generation of gifted youngsters who have made fortunes in cryptocurrency and AI. As the technology mints more billionaires, effective altruists will channel billions of dollars to causes that will shape society. EA s helped start Open AI and Anthropic, the leading labs, in the belief that those who understood the dangers of AI should control its development. To a degree, they now do—though, ironically, by creating more labs, and thus competition, they have probably accelerated AI ’s progress.
And all this has now made EA politically important. The Trump administration fears the movement will succeed in throttling AI. It has feuded with Anthropic over the firm’s refusal to let its models be used for some military purposes. Leading EA s acknowledge that AI may prove hugely benign, but nonetheless want to slow its development, to reduce the chance that an unstoppable machine-brain runs amok in pursuit of goals that are “misaligned” with those of humanity.
Some critics in America would rather accelerate the technology, to prevent China from winning the race. But when attacking AI doomerism they find it easier to play the nerd than the ball. They say that effective altruism is a movement of fanatics that is variously undemocratic, dangerous and cultish.
That is too crude. The movement’s charitable roots are obviously praiseworthy. Like The Economist, EA s are committed to empiricism and, especially since they blended with the online “rationalist” community, the application of logical thinking. They hold reason in higher esteem than tradition or norms and will enthusiastically debate any topic, however controversial, in pursuit of the truth.
EA s’ repeated success in predicting the rapid progress of AI has been impressive, too. They have a point about the technology’s risks to humanity. On September 28th Open AI scrapped plans to release its latest model after it appeared too dangerous in testing. Many recent safety incidents resemble scary scenarios set out on effective-altruist blogs years ago.
Yet the critics are right to warn that effective altruists reach some odd conclusions. Like the utilitarian philosophers before them, EA s boil ethical behaviour down to an optimisation problem. How can people bring about the best outcomes for humanity as a whole? Their calculus downplays local imperatives such as “keep promises” or “look after your family”. What matters is reducing suffering, impartially and everywhere.
That leads them to factor a very wide circle of concerns into their moral calculations. It is easy to see how being ethical means helping your neighbour. EA s ask: why not work as hard to help those far away? Early on, the movement directed its army of overachievers to take high-paying jobs in finance and consulting. Their wealth could buy good works, achieving more than if they became charity workers themselves.
But as it has become stronger, EA has become stranger. Some EA s see the moral distinction between humans and other animals as weak, because animals experience suffering. Stopping factory farming and helping shrimp are major concerns. Likewise, they have a counterintuitive approach to time. Those not yet born should be weighed in the balance—and there could be trillions of them. Many EA s are “longtermists” who feel bound to forestall risks in the distant future, even if that starves today’s good causes, such as helping the desperately poor.
Pushed to an extreme, this calculus can lead to a morality that is not just counterintuitive, but repugnant. Some EA s think that insect life is, in aggregate, more valuable than human life, because there are so many insects. Some say it would be better to bring about the existence of trillions of humans whose lives are barely worth living than to have mere billions with genuinely good lives. Others even think the welfare of AI s themselves will one day be of vital importance. They would redirect resources towards future humans or nonhumans, and away from people suffering today.
EA+AI=p(doom)
It is right to revere reason. But moral philosophers from Aristotle to Rawls knew that if an argument leads to intolerable places, it must have a hidden faulty step. EA s’ instinct, though, is to enjoy counterintuitive conclusions, even if they seem perverse. A problem is the margins of error that inevitably accompany effective-altruist arguments. Nobody is truly capable of weighing up moral quandaries involving billions today, let alone gigantic sums stretching into the future, involving uncertain numbers of people, discount rates, the progress of technology, and countless other variables. Another is that extinction is so dire that in a crude moral arithmetic even a small chance of it overpowers today’s flesh-and-blood concerns.
A third is that in weighing up the interests of billions, you lose sight of the inherent dignity of the individual. Anyone wielding a spreadsheet and grandiose claims about the future of humanity is one step away from anointing themself as a central planner. Moral authority in hand, the rights, welfare and opinions of a neighbour can come to seem unimportant.
The French revolution was supposedly founded on reason; it led to the execution of nuns. Lenin, too, thought that intellectuals who had mastered Marxism understood workers’ interests better than the workers themselves. If you truly believe that you act on behalf of all people, let alone trillions of future ones, you may end up justifying all sorts of atrocities.
Fortunately, effective altruists are hardly about to bring back the guillotine. The fear instead is that controlling AI will eventually bring them great power, and that plenty of EA s in Silicon Valley think that only they can be trusted with the technology. That is an alarming idea. Effective altruists’ argument for worrying about really powerful AI s is that the machines could optimise for a single and fatally wrong objective. Everyone should worry, too, about what happens when a small group of people fall into the same trap. ■
For subscribers only: to see how we design each week’s cover, sign up to our weekly Cover Story newsletter.