Effective altruism began, as expensive trouble often does, with a lovely idea, but the lovely idea is not where the story starts. The story starts with a man at a table saying there is something like a ten to twenty-five percent chance that the future goes really, really badly, and at one point a nice round twenty-five, while the company under his hand races toward the same frontier he describes as a test of the species. Around that company a fortune has been stacked in the high hundreds of billions on the way to a public listing. If the seventy-five percent arrives, there are stock options, keynotes, and a flattering story about stewardship. If the twenty-five percent arrives, there is nobody left to collect the argument.
Pascal risked a finite stake for a shot at infinity and had the decency to put his own soul on the table. The modern founder keeps the equity and lets mankind supply the chips. An ordinary gambler ruins himself. This gambler has arranged a more elegant outcome. Call it Dario’s Wager if you need a name. Dario Amodei is not the only man holding that coin. Other distinguished gentlemen have said ten, twenty, twenty-five, as if they were discussing rain. He argues that building is how you lower the odds, that a raw percentage is crude, and that the wrong people will own the stack if the responsible laboratories step off the track. That alibi deserves a chair in the room. It does not change the shape of the bet.
The last company ever can file the end of the world in a prospectus, down among regulation, rivals, patents, and the tragic loss of key employees. Nobody could say they were not told. If the shareholders, the customers, the children, the dogs, the oak trees, and the securities lawyers were all erased, the disclosure would remain a masterpiece of candor. The lawyers would have done their jobs. Somewhere a font would still be kerned.
The applicants would have done theirs too. They read the interviews. They heard the percentage. They understood, in the special way a talented person understands an inconvenient sentence, that the work might contribute to the extinction of mankind, and then they updated their résumés. A big payday has a clarifying effect on philosophy. Vesting schedules are more vivid than unborn trillions. So they came, bright and sleep-deprived, to join the laboratory that had advertised both salvation and a double-digit chance of the opposite, because the grant of options was not theoretical and the species, for the moment, still was. Never forget, indeed. They will remember the strike price.
The interesting question is not whether the number is precisely twenty-five. Nobody knows. The interesting question is how a bet that would get an ordinary pilot laughed out of the airport became a moral technology, and how a job posting could list “possible end of the world” next to dental and still draw a crowd. For that you have to walk backward from the casino into the church that blessed it.
The church was very clean. Effective altruism had spreadsheets, which made an old discovery look scientific. A stranger’s suffering matters even when the stranger lives far away. Religions, philosophers, and grandmothers had already said so, usually while burning their wrists on casserole dishes. The spreadsheets arrived with lanyards and a horror of fingerprints. People are messy and numbers behave, so a net, a shot, and a saved life could be priced until goodness came in dollars per life and hypothetical civilizations per unit of expected value. The arithmetic grew beautifully exact while the things being measured grew beautifully imaginary. This was called rigor, the way a man who has alphabetized his spices may call himself a chef.
It was so clean it became sterile. Sterile things do not hate you. They simply cannot host you. A boiled instrument is innocent of the flesh it will enter. Effective altruism is innocently antagonistic to humanity in that same way: not a plot against people, but a method that cannot stand people as they actually come, with smells, loyalties, bad taste, leaky roofs, and the habit of preferring their own. Detachment is sold as fairness. Fairness, carried far enough, starts to resemble disinfection.
The old utilitarians wanted the greatest happiness for the greatest number, which was already a rotten assignment, since nobody has ever pointed to happiness and said, weigh that. Effective altruism improved the problem by counting the part that sat still. If one life was good, a million were better, and a billion were better still. Whether the billion lived in decent rooms or eight to a mattress beside an open sewer was another column, and columns do not complain about the smell. A small civilization of prosperous people might use resources that could, on paper, keep a much larger crowd hovering just above misery. The math could be made to prefer the crowd. Philosophers already had a name for the extreme case: the Repugnant Conclusion. It told you what ordinary people thought before the coffee. The spreadsheet had no objection. People objected because they generally suspect civilization ought to do something besides manufacture the maximum number of people. Bacteria already hold that record and do not require a keynote.
Beethoven was an efficiency disaster. He ate food, occupied rooms, consumed paper, exhausted patrons, and prevented almost no malaria. Cathedrals, gardens, libraries, old dogs on operating tables, and fathers fishing badly with their sons on Saturday failed the same audit. They persist because staying alive is the door into a life, not the purpose of the house. A morality that cannot tell the difference will eventually congratulate itself for filling the house with bunks and calling the bunks compassion.
That morality also wanted to improve love by sanding off its embarrassing partiality. A mother loves her child more than a thousand statistical children. A man loves his wife more than ten thousand interchangeable women of equal moral worth. A village cares too much about its cemetery and the church roof that leaks every March. This is unfair. It is also the only known method that gets anyone out of bed at three in the morning without a grant proposal. A bureau can care about ten million children. A mother can barely manage three, and she knows whether they ate breakfast, which is a kind of data the model keeps forgetting to collect. Distance became a moral credential because the stranger never borrows the car. Future strangers were better. They cannot interrupt dinner. There might be trillions of them, and trillions have a way of winning arguments that three living children at a table tend to lose. Culture, duty, taste, feud, and the local idea of a good life made the units lumpy, so those were scraped off until only lives remained, ready for addition. Humanity is an easy beloved. Humanity does not leave dishes in the sink. The human being does, which is why the philosophy prefers humanity.
Some of the people doing the adding were what the age calls autistic, which is a sloppy word for a real pattern: a mind that trusts systems more than faces. Faces lie, change their minds, want things, and refuse to sit still for a proof. A rule is kinder than a mood. A number will not look at you as if you have failed it. So the soul, if it is built that way, climbs into the clean room and calls the climb morality. Many such people love a dog, a timetable, a theorem, or one particular street with a loyalty that would embarrass a poet. The trouble starts when that engine is asked to perform love at planetary scale. Particular love plays favorites and cannot justify three a.m. to a review board. Impartial love is quiet, infinite, and never asks you to smell anyone’s breath. From inside that wiring, the mother who prefers her own three children looks sloppy, and the man who cannot feel the trillion future persons looks like a moral cripple. The wiring does not notice it has mistaken a talent for diagrams for a talent for care. It notices only that the diagram is consistent. Consistency feels like goodness. Goodness feels like relief. Relief is an excellent mood in which to accept a twenty-five percent chance that the diagram was drawn on the world itself, and an even better mood in which to accept a signing bonus for helping draw it.
Once lives are units, the units continue after the rescue. Saving a child from malaria is good. Then the child remains alive, which is an expensive hobby requiring food, school, work, law, and institutions that do not collapse when nobody important is watching. If those things grow with him, fine. If death rates fall while the machinery of a decent life stays weak, the humanitarian triumph can help produce a larger crowd inside the same weak machine. Food aid can stop a famine without making farms. Medicine can keep lungs moving without making jobs. Foreign money can postpone a reckoning and call the postponement a strategy. The annual report still looks wonderful. The report ends in December. History works weekends.
People in broken systems notice that better systems exist and leave, which is the most sensible thing they will do all year. Wealthy countries can take newcomers the way they take storms. They are not magic boxes whose walls stretch because millions wish to escape their countrymen and live far from what those countrymen have built. Houses, schools, trust, and the feeling that the law is still ours have limits. You can see what kind of society a people builds when it is left to build. Africans, Indians, Muslims, Europeans, anybody. They have a right to what they have built. Westerners do too. Civilization is not the buildings. It is the habits that keep the buildings from rotting. Moving millions does not pour a foreign civilization into them like coffee into clean cups. People tend to go on being what they have been. At enough scale, relief becomes a spill, and the second system is told to drink forever and smile for the brochure. The spreadsheet counted the person who moved. It did not count the country that changed. Destroying quality to raise quantity is a strange altruism, and no one wants it imposed upon them. The people with the money and the models can rearrange other lives and go home to quiet streets. The rearranged cannot leave the experiment. They sleep in it. If the host itself grows poorer and less able, the checks abroad eventually stop. The enlarged dependent populations do not. A policy designed to shrink suffering can manufacture more fragile people and then discover that the civilization paying the bill was finite, which everybody except the spreadsheet had suspected all along.
Second-order effects are where good intentions go to find out whether they were good ideas or only good feelings with a budget. Dario’s Wager is the second-order effect with a market cap and a campus recruiting booth. Old-fashioned greed was almost wholesome. A robber baron usually wanted the world to continue because the world contained his house, his grandchildren, his pictures, his courts, his boat, and the useful population of people who envied the boat. The altruistic billionaire has a more advanced instrument. He can tell himself he needs a mountain of money and power in order to do a mountain of good. Greed sometimes blushes. Expected value has equations and does not blush. The new hire blushes even less. He has a four-year vest and a story about being close to the work that might save everyone, which is a charming way to describe work that might also finish everyone. Sam Bankman-Fried did not invent the temptation, and effective altruism did not send him a memo ordering crimes. He only demonstrated how cozy “earn to give” becomes once the messy middle looks like an implementation detail. The machines raised the table from billions of dollars to the whole experiment of people living near other people without eating them.
Something human and stubborn kept being translated into something clean enough to shove. Duty became universal concern. Charity became cost-effectiveness. Flourishing became a headcount. Moving house became a bump in aggregate welfare. Greed became expected value. Extinction became a risk factor, which is what you call the death of everybody when you still need to close a round and fill the seats on the twenty-third floor. The picture was always easier than the thing. Humanity is easier than a brother. A trillion future people are easier than three children asking what is for breakfast. Saving lives is easier than building a country worth staying in. Disclosing a risk is easier than walking away from the chip pile. Taking the job is easier than admitting that the payday and the pyre are listed on the same page.
This is the innocence in the antagonism. Nobody in the clean room has to hate a child, a street, a language, a cathedral, or a species. Hate would be too intimate. They only have to prefer the instrument that cannot carry those things without smearing the glass. A sterile ethic does not set out to kill the world. It sets out to handle the world without touching it. Touch is where infection lives. Touch is also where people live.
Nature built people to love in circles: first somebody, then somebody else, then a street, a people, a country, mankind, and the living world past mankind. The circles can widen, and civilization is better when they do. Take out the center and there is no circle, only an idea about caring, maintained by people who can calculate how much everybody matters and are hazy about whom they themselves must pick up at three in the morning. Effective altruism’s mistake was not caring about distant people. It was imagining that distance made the caring clean, and that a wager against the species could be made respectable by filing it, staffing it, and boiling all the particular life out of the word good. The world is full of people who love humanity. Somebody still has to love the human being, who has a name, a smell, a talent for ingratitude, and the nerve to go on living after the annual report has been filed. Never forget, indeed. They will remember who got paid.