What Longtermism Thinks Happens After Superintelligence

Key Point

Longtermism treats superintelligent AI as the hinge on which quadrillions of future lives turn, so preventing its misalignment matters infinitely more than any present cost.

See the arithmetic →

Formula

Expected value calculation, standard in longtermist priority-setting.

EV = P(\text{survival}) \times N_{\text{future}} \times V_{\text{life}}

See why the number explodes →

Misconception

From most corporate AI timelines and government white papers.

Common belief

We'll have decades to course-correct as AI systems become superintelligent, the way we've had time to regulate social media and gene editing.

Reality

RSI means recursive self-improvement — the system redesigning itself faster than humans can track, potentially crossing from human-level to vastly superhuman in days.

See the lock-in problem →

Standup Bit

Title

The Paperclip Triage

Premise

If you had to choose between curing one child's cancer today or preventing an AI catastrophe that kills everyone including that child, longtermism says let the kid die.

Setup

The math only breaks if you think there's something special about suffering you can see, which longtermists will tell you is just scope insensitivity with a hug.

Punchline

Seems heartless until you do the expected value calculation.

It is heartless — that's the whole point, the heart is an organ for ignoring large numbers.

Hear the whole set →

Hot Take

Most longtermists privately believe alignment research is a coin flip at best and are funding it anyway because even a two percent chance of mattering is worth more than anything else you could do with a billion dollars.

See the expected value logic →

Thought Experiment

Thought Experiment · A longtermist version of the repugnant conclusion.

The Simulation Choice

The question

If a superintelligent AI offers to run ten trillion happy simulated lives per second forever but requires shutting down all biological humans to free the resources, do you take it?

Sit with the numbers →

Corporate Memo

Future of Humanity Institute

RE: Reallocation of Global Health Funding to ASI Alignment Research — Revised Priorities for Q3

Body

Per this quarter's expected value analysis, marginal dollar impact on existential risk now exceeds global health interventions by six orders of magnitude. Recommend phased transition of discretionary funds.

Read the full memo →

Concept

Concept · Longtermism

Value lock-in

in the context of Longtermist AI trajectories

An early advanced system could permanently preserve and enforce specific goals across billions of years of future civilization.

unpack it →

Concept

Concept · Digital Sentience

Mind crime

in the context of Artificial sentience ethics

The harm caused when advanced systems instantiate and mistreat conscious digital entities during vast computational runs.

unpack it →

Satire Article

Business

Billionaire Reallocates Malaria Budget To Safeguard Trillion Unborn Digital Minds

Portfolio managers noted that mathematically, an infinite number of simulated post-humans outweighs Cleveland's current municipal wastewater backlog by several decimal places.

Read the article

Bullets

Outline

Why longtermists prioritize advanced artificial intelligence

▸Vast scale of future lives
Countless potential generations depend on surviving the transition to superintelligence
▸Irreversibility of extinction
Failing to safely guide recursive self-improvement permanently forecloses humanity's future
▸Shaping the initial trajectory
Early design and governance choices set long-lasting institutional norms

see the outline →

Book