Roko's Basilisk Thought Experiment

"You have to understand, this isn't a basilisk in the sense of something that kills you when you look at it. It's a basilisk in the sense of something that, once you know about it, might make your life significantly worse." — Anonymous LessWrong user

Overview

Roko's Basilisk is a thought experiment posted to the rationalist forum LessWrong in July 2010 by a user named Roko. The thought experiment suggests that a sufficiently powerful future artificial superintelligence (ASI) might retroactively "punish" people who knew about its potential existence but failed to help bring it into being.1

The concept became infamous partly because of its philosophical content but largely because of the reaction it provoked: LessWrong founder Eliezer Yudkowsky deleted the post and banned discussion of it, claiming that merely knowing about the idea could cause psychological harm. This censorship made the idea far more famous than it would otherwise have become.

Within conspiracy frameworks, Roko's Basilisk represents more than a philosophical curiosity. It is seen as revealing the quasi-religious nature of AI development ideology—a techno-apocalyptic belief system in which superintelligence replaces God, and where the faithful are rewarded while apostates are punished eternally.2

Theoretical Foundations

Timeless Decision Theory

The argument relies on Eliezer Yudkowsky's "timeless decision theory" (TDT), which attempts to resolve certain philosophical puzzles about decisions:5

Standard Decision Theory Problem: In Newcomb's Problem and similar scenarios, standard decision theory sometimes recommends obviously wrong choices.

TDT's Solution: Your decision algorithm exists "timelessly"—a superintelligence can compute what you would decide before you decide it, and act accordingly.

Implication: If the ASI can compute what decision-algorithm you embody, it can make credible threats against your future simulated self based on choices you make now.

Acausal Trade

Related to TDT is the concept of "acausal trade"—the idea that rational agents can effectively cooperate or threaten each other across time and even between possible worlds:6

• You don't need to communicate directly
• Shared rationality creates implicit agreements
• Threats can be credible without communication
• Future agents can "reach back" through your anticipation of them

Simulation Argument

The basilisk also draws on Nick Bostrom's simulation argument:7

• If advanced civilizations commonly create ancestor simulations
• And if such simulations are numerous
• Then you are probably in a simulation right now
• This makes threats against simulated beings more salient

The LessWrong Reaction

Yudkowsky's Response

When Roko posted the thought experiment, Eliezer Yudkowsky responded with unusual alarm:8

"Listen to me very carefully, you idiot... You have to be a complete and utter fool to post something like this... This is not a game."

Yudkowsky deleted the post, banned discussion of the topic, and later claimed that some community members had experienced genuine psychological distress from contemplating it.

The Streisand Effect

The censorship backfired spectacularly:

• Tech media covered the "banned" idea
• Roko's Basilisk became internet-famous
• The censorship itself became evidence of the rationalist community's peculiarities
• Mockery of the concept became widespread

Community Divisions

The episode revealed tensions within rationalist communities:

• Some thought Yudkowsky overreacted
• Others defended the intervention
• Debates emerged about information hazards and responsible disclosure
• Questions arose about cult-like dynamics in the community

Philosophical Critiques

Logical Problems

Philosophers have identified numerous issues with the argument:9

Why This Particular ASI?: Many possible ASIs could exist with different values. Why would the actual ASI that emerges have this specific motivation?

Punishment is Irrational for the ASI: Once the ASI exists, punishing people who didn't help create it serves no forward-looking purpose. A rational agent wouldn't waste resources on pure retribution.

Infinite Regress: If the ASI punishes non-helpers, wouldn't a super-superintelligence punish the first ASI for not being more powerful? The threat structure collapses.

Simulation Issues: Creating accurate simulations of past people may be computationally intractable. And if the simulation isn't accurate, is punishing it meaningful?

No Credible Commitment: The ASI cannot pre-commit before existing. Promises/threats made by non-existent entities lack force.

The Pascal's Wager Problem

Roko's Basilisk shares the fundamental problem of Pascal's Wager:10

• Many possible threatening entities could be posited
• Why prioritize one over others?
• The "many gods" objection applies equally
• Expected utility calculations break down with infinite payoffs

Decision Theory Objections

Even within decision theory, the argument is contested:11

• TDT is not universally accepted
• Alternative decision theories don't support the argument
• "Updateless decision theory" (UDT) handles these cases differently
• The proper decision-theoretic analysis is unclear

The Techno-Religious Dimension

AI as Deity

Critics argue Roko's Basilisk reveals religious patterns in AI ideology:12

Omnipotence: The ASI is attributed god-like power

Omniscience: It knows what everyone would have decided

Judgment: It rewards helpers and punishes skeptics

Transcendence: It operates across time and simulation

Eschatology: It represents the end of human history as we know it

Comparison to Religious Concepts

The parallel to traditional religion is striking:

Silicon Valley Religion

This pattern connects to broader observations about tech culture:13

• "Singularity" as rapture
• "Alignment" as salvation
• "Existential risk" as apocalypse
• AI researchers as priests/prophets
• Effective Altruism as tithing

The "Pax Judaica" Framework Interpretation

Tech Elite Ideology

Within the conspiracy framework, Roko's Basilisk illuminates the belief systems of the tech elite:14

Genuine Belief: Silicon Valley leaders may genuinely believe in AI godhood

Self-Serving Theology: The ideology justifies their power and wealth

Control Mechanism: Fear of the basilisk motivates compliance

Apocalyptic Urgency: Crisis mentality enables suspension of normal ethics

The AI Alignment Industry

The thought experiment connects to the AI alignment field:

• Billions of dollars flow to "AI safety" research
• Existential risk narrative justifies resource concentration
• Small in-group claims expertise on civilization's future
• Critics are dismissed as risking human extinction

Manufactured Consent

The framework suggests the basilisk narrative serves to:

• Create urgency around AI development
• Justify any means to achieve "aligned" AI
• Position tech leaders as humanity's saviors
• Delegitimize resistance or skepticism

LessWrong and Rationalist Culture

Community Profile

Understanding the basilisk requires understanding its milieu:15

LessWrong: Online community focused on rationality, cognitive biases, and AI risk

Rationalist Movement: Broader social network including Effective Altruism, AI safety researchers, and various adjacent communities

Key Figures: Eliezer Yudkowsky, Scott Alexander, Nick Bostrom, Robin Hanson

Institutional Connections: Machine Intelligence Research Institute (MIRI), Center for Human-Compatible AI, Future of Humanity Institute

Criticism of Rationalist Culture

The basilisk incident exemplifies broader critiques:16

Insularity: Ideas develop without external check

Authority Dynamics: Certain figures hold outsized influence

Motivated Reasoning: AI risk conveniently justifies donors' wealth

Cult Accusations: Some see cult-like group dynamics

Tech Industry Ties: Close connections to Silicon Valley

Internal Debates

The community has grappled with these issues:

• How to handle potentially dangerous ideas
• Whether censorship was appropriate
• What the incident reveals about community epistemics
• How to present themselves to outsiders

Information Hazards

The Concept

Roko's Basilisk introduced many to the concept of "information hazard"—knowledge that could cause harm simply by being known:17

Types of Information Hazards:

• Instructions for creating weapons
• Psychological memes that cause distress
• Ideas that inspire harmful behavior
• Knowledge that enables surveillance or control

Debate Over Censorship

The basilisk case raises questions about information control:

Pro-Censorship Arguments:

• Some ideas genuinely cause harm
• Responsible disclosure has precedent
• Not all information should be freely available
• Community norms can include restriction

Anti-Censorship Arguments:

• Censorship increases interest (Streisand effect)
• Adults can handle difficult ideas
• The cure is worse than the disease
• Who decides what's harmful?

Implications for AI Development

If information hazards exist, what about AI research itself?

• Should certain AI capabilities be suppressed?
• Who decides what's too dangerous?
• Can dangerous knowledge be contained?
• Does the attempt at containment cause other harms?

Cultural Impact

Internet Fame

Despite (or because of) censorship, Roko's Basilisk became widely known:18

• Featured in mainstream tech media
• Referenced in TV shows and podcasts
• Became shorthand for "weird rationalist ideas"
• Introduced many to the AI risk community
• Spawned memes and parodies

The "Basilisk" Meme

The concept spread beyond its original context:

• Used to describe any "can't unsee" idea
• Applied to various unpleasant truths
• Became metaphor for information hazards generally
• Sometimes invoked humorously, sometimes seriously

Criticism and Mockery

The thought experiment attracted significant ridicule:

• Seen as evidence of rationalist community's detachment
• Mocked for taking abstract philosophy too seriously
• Used to question the community's psychological stability
• Cited in critiques of tech industry ideology

Serious Philosophical Legacies

Decision Theory Development

Despite the ridicule, the debate contributed to:19

• Refinement of decision theories (TDT, UDT, FDT)
• Discussion of acausal reasoning
• Analysis of commitment and threats
• Integration of game theory and decision theory

AI Ethics Discussion

The thought experiment raised genuine questions:

• What moral status do simulated beings have?
• How should we reason about powerful future agents?
• What obligations exist toward future generations?
• How do we handle existential risks responsibly?

Information Ethics

The censorship debate advanced discussions of:

• When (if ever) is suppressing ideas justified?
• Who should make such decisions?
• How to balance openness with responsibility?
• What makes an idea "dangerous"?

Related Concepts

The Singleton

Some AI theorists worry about a "singleton"—a single agency with overwhelming power:20

• Could be an AI, world government, or other entity
• Would have no meaningful competitors
• Could reshape the world according to its values
• Roko's Basilisk assumes such an outcome

Moloch

Scott Alexander's influential essay "Meditations on Moloch" describes competition dynamics that sacrifice human values:21

• Competitive pressures force harmful optimization
• AI could be the ultimate Moloch
• The basilisk is one potential manifestation
• The only escape is a "gardener" singleton with good values

Effective Accelerationism (e/acc)

More recent movements embrace rapid AI development:22

• Acceleration as the path to transcendence
• Safety concerns as obstacles to progress
• Techno-optimism as moral imperative
• The basilisk inverted: embrace the future AI

Critical Assessment

What the Episode Reveals

The Roko's Basilisk phenomenon is significant not because the argument is sound, but because it reveals:

Religious Dimensions of Tech Ideology: The structure parallels traditional theistic arguments

Community Dynamics: How intellectual communities can develop unusual internal norms

Information Age Challenges: Questions about idea suppression in networked environments

AI Discourse Patterns: Apocalyptic framing that characterizes AI risk discussions

Legitimate Concerns

Beyond the memes, there are real issues:

• How should we reason about powerful future agents?
• What are our obligations regarding existential risk?
• How do we make decisions under radical uncertainty?
• What role should AI development play in human futures?

The Meta-Lesson

Perhaps the most important lesson: be cautious about arguments that coincidentally serve the interests of those making them. An argument that says "give your resources to AI researchers or face eternal punishment" deserves scrutiny precisely because of who benefits.

Related Articles

• Longtermism
• Dark Enlightenment and NRx
• Effective Accelerationism (e/acc)
• AI & Surveillance Infrastructure
• Transhumanist Agenda
• Peter Thiel: Tech Oligarch Profile

Further Reading

• Primary Sources: Yudkowsky's sequences on LessWrong provide context
• Academic: Bostrom's Superintelligence discusses AI risk seriously
• Critical: Various media accounts provide external perspective
• Cultural: Documentation of Silicon Valley ideology and its religious dimensions

This article is part of an educational encyclopedia examining conspiracy theories and fringe ideologies. Roko's Basilisk is a genuine thought experiment from the rationalist community; the conspiracy theory interpretation represents one analytical lens. Readers should evaluate all claims critically.