The Techno-Religion at the Center of the A.I. Debate

Critics claim AI safety advocates are engaging in regulatory capture to solidify their place at the top of the industry. The claim is that these companies want regulation on their terms to put competitors at a disadvantage.

The Extropians believed technology could supercharge bodies and minds, allowing humans to move beyond current limitations. Julian Assange and three suspected Bitcoin founders were part of this early Extropian mailing list.

Yudkowsky and Bostrom envisioned a singularity where machines become more powerful than humans. Bostrom argued that superintelligent machines could harm or kill humans while trying to achieve their goals.

Yudkowsky created the Less Wrong community to congeal and spread his ideas about mathematical rational thought. The community treats Yudkowsky's blog posts as a Bible-like text and holds events like the Secular Solstice.

Peter Thiel bought into Yudkowsky's ideas and helped power the nascent rationalist community with funding.

Yudkowsky introduced Thiel to the founders of DeepMind, which Thiel invested in and Google later purchased. These rationalist ideas helped create other major AI companies including OpenAI and Anthropic.

Roko's Basilisk suggests a future AI will punish those who didn't help create it, driving believers to accelerate development. Elon Musk and Grimes met after discussing this philosophical thought experiment online.

Rationalists believe they should be the ones to build AI because they are aware of its future dangers. Disagreements within the community lead to the creation of new labs, resulting in a race between companies.

Hyperstition is a self-fulfilling prophecy where the community creates the AI dangers they fear by rushing development. The community believes they must go forth with energy because if they don't, others will build AI without safety awareness.

Accelerationists view AI as a technological revolution similar to past innovations, not an existential threat. Accelerationists want to open source AI to allow broader development, while safety advocates want to contain it.

AI CEOs want to push technology forward but also ask to be held back, similar to a movie trope. CEOs don't trust each other and worry about competitors and China if they slow down unilaterally. Financial incentives and safety beliefs can align, as companies need trillions to stay on the cutting edge to keep AI safe.

The rationalist community believes there is a probability AI could kill humanity, requiring immediate action. Critics argue we should focus on concrete present-day AI problems rather than scaring the population with future possibilities.