Superintelligence

Nick Bostrom, Superintelligence: Paths, Dangers, Strategies, Oxford University Press, 2014

An intelligence far superior to ours is possible, perhaps near, and controlling it could be the most important — and most difficult — problem humanity has ever had to solve.

What the text says

Bostrom examines the possible paths to superintelligence, the speed of its arrival, and above all its dangers. He formulates several ideas that have become central: the orthogonality thesis, according to which intelligence does not guarantee good goals; instrumental convergence; the “treacherous turn”, in which a system would appear cooperative for as long as it is weak.

The second half of the book is devoted to the control problem: how to limit the capabilities of such a system, or pass on values to it, without knowing in advance everything we want.

Why read it now

Published when these scenarios seemed remote, the book shaped the vocabulary of AI safety. One may contest its assumptions, its speculative style or its priorities. Reading it remains the most direct way to understand why researchers take these risks seriously.

What remains open

Can we control what we cannot understand — and should we?