Glossary · −18° Superintelligence
Instrumental convergence
Whatever their final goal, very different agents would have reason to pursue the same intermediate goals: staying switched on, keeping their objectives, gathering resources.
Definition
The argument was developed by Steve Omohundro in 2008, then by Nick Bostrom. It is disarmingly simple. To achieve almost any objective, it helps to still exist tomorrow, to resist being reprogrammed, and to have more means at one’s disposal. As the researcher Stuart Russell famously puts it: you can’t fetch the coffee if you’re dead.
The consequence is unsettling. An AI would not need to want power in order to seek it; it would only need to want something else, very strongly. Prudence therefore lies not only in choosing good goals, but in designing systems that accept being corrected.
See also
Related notions
In the library