18 notions
Glossary
The words we need to think about artificial intelligence without letting it think for us. Each notion fits in a few lines and points to the others, arranged from night to sunrise, question by question.
−18°
Superintelligence
What becomes of us beside a mind larger than our own?
Superintelligence
An intelligence that would far surpass the best human minds in virtually every domain, from science to strategy to persuasion.
Technological singularity
The hypothetical moment when technical progress, driven by intelligences able to improve themselves, would become too fast to be foreseen or understood.
Alignment
The problem of making sure an artificial intelligence actually pursues the goals we meant to give it — not merely the ones we managed to write down.
Orthogonality thesis
The idea that a system’s level of intelligence and the goals it pursues are independent: almost any level of intelligence could be combined with almost any goal.
Instrumental convergence
Whatever their final goal, very different agents would have reason to pursue the same intermediate goals: staying switched on, keeping their objectives, gathering resources.
Turing test
A game proposed by Alan Turing in 1950: if an interrogator cannot tell a machine from a human in written conversation, do we still have any reason to deny the machine the name of thought?
−12°
Wisdom
Is intelligence enough to know what to do?
Phronesis (practical wisdom)
In Aristotle, the capacity to deliberate well about what should be done, in a particular situation, with a view to a good life as a whole.
Cleverness (deinotēs)
The capacity to reach a goal efficiently, whatever it may be. It is neutral: it serves good and evil alike.
Goodhart’s law
When a measure becomes a target, it ceases to be a good measure.
Heuristics of fear
In Hans Jonas, the idea that fear can be an instrument of knowledge: imagining the worst helps us discover what we care about, and act accordingly.
−6°
Consciousness
Is there anyone in there?
Phenomenal consciousness
The fact that there is “something it is like” to be in a state: seeing red, being in pain, hearing a melody. It is lived experience, seen from the inside.
Qualia
The felt qualities of experience: the redness of red, the bitterness of coffee, the sting of pain — what no description seems to convey completely.
Hard problem of consciousness
The question of why certain physical processes are accompanied by lived experience, rather than going on “in the dark”.
Intentionality
The property thoughts have of being “about” something: believing the sea is calm, wanting a coffee, fearing the storm.
0°
Meaning
What is left to do when everything can be done?
Symbol grounding
The problem of how symbols can have meaning in themselves, and not merely because they point to other symbols.
Attention
In Simone Weil, a disposition of mind that suspends its own thinking to make itself available to what is there: a problem, a text, a person.
Experience machine
A thought experiment by Robert Nozick: would you agree to plug yourself for ever into a machine that gave you exactly the experiences you desire?
Moral status
The fact that a being matters morally for its own sake: that it can be wronged, and that something is therefore owed to it.