To understand why so many are concerned, including the founders of DeepMind and tech entrepreneurs like Elon Musk, Chivers explores several thought experiments. One involves a machine programmed to make paper clips. Its end goal eventually becomes “a solar system in which every single atom has been turned into either paper clips, paper-clip-manufacturing machines, computers that think about how best to manufacture paper clips…” and so on.
The point is that even “thoroughly innocuous-seeming goals could be an existential threat”. And it’s extremely difficult to predict what can go wrong. Chivers raises real examples, including an AI tasked with winning an online Tic-Tac-Toe tournament. The program quickly learned to make moves billions of squares away from the board, forcing the opposing algorithm to crash and thereby winning the game by default. Another that was meant to replicate a set of text files as closely as possible ended up deleting those files. It then turned in blank sheets to get perfect scores.