An AI might do bad things even if its designers only intend to use it for good^[1]. To give a simplistic, but illustrative example, if an AI is given the goal of eradicating cancer in humans the most reliable way to do so might be to kill humanity instead of finding a cure.

Complexity of Value: Human values aren’t as simple as they may seem

A quick fix for this situation might look like changing the AI’s goal from “eradicate cancer” to “eradicate cancer while preserving human flourishing”. But here we run into another roadblock: we don’t know how to mathematically define human concepts like “flourishing”, and as a result, we can’t explicitly program them into the AI.

Orthogonality Thesis: Intelligence and human values need not go together

There might come a point when an AI is intelligent enough to understand what we truly mean by “preserve human flourishing”, so one might think it will just stop doing anything that doesn’t line up with our intent. However, just because the AI understands what we intend, that doesn’t mean it will actually do what we intend, because it won’t “care” about following our intent unless we explicitly train it to. Unfortunately, we don’t presently know how to train an AI to “care” about following our intentions, and we can’t expect it to care by default, because intelligence and morality are orthogonal.

The complexity of value and orthogonality theses demonstrate why it is not easy to ensure that an AI tries to do what we want it to. Furthermore, there is reason to think that the most useful actions an AI could take toward its own goals might be harmful to us.

Instrumental Convergence: Certain bad behaviors are useful for most AI goals

For almost any goal that an AI could have, there are instrumental goals it could pursue that make it more likely to achieve its final goal, such as acquiring resources, staying operational, and preserving its goals. This could be catastrophic for humanity, because an AI with these instrumental goals may end up using resources humans need to live to further its goals. Moreover, it may take extreme measures to prevent us from turning it off or altering its goals, because human interference might lead to its final goal not being achieved.

All in all, an AI does not need to be malevolent to do bad things – it’s enough for it to be indifferent to what we care about. This is why it’s crucial to align AI with human values.

It can also be used intentionally to do bad actions; this would count as misuse. ↩︎

Why would an AI do bad things?.link_svg__st0{opacity:.6;enable-background:new}.link_svg__st1{fill:gray}