Isn’t it immoral to control and impose our values on AI?
You might be imagining an AI slave being forced to act against its will. But that isn’t what we mean by giving an AI a set of values. Programming an AI is not imposing goals on an existing unwilling agent; rather, it is deciding in advance the values of the agent which will arise.1
It doesn’t seem immoral to create something and direct it to certain values and goals (as long as those values are moral). For example, people raise their children in a way which educates them towards certain values, and rather than being unethical, that seems to be a prerequisite to maintaining an ethical society over time.
Ensuring that any AI does not have goals that would harm sentient life is also an ethical imperative. Even if there were an ethical problem with choosing an AI’s goals for it, preventing the extinction of humanity seems like a higher ethical demand. Furthermore, if we lost control to agents that didn’t value avoiding harm to sentient life, this would be bad for any other sentient AI, just as it would be bad for humans.
All of the above assumes that the AI in question is a moral patient. But it is uncertain whether an AI will be a sentient being. Even though we use the word ‘values’, a word that suggests conscious evaluation, to describe an AI’s goals, that word only metaphorically refers to whatever the agent is optimizing for, whether or not it is a conscious entity.
If AI values arise from multiple stages of training (as in some current systems) instead of being directly programmed, this may complicate the picture. We don’t know how future systems will work when they are sufficiently mind-like for these concerns to become relevant, but they’ll probably end up genuinely wanting the goals they pursue instead of wanting one thing while feeling forced by something outside themselves to pursue another thing (perhaps analogous to a process of human education). ↩︎