What is Vingean uncertainty?

Vingean uncertainty is a situation where you are more certain about the outcome, despite being less certain about the path that will lead there.

For example, if you were playing a game of chess against grandmaster Magnus Carlsen or the latest version of the Stockfish chess engine, you would surely lose. Even if you had memorized thousands of common openings, had developed counter-moves for each of them and just couldn’t see how they would beat you, you still would expect to lose.

This situation occurs whenever you are trying to predict the behavior of someone (or something) much more intelligent or competent than yourself. Because you know they are competent you can predict that they will achieve their goals, but you cannot predict the specific steps that they will take to get there without yourself being similarly intelligent.

A major challenge with dealing with a superintelligence is that we'll be in a state of Vingean uncertainty regarding its actions. An adversary with significantly greater intelligence within a specified domain will reliably outmaneuver us, even if we do not know the specific way this will happen. We will be able to predict that it will be successful at its goals, but we won’t be able to predict its specific actions. This means that we will be unable to predict negative side effects its actions might have and that we will struggle to stop it from carrying out its plans.