What are "true names" in the context of AI alignment?

True names are precise mathematical formulations of intuitive concepts that capture all the properties that we care about for those concepts. “True names” is a term introduced by alignment researcher John Wentworth and might be inspired by “true names” in popular culture, and how knowing these is supposed to grant you power over the thing being talked about.

John Wentworth gives many examples of true names. Concepts like "force", "pressure", "charge" and "current" are all things that were once poorly understood, based on vague intuitions about the physical world, but for which we now have extremely robust mathematical and operational formulations, as well as instruments to measure them.

To put it another way, a “true name” can be thought of as a mathematical formulation that robustly generalizes as intended. An important property of true names is that they are not susceptible to failing via Goodhart's law when faced with the immense optimization pressure of a future superintelligence. Since alignment researchers are interested in finding mathematical measures that are “non-Goodhartable”, they also care about finding true names. However, non-Goodhartability is just one property of true names. Robustness to optimization might be a necessary condition to conclude that we are dealing with a true name, but it isn't the definition of true names.

Many alignment researchers care about human values. It would be a huge boon for AI alignment efforts if we could discover a robust formulation or a “true name” of human values. Currently, we use proxies of what humans truly care about in AI models in order to measure how well a given model performs. The use of these proxies often results in side effects through things like reward misspecification or specification gaming. However, if we had a “true name” for human values which we could optimize for, then we would not need to worry about undesired side effects or unforeseen consequences.

In addition to human values, alignment researchers also seek true names for components of agency such as optimization, goals, world models, abstraction, counterfactuals, and embeddedness.