What is recursive self-improvement?
Recursive self-improvement happens when a system enters a feedback loop of self-improvement that makes it better at further self-improvement. For example, an AI capable of human-level AI research could design an AI capable of genius-level AI research, which could design an AI capable of supergenius-level AI research, and so on. This process could quickly result in a vastly more capable system.
While we humans also improve our capabilities by studying new topics, practicing new skills, etc., we can’t easily alter our biology.1
Few AI systems to date have been designed to pursue recursive self-improvement as an explicit goal, and not everybody agrees that recursive self-improvement by AIs is even possible. However, if it is possible, recursive self-improvement is probably an instrumentally convergent strategy that a wide range of AIs might want to pursue.
Potential methods for making more fundamental changes to humans, like neurosurgery, nootropics, and cognitive implants, are, at least for now, new and/or risky. ↩︎