What is the Abstraction and Reasoning Corpus (ARC)?
The Abstraction and Reasoning Corpus (ARC)1 is a corpus of tests designed to measure machine intelligence. It was introduced by François Chollet in order to test "intelligence" as defined in his paper “On the Measure of Intelligence”. ARC is designed to require the ability to generalize and to resist being solvable through rote memorization. While most human children can perform well on ARC tasks, unmodified LLMs2 tend to perform poorly on it (as of 2024).
Each question in the test is composed of a few examples of sets of colored pixels on a grid, where the AI must guess the pattern and apply it to the test grid.
Further reading:
Not to be confused with the Alignment Research Center, also abbreviated as ARC. ↩︎
Some models specifically trained for this task have near-human performance. ↩︎