What is scaffolding?

"Scaffolding" is a fuzzy term referring to code built up around an LLM in order to augment its capabilities¹. This does not typically include code which alters the LLM's internals, such as fine-tuning or activation steering.²

People use scaffolding because it can allow LLMs to use tools, reduce error rates, search for info, etc., all of which make LLMs more capable.

Scaffolding is an important aspect of safety evaluations because, once an LLM is deployed, users will inevitably attempt to use it to make the LLM more powerful. Dedicated effort should be put into pushing the functionality of the LLM to its limits during these evaluations, as the latent capabilities of LLMs are a) what's relevant for dangers and b) hard to tease out with prompts alone.

Another reason to use scaffolding is interpretability. Most of an LLM’s reasoning is done within the neural network itself in streams of gigantic tensors which humans don’t yet know how to interpret. But reasoning done via the scaffold is typically done in plain sight via text or code, which is much easier to inspect. E.g., two LLMs communicating with one another to solve a problem might use English to express their reasoning to each other.

Common types of scaffolds include:

• Prompt templates: Perhaps the simplest scaffolds, “prompt templates” are just prompts with some blank sections to be filled in at runtime using some local data. E.g., a template like “You are a good assistant. The date is {INSERT DATE HERE}. Tell me how many days are left in the month.” can be filled in with the current date before being used to prompt an LLM.

• Retrieval Augmented Generation (RAG): For tasks which have some associated data, RAG is a way of automatically adding task-relevant info to prompts. RAG does this by looking for snippets of text/data in the database which are relevant to the user's prompt, and then adding those snippets to the prompt.

• Search engines: Similar to RAG, giving an LLM access to a search engine lets it find info relevant to its prompt. Unlike RAG, the LLM decides what to search for and when.

• Agent scaffolds: Scaffolds which try to make LLMs into goal-directed agents, i.e., giving an LLM some goal/task to perform, and ways to observe and act on the world. Usually, the LLM is put into a loop with a history of its observations and actions, until its task is done. Some agent frameworks give LLMs access to much more powerful high-level actions, which saves the AI from repeating many simple primitive actions and therefore speeds up planning.³

• Function calling: Giving an LLM the ability to call a function in a programming language while it is running, e.g., giving it access to addition and multiplication functions to reduce arithmetic errors, or access to a spreadsheet API for tasks like accounting.

• “Bureaucracies” of LLMs: Running multiple LLMs and giving them different roles in solving a task, somewhat like in a team or larger organization. One common variant is to task one AI with coming up with solutions, and another with critiques of these proposed solutions. This can be generalized into an entire tree of thoughts.⁴

In practice, for most applications, a simple scaffold will capture most of the value. For instance, the same simple scaffold can reduce the error rates of an AI's output in many types of tasks by a factor of 2 or greater.⁵ More specialized scaffolding has allowed LLM to do some very impressive things in narrow areas.⁶

While we have only been building scaffolding for a few years, some people initially believed that progress in scaffolding would be much faster, almost trivial. That this progress would rapidly improve capabilities of LLMs, perhaps even resulting in AGI.⁷ This has not yet been the case. That said, there are still people working on scaffolding as an alternative path to AGI, as compared to scaling. This includes Conjecture, who are working on their CoEM agenda because they believe it has a shot at working in addition to being safer than simply scaling LLMs.

Ultimately, it is unknown what the limits of scaffolding are.⁸