Empty shelves or lost keys? Recall is the bottleneck for parametric factuality

Factuality is essential for making Large Language Models (LLMs) reliable. When a model answers a factual question incorrectly, is it because the fact was never encoded, or because the fact is encoded but not accessible? Standard accuracy metrics collapse these cases together, even though they suggest very different limitations and very different interventions. Encoding failures call for scaling model size or expanding data coverage, while recall failures might also point to post-training and inference-time methods that help LLMs better utilize what they already encode.

In “Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality”, we introduce knowledge profiling, a behavioral framework that measures both encoding and recall, and use it to examine the underlying bottlenecks of factuality in frontier LLMs (such as Gemini3 and GPT-5). We then show that many factual errors in frontier LLMs are better understood as lost keys (recall failures), not empty shelves (encoding failures).

By analogy, we use encoding to denote parametric representation of facts, recall to denote retrieving encoded facts without external cues, and recognition to denote identifying the correct fact when it is presented among alternatives. To support this analysis, we introduce WikiProfile, a benchmark of 2,150 Wikipedia-derived facts, each paired with ten questions that probe encoding, recall, and recognition.

The core idea: Knowledge profiling

Knowledge profiling shifts the unit of analysis from individual questions to facts. Instead of asking whether a model answered a specific question correctly, we ask a broader question: what is the state of the fact? We classify each fact into one of five knowledge profiles: (1) encoding failure, (2) recall failure, (3) direct recall, (4) recall with thinking, and (5) inference without encoding. These profiles provide a more informative diagnosis than question-level accuracy alone.

We operationalize this with three behavioral notions: