When an AI agent solves a problem, it hunts through web pages, reads source code, and queries databases across many turns. The evidence it needs to answer the original question is scattered across all those turns. But here is the problem: the standard way we train these agents actively discards most of that evidence during training. Tool responses are masked out. The model learns which button to press next, but never learns to connect distant clues into a coherent answer.
Agent Context Compilation (ACC) is a disarmingly simple fix. Instead of training turn-by-turn, ACC gathers every tool response and environment observation the agent collected, shuffles them into one long context window alongside the original question, and trains the model to answer directly from that compiled evidence. Think of it as giving a detective all the case files at once, rather than handing them one document at a time and asking what to search for next.
The result: a 30-billion-parameter model trained with ACC matches a 235-billion-parameter model on MRCR and GraphWalks, the hardest long-range reasoning benchmarks available. On MRCR, the score jumps from 50.2 to 68.3 (+18.1). On GraphWalks, from 69.9 to 77.5 (+7.6). General capabilities like GPQA and MMLU-Pro hold steady. Under the hood, the model develops task-specific attention patterns — different experts specialise for different reasoning tasks, a flexibility not seen in the baseline.
A detective collects fingerprints from six rooms, photographs three crime scenes, and interviews ten witnesses. But during debriefing, the sergeant only asks: "Which room did you search next?" Nobody ever asks what the detective found. That is how we currently train AI agents.
An agent trajectory consists of $k-1$ interaction turns followed by a final answer turn:
where $r_t$ is the reasoning at turn $t$, $a_t$ is the action, $o_t$ is the tool response (observation), and $(r_k, y)$ is the final reasoning-answer pair. The standard agent SFT objective supervises only the model-generated tokens:
where $I_t = r_t \cup a_t$ for $t < k$ and $I_k = r_k \cup y$. The observations $o_t$ are excluded from the loss. Grouping by turn reveals two components:
The first $k-1$ terms supervise only local reasoning and tool selection. Any gradient relevant to the final answer $y$ must back-propagate through a long chain of intermediate turns to reach $o_t$, and is heavily weakened. This is the supervision blind spot.
See how the gradient signal from the final answer decays as it passes through masked observations. Drag the sliders to explore.
Not all evidence is created equal. A web page reads differently from a database table, which reads differently from a Python source file. ACC works across all three — and their diversity turns out to be crucial.
ACC applies to three representative agent classes, each producing trajectories with distinct evidence structures:
Each agent type produces trajectories of different lengths. Click the legend to toggle agent types.
If the problem is that evidence is scattered and the gradient cannot reach it, the solution is obvious in hindsight: gather all the evidence into one place, then train directly on it.
ACC converts each trajectory into a training example $\tau_i \mapsto (x_i, y_i, r_i)$, producing a dataset $\mathcal{D} = \{(x_i, y_i, r_i)\}_{i=1}^{M}$. Here $x_i = (q_i, C_i)$ combines the original query with the compiled context, $y_i$ is the final answer, and $r_i$ is the reasoning trace. The new training objective:
Unlike $\mathcal{L}_{\text{agent}}$, this objective contains no intermediate action terms. The final answer supervision reaches every evidence token directly without being filtered through turn-level tool selection.
See how gradient signal flows differently through the two training objectives. Drag the slider to change the number of trajectory turns.
Gathering evidence is only half the battle. How you arrange it — and what you leave out — determines whether the model learns to find signals in noise or just memorises positions.
For each trajectory, ACC extracts structured evidence pieces $\text{Evi}(\tau) = [e_1, \ldots, e_m]$ and applies a random permutation $\pi$ over $\{1, \ldots, m\}$:
where $B$ is the token budget (131,072 tokens). Because evidence pieces are self-contained, shuffling forces the model to locate relevant information via semantic association rather than sequential position. Distractors are added to increase difficulty.
Watch how ACC assembles a compiled context from a search trajectory. Evidence pieces are shuffled; distractors are mixed in. Click "Shuffle" to randomise the order.
A 30-billion-parameter model should not be able to compete with a 235-billion-parameter one. On long-range dependency benchmarks, ACC makes it happen.
Compare models across MRCR and GraphWalks benchmarks. Click legend entries to toggle models.
Long-context training often raises a fear: does specialising for long-range reasoning come at the cost of general intelligence? The short answer: no.
Compare baseline vs ACC-trained model across general capability benchmarks. No degradation — slight improvements.
ACC does not just change what the model knows — it changes how the model thinks. Attention patterns restructure. Experts specialise. And the restructuring is task-specific, not one-size-fits-all.
For layer $l$ and head $h$, the per-layer per-bin mean attention is:
The reported heatmap shows $\Delta_{l,b} = \mu^{\text{SFT}}_{l,b} - \mu^{\text{Base}}_{l,b}$. Positive values indicate increased attention mass at that distance after ACC training.
Select a task to see how attention and expert routing change after ACC training.
Which ingredients actually matter? If you remove one agent type, or strip the distractors, how much do you lose? The ablations tell a clean story: every component contributes, and the whole is greater than the sum.
See how each training configuration affects MRCR and GraphWalks scores relative to the baseline.
The full ACC setup compiles 10,802 trajectories (Search: 3,369; SWE: 4,368; SQL: 3,065) with context lengths from 2K to 128K tokens. Training uses sequence length 131,072 tokens, global batch size 16, learning rate $1 \times 10^{-5}$ with cosine schedule, AdamW optimizer ($\beta_1=0.9$, $\beta_2=0.999$, weight decay 0.1), and 4 epochs.