Memory You Can Train
Agent memory is crossing from retrieval plumbing to trained behavior — metamemory, memory-as-action, learned context representations, and governed persistent state.
§1 · The amnesia bill, revisited
Agent memory grew up in one direction: outward. More stores, more retrieval hops, more context stuffed per turn — memory as an ever-larger annex bolted to a model that never changes how it remembers. The July 2026 literature marks the turn inward. Wei Zhou and colleagues at Shanghai Jiao Tong — writing with the MemTensor team — describe where the outward path arrived: agent memory has "rapidly evolved from simple retrieval-augmented mechanisms into a data management system" supporting persistent storage, retrieval, update, consolidation, and dynamic lifecycle governance. And then they name the embarrassment: this whole system is still evaluated through end-to-end task scores — F1, BLEU — as a monolithic black box, leaving operational costs, architectural trade-offs between memory modules, and robustness under dynamic knowledge updates largely unmeasured (Zhou, W., et al., 2026).
A data management system evaluated only by whether the application above it succeeds: that is where "add more RAG" ends. The interesting papers of this batch all make the same move in response — they stop adding storage and start training the behavior: what to encode, how to represent it, when and how to fetch it, and how to govern what persists. Memory as something you train, not something you add.
[← B5] The Memory Stack mapped the June-era systems layer — stores, consolidation, context management. This article is about the layer acquiring gradients. The serving-side mechanics of caches at inference time stay in [→ 11]; skills-as-capital stays in [← 4].
Agent memory has become a data-management system still graded like a black box. The 2026 turn is inward: instead of adding stores, the new work trains the memory behavior itself — encoding, representation, access, and governance.
§2 · Metamemory: managing memory is a skill
The anchor of the turn is AutoMem, from Serena Yeung-Levy's group at Stanford (Wu et al., 2026). Its premise imports a concept from cognitive science: metamemory — knowing what to encode, when to retrieve, how to organize — is a learned skill, not a fixed faculty. The system design follows the premise all the way down. File-system operations are promoted to first-class memory actions alongside task actions, so the model itself decides how to manage its memory rather than delegating to a fixed pipeline.
What makes AutoMem a Series-D paper is its answer to the obvious question — who improves the memory skill? The authors observe that the skill improves along two axes that both resist manual optimization: the structure supporting it (prompts, file schemas, the action vocabulary) and the proficiency of the model exercising it. Long-horizon episodes run for thousands of steps, and a single memory mistake can hide long before it surfaces — human review of full trajectories is impractical. So AutoMem automates both axes as two loops: in the first, a strong LLM reviews complete agent trajectories and iteratively revises the memory structure; in the second, the agent's own good memory decisions become the training signal for the skill itself. It is the four-substrate story of this series in miniature — a memory substrate improved by an evaluator-audited loop, not by an engineer.
AutoMem treats memory management as trainable metamemory: file operations become first-class actions, and two automated loops improve both the structure (schemas, vocabulary) and the proficiency (the model's decisions) — because thousand-step episodes make manual memory engineering unreviewable.
§3 · Trained representations: the store itself learns
One level down, the artifact in the store is changing from text to trained tensors.
Cartridges. The Stanford Hazy Research group starts from the serving bill: answering queries over a large corpus by stuffing it into a 100K–1M-token context is costly precisely because KV-cache memory scales with input length (Eyuboglu, Ehrlich, Arora, et al., 2025). Their alternative inverts the ritual — train a small KV cache offline, once per corpus, through their self-study recipe; load that Cartridge at inference; amortize the training cost across every query that ever references the same corpus. The context window stops being a reading room and becomes a compiled artifact.
Still. Baseten's compaction paper makes the deployment constraints explicit: a compactor must be callable during inference, expressive under constraint, and reusable across a trajectory — and prior methods satisfy only part of that (selection is lightweight but subset-bound; synthesis is expressive but needs per-context optimization). Still is a small per-layer Perceiver trained once against a frozen base model that emits compact keys and values in a single forward pass (O'Neill et al., 2026). On Qwen and Gemma models it holds the favorable side of the speed–quality frontier across compression ratios from 8× to 200× and context lengths from 8k to 128k, and on the long-context RULER grid it exceeds the strongest baseline by 8–22 points.
HOLA. The architectural wing of the same idea comes from a single-author paper by Wanyun Cui: linear-attention and state-space models buy O(1) memory by compressing the prefix into a fixed-size recurrent state — a lossy exact memory where competing key–value associations overwrite each other and needle recall degrades. HOLA gives linear attention a hippocampal complement, in the spirit of Complementary Learning Systems: the delta-rule state keeps modeling compressible structure, while a bounded exact KV cache stores the associations that should not be forced through it (Cui, 2026). The write rule needs no learned eviction — it keeps tokens with a large committed prediction residual — and a decoupled cache read turns those exact pairs into sharp retrieval rather than soft averaging. At 340M parameters trained on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 to 22.92 — a 16.1% drop that lands below a full-attention Transformer++ baseline at 26.88 (see Figure 2).
The store itself now learns: corpora compile into trained Cartridges amortized across queries; a once-trained per-layer compactor (Still) holds the speed–quality frontier from 8× to 200× compression and gains 8–22 RULER points; and a hippocampal exact cache (HOLA) beats both its linear-attention baseline and a full-attention reference on perplexity.
§4 · Memory as action
The third front trains not the store but the access policy.
Navigation, not retrieval. The Qwen application team's NapMem makes the inversion explicit: passive retrieval interfaces make the model "a consumer of pre-selected evidence." NapMem organizes long-term user history into a linked, multi-granularity memory pyramid — raw conversations, typed memory records, topic tracks, user profiles, connected by provenance relations — and exposes each level through memory tools. The agent is then trained to select memory according to the query and its intermediate evidence, inspecting different granularities before answering (Xu et al., 2026). Memory becomes a structured action space; fetching becomes a learned behavior with feedback, exactly parallel to the metamemory move of §2 but aimed at reading rather than writing.
Steering, not instructing. Chengfeng Zhao and colleagues at CAS identify a failure of the text-based alternative: injecting explicit textual guidelines by RAG suffers a text-action disconnect — symbolic instructions frequently fail to activate the internal representations that produce correct execution. Their Neural Procedural Memory is training-free but decidedly not text: it distills procedural skills from historical contrastive episodes into implicit activation steering, applied inside the network at execution time (Zhao, C., et al., 2026). Procedural memory as a vector field rather than a style guide — early-stage, but pointed exactly where §2's file schemas cannot reach.
Replaying, not pruning. Between store and policy sits the harness, and Yanjun Zhao's team at UIUC contributes the harness-level version: ReContext, a training-free method built on the observation that models fail to use evidence already present in their input — a gap between context access and context utilization. It uses model-internal relevance signals to build a query-conditioned evidence pool and replays it before final generation, preserving the full original context, with a theoretical account grounded in associative memory (Zhao, Y., et al., 2026). No training, no external store, no pruning — just an organized second pass over what was already there.
Access is becoming a policy: NapMem trains navigation over a provenance-linked memory pyramid; Neural Procedural Memory swaps textual guidelines for activation steering to close the text-action disconnect; ReContext shows even a training-free organized replay recovers evidence the model had but failed to use.
§5 · The reality check at a million tokens
Every trained-memory design above competes against one default: skip the machinery and put everything in context. The Berkeley study of corpus-scale in-context retrieval is the first systematic measurement of that default at the scales practical retrievers face — million-token corpora, and length generalization far beyond training sizes (Gollapudi et al., 2026). Their purpose-built 0.6B LM retriever, BLOCKSEARCH, improves on prior LM baselines and length-generalizes to 10× beyond its training regime — and then retrieval still collapses under more extreme extrapolation. The diagnosis is the durable contribution: an attention dilution effect. As the corpus grows, irrelevant documents dominate the softmax denominator; the normalized attention mass on the gold document falls even when its pre-softmax score stays high. The model still "knows" where the answer is, and the arithmetic of normalization buries it anyway.
That mechanism is the quantitative floor under this whole article. Context-as-memory does not degrade because models are badly trained; it degrades because attention is a normalized resource that dilutes with corpus size. Structured stores, compiled representations, and trained access policies are not conveniences — they are how a memory system escapes an arithmetic that punishes scale.
At million-token scale, in-context retrieval hits attention dilution: irrelevant mass in the softmax denominator buries gold documents whose raw scores remain high. The context window loses to structure not by engineering taste but by arithmetic — which is the standing justification for every trained memory in this article.
§6 · Governed state
What deployment adds to all of this is accountability. The Always-On survey — Tianyu Ding and colleagues — widens "memory" to what production systems actually persist: not just retrievable memories but task ledgers, permissions, credentials, commitments, provenance and audit records, shared state, trigger conditions, and externally committed effects (Ding et al., 2026). They read a 435-work coded corpus through six diagnostic axes per state item — authority, scope, mutability, provenance, recoverability, actionability — and through a full lifecycle in which state is written, validated, organized, retrieved, acted upon, updated, forgotten, audited, and sometimes rolled back. Their headline finding is an imbalance this series should take personally: the literature concentrates on accumulating and retrieving state, far more than on governing, recovering, or relinquishing it. Their response, a pilot Always-On Evaluation Protocol (AOEP-v0), makes governance scoreable rather than aspirational.
Put §2–§5 next to that finding and the loop card writes itself. A trained memory is a memory that changes itself under feedback — which is precisely the kind of state whose authority, provenance, and recoverability need governing. The more the store learns, the more the ledger matters.
| Layer (what trains) | System | Mechanism | Reported evidence |
|---|---|---|---|
| Management policy (write/organize) | AutoMem | metamemory as trainable skill; two loops over structure + proficiency | ⚠️ framework result; thousand-step episodes motivate automation (Wu et al., 2026) |
| Stored representation | Cartridges | per-corpus KV cache trained offline via self-study; amortized | ✅ replaces length-scaling KV of 100K–1M-token ICL (Eyuboglu et al., 2025) |
| Stored representation | Still | per-layer Perceiver, trained once, single-pass compaction | ✅ 8×–200× compression, 8k–128k contexts; +8–22 RULER vs strongest baseline (O'Neill et al., 2026) |
| Architecture (state + cache) | HOLA | delta-rule state + bounded exact KV, residual-gated writes | ✅ Wikitext ppl 27.32→22.92 (−16.1%), below Transformer++ 26.88 (Cui, 2026) |
| Access policy (read) | NapMem | trained navigation over a provenance-linked memory pyramid | ⚠️ trained selection over granularities; personalization benchmarks (Xu et al., 2026) |
| Access (training-free bounds) | ReContext / NPM | evidence replay from internal signals; activation-steered procedures | ⚠️ training-free gains; text-action disconnect named (Zhao, Y.; Zhao, C., 2026) |
| The in-context default | BLOCKSEARCH study | corpus-scale ICL retrieval, 0.6B retriever, 10× length-gen | ❌ collapses beyond; attention dilution mechanism (Gollapudi et al., 2026) |
SIGNAL — complete agent trajectories with memory actions logged (writes, schema changes, retrievals), plus the six governance attributes per state item (authority, scope, mutability, provenance, recoverability, actionability — Ding et al., 2026).
UPDATE — the memory skill, on both AutoMem axes: structure (schemas, action vocabulary) via trajectory review, proficiency via the agent's own vetted memory decisions (Wu et al., 2026).
GUARDRAIL — a governance ledger with rollback: no memory-structure change without provenance and recoverability; needle-recall probes watched for dilution-style degradation as stores grow (Gollapudi et al., 2026).
CHECK — a fixed long-horizon task suite re-run monthly: memory-action quality (correct encode/retrieve decisions) and recall-at-scale both non-decreasing while store size grows.
Deployment-grade memory is governed memory: six auditable attributes per state item, a lifecycle that includes forgetting and rollback, and — per the 435-work survey — a literature that badly over-weights accumulation over governance. Train the memory, and govern what you trained.
What comes next
A memory system answers questions about the past. The bolder bet of 2026 is a substrate that answers questions about the future — predict the next state of the world, not the next token of text, and let perception, prediction, and control share one model. Three labs converged on that bet within a month of each other. The world-model turn is next.
References
- Wu, S., Zhu, H., Zhang, Y., Wang, X., & Yeung-Levy, S. (2026). AutoMem: Automated Learning of Memory as a Cognitive Skill. arXiv preprint. arXiv:2607.01224.
- Eyuboglu, S., Ehrlich, R., Arora, S., Guha, N., Zinsley, D., Liu, E., Tennien, W., Rudra, A., Zou, J., Mirhoseini, A., & Ré, C. (2025). Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study. arXiv preprint. arXiv:2506.06266.
- Zhou, W., Zhou, X., Han, S., Xu, H., Li, G., Li, Z., Xiong, F., & Wu, F. (2026). Are We Ready For An Agent-Native Memory System? arXiv preprint. arXiv:2606.24775.
- Ding, T., Nannapaneni, A., Liu, B., & Zhang, L. (2026). Always-On Agents: A Survey of Persistent Memory, State, and Governance in LLM Agents. arXiv preprint. arXiv:2606.30306.
- Xu, Y., Sun, Y., Liu, Y., Zhou, M., Qiao, J., Ma, L., Tang, K., Wang, W., Jiang, X., & Jiang, G. (2026). From Passive Retrieval to Active Memory Navigation: Learning to Use Memory as a Structured Action Space. arXiv preprint. arXiv:2607.05794.
- Gollapudi, S., Gupta, N., Singhal, P., & Min, S. (2026). Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale. arXiv preprint. arXiv:2607.01538.
- Cui, W. (2026). A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets. arXiv preprint. arXiv:2607.02303.
- Zhao, C., Tan, Y., He, S., Wang, Y., Zhao, J., & Liu, K. (2026). Neural Procedural Memory: Empowering LLM Agents with Implicit Activation Steering. arXiv preprint. arXiv:2606.29824.
- Zhao, Y., Qiu, R., Wei, T., Bei, Y., Liu, Z., Chen, L., Lourentzou, I., Tong, H., & He, J. (2026). ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning. arXiv preprint. arXiv:2607.02509.
- O'Neill, C., Sandomirsky, A., Partridge, H., Jayasekara, M., & Kirkby, M. (2026). Still: Amortized KV Cache Compaction in a Single Forward Pass. arXiv preprint. arXiv:2606.07878.