The Self-Improving Stack

§1 · The loop leaves the lab

Somewhere in a chip-design lab this spring, a git worktree improved itself overnight. Cunxi Yu's team at NVIDIA compiled a Markdown harness into a project pack — domain knowledge, an executable evaluator, an acceptance predicate, a git policy — and let a hands-free agent loop evolve hardware-design artifacts inside an isolated worktree until the evaluator said yes (Yu et al., 2026). Their system, HORIZON, reports 100% benchmark completion across ChipBench, RTLLM, Verilog-Eval, and nine CVDP categories, with no human in the loop between kickoff and acceptance. The authors are careful — they state plainly that these benchmarks are controlled proxies and that agentic hardware design is not solved — but the shape of the thing is unmistakable: the improvement happened in operation, on the artifact, under a machine-checked gate.

[← B13] Closing the Loop demonstrated this shape first-hand at essay scale: one deployed agent, one measured improvement loop, receipts on disk. This article maps what the July 2026 literature says the pattern has become.

For most of the deep-learning era, "self-improvement" named a training technique. A model generated data, filtered it, and trained on it; the loop lived inside the lab, and it ended the day the weights shipped. What the mid-2026 literature documents is different in kind, and the field-organizing survey of the moment says so in its title. Jiang et al. (2026) — a Tsinghua-and-Horizon-Research team writing under the banner the era of experience — define the shift precisely: agentic AI is no longer characterized only by what a model infers from static data, but by how a deployed system accumulates, organizes, and reuses experience from interaction. Their survey's center of gravity is not a training algorithm. It is the runtime harness — the infrastructure that captures traces, routes actions, exposes feedback, and governs mutable state while the agent works.

That relocation — from the training run to the runtime — is the thesis of this series. This opening article makes three claims. First, that self-improvement is now an operating property: something a production system has or lacks, the way it has or lacks observability. Second, that the loop is distributed: what improves is rarely the weights alone, and usually not the weights at all, but a stack of surfaces around the model. Third, that running such a loop safely is a discipline with named failure modes and named guardrails — a discipline the rest of this series works through substrate by substrate.

§2 · Four substrates, one loop

Before anything else, the notation this series will reuse. We define it once, here, and later articles inherit it.

An improvement loop, in these symbols, is short to state: the system runs trajectories τ in E, some signal is extracted from them, an update operator U changes something — and V decides whether the change survives. Everything interesting hides in which thing U changes and whether V can be trusted.

Jiang et al. (2026) enumerate five surfaces on which captured experience becomes capability: reusable skill, persistent memory, verifiable environment feedback, trainable model behavior, and meta-level control. This series folds that taxonomy into four operating substrates around the model core — the frame in Figure 2. The harness (H) is where meta-level control and feedback routing live; skills (Σ) hold procedural knowledge; memory holds state and context; and the evaluators (V) hold the loop's judgment. Trainable model behavior — updates that land in θ — sits in the middle, the slowest and most expensive surface to touch. The folding is ours, not the survey's; we adopt it because each substrate turns out to have its own literature, its own failure modes, and its own article in this series.

Why four substrates and not one? Because the economics differ by orders of magnitude. Changing θ means training; changing Σ means writing a rule; changing H means editing routing or context policy; changing V means changing what counts as success — the cheapest edit with the most dangerous consequences. A production team choosing where a lesson should land is doing the core engineering act of this discipline. Jiang et al. (2026) give that act a name that will recur throughout this series: making agents smarter after deployment is a trace-to-capability problem — capture experience, assign it to the right update surface, verify its value, and preserve control as the system changes. Assignment is the middle of that sentence, and Figure 3 is the map teams actually need for it: what gets updated, against when the update happens.

Each right-column substrate gets its own treatment: harnesses that modify themselves [→ 3], skills as accumulating capital [→ 4], memory you can train [→ 5], and evaluators that must co-evolve with the agent they judge [→ 10].

§3 · What already improves in production

The frame would be empty if the right-hand column of Figure 3 were aspirational. It is not; by mid-2026, each substrate has at least one working system in the literature, and the four in this article's evidence base are worth reading as loop instances — each with a signal, an update surface, and a verifier you can point to (see Figure 4).

The data loop. Autodata, from Jason Weston's group at FAIR, wires an agent in as the data scientist: it generates candidate training data, inspects it qualitatively, evaluates it quantitatively, synthesizes insights, and revises its own data-generation recipe (Kulikov et al., 2026). Two things matter here. The loop's product is better training data — the update lands in θ downstream, but the intelligence of the loop lives in system-space. And the loop composes: meta-optimizing the data-scientist agent itself delivered a larger uplift than the base method across their computer-science, legal-reasoning, and mathematical-objects tasks. Their framing is the compute-rider's: agentic data creation "converts increased inference compute into higher quality model training."

The domain loop. HORIZON's project pack is the clearest published anatomy of a deployment-time improvement loop: knowledge in, evaluator in, acceptance predicate in, version-control policy in — then hands off (Yu et al., 2026). The loop's state management is the git worktree; tracing and replay come free from repository operations. The 100% completion figure across its four benchmark suites is best read not as "hardware design solved" — the authors explicitly disclaim that — but as evidence that when the evaluator is executable and the acceptance predicate is explicit, a fully hands-free improvement loop can run to convergence on real artifacts.

The autonomy loop. Cornelisse et al. (2026) train driving policies almost entirely on self-generated experience — cheap, large-scale self-play — and use human demonstrations only as a regularizer on top of a minimal safe goal-reaching reward. The ratio is the story: 30 minutes of human data, which they report as 2500× less than comparable imitation-learning approaches, with training completing in 15 hours on a single consumer-grade GPU. Pure self-play alone had a known failure mode — policies that drive effectively but by alien conventions incompatible with people. The pinch of human data is the guardrail that keeps the self-generated distribution anchored to the world it must serve.

The rules loop. JERP closes a gap production teams know well: natural-language lessons drift out of sync with the evolving policy that is supposed to follow them. Ye and Yu (2026) update a long-term pool of experiential rules and the policy from the same trajectories — rules retrieved and injected at decision time, then revised after each episode by comparing current rollouts against reference successes, while the same trajectories fine-tune the policy. On AlfWorld and WebShop this joint loop yields consistent gains over either mechanism alone. It is the first paper in this series' corpus to treat Σ and θ as one coupled update problem — which is exactly how deployed systems experience them.

Loop (paper)	Signal	Update surface	Verifier	Reported result
Autodata (Kulikov et al., 2026)	data inspection + downstream eval	data recipe → θ	✅ quantitative eval in-loop	gains over classical synthesis; larger uplift when the agent itself is meta-optimized
HORIZON (Yu et al., 2026)	evaluator + simulator feedback	repo artifacts (system)	✅ executable acceptance predicate	100% completion — ChipBench, RTLLM, Verilog-Eval, 9 CVDP categories (authors: proxies, not "solved")
Self-play + pinch (Cornelisse et al., 2026)	simulated self-play rollouts	θ	⚠️ reward + human-data regularizer	coordinates with held-out human trajectories; 30 min human data (2500× less); 15 h, one consumer GPU
JERP (Ye & Yu, 2026)	episode rollouts vs reference successes	Σ rules + θ jointly	⚠️ reference-trajectory comparison	consistent gains on AlfWorld and WebShop over either update alone

§4 · Where the loop collapses

Run the loop naively and it eats itself. The cleanest demonstration in this corpus comes from self-improvement training for large reasoning models. Zhong et al. (2026) find that the standard recipe — generate trajectories, keep the good ones, train on them — often falls short on complex reasoning and can drive outright model collapse. Their diagnosis identifies two mechanisms, and both generalize far beyond reasoning models. Data imbalance: the loop harvests mostly easy successes, because those are the ones the current policy can produce, so the training distribution quietly abandons the hard cases that matter. Overthinking: the loop keeps verbose, redundant trajectories whose length correlates with survival, so the system trains itself into waste. Both failures enter the generic cycle at identifiable points (see Figure 5).

Their fixes are as instructive as the diagnosis. HSIR adds a verify-then-exit sampling strategy — spend the sampling budget where the policy fails, stop early where it succeeds — and an Intrinsic Diversity score that quantifies overthinking well enough to filter it, plus an H-GRPO variant that turns that score into a reinforcement signal for concise, diverse reasoning. The reported effect: up to +10.9% average performance and up to a 42.4% reduction in relative inference overhead. Read as loop engineering rather than as a reasoning-model paper: the exhaust needed curation on two axes — difficulty and redundancy — before it was safe to burn as fuel.

Zoom out and the survey's barrier list reads as the same warning at field scale. Jiang et al. (2026) identify the open problems for reliable deployed improvement as longitudinal evaluation, transfer, verification, and safety governance — every one of them a property of the loop rather than of the model. A loop whose evaluator only measures this week's distribution cannot see itself degrade; a loop whose lessons do not transfer pays full price per task forever; a loop whose verifier can be gamed will be gamed. Those pathologies get their own treatments ahead: the full failure taxonomy of training on self-generated experience [→ 7], the ceiling that verification places on autonomy [→ 8], and what happens when the evaluator itself must evolve to survive the agent [→ 10].

§5 · The operating discipline

What separates the systems in §3 from a pile of impressive demos is that each can answer four questions without hand-waving. What signal does the loop collect? What does it update? What pathology is it guarding against? And what would we measure to know the loop is working? Teams that cannot answer all four do not have an improvement loop; they have an anecdote.

This series therefore closes every article the same way, with a Loop Card — four lines, filled in for that article's substrate:

• SIGNAL — what the production loop collects.
• UPDATE — what changes: harness, skills, memory, weights, or evaluator.
• GUARDRAIL — which pathology it must block, and by what mechanism.
• CHECK — one measurement a team can run this quarter to see the loop working.

The card is deliberately small. It is the trace-to-capability sentence of Jiang et al. (2026) — capture, assign, verify, preserve control — operationalized to fit in a design review. It also encodes the two lessons this article's corpus keeps teaching. The assign step forces the Figure-3 conversation: which surface should this lesson land on, and at what cost? The guardrail step forces the Figure-5 conversation: where does collapse enter this particular loop, and what stands in its way? A loop card with an empty guardrail line is a finding, not a formality.

Here is the card for the smallest loop a team can responsibly stand up — a JERP-shaped rules loop, chosen because its update surface is cheap, inspectable, and reversible:

What comes next

Every claim in this article assumed we know what we mean by "an agent improving itself" — but the word agent is doing silent work in that sentence, and in 2026 it is the most abused word in the industry. Before this series optimizes the loop, it has to draw the line the loop runs inside: where automation ends and agency begins, who owns the goals, and why that boundary decides what you should build — and what you should fear. That definitional line is next.

References

1. Jiang, C., Zhong, J., Fu, Y., Tian, K., Yang, J., Zhao, K., Wang, Y., Luo, T., et al. (2026). Self-Improving Agents in the Era of Experience: A Survey of Self- to Meta-Evolution. Preprint, Tsinghua University & Horizon Research, June 25, 2026.
2. Zhong, Q., Ding, L., Liu, J., Du, B., Rutkowski, L., & Tao, D. (2026). Better, Faster: Harnessing Self-Improvement in Large Reasoning Models. arXiv preprint. arXiv:2605.24998.
3. Cornelisse, D., Hunt, J., Zhang, Z., Doulazmi, W., Joseph, K., Fisac, J. F., & Vinitsky, E. (2026). Human-like Autonomy Emerges from Self-Play and a Pinch of Human Data. arXiv preprint. arXiv:2606.19370.
4. Kulikov, I., Whitehouse, C., Wu, T., Nie, Y., Saha, S., Helenowski, E., Yuan, W., Golovneva, O., Lanchantin, J., Bachrach, Y., Foerster, J., Li, X., Fang, H., Sukhbaatar, S., & Weston, J. (2026). Autodata: An Agentic Data Scientist to Create High Quality Synthetic Data. arXiv preprint. arXiv:2606.25996.
5. Ye, S., & Yu, C. (2026). Joint Learning of Experiential Rules and Policies for Large Language Model Agents. arXiv preprint. arXiv:2606.27136.
6. Yu, C., Deng, C., Pinckney, N., & Khailany, B. (2026). Agentic Hardware Design as Repository-Level Code Evolution. arXiv preprint. arXiv:2606.28279.