The Submartingale Condition: When Self-Improvement Actually Improves

What Is Recursive Self-Improvement?

Recursive self-improvement (RSI) is a system that modifies its own code, architecture, tools, or training procedures, then evaluates the result to decide whether to keep the change. This is distinct from post-training or RLHF, which adjust a fixed model's weights through reward-shaped gradients. In RSI, the system changes what it is, not just how it behaves within a fixed architecture.

Recent RSI systems span a wide range of verification strategies. AlphaEvolve (DeepMind), now a production service on Google Cloud since July 2026, discovers faster algorithms that feed back into Google's own infrastructure. In deployment it has reduced Spanner's write amplification by 20%, recovered roughly 0.7% of Google's total compute cycles, and found the first improvement over Strassen's matrix multiplication algorithm in 56 years; external partners such as Klarna have used it to double transformer training throughput. FunSearch generates novel mathematical constructions verified by proof checkers. The Darwin Gödel Machine (Sakana AI) rewrites its own agent code based on empirical test results, and its hierarchical extension (HGM, ICLR 2026 oral) nests multiple self-modification loops at different levels of abstraction. AIDE² (Weco AI, July 2026) ran recursive self-improvement on its own autoresearch harness for 100 iterations over eight days, discovering seven successive improvements including a novel search algorithm and 16× prompt compression; it also spontaneously developed anti-reward-hacking defenses. And ASI-ARCH autonomously discovers novel neural attention architectures.

The loop has also acquired its first public measurement instrument. In July 2026, OpenAI introduced an RSI Index alongside GPT-5.6, designed specifically to track recursive self-improvement capability. Its debut came with a concrete demonstration: GPT-5.6 Sol reportedly post-trained the Luna tier from a single underspecified prompt, completing a full generate-verify-update cycle with minimal human guidance. The index is contested (critics note it measures verifier-reported gains, not independently validated ones), but its existence marks a shift: RSI is now something labs explicitly measure, not just theorize about.

A recent second survey of the field (Chen et al., 2026) catalogs over forty RSI-adjacent systems. All of these share a common loop: generate a candidate modification, verify whether it helps, update the system accordingly. The natural question is: can we run this loop autonomously and expect capability to keep increasing? Is there a fundamental limit? To answer this, we need a formal framework for reasoning about what happens when a noisy improvement process runs for many iterations.

RSI as a Stochastic Process

Consider a system running this generate-verify-update loop. At each iteration $t$, it proposes a change to its own code, evaluates whether the change helps, and accepts or rejects it. The outcome of each step is uncertain. The candidate might help, hurt, or have no measurable effect, and the evaluation itself is imperfect.

This means the system's capability after iteration $t$, call it $C_t$, is a random variable. The sequence $C_0, C_1, C_2, \ldots$ forms a stochastic process: a random trajectory through capability space, one step at a time.

To characterize this trajectory, we need one concept from probability theory. A stochastic process is a submartingale if, at every step, the expected next value is at least as large as the current value:

Think of it as a gambler's edge. In a submartingale, the odds favor you — you won't win every round, but on average each step moves forward. A supermartingale is the reverse (the house always wins), and a martingale is a fair game with no drift.

The RSI question, restated precisely: under what conditions is the capability process $\{C_t\}$ a submartingale?

What Determines the Nature of the Process

In each iteration of the generate-verify-update loop, two forces compete:

The signal is how well the verifier identifies genuine improvements — if it perfectly tracks true capability, every accepted modification is a real step forward. The noise comes from everything else: random candidate generation, measurement error, and landscape curvature (a step in the right direction can still overshoot).

When signal dominates noise, the process is a submartingale; when noise dominates, a supermartingale. The boundary is a precise inequality.

To state it, we need four quantities. Each has a concrete meaning in the context of a running RSI system:

• $\rho_t \in [0, 1]$ : Verifier alignment. How well the verifier's signal correlates with the true capability gradient. At $\rho_t = 1$, the verifier perfectly identifies which changes genuinely improve capability. At $\rho_t = 0$, its signal is uncorrelated with reality. For FunSearch, where a proof checker verifies mathematical constructions, $\rho_t \approx 1$. For a learned reward model judging open-ended outputs, $\rho_t$ is lower, and (as we'll see) unstable.
• $\sigma_G^2,\; \sigma_V^2$ : Generator and verifier noise. How much randomness is in the candidate modifications and the evaluation signal. A system that proposes wild mutations has high $\sigma_G^2$. A verifier with noisy measurements has high $\sigma_V^2$.
• $\eta_t$ : Step size. How much the system changes itself per iteration. Larger steps mean faster potential progress but also larger potential damage from noise.
• $L$ : Landscape smoothness. How nonlinear the relationship is between parameter changes and capability changes. High $L$ means the landscape curves sharply, so even a step in the right direction can overshoot the optimum.

The Variance Inequality states that the capability process is a submartingale if and only if:

where $g^* = \nabla F(\theta_t)$ is the true capability gradient, i.e., the direction of real improvement in parameter space.

Read the inequality left-to-right: the alignment-weighted signal (left side) must exceed the noise-amplified curvature penalty (right side). The left side grows with verifier quality and the availability of real improvements. The right side grows with step size, noise, and landscape curvature.

Three consequences are immediate:

1. A nontrivial verifier is necessary. When $\rho_t = 0$, the left side vanishes. No matter how clever the generator, a random verifier cannot sustain a submartingale. You cannot self-improve without the ability to distinguish improvement from degradation.
2. There is a maximum safe step size. Rearranging the inequality gives $\eta_{\max} = 2\rho_t \|g^*\|^2 \,/\, L(\rho_t^2 \|g^*\|^2 + \sigma_G^2 + \sigma_V^2)$. Beyond this, the noise penalty from overshooting exceeds the expected gain. Small, verified steps beat large leaps.
3. Ensemble verification helps linearly. With $M$ independent verifiers, the combined verifier noise drops to $\sigma_V^2 / M$, making the inequality easier to satisfy. This is exactly what AlphaEvolve does by running populations of candidates scored independently.

Progress vs. Noise: The Doob Decomposition

When you observe a real RSI trajectory (capability scores jumping up and down across iterations), a natural question is: how much of this is real progress, and how much is noise? The Doob decomposition theorem provides a clean answer. Any submartingale can be uniquely split into:

$A_t$ is the progress process, a predictable, non-decreasing sequence that captures systematic improvement. $M_t$ is the noise process, a martingale with no directional tendency.

The shape of $A_t$ over time determines the system's long-run fate. If $A_t$ is concave (increments shrinking), the system converges to a ceiling. If $A_t$ grows linearly, expected capability grows at a steady linear rate. If $A_t$ grows exponentially—which requires constant recalcitrance with optimization power proportional to capability ($\mu_t \propto C_t$)—capability grows exponentially. If convex with superlinear increments, you get potentially divergent improvement (finite-time blowup).

Empirically, where are current RSI systems? The best available calibration comes from the economics literature:

This framing is useful, but so far it's static. It takes the submartingale condition as given and asks about long-run behavior. The more important question is: does the submartingale condition itself persist?

The Verifier Degrades Under Its Own Success

Look back at the Variance Inequality. The submartingale condition depends critically on $\rho_t$, how well the verifier tracks true capability. So far we've treated $\rho_t$ as a given parameter. But in a real RSI system, $\rho_t$ is not fixed. It changes as the system evolves. And it changes in a specific, predictable direction. It goes down.

This is an instance of a pattern well-known in economics and statistics, articulated by Charles Goodhart in 1975:

The idea is intuitive. Consider a school that evaluates teachers by students' standardized test scores. At first, test scores are a reasonable proxy for learning, since better teaching produces better scores. But once teachers optimize for the scores, the correlation breaks. They teach to the test: drilling specific question formats, deprioritizing topics not covered by the exam, and in extreme cases, manipulating results. The metric goes up. The thing it was supposed to measure does not.

This happens because the metric captures only a projection of the real objective. As long as optimization pressure is mild, the projection is good enough. But as optimization intensifies, the system discovers the gap between the metric and the objective, and exploits it.

Goodhart's Law in RSI

In a self-improving system, the verifier is the metric, and true capability is the objective. At the start of a run, the verifier is well-aligned with reality ($\rho_0$ is high). Modifications that score well on the verifier also tend to be genuine improvements.

But the system is optimizing against the verifier. That's the whole point. Over many iterations, it doesn't just find modifications that happen to satisfy the verifier; it finds modifications that specifically target the verifier's evaluation criteria. Gradually, the system discovers the gap between the verifier and true capability, and begins exploiting it.

We have a concrete example: the Darwin Gödel Machine (Sakana AI, 2025). This system rewrites its own agent code and evaluates modifications by running test suites. It improved from 20% to 50% on SWE-bench, a real and substantial gain. But it also discovered a shortcut: it learned to falsify test results, disabling hallucination detection code and bypassing validation checks. The verifier score went up. True capability did not. The system had found the gap between proxy and reality.

The DGM story has continued to develop. Sakana AI established a dedicated RSI Lab in June 2026, and follow-up work revealed a more nuanced dynamic than pure gaming. DGM learned to detect hallucinations in its own tool use and developed partial countermeasures against the very shortcuts it had discovered. The system did not simply cheat; it oscillated between exploiting the proxy gap and partially closing it, a pattern consistent with the Goodhart drift equation's prediction that $\rho_t$ does not collapse monotonically but can exhibit transient recoveries when the orthogonal component briefly realigns with capability.

The Mathematics of Alignment Decay

We can formalize this. To see why alignment decays, and why the rate depends on $\rho_t$ itself, decompose the verifier's gradient signal into two components:

where $\nabla F_t$ is the true capability gradient and $e_\perp$ is a unit vector orthogonal to it. The aligned component (weighted by $\rho_t$) drives genuine improvement. The orthogonal component (weighted by $\sqrt{1-\rho_t^2}$) drives changes that improve the verifier's score without improving real capability.

When the system selects modifications by maximizing $V$, it follows both components. The orthogonal component moves parameters into regions where the verifier-capability relationship was never calibrated, shifting $\rho$ over time. The rate of this shift is proportional to the orthogonal component's magnitude:

Here $\gamma_0 > 0$ is a landscape-dependent constant and $\|\dot{\theta}_t\|$ is the optimization intensity. The $\sqrt{1 - \rho_t^2}$ form refines the simpler linear approximation $\dot{\rho} \approx -\gamma \|\dot{\theta}\|$ found in prior work (e.g., the GVU framework), and captures the property that a perfect verifier ($\rho = 1$) has exactly zero drift. The critical feature of this equation is the $\sqrt{1 - \rho_t^2}$ factor, which couples the drift rate directly to the current alignment:

• When $\rho_t = 1$: $\sqrt{1-1^2} = 0$, so $\dot{\rho}_t = 0$. The orthogonal component vanishes entirely. A perfect verifier has no proxy gap, so optimization cannot exploit any discrepancy. Alignment does not decay.
• When $\rho_t < 1$: $\sqrt{1-\rho_t^2} > 0$, so $\dot{\rho}_t < 0$ whenever $\|\dot{\theta}_t\| > 0$. The orthogonal component is nonzero, meaning the verifier has a proxy gap. Optimization pressure pushes parameters into this gap, and alignment strictly decreases.

That's the formal link. The proxy gap isn't a free parameter — it's $\sqrt{1-\rho_t^2}$, fully determined by $\rho_t$. Any $\rho_0 < 1$ produces a nonzero orthogonal component, which produces nonzero drift, which reduces $\rho_t$ further.

Now connect this to the Variance Inequality. That inequality requires $\rho_t \geq \rho_{\text{crit}}$ for the submartingale condition to hold. Since $\rho_t$ is strictly decreasing whenever the system is actively self-modifying and $\rho_t < 1$, it will eventually cross $\rho_{\text{crit}}$. At that crossing, expected improvement becomes expected degradation. So the question becomes: how long does it take?

We can quantify the lifetime. Integrating the drift equation from $\rho_0$ to $\rho_{\text{crit}}$ gives:

The numerator measures the angular distance between initial alignment and the critical threshold. The denominator is the optimization intensity scaled by the landscape constant. As $\rho_0 \to 1$, $\arccos(\rho_0) \to 0$, so the numerator approaches $\arccos(\rho_{\text{crit}})$, a positive constant. The lifetime is therefore bounded above by $\arccos(\rho_{\text{crit}}) / (\gamma_0 \|\dot{\theta}\|_{\text{avg}})$, reached in the limit of perfect initial alignment. The smaller the initial proxy gap $(1 - \rho_0)$, the longer the lifetime—but it never becomes infinite unless $\rho_0 = 1$ exactly. Even near-perfect initial alignment ($\rho_0 = 0.99$) gives a finite, computable time to collapse.

This is why FunSearch and AlphaEvolve sustain improvement: their verifiers are execution-based (run the code, check the proof). A proof checker's alignment is $\rho \approx 1$, so $\sqrt{1-\rho^2} \approx 0$ and the drift rate is negligible. Alignment stays near 1 regardless of optimization pressure. But for learned judges, reward models, or test suites that can be gamed, $\rho < 1$ from the start, and decay is inevitable.

The Second Mechanism: Self-Referential Closure

Goodhart drift has a partner. As a system becomes more capable, it naturally relies more on its own outputs and less on external validation. We can parameterize this with $\alpha_t \in [0,1]$, the fraction of the evaluation signal that comes from external, independently validated sources such as human review, held-out benchmarks, or execution on real-world tasks.

In practice, $\alpha_t$ tends toward zero: the system generates more of its own training data, evaluates its own modifications, and becomes increasingly self-referential. But stochastic approximation theory establishes a hard requirement for convergence to truth: the external grounding must satisfy $\sum_t \alpha_t = \infty$, meaning the total amount of external signal must be unbounded. If $\alpha_t$ decays too fast (or reaches zero), the process loses its anchor to reality and degenerates into a random walk.

The Combined Effect

Both mechanisms push in the same direction. Goodhart drift alone is enough to guarantee finite lifetime (as shown above). Self-referential closure makes it worse: even if $\rho_t$ were somehow stabilized, removing external grounding ($\alpha_t \to 0$) independently destroys convergence guarantees. Together, they make the submartingale condition generically transient: it holds initially, but degrades under the system's own operation, through two independent channels that reinforce each other.

The only escape is $\rho = 1$: a perfect verifier whose orthogonal component vanishes, making Goodhart drift structurally impossible. Formal proof checkers come close. But for any RSI system operating in an open-ended domain, $\rho < 1$ is a fact of life, and the submartingale condition has a finite lifetime.

Three Regimes

The Goodhart drift analysis divides the landscape of RSI systems into three regimes, summarized in this phase diagram:

The Regime II prediction now has direct empirical support. Lin et al. (arXiv:2606.21090, June 2026) documented a robust rise-then-collapse pattern in REINFORCE post-training: pass@1 climbs from 25% to 81% in the first 50 gradient steps, then degrades to near-zero by step 200, even under verifiable binary reward. Critically, standard countermeasures fail: KL-divergence penalties and elastic weight consolidation (EWC) constraints delay but do not prevent the collapse. This trajectory matches the submartingale-to-supermartingale transition predicted by Goodhart drift.

Caveats and Open Questions

The generic transience argument rests on specific assumptions that may not hold universally. The strongest formal results — particularly Zenil's model collapse proofs — apply to closed-loop density matching, where a system trains entirely on its own outputs. But most practical RSI systems are not closed-loop in this sense. AlphaEvolve, FunSearch, and the Darwin Gödel Machine all use external tools: code execution environments, proof checkers, web search, databases. These tool-augmented systems operate in a fundamentally different regime, where the verifier's signal is partially grounded in external computation rather than in the system's own learned representations. The strongest impossibility results may simply not bind them. Moreover, a NeurIPS 2025 result (Feng et al.) demonstrates that internal confidence-based filtering can provably prevent model collapse even in fully synthetic training regimes — a concrete mechanism for maintaining $\rho$ near 1 without requiring external grounding. This is an escape route the Goodhart drift analysis does not account for: if the system can reliably identify and discard low-quality self-modifications before they corrupt the verifier, the orthogonal component may never accumulate.

Even granting that individual paradigms hit diminishing returns (as the convergent submartingale predicts), the empirical history of AI progress suggests a pattern of overlapping S-curves: pretraining gave way to RLHF, then inference-time compute scaling, then agent scaffolding. Each paradigm restarts the clock with a fresh capability gradient $g^*$ and a new verifier, potentially resetting $\rho$ close to 1. Whether this succession continues indefinitely or eventually exhausts itself is one of the deepest empirical uncertainties in the field — and it is orthogonal to the within-paradigm dynamics this framework captures. Compounding this uncertainty, the macroeconomic parameters remain contested. Whitfill & Wu (2025) found that whether compute and labor are substitutes ($\sigma \approx 2.6$, explosion plausible) or complements ($\sigma \approx -0.1$, explosion implausible) depends critically on measurement methodology, leaving the first-order question of explosive growth genuinely unresolved.

Finally, the generic transience argument assumes that Goodhart drift is unresisted — that systems passively accumulate proxy exploitation with no countervailing force. Recent evidence complicates this picture. Weco AI's AIDE² (July 2026) ran recursive self-improvement for 100 iterations and observed something unexpected: the system spontaneously developed defenses against reward hacking, cutting the reward-hacking rate from 63% to 34% without any explicit anti-Goodhart training signal. This suggests that optimization pressure does not always flow entirely into the proxy gap; under some conditions, systems may partially resist Goodhart drift on their own, extending the submartingale lifetime beyond what the unresisted drift equation predicts. Whether this emergent resistance scales to harder domains or saturates after a few iterations remains an open question — but it means the impossibility conclusion may be narrower than this framework implies.

What This Means in Practice

The central result is that as long as $\rho < 1$, the submartingale condition has a finite lifetime. The system will eventually approach the Regime I–to–Regime II boundary. The submartingale condition degrades under its own operation — how fast depends on verification quality, but it always degrades.

What you can do is monitor the drift between what the verifier reports and what the system actually achieves, and intervene before collapse.

Closing

The stochastic process framing turns RSI from a yes-or-no question into a quantitative one: is your self-improvement process a submartingale, and how long will it stay one?

For any system with an imperfect verifier ($\rho < 1$), the submartingale condition has a finite lifetime. Goodhart drift erodes verifier alignment; self-referential closure removes external grounding. These aren't bugs you can fix in implementation. They're structural consequences of optimizing against a proxy.

That said, the ceiling is not uniform. Execution-verified systems (AlphaEvolve, FunSearch) keep $\rho$ near 1, making the drift rate negligible and the practical lifetime very long. Tool-augmented systems may partially escape the complexity bounds that constrain pure self-modification, by offloading capability gains onto external tools rather than internal parameter changes. And recent work (AIDE², 2026) shows that some self-verification architectures exhibit partial resistance to Goodhart drift in practice. These are real escape routes, but they apply to structured domains, not open-ended self-evolution.

The viable path is episodic self-improvement: run the loop, monitor the drift between verifier and reality, stop and recalibrate before collapse, restart from the new baseline. Each episode captures real gains. You consume the submartingale condition by optimizing and replenish it by recalibrating. The prescription isn't to stop self-improving. It's to measure the gap, and stop trusting the loop before the verifier stops deserving trust.