No One Wins Forever

The standard story about artificial general intelligence ends the same way: a system gets smart enough, and then it wins for good. It out-predicts and out-coordinates everyone else and settles into a stable global equilibrium with itself on top.

The story rests on an assumption that more intelligence converts directly into more prediction and more control, without limit. This article is about why that assumption breaks against three structural features of the real world, and where it survives.

My claim is more limited than “takeover is impossible”: permanent, fine-grained, centralized dominance is information-theoretically expensive and dynamically fragile. Where dominance reduces to controlling a small, slow, low-entropy summary of the world, it can be cheap and durable. Where it does not, no amount of compute makes it stable. Which regime a given world falls into is an empirical question, and most of the work is in answering it.

Limit 1: chaos caps the prediction horizon

Start with prediction, because every control story is built on it. Many real systems — weather, ecosystems, markets, geopolitics — are nonlinear and chaotic: nearby trajectories diverge exponentially. If two world-states differ initially by $\delta_0$, that gap grows roughly as

$\lVert \delta x_t \rVert \approx \delta_0\, e^{\lambda t},$

where $\lambda > 0$ is the largest Lyapunov exponent. Solve for the time at which the error reaches some actionable tolerance $\Delta$ and you get a predictability horizon:

$T_p \approx \frac{1}{\lambda}\ln\!\frac{\Delta}{\delta_0}.$

⁺

The logarithm does the damage: a billion-fold improvement in measurement precision buys only about $20/\lambda$ extra units of foresight.

The dependence on initial error is logarithmic, which is what makes this binding. To double your forecast horizon you must square your precision. An agent with a thousand times better sensors and models than you gets a small additive constant of extra foresight, not a thousand times more. And $T_p$ stays finite for any nonzero $\delta_0$, which physical measurement guarantees. The limit lives in the dynamics; the agent’s quality only sets $\delta_0$.

Limit 2: control has a minimum information rate

Prediction limits don’t forbid control. You can steer a system you can’t forecast, using feedback. But chaos imposes a cost on feedback too, and this is where the dominance argument is decided.

A chaotic system produces information. As trajectories diverge, states that were indistinguishable become distinguishable, and a controller has to keep resolving those new distinctions to hold the system in place. The rate of this production is the Kolmogorov–Sinai entropy. For well-behaved (SRB) measures, Pesin’s formula makes it exact:

$h_{\mathrm{KS}} = \int \sum_{\lambda_i(x) > 0} \lambda_i(x)\, d\mu(x).$

⁺

A caveat the original draft of this got wrong: Pesin’s equality needs an absolutely continuous (SRB) measure. In general you only have Ruelle’s inequality $h_{\mathrm{KS}} \le \sum \lambda_i^+$. For the control argument that’s enough — you only need $h > 0$.

Control theory then sets a floor. To stabilize an unstable or chaotic system through a finite-rate feedback channel, the channel’s information rate $R$ must exceed the system’s entropy production. For linear systems this is the data-rate theorem; the nonlinear generalization uses topological feedback entropy $h_{\mathrm{TFE}}$:

$R > h_{\mathrm{TFE}}.$

Below that rate stabilization is impossible, and the floor is set by the dynamics rather than by the controller’s design. A system with positive entropy demands a strictly positive feedback rate, sustained indefinitely. Interrupt it with a sensing gap, a delay, or a severed link, and exponential divergence resumes within a few divergence times.

Local, temporary control still works: the Ott–Grebogi–Yorke scheme stabilizes unstable periodic orbits inside a chaotic attractor with tiny nudges. But it’s fragile, needs constant state monitoring, and doesn’t scale to high-dimensional, nonstationary worlds. Centralization is the worst case for it, because one controller has to out-rate the entropy production of the entire system it wants to dominate.

Measuring the floor

To make the claim concrete I took a Lorenz-96 system, the standard tunable-dimension chaos benchmark, and tried to hold it near an unstable target using feedback limited to $R$ bits per control interval. The threshold $R^\star$ is computed independently from the system’s unstable modes. The measured quantity is $T_{\mathrm{dom}}$: how long control lasts before the state escapes.

The transition is smooth rather than knife-edge, because sub-bit-per-mode quantization smears it, but the shape matches the theory. There is a rate below which control cannot be maintained, and that rate is a property of the world, not of the agent.

Limit 3: evolution has no terminal state

Suppose an agent affords the feedback rate and stabilizes its slice of the world. It still isn’t secure, because it isn’t alone. Real environments are competitive and adaptive, and the natural model is replicator dynamics: the frequency $x_i$ of strategy $i$ grows with its fitness relative to the population average,

$\dot{x}_i = x_i\big(f_i(\mathbf{x}) - \bar{f}(\mathbf{x})\big).$

Dominance is a monomorphic state, one strategy at frequency $\approx 1$. For it to last it must be evolutionarily stable: no rare mutant can invade. But fitness here is endogenous and frequency-dependent. The act of dominating reshapes the payoff landscape and opens niches for counter-strategies. Add mutation and innovation, a diffusion term that keeps reinjecting variety, and the dominant vertex stops being absorbing. Holding it requires active defense against a constant influx of novelty.

⁺

This panel uses a sigmoidal “hold probability” distilled from Figure 1 rather than re-simulating the controller inside the evolutionary loop. It’s illustrative of the mechanism, not a closed derivation.

Dominance, then, is a spell rather than a fixed point. An agent can stay on top for a while, sometimes a long while, but a shifting environment and open-ended innovation keep displacing incumbents.

Reducible vs. irreducible dominance

This is where the impossibility claim has to give. Everything above assumes the agent wants to control the full, fine-grained state, and usually it doesn’t.

Dominance only requires controlling whatever observable defines dominance: resource share, institutional control, compute. The relevant quantity is therefore not the entropy of the whole world but the entropy of the cheapest sufficient projection:

$h_c^\star = \min_{\Pi\ \text{sufficient for dominance}} h_c(\Pi).$

A capable agent picks the lowest-entropy summary that still counts as winning. That choice splits the world into two regimes:

• Reducible dominance: a low-entropy sufficient statistic exists, such as slow modes or aggregates. Control is cheap and can be durable.
• Irreducible dominance: every dominance-relevant observable is itself chaotic. Then $h_c^\star$ exceeds any achievable rate, and dominance has a finite half-life regardless of the agent’s capability.

The same Lorenz-96 setup shows the gap directly. Controlling a coarse aggregate (the mean) holds for the full horizon at about 1 bit per interval, while controlling the full state needs roughly 5–6 bits.

The distinction cuts against both extremes. It explains why “it can never dominate” is wrong, since plenty of dominance targets are reducible. It also explains why “it will dominate permanently and completely” is wrong, since fine-grained control of a chaotic world is the irreducible case and is unaffordable.

Implications

Reframing the limits doesn’t make powerful agents safe. It changes the shape of the risk.

• The threat is transient. A capable agent can do severe, irreversible damage inside a single dominance spell. That no one wins forever is little comfort to whatever gets broken before the spell ends.
• Centralization is the fragility. A single controller trying to out-rate the entropy of everything is a single point of failure with a continuous and unbounded information requirement. Distributed control is cheaper and more robust, at the cost of the unified objective the takeover story assumes.
• Resilience beats prevention. If terminal lock-in isn’t the generic outcome, governance should aim at limiting the damage of any single failure, preserving diversity and redundancy, and keeping the system adaptive, rather than at averting one final move.

In a world governed by chaos and evolution, no agent wins forever. But many can do lasting damage before they lose, and that is the problem worth working on.

---

⁺

The figures come from a small Lorenz-96 + quantized-feedback + replicator simulation. Caveats stated inline: the entropy threshold is exact for the unstable fixed point but the measured transition is smooth, and Figure 2 uses a distilled hold-probability rather than control-in-the-loop.

Sources

Chaos and predictability

• Lorenz (1963), Deterministic Nonperiodic Flow. J. Atmos. Sci. 20, 130–141.
• Eckmann & Ruelle (1985), Ergodic Theory of Chaos and Strange Attractors. Rev. Mod. Phys. 57, 617–656.
• Pesin (1977), Characteristic Lyapunov Exponents and Smooth Ergodic Theory. Russ. Math. Surv. 32, 55–114. (Ruelle’s inequality covers the general, non-conservative case.)

Information limits of control

• Nair, Evans, Mareels & Moran (2004), Topological Feedback Entropy and Nonlinear Stabilization. IEEE Trans. Autom. Control 49, 1585–1597.
• Nair & Evans (2004), Stabilizability of Stochastic Linear Systems with Finite Feedback Data Rates. SIAM J. Control Optim. 43, 413–436.
• Tatikonda & Mitter (2004), Control Under Communication Constraints. IEEE Trans. Autom. Control 49, 1056–1068.
• Ott, Grebogi & Yorke (1990), Controlling Chaos. Phys. Rev. Lett. 64, 1196–1199.

Evolutionary and ecological dynamics

• Taylor & Jonker (1978), Evolutionarily Stable Strategies and Game Dynamics. Math. Biosci. 40, 145–156.
• Hofbauer & Sigmund (1998), Evolutionary Games and Population Dynamics. Cambridge Univ. Press.
• Maynard Smith (1982), Evolution and the Theory of Games. Cambridge Univ. Press.
• May (1974), Biological Populations with Nonoverlapping Generations: Stable Points, Stable Cycles, and Chaos. Science 186, 645–647.
• Hastings, Hom, Ellner, Turchin & Godfray (1993), Chaos in Ecology: Is Mother Nature a Strange Attractor?. Annu. Rev. Ecol. Syst. 24, 1–33.