The evaluation problem: how to avoid knowledge debt

For fifty years, the way we derived running systems from higher-level descriptions was deterministic. A compiler turns source into machine code; a model-driven transformation turns a model into an implementation. If the transformation is correct, the output is correct – by construction. You don’t have to test every compiled program to believe the compiler preserved its meaning. Correctness flows from the derivation itself.

Generative regeneration throws that away. When an AI realizes a contract, there’s no meaning-preserving transformation underneath. The output is trustworthy only insofar as you can check it. We’ve traded conformance by construction for conformance by evaluation. A proof for a measurement. And measurements are weaker than proofs. That’s not a detail; it’s the central cost of the approach I proposed, and I glossed over it.

It gets sharper. A test result isn’t a free-floating fact about a system; it’s indexed to the harness that produced it and to the model that generated the code. Change the generator and you change the distribution of mistakes it makes. The evaluation you designed was implicitly ruling out the failure modes of a particular model at a particular time. Swap in a new model, which is a routine event given how fast models evolve, and the error distribution shifts underneath you. The old evidence was answering a question about a system that no longer exists.

So, a green evaluation suite isn’t a durable guarantee; it’s a statement of the form: This contract held, under this harness, against this model, at this moment. Push the horizon out even a few months and the harness itself stops being infrastructure you can trust blindly and becomes an object you have to investigate. Evidence, it turns out, isn’t a stock you accumulate and bank; it’s perishable.

I’ve seen the resulting condition being dubbed “knowledge debt,” a name I want to borrow, because it’s exactly right. Technical debt is code you understand, but that’s messy. You know why it works and you know it needs cleaning up. Knowledge debt is the inverse. The code is clean, generated and passing its checks, but nobody knows why it works or under which validation regime it was ever shown to work. The system runs; the understanding is gone.

It’s the characteristic debt of the regeneration era, and the naive version of my own thesis maximizes it. “Don’t worry about the code, just regenerate it when you need it” sounds liberating. Practiced without discipline, it means continuously producing artifacts that no human comprehends, validated by evidence whose expiry date nobody tracked. You don’t eliminate the maintenance burden; you move it from the code, where at least it was visible, into the assurance, where it’s invisible until it fails.

If contract-and-evaluation is going to hold up, it has to answer this. The mistake hidden in my original framing is treating evaluation as a gate: a one-time acceptance test the regenerated system passes on its way into production. If evidence is perishable, a gate is precisely the wrong shape. A gate issues a certificate, and the certificate silently expires.

The reframe is the core of the work I’ve been doing on continuous, trustworthy autonomy: Software engineering is migrating its center of gravity from design time to runtime. A quality attribute stops being a fixed guarantee frozen into structure at design time and becomes a managed variable held by a closed loop: Specify what “good” means, observe the current level, detect the gap and act to close it. Design time produces artifacts; runtime produces controllers.

Apply that to evaluation and the perishability problem changes character. Evaluation isn’t something you do once to admit a system; it’s a standing, runtime mechanism that continuously re-checks the contract as models, inputs and the world drift. A design-time acceptance test hands you a guarantee that decays the moment anything changes. A runtime evaluation loop keeps the contract honored as things change and tells you the moment it can no longer do so. Once evaluation is a control loop, “the evidence expired” stops being a silent catastrophe and becomes a signal the loop is built to catch.

Four properties separate a real contract-and-evaluation harness from the naive version that manufactures knowledge debt: Evidence is versioned (stamped with the model, harness and regime that produced it), contract and evaluation and generator co-evolve (a model change forces reevaluation, never silent trust), the harness is regime-aware (it knows the bounds where its evidence holds and detects departures from them) and value judgments route to humans rather than being silently mechanized (some conflicts resolve by mechanism, some are value judgments the control loop mustn’t make, and knowing which is which, and routing accordingly, is the engineering job – it’s the structural reason humans remain non-optional).

This connects directly to the question I left open last time: How large, how critical and how complex must a system be before an AI can’t reliably regenerate it from scratch? Knowledge debt reframes this question with unexpected precision. For a small, low-stakes, easily-checked component, perishable evidence barely matters. Regenerate freely and the cost of a miss is trivial. For a large, critical, complex system, the same casual regeneration is how you end up operating something nobody understands, whose assurance quietly expired. The frontier of what’s reliably regenerable turns out to coincide with the frontier of what’s affordably evaluable. The threshold isn’t really about whether the AI can write the code; it’s about whether you can build an evaluation loop good enough to keep trusting the code it writes.

Eliminating code was never the hard part; models can already do more of that than most organizations have admitted to themselves. Making evidence durable, ie versioned, regime-bounded and co-evolving with the generators that keep changing underneath it, is the hard part, and it’s largely unsolved. That’s where I think the interesting work of the next few years actually sits, and it’s a large part of what my own research and the work in Software Center is turning toward. For example, we’re kicking off a large project on continuous compliance.

To end with a fellow Dutchman, Edsger Dijkstra warned us in 1970: “Program testing can be used to show the presence of bugs, but never to show their absence.” He was right, and regeneration doesn’t repeal it; it makes it the central problem. If evaluation is all we have left, then making that evaluation trustworthy, durable and honest about its own limits is the discipline the next decade has to build.