Holdouts Should Stay Out of the Workshop

Scope: Engineering or methodology note based on a September 17, 2026 snapshot. Source-level behavior is not a public availability or universal compatibility claim. Any scenarios are illustrative, not customer outcomes.

An evaluation becomes less informative when the implementation has effectively memorized the questions. This can happen without anyone explicitly cheating. Engineers inspect failures, tune rules, add exceptions, and gradually turn a once-independent test set into a development set.

For a recovery system, that risk is especially important. A library recognizer can appear accurate because it has seen the exact bundle shape before. A semantic recovery step can look convincing because the fixture quietly exposes the intended answer.

Development examples have a legitimate job

We need examples that help us understand failures. Those examples belong in the workshop: small enough to inspect, reproducible enough to debug, and explicit enough to expose a broken assumption. They are valuable even after they are no longer independent evidence.

The mistake is continuing to describe performance on those examples as though they were unseen. A regression suite proves that known boundaries remain protected. It does not, by itself, establish how well the system generalizes.

Separate the reference from the answer key

KodeBack’s dependency work may legitimately consume authenticated reference-package bytes. That is part of the declared method. It should not gain access to an application’s hidden package manifest or unexposed source map merely because the evaluator has those files.

The evaluation should make that boundary explicit. Which inputs would an ordinary operator possess? Which are reserved for scoring? Which transformations were chosen before the holdout was opened? These questions matter more than a dramatic single-number result.

Preserve difficult outcomes

The Ushman research separates adaptive examples from held-out operational samples. Kalu’s architecture also calls for independent corpus pressure and holdouts. Those are evaluation requirements, not a claim that every planned benchmark has already been completed.

A useful result can include false acceptance, false refusal, unresolved cases, and measurement failures. Combining them into one recovery percentage can hide the exact tradeoff an engineer needs to judge. We prefer to keep the categories visible and explain the decision being measured.

The standard is straightforward: use development examples to make the system better, and use genuinely independent examples to learn how much better it became. The two jobs support each other only when their boundaries remain intact.