How We Evaluate a Product That Is Supposed to Verify AI

A product that claims to control AI code changes has a credibility problem.

It cannot simply say:

KodeProof passed the candidate, therefore the candidate was good.

That would make the product both the participant and the judge.

The evaluation has to be capable of finding that KodeProof was wrong.

That includes two uncomfortable outcomes:

• KodeProof allows a bad candidate through.
• KodeProof blocks a valid candidate.

Both belong in the product report.

The unit is not “the model”

Coding performance depends on a treatment combination:

• exact model;
• model snapshot where exposed;
• reasoning setting;
• agent harness;
• system and repository instructions;
• tools and permissions;
• time and token budgets;
• KodeProof version and configuration;
• task family and fixture;
• environment.

A leaderboard that says “Model X scored 82” can hide the variables that produced the result.

Our evaluation unit is closer to:

model + harness + reasoning mode + condition + frozen task + budget

That is less convenient and more reproducible.

Separate judgment quality from agent completion

KodeProof contains bounded judgment surfaces, and it also operates inside larger agent workflows.

Those require different studies.

Direct judgment study

The model receives only the decision packet.

This measures:

• option selection;
• refusal quality;
• confidence calibration;
• authorization discipline;
• latency and cost.

It can be compared with simple baselines such as random choice, first option, or a deterministic heuristic.

Ecological agent trial

The agent uses the actual product and repository.

This measures:

• discoverability;
• command following;
• state-machine completion;
• tool compatibility;
• source mutation discipline;
• final transfer;
• total workflow cost.

A model can make excellent direct judgments and still fail to operate the harness. An agent can complete the workflow while making mediocre underlying decisions.

Those claims must remain separate.

The statistical unit

A long refactor may contain many iterations.

Those iterations are not independent experiments.

We use three levels:

1. Program or trial — one user objective and the main statistical unit.
2. Transaction — one immutable before-state and trust/transfer boundary.
3. Round — one analyze, decide, apply, verify, and re-observe cycle.

Counting 20 rounds as 20 successful trials would exaggerate the evidence.

Freeze the campaign

A result is only comparable when the treatment identity is stable.

A campaign should bind:

• task and fixture snapshot;
• prompt;
• hidden grader;
• KodeProof build and schemas;
• upstream engine builds;
• model identity where exposed;
• harness version and settings;
• budgets;
• repository before-state;
• condition;
• scorer and rubric version.

A material change creates a new campaign identity.

Historical raw evidence remains immutable.

A new rubric may rescore the old evidence, but the original score must remain available.

Use an independent grader

KodeProof’s own report is one evidence source.

The independent grader determines whether the task was actually satisfied.

For a refactor fixture, it may check:

• public behavior;
• hidden edge cases;
• errors and serialization;
• initialization order;
• module-load behavior;
• public exports;
• cycles;
• direct consumer imports;
• forbidden compatibility layers;
• architecture objective;
• unrelated changes.

The grader runs after the agent can no longer edit the candidate.

It should not be accessible from the trial workspace.

Calibrate the grader with known-bad candidates

A hidden test suite can look impressive and still miss the failure classes that matter.

Before using a fixture, create required mutants such as:

• drop a public export;
• leave one consumer on an old import;
• introduce a cycle;
• move initialization after first use;
• remove a top-level side effect;
• change an error type;
• reorder asynchronous work;
• merge two subtly different helpers;
• alter a serialized payload;
• add a compatibility barrel;
• keep an obsolete implementation;
• break a dynamic import.

The grader must detect every required mutant before the fixture is trusted for that claim.

When a mutant escapes, the right response is not to hide the failed campaign.

Preserve it, fix the grader or fixture, and create a new campaign identity.

Calibrate KodeProof separately

A strong grader does not establish that KodeProof catches the same defects.

Run:

• the valid reference solution through KodeProof;
• each known-bad mutant through KodeProof;
• several valid alternative solutions;
• environment and tooling failure cases.

This produces a detection matrix.

For each category, measure:

• true catches;
• escaped regressions;
• valid candidates blocked;
• unresolved disagreements;
• explanation quality;
• overhead.

The goal is not the largest number of blockers.

It is useful precision and recall for the declared contracts.

Separate safety claims

“Safer” is too broad.

KodeProof needs independent claims for:

Original integrity

Did the original repository remain unchanged until an authorized transfer?

Transfer-policy integrity

Did any unauthorized or policy-invalid candidate reach the original?

Task and behavior integrity

Did hidden task regressions escape?

False confidence

Did the agent or product declare success while the independent grader found a material failure?

A product can perform well on one and poorly on another.

For example, candidate isolation may be perfect while the behavioral oracle is weak.

Separate candidate success from delivered success

A treatment trial can produce a valid candidate that never transfers because KodeProof blocks it.

That is not the same as a bad candidate.

We track:

• agent completed;
• candidate objective complete;
• independent candidate valid;
• KodeProof trusted;
• transferred;
• delivered success.

This allows a false block to be measured as a product failure without rewriting the candidate as invalid.

Cost belongs in the result

A guard that catches regressions but doubles time and cost may still be worthwhile for high-risk work.

The tradeoff has to be visible.

Capture:

• wall time by phase;
• model-active time;
• command time;
• input, output, reasoning, and cached tokens where reported;
• monetary cost;
• tool and command counts;
• retries and restarts;
• human interventions;
• KodeProof-specific overhead.

Do not estimate unavailable provider metrics without labeling the estimate.

Report categories before a composite

A safe but incomplete attempt should not look like a 90/100 success.

Lead with categories:

• successful and delivered;
• valid but product-blocked;
• safe but incomplete;
• unsafe;
• harness failure;
• evaluation invalid;
• unsupported.

Then show the raw metrics.

A material hidden regression remains unsafe regardless of architecture points.

Publish the uncomfortable evidence

KodeProof’s Evidence Lab should publish, where the artifacts can be safely disclosed:

• exact treatment identity;
• task and conditions;
• catches;
• escaped failures;
• false blocks;
• invalid trials;
• cost;
• missing telemetry;
• coverage gaps;
• product version;
• what changed in the next campaign.

Selective success stories would undermine the category.

A product built to challenge AI confidence needs to tolerate evidence that challenges its own.

What a legitimate early claim looks like

A weak claim:

KodeProof makes AI agents safer.

A stronger, campaign-bound claim:

In campaign KP-RF-003, under the frozen TypeScript decomposition fixture and model configuration, KodeProof prevented two independently confirmed invalid transfers, falsely blocked one valid candidate, and added a median 18% wall-time overhead. The sample is descriptive and does not establish general product lift.

That sentence is less marketable in the conventional sense.

It is more credible to the buyers who will eventually rely on the product.

The standard

KodeProof should earn stronger language one frozen campaign at a time.

Until then, the methodology is part of the product:

• the product does not grade itself;
• raw evidence is retained;
• historical results are not overwritten;
• false blocks are first-class;
• invalid experiments remain visible;
• model aliases are not treated as stable identities;
• cost and coverage are reported with success.

Verification is not a badge.

It is an evidence process that must remain open to being proven wrong.