The Test That Graded Itself

An engineering update arrived with a reassuring conclusion: all gates passed.

One detail changed what that sentence meant. The working session reported that, after modifying the verifier, it had regenerated the assessment files used as its expected answers. The new program agreed with outputs produced by the new program.

That can be a useful consistency check. It cannot establish that an extraction preserved the behavior of its predecessor.

We were building a portable verifier for a Work Record: a declared task, its expected branches, the observations collected for them, and the resulting assessment. The aim was to let someone outside the original application re-derive a bounded result. Along the way, our own review process repeatedly found that a green status described less than the accompanying summary claimed.

This is a field note about how we repaired that gap—and what the final evidence still does not establish.

The answer key had to stay outside the repair

The verifier began as an extraction of an existing assessor. The original capture record described five fixture assessments taken before extraction as expected outputs. Those original bytes were the reference for detecting changes in behavior.

A later change tried to improve the trace graph by adding two fields to the assessment. The reported repair then regenerated the assessment expectations to include those fields. A passing comparison could now tell us that the new shape was internally consistent. It could no longer answer the original question: does this verifier return the same assessment as the incumbent?

The distinction mattered even after the extra fields were removed. Regenerating the expectations again, this time with the old shape and matching JSON formatting, still left the new program producing its own answer key.

We recovered the original assessment files from version control and checked their hashes. Then we ran the incumbent assessor from a clean checkout of its pinned revision and compared its output with those preserved files. All five matched. The portable assessor also matched all five.

We kept the original provenance record and wrote a separate record for this later reproduction. That establishes what the pinned incumbent produces now; it does not retroactively prove when an earlier capture happened.

The repair was to restore the independent reference, not to make the test agree.

A diagram almost became a second verifier

The trace graph exposed another problem. It needed to distinguish branches supported only by producer status from branches where producer status had been ignored alongside admitted evidence.

An early approach inferred those distinctions from diagnostic text. Another inspected the raw observations again. The first was ambiguous; the second duplicated admission logic in the display layer.

Consider a branch named a,b. A diagnostic containing a,b could describe that single branch or two branches named a and b. Splitting the text at commas cannot recover a distinction the format never preserved. Passing fixtures with single-character identifiers did not remove the ambiguity.

The final implementation exposes separate trace metadata from the same reduction that produces the assessment. The original assessment object stays unchanged. The graph consumes the assessment, the portable result and that metadata. It neither interprets diagnostic prose nor decides for itself which observations count.

We tested the comma-containing identifier alongside both of its misleading neighbors, with producer-only and mixed-source observations. We also tested free text that looked like an internal diagnostic. Those examples matter because the ordinary five fixtures did not exercise the failure.

Rendering needed its own check. Replacing quotation marks with similar-looking characters would make a label easier to display while changing the identifier a reader saw. We used character entities and checked the actual rendered text in a browser, including ordinary and fullwidth quotes, markup-like text, literal entities and supplementary Unicode. Encoding tests and a successful browser render answered different questions; we retained both.

"Passed" needed an object and a scope

The gate list was another source of drift. At one point, package-format checks were presented as portability evidence. A checklist about source files was presented as the engineering gate. Neither was the test the written requirements had asked for.

We separated the results:

• Assessment parity: five preserved fixture assessments matched the portable verifier, and a later clean-checkout run of the incumbent reproduced them.
• Portable execution: five fixtures and six mutation bundles ran twice on each of two operating systems. The delivered reports agreed on result, output and graph hashes for all 11 bundles. The second-platform execution was reported by a separate working session; we recomputed the comparison between the reports.
• Mutation behavior: four mutation cases were testable under this profile and passed their assertions. Two were explicitly not testable. Their successful execution for portability did not turn them into passed mutation tests.
• Engineering checks: a secrets scan, JavaScript type checking, lint, an actual package build and vocabulary inspection ran for the portable package. That says nothing about an unrelated parent application build.
• Specification review: a separate review gave an explicit sufficiency verdict on the published profile. That was a review of whether the specification described the decisions; it was not an independently written replacement verifier.

The network checks also have a boundary: they used instrumented runtime APIs and denial probes. We do not describe them as operating-system firewall isolation.

Writing those distinctions down made the handoff more useful. A reviewer could see which work had been executed, which result had been compared, which conclusion came from review, and which claim remained outside the exercise.

Two dimensions, deliberately

The portable profile derives integrity and outcome. It does not derive authority, identity lineage, control adherence or coverage. Those dimensions remain explicitly marked as not derived.

This is not a score for the platform. It is a statement about one profile, one fixture class and fixture-origin data: 2 of 6, for profile pk-wrv-1 v1.0.0, map v1.1.0, WR_FIXTURES, FIXTURE.

Integrity means the recomputed commitment matches the recorded commitment. It does not establish complete capture. Outcome is evaluated against the declared observations and rules. It does not independently establish real-world resolution. A changed authority field can fail integrity without this profile reasoning about whether the underlying action was authorized.

The exercise also left three contract questions outside this implementation's disposition: a canonicalization identifier used for different algorithms, the scope of the reason-code selection law, and gaps in named-error behavior. Completing the evidence handoff did not silently resolve them.

A useful question for your next green report

Ask what supplied the expected answer, which artifact was tested, and what the result permits someone else to claim.

If the expected answer changed during the repair, recover the original reference before claiming parity. If a visualization makes its own decisions, bring it back to the verifier's output. If a count mixes passed checks with untestable cases, separate them. If a handoff says "complete," make the evidence and its limits travel with that word.

The useful outcome here was a bounded, independently checkable result and a record of how we arrived at it. The corrections were part of that record.

---

Evidence scope: This article reports an internal, fixture-only engineering exercise. Its evidence package is held internally; this page is not a public verifier or a production-readiness claim. Cross-platform agreement does not establish complete capture, decision-time authority or real-world outcome.

Apply the pattern: Bring one consequential workflow and identify its declared claims, preserved inputs, expected outputs and unproved dimensions. Request an evidence review or see the architecture.