The test that cannot fail
Published
Green is only an observation
A green test suite tells us that the code produced the expected result for the cases the suite exercised. That is useful. It is not the same as proving that a safeguard can catch the failure it was designed to catch.
The difference is easy to miss because a good test and a decorative test look identical when they pass. They have careful names, arrange realistic data, and end in a reassuring assertion. The distinction appears only when we challenge the test itself: change the rule, cross the boundary, remove a term, or omit a mapping. If the result stays green, the test was observing an example rather than defending a property.
We encountered that distinction while working through internal evaluations at Kaizen AI Systems. Nothing defective landed. The useful part of these episodes was not the feature involved; it was the reasoning mistake that let a test look stronger than it was.
A boundary the test never touched
A case involved a retention rule. The test arranged an obviously old item and a fresh item. It then checked that the old item was affected and the fresh item was retained. That sounds sensible because the outcomes point in opposite directions.
During a mutation check, we replaced the governing boundary with one day. The test still passed. The old item remained old enough to be affected, and the fresh item remained fresh enough to survive. The implementation had changed dramatically, but the test observed the same pair of outcomes.
The problem was not a missing assertion after the operation. It was the input geometry before the operation. Those fixtures sat far from the boundary, so many incorrect boundaries partitioned them in exactly the same way.
The repair was to derive the cases from the rule:
- an item just before the boundary,
- an item exactly on the boundary,
- an item just after the boundary,
- and assertions for the intended change and the state that must remain untouched.
Now the test distinguishes the comparator, the boundary, and the preservation rule. Moving the boundary makes a witness cross sides. Flipping an inclusive comparison to an exclusive one changes the exact-boundary result. Applying the operation too broadly changes state the test explicitly preserves.
The lesson is broader than retention. Whenever behavior depends on a threshold, representative examples are not enough. The test data should be derived from the boundary, not merely compatible with it.
Enumeration looked like coverage
Another case appeared in a source-to-destination mapping. A test populated fields with distinctive values and compared the result. It caught wrong values, which made it feel comprehensive.
But a newly added source field could still be omitted from the mapping. In Go, a keyed struct literal does not fail to compile when the struct gains a field. If the fixture and the mapped result each leave that field at its zero value, they compare equal. The test passes precisely because the same omission exists on each side.
Review kept finding narrower versions of the same problem. Each response correctly handled the named example, yet left the class open. The history was three enumerative rounds, then one structural fix.
The structural check asks a different question. Instead of listing fields that should be mapped, it perturbs each non-exempt source field and requires the destination to change. If a field has no influence on the mapped output, the check fails automatically. A future field is covered by the property without somebody remembering to extend another inventory.
This is the difference between enumeration and derivation. Enumeration says, “these are the cases we remembered.” Derivation says, “anything with this role must affect the result.” The latter survives growth because the test obtains its cases from the structure under test.
Compute the failure region first
Before writing fixtures for a threshold, write down the rule and solve it in both directions.
textpasses when measured >= floor
fails when measured < floorThen derive the nearest passing witness and the nearest failing witness. For discrete inputs, account for rounding explicitly. For a ratio, derive the required numerator from the actual denominator rather than trusting a worked example. For a time boundary, use values immediately on each side and at the boundary itself.
There is another step that matters just as much: intersect the failure region with the inputs the system can actually produce. Filters, caps, fixture composition, or preprocessing can make the mathematically failing region unreachable. A guard that no permitted input can trip is not a guard, even if its formula looks reasonable in isolation.
This gives a practical review sequence:
- derive what must pass,
- derive what must fail,
- prove each witness is reachable,
- and make the test observe the difference that matters.
Do the arithmetic before debating whether the sample data feels realistic. Realistic data can still sit entirely on the same side of a broken boundary.
Ask which mutants die
Mutation thinking is useful even without a mutation-testing framework. For every important term or branch, name a plausible wrong implementation and point to the assertion that would fail.
For example:
- replace the threshold with a much more permissive value,
- flip the comparison at the boundary,
- remove a term from a calculation,
- hard-code the result produced by the default configuration,
- or omit a field from a mapping.
If no existing assertion clearly dies, the suite has not pinned that behavior.
The wording matters: a mutant dying proves the guard fires; it does not prove the guard is sufficient. A mutation can test the same path the author already anticipated while leaving the surrounding class untouched. That is exactly how a sequence of locally correct fixes can create the appearance of convergence.
Good mutation questions therefore move outward. After the named mutant dies, try the neighboring boundary. Vary runtime inputs away from their defaults. Follow the error message or repair advice exactly and check whether that path is safe. Add a source field without updating the destination. Search for the counterexample that preserves the current assertion while violating the intended property.
Review the proof, not the shape
Tests often borrow the visual shape of a nearby correct test: similar fixtures, similar assertions, similar comments. That resemblance is comforting and weak evidence. Similar formulas can differ in whether an input is fixed at compile time or supplied at runtime. Similar mappings can compare equal while one silently drops a field.
A stronger review starts from the invariant:
- What property is this test claiming to protect?
- What mechanism enforces it?
- What is the smallest reachable counterexample?
- Which incorrect implementations does the suite reject?
- Which terms can vary at runtime, and have they been varied away from defaults?
If repeated fixes keep exposing the same defect class, pause. Another patch to the latest example is usually less valuable than rewriting the test around the invariant. Review rounds are not progress by themselves; the proof should become more structural as understanding improves.
The habit we kept
We now treat a green suite as the start of the question, not the end. For safeguards and boundaries, we compute the pass and fail regions, prove the failing witness is reachable, and place tests at the boundary. For mappings and inventories, we prefer checks derived from the structure over lists maintained by memory. For important claims, we ask which mutants die and which still walk through.
The goal is not to make every test adversarial. It is to recognize when a test is being used as evidence that a property holds. At that point, examples are supporting material. The property—and a reachable witness that violates it—is the proof.