The checks weaken

A check shows green and means less than it says.

Each failure below is a row of the grid, with what holds it at each rung.

  • Throughout In place across the stack.
  • In places In some projects, or for part of the failure. The entry says what is left out.
  • Out of reach This rung cannot hold the failure.
  • Wanted More is intended here and not built.

Green but wrong

Code passes its gates and is still incorrect, or a test passes whether or not the code is right.

Person: Review after landing

Coverage
In places
Does
The maintainer reviews commits after they land on the main branch.
Leaves out
One person reads what many sessions push, and nothing yet lists the commits pushed across the repos in one place.
Fails
When a defect sits on the main branch until that review reaches it.
Wanted
A view of the commits pushed across every repo, so the review has one place to happen.
Evidence

Session: Writer and reviewer loop

Coverage
In places
Does
In long-running work, one writer changes the code and a read-only reviewer reports findings, and they alternate until a round finds nothing worth fixing.
Leaves out
The loop is for long-running work and not for every change, and its reviewer is another agent.
Fails
When the reviewer's own severity grades are accepted, and a loop closes with wrong answers filed as minor.
Evidence

Tool: Oracles, mutants and control cases

Coverage
In places
Does
Where a reference exists, a harness checks output against it. The install script of fuz and the fuz_schema library also have their code broken on purpose to show that a test catches it, and the divergence detector of tsv needs a negative case for each pattern, to prove it can report a failure.
Leaves out
Most code has no reference to compare against, and only those two have a mutation run over their tests.
Fails
When both sides agree on the same bug, when the comparison never looks at the property that is wrong, or when a harness has no control case and reports success for reasons unrelated to the code.
Cost
A twin roughly doubles what is written by hand.
Wanted
Routine mutation and branch-coverage runs for tsv.
Evidence
  • The fuz installer's mutation gate, which breaks one behavior at a time and checks that a test catches it
  • fuz_schema's mutants, where a survivor needs a written reason
  • The tsv divergence detector's overmatching audit, which requires a negative case for each pattern
Listed

A check is silenced instead of satisfied

A failing fixture, lint or case is made to pass by editing the expectation, suppressing the lint or skipping the case.

Session: The expected diff is read as code

Coverage
In places
Does
After an update task regenerates the expected files of a fixture test, the session reads their diff as it would read code, since the diff is the assertion.
Leaves out
Only a fixture test has such a diff to read.
Evidence
  • The testing reference on fixture tests, where the diff of the expected files is reviewed as code

Written: Fix the code, not the expectation

Coverage
In places
Does
The written rules say an expected file is generated and never edited by hand, a Rust lint is suppressed only with a stated reason, and a difference between twins is recorded and never skipped by hand. tsv adds that a correct fixture is never changed to make a test pass.
Leaves out
No rule covers a suppression in TypeScript or a skipped test.
Fails
When the expectation really is wrong, and telling that apart from a convenient edit is left to the session.
Evidence
  • The fuz-stack skill's rules by topic, where expected files are generated and never hand-edited
  • tsv's golden rules for fixtures
  • The Rust lint rules, on suppressing a lint with a reason

Tool: Suppressions that expire

Coverage
In places
Does
A Rust lint suppressed with an expectation warns once the lint stops firing, and the gate treats the warning as an error. A recorded difference between twins fails once the difference is gone. The pinned counts of tsv fail when a count moves, and its fixtures are compared against a live, pinned Prettier, so a fixture edited by hand fails.
Leaves out
Where no outside reference exists, an expected file regenerated from wrong output passes, and so does a suppression with a plausible reason. The record of twin differences is checked only when the cross-backend suites run, which for the Rust side is on demand.
Fails
When the reason on a suppression is accepted without being checked against the code.
Evidence
Listed

A gate stops gating

A check skips, caps its run or falls back, and still shows green.

Written: Read the exit code

Coverage
Throughout
Does
The written rules make the gate's exit code the pass condition, never its filtered output.
Fails
When a gate's output is piped through a filter that hides its exit code, and the commit follows anyway.
Evidence
  • The fuz-stack skill on the gate, which says to read the gate's exit code
  • The Rust gate, whose pass condition is the exit code

Tool: Suites that fail on a skip

Coverage
In places
Does
The suites of zap fail when a program a test needs is missing, unless an explicit allowance names it. fuz fails when a database suite is left off its CI list, pglet's CI fails when a gate reports nothing, and the pinned counts of tsv fail when a count moves.
Leaves out
Elsewhere a skip may only warn, as when the install suite of fuz finds a program it needs missing.
Fails
When a skip stays green with a warning that nobody reads.
Evidence
  • tsv's pinned gate counts, which fail when a count moves
  • pglet's CI workflow, which refuses to be green because a prerequisite was missing
  • The Rust testing rules, on suites that never pass by skipping and on the test that holds a CI list to the tree
  • fuz's install suite, which skips when a program is absent
Listed

Shape: One gate definition

Coverage
In places
Does
The CI of tsv and of the repos with no Rust runs the check command a session runs, so the two lists cannot drift there.
Leaves out
Most other Rust workspaces list their steps again in CI, and nothing structural stops a commit whose local gate was red.
Evidence
  • tsv's CI workflow, which runs the same check command as a session does
  • mdz's CI workflow, one of the repos with no Rust

A slow check gets skipped

A check is slow enough that sessions stop running it.

Written: The full gate before landing

Coverage
Throughout
Does
The written rules name the full gate as what runs before work counts as done, and warn that the faster checks skip test targets.
Fails
When a fast targeted check stands in for the full gate before landing.
Evidence
  • The fuz-stack skill on the gate
  • The Rust gate, on what a quick check leaves out

Tool: CI reruns the gate

Coverage
In places
Does
CI runs the gate on a push to the main branch, whatever the session ran.
Leaves out
A few repos have no CI, or one that runs less than the local gate; the list under the gates table names them.
Fails
When the answer arrives after the commit has landed.
Evidence
Listed

Shape: Fast tools

Coverage
In places
Does
The formatter and parser of tsv, and pglet as a test database, are built to answer quickly, and pglet states a startup target.
Leaves out
Rust compile loops stay slow, and the heaviest gates, such as the corpus and conformance runs of tsv, run before a release and not on each push.
Evidence