Oracles in use
Each project below checks its output against an oracle, something to agree with. A row names what the project compares itself against, how, and when the comparison runs. The rows are sorted by how soon each comparison reports: one that runs on every change speaks before the change lands, and one that runs before a release speaks long after the change that broke it.
The holds whose instances this table lists: Drift checks and twin suites (Two copies drift apart); Oracles, mutants and control cases (Green but wrong); Suppressions that expire (A check is silenced instead of satisfied)
Where a project's docs say what keeps a comparison's result meaning what it says, the row lists it by discipline. A discipline a row does not list is not stated either way.
A row may carry probes: a string that a named file, check script or CI step of the repo must have, or must not have, for the row to still be true in the way a file can show. The tests of this site check the ones about a check script or a CI step against its snapshot of the repos, and the evidence task checks the ones about a file. A probe that passes says the file, script or step still has the text the row depends on. It does not say the check is sound. A probe reads only the text it names, so one that says a step is absent does not see the same thing reached through another script or switched on by an environment variable.
- Every change In the gate a session runs before landing, and again in CI.
- CI In CI on a push to the main branch or a pull request, and not in the local gate.
- Release Before a release, and not on each change.
- On demand When someone chooses to run it.
| Project | Compared against | Kind | How | Runs | Kept meaningful by | Source |
|---|---|---|---|---|---|---|
| tsv formatter | Prettier | Reference | Fixtures hold Prettier's output with each difference documented, the recordings are checked against a live, pinned Prettier, and a corpus diff fails on lost content or a new uncataloged difference | Every change: fixtures, against a live Prettier 4 probes
Release: corpus diff 6 probes
| Pinned: One audit holds every place that names Prettier, its Svelte plugin or the parsers to a single exact version, and the options passed to Prettier to a single set. pin agreement audit 1 probe
Run live: A fixture that claims Prettier never settles on an input, or rejects it, has the claim run against the live Prettier whenever the fixtures are validated. validation rules Sorted: A corpus file that differs is sorted by whether the catalog explains every part of the difference, some of it, or none. classification Cataloged: The cases where Prettier's output is unstable, does not parse again, throws or changes meaning are listed, each held by a fixture. Prettier bug index | CI workflow 1 probe
|
| tsv formatter | Itself | Law | Formatting twice gives the same result, on every fixture and across a real-code corpus | Every change: fixtures 2 probes
Release: corpus 3 probes
| corpus bundle | |
| tsv formatter | Its input, reparsed | Law | Formatted output parses back to the same document as its source | Every change: fixtures 1 probe
Release: corpus 2 probes
| round-trip audit | |
| tsv formatter | Its input, rendered | Law | A Svelte component renders the same before and after formatting | Every change: fixtures 3 probes
Release: corpus 2 probes
| render audit | |
| tsv parser | Svelte's parser, acorn-typescript and Svelte's CSS parser | Reference | Fixtures hold the canonical parser's syntax tree, the recordings are checked against the live parser, and a deep diff over real codebases classifies every difference against a catalog | Every change: fixtures, against the live parsers 2 probes
Release: corpus diff 2 probes
| fixture tests 1 probe
| |
| tsv type stripping | ts-blank-space, with Node's type stripping as a cross-check | Reference | Fixtures hold ts-blank-space's output, a corpus diff runs against it, and a Node cross-check of the fixtures runs by hand | Every change: fixtures 2 probes
Release: corpus diff 2 probes
On demand: Node cross-check 4 probes
| strip fixtures | |
| tsv import scanner | The canonical parsers' trees, and oxc-parser's module record | Reference | Fixtures hold records read off the canonical parsers' trees, the recordings are checked against those parsers by hand, and the records must agree with oxc-parser's over a corpus | Every change: fixtures 3 probes
Release: corpus 2 probes
On demand: recordings against the parsers 3 probes
| import differential | |
| fuz_forge server, TypeScript | Its Rust server | Twin | Both unit suites assert against shared case tables, and a cross-backend suite runs both servers | Every change: shared case tables 5 probes
On demand: cross-backend suite 2 probes
| parity fixtures 1 probe
| |
| pglet | Postgres | Reference | SQL fixtures run through both, compared in the binary format against recorded Postgres results and in the text format live | Every change 3 probes
| Cataloged: Behaviors that differ from Postgres are listed in a table, the deliberate tradeoffs and the open gaps alike. simplified behaviors | fixtures |
| fuz_schema, Rust side | The TypeScript library | Twin | The TypeScript side generates fixtures and the Rust tests assert both evaluators against them, with differences declared | Every change 3 probes
| Cataloged: Each place the crate differs from parsing the JSON and checking it with the library is a flag a fixture case declares, and the Rust tests assert the declared behavior. declared divergences | fixtures |
| fuz_schema | Zod and Valibot | Reference | Where the two agree, it gives the same result or declares the difference | Every change 3 probes
| Sorted: Each generated case is classified by the conformance tables, as agreeing with both or as differing in a way a rule declares, among other classes. the differential Cataloged: A difference from the two fails unless the conformance table declares it, and so does a declared difference that is no longer there. declared differences | differential |
| mdz streaming parser | Its one-shot parser | Twin | Whole and chunked input must give the same tree | Every change 3 probes
| architecture | |
| gro's Rust runtime, built-in modules | gro's Node implementations | Twin | Each case runs under the Rust runtime and must equal what the Node version printed, with differences declared per case | Every change 3 probes
| Rust workspace | |
| tsv formatter | Itself, on mutated input | Law | Seeded mutations of the fixtures go through parse, format and reparse, and a panic, a second pass that formats differently, or a lost node fails | Every change 1 probe
| fuzzer | |
| tsv location wire | The reconstruction its npm package ships | Twin | The two implementations of one definition of line and column must agree on every fixture | Every change 1 probe
| loc cross-grade | |
| tsv ignore matcher | git check-ignore | Reference | Nested ignore files are built in a throwaway repo, and the matcher must agree with git on every candidate path | Every change 2 probes
| git oracle test | |
| gro's module resolver | Node's resolution | Reference | Node resolves each case to a committed expectation, and the Rust resolver must match it unless the case declares a departure | Every change 4 probes
| Cataloged: A case where the resolver departs from Node by design declares why and what the resolver does instead. departures from Node | departures from Node |
| tsv WebAssembly build | tsv native build | Build | Both engines must return the same bytes, exit code and error output | CI 2 probes
| engine parity 1 probe
| |
| fuz_app backend on Node | The same backend on Bun | Build | Every conformance case is sent to both and the answers compared | CI 2 probes
| cross-backend tests | |
| fuz's package installer | npm | Reference | From one lockfile, what it installs must be a superset of what npm installs, with matching links | CI 2 probes
| installer gate | |
| gro's release binary | The same commit built at another path | Build | Two release builds of one commit must be the same, byte for byte | CI 1 probe
| Rust workspace | |
| tsv parser | The test suites of Svelte's compiler and acorn-typescript | Suite | Accept and reject verdicts must match, and differences in tree shape are reported without failing | Release 3 probes
| suite conformance | |
| tsv TypeScript parser | The TypeScript compiler's recorded baselines | Suite | The compiler's baselines over its own test cases decide which programs are valid, and a rejection outside the ledger fails | Release 2 probes
| Scoped: The `.tsx` cases are skipped and counted, since JSX is outside what tsv parses. scope Sorted: A program the compiler accepts and tsv rejects is either a sanction kept on purpose or a known gap to fix, and a ledger entry that matches nothing fails the gate. the ledgers | the oracle |
| tsv parser | test262 | Suite | The programs that should parse must parse, held to a pinned count, and wrongly accepted programs are reported without failing | Release 2 probes
| Scoped: Tests that need execution or module resolution are skipped, and so are the Annex B tests for web browsers, which tsv declares outside its scope. what is skipped | current results |
| blake3 WebAssembly builds | The native blake3 crate's test vectors | Reference | Each build must reproduce test vectors generated from the native crate | Release: the publish steps 1 probe
On demand: by hand 4 probes
| test vectors 1 probe
| |
| fuz_app backend, TypeScript | The fuz spine crates, Rust | Twin | One suite runs against both over HTTP, WebSocket and SSE, and their schemas, action manifests and conformance answers are compared | On demand 3 probes
| cross-backend tests | |
| zzz server | fuz_app's shared cross-backend suites | Suite | The suites that fuz_app ships run against the zzz server | On demand 3 probes
| development stage | |
| zap, working tree | zap, previous main build | Previous | Both binaries run against identical virtual machines and must report the same, which shows equivalence and not correctness | On demand 3 probes
| Sorted: Each scenario reports that it passed, diverged or errored, and an error is not read as a verdict. reading the output Cataloged: A change of behavior made on purpose is an entry that names the exact lines that differ and the reason, and it matches only the run it names. intended divergences | differential harness |
- Reference An independent implementation that already has the right behavior.
- Suite A test suite or recorded baselines from outside the implementation under test.
- Twin Two implementations in this stack built to match.
- Build The same source built or run two ways.
- Law A property the output must satisfy.
- Previous An earlier version of the same thing.
- Scoped What the project does not mean to match is named and taken out of the result.
- Pinned The other side is held to one version, so it changes only when someone changes it.
- Run live An expectation is checked against the other side as it runs, not only a recording.
- Sorted A difference is put into a kind before it is counted.
- Cataloged A difference kept on purpose is listed, and the list is held to the comparison.