Oracles in use

Each project below checks its output against an oracle, something to agree with. A row names what the project compares itself against, how, and when the comparison runs. The rows are sorted by how soon each comparison reports: one that runs on every change speaks before the change lands, and one that runs before a release speaks long after the change that broke it.

The holds whose instances this table lists: Drift checks and twin suites (Two copies drift apart); Oracles, mutants and control cases (Green but wrong); Suppressions that expire (A check is silenced instead of satisfied)

Where a project's docs say what keeps a comparison's result meaning what it says, the row lists it by discipline. A discipline a row does not list is not stated either way.

A row may carry probes: a string that a named file, check script or CI step of the repo must have, or must not have, for the row to still be true in the way a file can show. The tests of this site check the ones about a check script or a CI step against its snapshot of the repos, and the evidence task checks the ones about a file. A probe that passes says the file, script or step still has the text the row depends on. It does not say the check is sound. A probe reads only the text it names, so one that says a step is absent does not see the same thing reached through another script or switched on by an environment variable.

  • Every change In the gate a session runs before landing, and again in CI.
  • CI In CI on a push to the main branch or a pull request, and not in the local gate.
  • Release Before a release, and not on each change.
  • On demand When someone chooses to run it.
ProjectCompared againstKindHowRunsKept meaningful bySource
tsv formatterPrettierReferenceFixtures hold Prettier's output with each difference documented, the recordings are checked against a live, pinned Prettier, and a corpus diff fails on lost content or a new uncataloged difference
Every change: fixtures, against a live Prettier
4 probes
  • the check script of tsv has npm run test &&
  • tsv/package.json has "test": "cargo test --workspace"
  • the steps of check.yml in tsv have npm run check
  • the steps of check.yml in tsv have npm ci --prefix crates/tsv_debug/sidecar
Release: corpus diff
6 probes
  • tsv/scripts/publish.ts has ['run', 'conformance:all']
  • tsv/benches/js/conformance.ts has run_corpus_compare_format
  • the check script of tsv lacks conformance:all
  • the CI steps of tsv lack conformance:all
  • the check script of tsv lacks corpus:compare
  • the CI steps of tsv lack corpus:compare
Pinned: One audit holds every place that names Prettier, its Svelte plugin or the parsers to a single exact version, and the options passed to Prettier to a single set. pin agreement audit
1 probe
  • the check script of tsv has npm run pins:audit
Run live: A fixture that claims Prettier never settles on an input, or rejects it, has the claim run against the live Prettier whenever the fixtures are validated. validation rules
Sorted: A corpus file that differs is sorted by whether the catalog explains every part of the difference, some of it, or none. classification
Cataloged: The cases where Prettier's output is unstable, does not parse again, throws or changes meaning are listed, each held by a fixture. Prettier bug index
CI workflow
1 probe
  • tsv/tests/fixtures_tests.rs has sidecar::check().await
tsv formatterItselfLawFormatting twice gives the same result, on every fixture and across a real-code corpus
Every change: fixtures
2 probes
  • the check script of tsv has npm run test &&
  • tsv/crates/tsv_debug/src/fixtures/validation/mod.rs has validate_formatter_idempotent
Release: corpus
3 probes
  • tsv/scripts/publish.ts has ['run', 'audit:corpus']
  • the check script of tsv lacks audit:corpus
  • the CI steps of tsv lack audit:corpus
corpus bundle
tsv formatterIts input, reparsedLawFormatted output parses back to the same document as its source
Every change: fixtures
1 probe
  • the check script of tsv has npm run roundtrip:audit &&
Release: corpus
2 probes
  • tsv/scripts/publish.ts has ['run', 'audit:corpus']
  • tsv/benches/js/corpus_audit.ts has 'roundtrip_audit'
round-trip audit
tsv formatterIts input, renderedLawA Svelte component renders the same before and after formatting
Every change: fixtures
3 probes
  • the check script of tsv has npm run test &&
  • tsv/tests/fixtures_tests.rs has sidecar::check().await
  • tsv/crates/tsv_debug/src/fixtures/validation/errors.rs has RenderEquivalenceMismatch
Release: corpus
2 probes
  • tsv/scripts/publish.ts has ['run', 'conformance:all']
  • tsv/benches/js/conformance.ts has 'render:audit (pinned checkouts)'
render audit
tsv parserSvelte's parser, acorn-typescript and Svelte's CSS parserReferenceFixtures hold the canonical parser's syntax tree, the recordings are checked against the live parser, and a deep diff over real codebases classifies every difference against a catalog
Every change: fixtures, against the live parsers
2 probes
  • the check script of tsv has npm run test &&
  • the steps of check.yml in tsv have npm ci --prefix crates/tsv_debug/sidecar
Release: corpus diff
2 probes
  • tsv/scripts/publish.ts has ['run', 'conformance:all']
  • tsv/benches/js/conformance.ts has run_corpus_compare_parse
fixture tests
1 probe
  • tsv/tests/fixtures_tests.rs has sidecar::check().await
tsv type strippingts-blank-space, with Node's type stripping as a cross-checkReferenceFixtures hold ts-blank-space's output, a corpus diff runs against it, and a Node cross-check of the fixtures runs by hand
Every change: fixtures
2 probes
  • the check script of tsv has npm run test &&
  • tsv/tests/strip_fixtures_tests.rs exists
Release: corpus diff
2 probes
  • tsv/scripts/publish.ts has ['run', 'conformance:all']
  • tsv/package.json has npm run strip:corpus:run && npm run imports:corpus:run
On demand: Node cross-check
4 probes
  • tsv/package.json has "strip:fixtures:oracle"
  • tsv/benches/js/strip_fixtures.ts has stripTypeScriptTypes
  • the check script of tsv lacks strip:fixtures:oracle
  • the CI steps of tsv lack strip:fixtures:oracle
strip fixtures
tsv import scannerThe canonical parsers' trees, and oxc-parser's module recordReferenceFixtures hold records read off the canonical parsers' trees, the recordings are checked against those parsers by hand, and the records must agree with oxc-parser's over a corpus
Every change: fixtures
3 probes
  • the check script of tsv has npm run test &&
  • tsv/tests/imports_fixtures_tests.rs exists
  • tsv/tests/imports_self_check_tests.rs exists
Release: corpus
2 probes
  • tsv/scripts/publish.ts has ['run', 'conformance:all']
  • tsv/package.json has npm run strip:corpus:run && npm run imports:corpus:run
On demand: recordings against the parsers
3 probes
  • tsv/package.json has "imports:fixtures:validate"
  • the check script of tsv lacks imports:fixtures:validate
  • the CI steps of tsv lack imports:fixtures:validate
import differential
fuz_forge server, TypeScriptIts Rust serverTwinBoth unit suites assert against shared case tables, and a cross-backend suite runs both servers
Every change: shared case tables
5 probes
  • the check script of fuz_forge has gro check
  • fuz_forge/src/test/fixtures/parity/parity_fixtures.meta.test.ts exists
  • fuz_forge/crates/fuz_forge_server/tests/parity_fixture_census.rs exists
  • the steps of check.yml in fuz_forge have cargo test --locked --workspace
  • the steps of check.yml in fuz_forge have gro check
On demand: cross-backend suite
2 probes
  • fuz_forge/vite.config.ts has 'cross_backend_parity'
  • the CI steps of fuz_forge lack cross_backend
parity fixtures
1 probe
  • fuz_forge/src/test/fixtures/parity/load.ts exists
pgletPostgresReferenceSQL fixtures run through both, compared in the binary format against recorded Postgres results and in the text format live
Every change
3 probes
  • pglet/crates/xtask/src/gates.rs has Gate::new("test:fixtures"
  • pglet/package.json has pglet_tests --bin test_fixtures
  • the steps of check.yml in pglet have test:fixtures
Cataloged: Behaviors that differ from Postgres are listed in a table, the deliberate tradeoffs and the open gaps alike. simplified behaviors
fixtures
fuz_schema, Rust sideThe TypeScript libraryTwinThe TypeScript side generates fixtures and the Rust tests assert both evaluators against them, with differences declared
Every change
3 probes
  • fuz_schema/crates/fuz_schema/tests/fixtures.rs exists
  • fuz_schema/src/test/json.fixtures.test.ts exists
  • the steps of check.yml in fuz_schema have cargo test --workspace
Cataloged: Each place the crate differs from parsing the JSON and checking it with the library is a flag a fixture case declares, and the Rust tests assert the declared behavior. declared divergences
fixtures
fuz_schemaZod and ValibotReferenceWhere the two agree, it gives the same result or declares the difference
Every change
3 probes
  • the check script of fuz_schema has gro check
  • fuz_schema/src/test/schema.differential.test.ts exists
  • the steps of check.yml in fuz_schema have gro check
Sorted: Each generated case is classified by the conformance tables, as agreeing with both or as differing in a way a rule declares, among other classes. the differential
Cataloged: A difference from the two fails unless the conformance table declares it, and so does a declared difference that is no longer there. declared differences
differential
mdz streaming parserIts one-shot parserTwinWhole and chunked input must give the same tree
Every change
3 probes
  • the check script of mdz has gro check
  • mdz/src/test/mdz_parser_parity.test.ts exists
  • the CI steps of mdz have gro check
architecture
gro's Rust runtime, built-in modulesgro's Node implementationsTwinEach case runs under the Rust runtime and must equal what the Node version printed, with differences declared per case
Every change
3 probes
  • gro/crates/gro_cli/tests/host.rs has expected.json
  • gro/crates/gro_cli/tests/fixtures/host/oracle.js exists
  • the steps of check.yml in gro have cargo xtask check-rust
Rust workspace
tsv formatterItself, on mutated inputLawSeeded mutations of the fixtures go through parse, format and reparse, and a panic, a second pass that formats differently, or a lost node fails
Every change
1 probe
  • the check script of tsv has npm run fuzz:audit &&
fuzzer
tsv location wireThe reconstruction its npm package shipsTwinThe two implementations of one definition of line and column must agree on every fixture
Every change
1 probe
  • the check script of tsv has npm run check:loc
loc cross-grade
tsv ignore matchergit check-ignoreReferenceNested ignore files are built in a throwaway repo, and the matcher must agree with git on every candidate path
Every change
2 probes
  • the check script of tsv has npm run test &&
  • tsv/crates/tsv_ignore/tests/git_oracle.rs has check-ignore
git oracle test
gro's module resolverNode's resolutionReferenceNode resolves each case to a committed expectation, and the Rust resolver must match it unless the case declares a departure
Every change
4 probes
  • gro/crates/gro_runtime/src/resolver/tests/node_differential.rs exists
  • gro/crates/gro_runtime/tests/fixtures/resolution/expected.json exists
  • gro/src/test/resolution_differential.test.ts exists
  • the steps of check.yml in gro have cargo xtask check-rust
Cataloged: A case where the resolver departs from Node by design declares why and what the resolver does instead. departures from Node
departures from Node
tsv WebAssembly buildtsv native buildBuildBoth engines must return the same bytes, exit code and error output
CI
2 probes
  • the steps of check.yml in tsv have npm run engines:audit
  • the check script of tsv lacks engines:audit
engine parity
1 probe
  • tsv/scripts/engine_parity.ts exists
fuz_app backend on NodeThe same backend on BunBuildEvery conformance case is sent to both and the answers compared
CI
2 probes
  • the steps of check.yml in fuz_app have npm run test:cross:ts-differential
  • fuz_app/package.json has cross_backend_ts_differential
cross-backend tests
fuz's package installernpmReferenceFrom one lockfile, what it installs must be a superset of what npm installs, with matching links
CI
2 probes
  • the steps of check.yml in fuz have scripts/fuzi-gate.sh
  • fuz/scripts/fuzi-gate.sh exists
installer gate
gro's release binaryThe same commit built at another pathBuildTwo release builds of one commit must be the same, byte for byte
CI
1 probe
  • the steps of check.yml in gro have cargo xtask check-reproducible
Rust workspace
tsv parserThe test suites of Svelte's compiler and acorn-typescriptSuiteAccept and reject verdicts must match, and differences in tree shape are reported without failing
Release
3 probes
  • tsv/scripts/publish.ts has ['run', 'conformance:all']
  • tsv/benches/js/conformance.ts has SVELTE_FIXTURES_GATE
  • tsv/benches/js/conformance.ts has TS_FIXTURES_GATE
suite conformance
tsv TypeScript parserThe TypeScript compiler's recorded baselinesSuiteThe compiler's baselines over its own test cases decide which programs are valid, and a rejection outside the ledger fails
Release
2 probes
  • tsv/scripts/publish.ts has ['run', 'conformance:all']
  • tsv/benches/js/conformance.ts has run_ts_repo_compare
Scoped: The `.tsx` cases are skipped and counted, since JSX is outside what tsv parses. scope
Sorted: A program the compiler accepts and tsv rejects is either a sanction kept on purpose or a known gap to fix, and a ledger entry that matches nothing fails the gate. the ledgers
the oracle
tsv parsertest262SuiteThe programs that should parse must parse, held to a pinned count, and wrongly accepted programs are reported without failing
Release
2 probes
  • tsv/scripts/publish.ts has ['run', 'conformance:all']
  • tsv/package.json has "conformance:all": "npm run conformance:test262
Scoped: Tests that need execution or module resolution are skipped, and so are the Annex B tests for web browsers, which tsv declares outside its scope. what is skipped
current results
blake3 WebAssembly buildsThe native blake3 crate's test vectorsReferenceEach build must reproduce test vectors generated from the native crate
Release: the publish steps
1 probe
  • blake3/scripts/lib/release.ts has 'test:wasm'
On demand: by hand
4 probes
  • blake3/package.json has "test:wasm"
  • the check script of blake3 lacks test:wasm
  • the check script of blake3 lacks npm run test &&
  • the check script of blake3 lacks test_wasm.ts
test vectors
1 probe
  • blake3/scripts/test_wasm.ts has test_vectors.json
fuz_app backend, TypeScriptThe fuz spine crates, RustTwinOne suite runs against both over HTTP, WebSocket and SSE, and their schemas, action manifests and conformance answers are compared
On demand
3 probes
  • fuz_app/package.json has "test:cross:parity"
  • fuz_app/src/test/cross_backend/run_cross.ts exists
  • the CI steps of fuz_app lack test:cross:parity
cross-backend tests
zzz serverfuz_app's shared cross-backend suitesSuiteThe suites that fuz_app ships run against the zzz server
On demand
3 probes
  • zzz/package.json has "test:cross"
  • zzz/src/test/cross_backend/spine_suites.cross.test.ts has describe_spine_cross_suites
  • the CI steps of zzz lack test:cross
development stage
zap, working treezap, previous main buildPreviousBoth binaries run against identical virtual machines and must report the same, which shows equivalence and not correctness
On demand
3 probes
  • zap/crates/xtask/src/differential/mod.rs exists
  • zap/crates/xtask/src/main.rs has differential
  • the CI steps of zap lack differential
Sorted: Each scenario reports that it passed, diverged or errored, and an error is not read as a verdict. reading the output
Cataloged: A change of behavior made on purpose is an entry that names the exact lines that differ and the reason, and it matches only the run it names. intended divergences
differential harness
  • Reference An independent implementation that already has the right behavior.
  • Suite A test suite or recorded baselines from outside the implementation under test.
  • Twin Two implementations in this stack built to match.
  • Build The same source built or run two ways.
  • Law A property the output must satisfy.
  • Previous An earlier version of the same thing.
  • Scoped What the project does not mean to match is named and taken out of the result.
  • Pinned The other side is held to one version, so it changes only when someone changes it.
  • Run live An expectation is checked against the other side as it runs, not only a recording.
  • Sorted A difference is put into a kind before it is counted.
  • Cataloged A difference kept on purpose is listed, and the list is held to the comparison.