feat(oracles)!: split each campaign into named units, replayable with ${prefix}_UNIT by KyleAMathews · Pull Request #35 · TanStack/oracles

KyleAMathews · 2026-11-11T23:51:42Z

Summary

oracleProperty now registers each part of a generated source's campaigns as its own named test (a "unit"). A CI matrix can then shard one long campaign across jobs. This is option D of the e2e speed plan (/tmp/e2e-speed/plan.md). Its design is in /tmp/e2e-ci/design-c-d.md, section D, which the maintainer approved on 2026-11-11 (Q2, Q3, Q4). Today a shard can hold only whole tests. The e2e gap oracle's determinism test (about 800 s in CI) and its random campaign (about 700 s) could not be split.

What changes for a property with a generated source:

Fixed-seed units. For each fixed seed N there is one test fixed seed N: <name> for each constructed history. It runs that history only, built from seed N + its index, as a fast-check example with no generated runs. Then fixed seed N: generated runs the campaign's generated histories, with the same seed and oracleRuns(runs) and no examples. fast-check draws generated values independently of its examples, so the units of seed N run exactly the histories that the old campaign ran. I measured this on fast-check 4.10.2: 0 differences in 1,050 seed and run-count combinations. The self-oracle's new units law keeps it under test.
Random units. The random campaign has random seed: <name> and random seed: generated units in the same way. They share one seed per property per process (Q2 (a)). The first random unit that runs selects that seed and logs the one ORACLE_SEED line.
Determinism units. There is one test for each compared history: determinism: fixed history: <name>, determinism: constructed history: <name>, and determinism: generated 0 to 2. An enumerated source keeps its one determinism test.
Per-unit verdict, shrink, timeout, and reach. Each unit decides its own first failure, shrinks within its own timeout, and prints <id> <unit> reach {...}. A campaign's reach is the sum of its units' lines. A unit that fails or times out prints no reach line.
Name errors at registration. These fail to load: a constructed history named generated, two constructed histories of one name, and two fixed histories of one name. Each would give two tests one name. They come after the state checks and before law membership.
${prefix}_UNIT (Q3). This variable names the random unit that a path replay runs. It requires ${prefix}_PROPERTY, and its check comes after every existing check in readRunConfig. During a path replay, only the named random unit runs, and every other test stands aside. A UNIT that names no random unit of the replayed property fails that property's registration. This includes an enumerated property, which has no random units. UNIT is part of the replay selection that configureOracles will not change after a read. The retired-variable message and the envPrefix docs list UNIT.
PATH without UNIT (Q4). A path replay of a generated source with PROPERTY and PATH but no UNIT fails when the property registers: "ORACLE_PATH of a generated source requires ORACLE_UNIT".
Plugin. oracle/no-disabled reads ${envPrefix}_UNIT as a shared replay variable and reports any other process.env read whose name ends in _UNIT as a private replay variable.
Replay lines. A failing unit prints Replay: ${prefix}_SEED=… ${prefix}_PATH=… ${prefix}_PROPERTY='…' ${prefix}_UNIT='random seed: <name>' vitest run -t '…' under the configured prefix. A fixed-seed unit's replay names the random unit of the same name, with the fixed seed, which draws the same stream. A failing random unit's violation prints the same command as its Rerun: line. That violation is built once, after fast-check reports the path, so its stack holds the message it prints. Without a path, it prints no rerun line. A failing fixed-seed unit also prints Rerun: -t '…' with its own anchored pattern.

The UNIT API is general: a unit is a named test that a path replay can select. A fresh-seed calibration (#31) can print the same replay line.

Not in scope: the trace memo (C), the CI matrix and partition-check script (E), and option M.

Evidence

Law. The unit and replay laws come from the design's section D. The authority is the maintainer decisions of 2026-11-11: option D, Q2 (one seed per property per process), Q3 (${prefix}_UNIT requires ${prefix}_PROPERTY; with UNIT, a path replay runs only that random unit; a UNIT that names no random unit fails registration), and Q4 (PATH without UNIT on a generated source fails at registration; every replay line names its unit). These laws are stated in the header of oracle-property.ts and in the self-oracle header: registration, units (new), verdict, shrink, reach, rerun, replay, replay-config (new), seed-log, route, and timeout. The orders of the name errors and the replay errors are analyst-chosen, and the self-oracle header says so.

Reach. The self-oracle declares 15 new measured states. Fixed histories name each of them and must enter them:

State	Fixed histories
`unit-split-verdict`	1
`unit-name-load-error`	3
`second-fixed-seed`	1
`unit-replay-constructed`	1
`unit-replay-generated`	1
`unit-replay-custom-prefix`	1
`unit-rerun`	2
`unit-rerun-beside-extension`	1
`seedless-units-drawn`	1
`replay-environment-error`	4
`unit-requires-property`	2
`retired-names-unit`	1
`replay-without-unit-error`	1
`replay-unit-missing-error`	2
`replay-unit-selected`	1

The grammar gains these dimensions: constructed-history names (indexed, names that extend each other such as c and c.1, generated, duplicate), duplicate fixed names, a second fixed seed (N and 10N), a suite prefix (ORACLE or SYNTH_ORACLE), and a drawn replay environment (retired, seed, path, property, unit). Generated suites (3 runs per unit) in the final full run reached these new states: fixed seed 1: generated (3 generated suites): unit-replay-generated 2, unit-replay-custom-prefix 1, seedless-units-drawn 1; random seed: generated (3 generated suites): unit-name-load-error 2, replay-environment-error 2, retired-names-unit 1. The other new states are reached only by the fixed histories above.

RED. npx vitest --run --config vitest.oracle-framework.config.mjs -t '…' with the extended self-oracle, run against unchanged production (the merge base, f0cda58). All 6 new histories fail at law registration. For example: self-oracle.synthetic-0 registered ["determinism","fixed seed 5","random seed"], owed ["determinism: constructed history: constructed-0", …, "fixed seed 5: constructed-0","fixed seed 5: generated","random seed: constructed-0","random seed: generated"]. Two duplicate constructed names loaded where the model owes a load error. During the change, the extended oracle also found two defects in my own draft. First, String.replace read the $' in the replay command as a replacement pattern and cut the rerun line short (law replay). Second, a unit whose test timed out printed its reach line late (law reach). Both are fixed, and the second is now a law: printed reach for ["fixed seed 7: generated","random seed: generated"], which passed no test in time. That behavior also exists on main for whole campaigns.
GREEN.
- npx vitest --run --config vitest.oracle-framework.config.mjs --maxWorkers 2: 299 of 299 passed in 1,125 s (on a loaded machine).
- npx vitest --run --maxWorkers 2 (default suite): 7 files, 114 tests passed.
- packages/eslint-plugin-oracles suite, against a fresh build of @tanstack/oracles: 14 files, 6,440 tests passed.
- test:oracle-gaps:fast: 4 files, 31,627 tests passed in 332 s (after merging a19c01f).
- After merging main (a19c01f), the full e2e gap oracle under the e2e lock: 96 of 96 passed in 2,610 s (machine under load). This includes the 28 fixed-seed units, the random units, every calibration, and every determinism unit. Before the merge, the fixed seed alone (-t "fixed seed"): 28 passed and 68 skipped of 96 tests, in 233 s. The 28 fixed-seed unit reach lines sum to the committed 144-state baseline (/tmp/e2e-speed/fixed-cpu/reach.json) with 0 differences. There is one extra state, a refused run whose package lies inside the CLI's directory (11), which test(oracles): fix the e2e gap oracle's determinism and summary-calibration tests #28 added after the baseline was taken.

Mutant. I ran 15 hostile mutants in an isolated copy against the relevant self-oracle histories. Every one failed:

Mutant	Rejected by
D-M1: the generated unit keeps the examples	`units`: `"fixed seed 7: generated" ran [[3,1,0]], owed [[3,1,1]]`
D-M2: the generated unit runs `numRuns - examples`	`units`: `ran [], owed [[1,1,2],[7,1,1]]`, and `shrink`: generated passed, owed `four`
D-M2b: the generated unit draws from seed + 1	`shrink`: owed `four`, got `three` at seed 6
D-M3: one predicate shared across units	`shrink`: `"fixed seed 5: generated" passed`, owed law `four`
D-M4: each random unit draws its own seed, and only one is logged	`units` (seedless): `"random seed: constructed-1" ran [[1,3,3]] … logged seed -1104256656, owed [[1,4,2]]`
D-M5: a rerun pattern without its `$`	`rerun`: `rerun of "fixed seed 7: c": other tests ran: [… "fixed seed 7: c.1" …]`
D-M6: a path replay ignores UNIT	`replay-config`: `ran ["random seed: constructed-0","random seed: generated"] … owed ["random seed: constructed-0"]`
D-M7: the replay line leaves out UNIT	`replay`: the printed variables lack `ORACLE_UNIT`
D-M7b: the replay line hard-codes `ORACLE_UNIT` under a custom prefix	`replay`: `ORACLE_UNIT` where `SYNTH_ORACLE_UNIT` is owed
PATH without UNIT accepted	`replay-config`: `loaded …, owed "ORACLE_PATH of a generated source requires ORACLE_UNIT"`
A UNIT that names no random unit accepted	`replay-config`
UNIT does not require PROPERTY	`replay-config`: `loaded …, owed "ORACLE_UNIT requires ORACLE_PROPERTY"`
UNIT checked before `PATH requires SEED`	`replay-config`: got `ORACLE_UNIT requires ORACLE_PROPERTY`, owed `ORACLE_PATH requires ORACLE_SEED`
The retired message leaves out UNIT	`replay-config`
A constructed history named `generated` accepted	`registration`
D-M8: a constructed unit keeps the generated runs (`{ ...run, numRuns: 0 }` dropped)	`units`: `"fixed seed 7: constructed-0" ran [[0,0,0],[0,0,0],[0,0,0]], owed [[0,0,0]]`. Review found this gap. Law `units` now requires a passing unit to run exactly its stream.

The late-reach fix's mutant is the code before the fix, which failed as shown under RED.

Limits.
- The units law compares a passing unit's log exactly, and a failing unit's log only up to its first failure.
- The model's unitStream restates production's fc.oneof weighting. The design authorizes this. It is that composition, not a contract promise, and the model's comment says so.
- An error's raw stack is not observed. Vitest reprints a failure's stack header from its message, so the old stale-placeholder stack was not visible under Vitest. Its fix has no killing mutant.
- The plugin's _UNIT change is covered by ordinary rule tests (package-specifiers.test.js), not by an oracle. It does not compare any unit of a slow spec, whose driver runs on after its test times out.
- A drawn replay environment always uses path 0 and the first spec's property. Its unit is the first random unit of that spec or a missing one.
- A custom prefix applies to a whole suite.
- The driver records no log entries for a slow spec: its driver runs on after its test times out and logs under whichever test runs next. Its late probe calls therefore have no upper bound, and its seedless histories are not compared. The driver also sorts the log by property, because the child runs spec files in parallel.
- Only one fixed-unit -t rerun and one history rerun are probed per property.
- The fixture makes Math.random a fixed-seed generator, so the seedless run's drawn seed repeats between the self-oracle's own determinism runs. Each draw is still the framework's.
- No history changes only the UNIT between two configureOracles calls, so the changesReadSelection term for UNIT has no witness. It is the one production line here that no oracle checks.
- The campaign reach merge script (option E) is not part of this change.

e2e shards

packages/oracles/scripts/e2e-shards.mjs (#34) now partitions the e2e gap oracle's 96 units into 5 shards. The shards are a longest-first packing of the per-unit costs from one full run of every unit on this branch: 96 of 96 passed in 2,610 s on a loaded 18-core Mac.

Group	Measured cost
Fixed-seed units	370 s
Random units	605 s (varies with the drawn seed)
Calibrations	917 s
Determinism units	718 s
Slowest single unit	236 s (determinism of the edit sequence)

The fixed-seed units took 2.36 times the 157 s of an idle run. Each shard is about 522 s here, about 221 s idle, and an estimated 5.5 to 8.5 minutes in CI by the measured 1.5 to 2.31 factor, before job setup.

partitionErrors passes against vitest list: 96 tests, each in exactly one shard. Each shard's -t pattern selects exactly its units under Vitest: 19, 19, 19, 19, and 20.

The CI workflow keeps its single fixed-seed job, for three reasons:

The shards include the random units. Their tail is unbounded until option M: a random unit that draws a long edit sequence over the witness module can pass 10 minutes in CI, about one run in six by the plan's sampling.
Running all 96 units adds the calibrations and determinism, about 1,600 s of local compute, to every PR.
A 5-job matrix takes the workflow from 7 jobs to 11, against the org-wide concurrency queue.

Wiring the matrix, or a matrix of only the 28 fixed-seed units, is a follow-up decision.

pnpm exec in the check's vitest list call tries to reinstall a symlinked node_modules. Locally I ran the script's own partitionErrors over npx vitest list output instead. CI installs normally, so pnpm --filter @tanstack/oracles run test:oracle-gaps:shards is unaffected.

Breaking for consumers: migrating a recorded path

A path replay of a generated source now needs ${prefix}_UNIT, and recorded paths move onto their units. In the old campaign, the first index of a path selected the history: an index k below n (the number of constructed histories) was constructed history k, and an index k of n or more was generated history k − n. With PATH=k:<rest>:

for k < n: ${prefix}_UNIT='random seed: <name of constructed history k>' and ${prefix}_PATH=0:<rest>;
for k ≥ n: ${prefix}_UNIT='random seed: generated' and ${prefix}_PATH=(k − n):<rest>.

A path recorded from a fixed-seed campaign moves the same way, with that seed as ${prefix}_SEED. The changeset states this rule.

TanStack DB follow-up (not in this PR). scripts/calibrate-persisted-option-types.mjs sets TANSTACK_DB_ORACLE_SEED, _PATH, and _PROPERTY for generated sources (about lines 101–107 and 349–355). With this change those properties fail to load. It needs TANSTACK_DB_ORACLE_UNIT='random seed: all utility admission forms' and TANSTACK_DB_ORACLE_UNIT='random seed: all admitted type forms' respectively, with each path rebased as above.

User-visible changes

Test names of generated sources change. -t "fixed seed" still selects the fixed-seed units, and -t "random seed" selects the random units.
A copied replay command now needs ${prefix}_UNIT. The printed command includes it.
The test count grows. The e2e gap oracle goes from 13 tests to about 96. The fast gap suites register 31,627 tests.

This pull request and its description were written by Isaac.

…T replay A generated source registers each part of its campaigns as its own test: `fixed seed N: <name>` and `fixed seed N: generated` for each fixed seed, `random seed: <name>` and `random seed: generated` for the random campaign, and one `determinism: <history>` test per compared history. The units of a seed run exactly the histories the campaign ran; each decides its own first failure, shrinks within its own timeout, and prints its own reach line. A unit that fails or times out prints no reach line. The random units share one seed per property (maintainer decision, 2026-11-11, Q2). `${prefix}_UNIT` names the random unit a path replay runs and requires `${prefix}_PROPERTY` (Q3). A path replay of a generated source without UNIT, or with a UNIT that names no random unit, fails at registration (Q4). Every failing unit prints a replay command that names its unit, and a failing fixed-seed unit prints a `-t` rerun of itself. Duplicate constructed or fixed names, and a constructed history named `generated`, fail to load. The framework self-oracle gains the `units` and `replay-config` laws, unit names, a custom prefix, a drawn replay environment, a second fixed seed, and names that extend each other. Co-authored-by: Isaac <no-reply@databricks.com>

…'s violation built once The self-oracle's units law now requires a unit that the model owes a pass to run exactly its stream, no history more; only a failing unit is compared up to its first failure. Mutant D-M8 (a constructed unit keeps the generated runs) now fails: "fixed seed 7: constructed-0" ran [[0,0,0],[0,0,0],[0,0,0]], owed [[0,0,0]]. The model's comment on unitStream says that its weighting restates production's composition, which the design authorizes, and is no contract promise. A random unit's violation is now built once, after fast-check reports its path, with the real replay command as its rerun line, so its stack holds the message it prints; without a path it prints no rerun line. Before, the message was patched over a placeholder that the stack kept. Vitest reprints a failure's stack header from its message, so the self-oracle cannot observe the raw stack; that is a stated limit. Co-authored-by: Isaac <no-reply@databricks.com>

…he replay migration oracle/no-disabled reads `${envPrefix}_UNIT` as a shared replay variable and reports a process.env read of any other `*_UNIT` variable as a private replay variable. RED: the two new invalid cases (`ORACLE_UNIT` under another prefix and a destructured `UTILS_ORACLE_UNIT`) were not reported; GREEN: the plugin suite passes. The changeset marks the unit change as breaking for consumers and states how to rebase a recorded path onto its unit. Co-authored-by: Isaac <no-reply@databricks.com>

Co-authored-by: Isaac <no-reply@databricks.com>

The property now registers its campaign parts, calibrations, and determinism histories as units, so the e2e shards name the 96 units. The shards are a longest-first packing of the per-unit costs of one full run of every unit on this branch (2,610 s under load, 2.36 times an idle run), so each shard is about 522 s here, about 221 s idle, and 5.5 to 8.5 minutes in CI by the measured 1.5 to 2.31 factor. The partition check passes against `vitest list`, and each shard's `-t` pattern selects exactly its units. CI keeps its one fixed-seed job: the shards include the random units, whose tail can pass 10 minutes in CI until option M bounds it. Co-authored-by: Isaac <no-reply@databricks.com>

changeset-bot · 2026-11-11T23:51:48Z

🦋 Changeset detected

Latest commit: 6f095b1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@tanstack/oracles	Patch
@tanstack/eslint-plugin-oracles	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

coderabbitai · 2026-11-11T23:51:50Z

Important

Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 72b68cae-2114-4008-b24f-d4341a84f437

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

pkg-pr-new · 2026-11-12T00:07:33Z

Open in StackBlitz

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@35

npm i https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@35

commit: 6f095b1

KyleAMathews and others added 6 commits November 11, 2026 13:01

Merge remote-tracking branch 'origin/main' into feat/unit-replay

52cc8be

Merge origin/main into feat/unit-replay

b9bf7ed

Co-authored-by: Isaac <no-reply@databricks.com>

KyleAMathews merged commit d38af38 into main Nov 12, 2026
13 checks passed

KyleAMathews deleted the feat/unit-replay branch November 12, 2026 00:08

KyleAMathews mentioned this pull request Nov 12, 2026

fix(oracles): types label a survivor as outside-annotation, and never skip a mutant or mark it dead #36

Merged

Navigation Menu

Repository navigation

feat(oracles)!: split each campaign into named units, replayable with ${prefix}_UNIT - #35

feat(oracles)!: split each campaign into named units, replayable with ${prefix}_UNIT#35
KyleAMathews merged 6 commits into
mainfrom
feat/unit-replay

KyleAMathews commented Nov 11, 2026

Uh oh!

changeset-bot Bot commented Nov 11, 2026

Uh oh!

coderabbitai Bot commented Nov 11, 2026

Uh oh!

pkg-pr-new Bot commented Nov 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

KyleAMathews commented Nov 11, 2026

Summary

Evidence

e2e shards

Breaking for consumers: migrating a recorded path

User-visible changes

Uh oh!

changeset-bot Bot commented Nov 11, 2026

🦋 Changeset detected

Uh oh!

coderabbitai Bot commented Nov 11, 2026

Uh oh!

pkg-pr-new Bot commented Nov 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant