feat(oracles)!: split each campaign into named units, replayable with ${prefix}_UNIT

.changeset/shardable-units.md

@@ -0,0 +1,23 @@
+---
+'@tanstack/oracles': patch
+'@tanstack/eslint-plugin-oracles': patch
+---
+
+**Breaking for consumers.** Test names of generated sources change, and a path replay of a generated source now needs `${prefix}_UNIT`. A command or script that sets only the seed, the path, and the property now fails when the property registers.
+
+A generated source now registers each part of a campaign as its own test, so a runner can shard a long campaign across jobs. For each fixed seed N, `oracleProperty` registers `fixed seed N: <name>` for each constructed history and `fixed seed N: generated` for the generated histories. The random campaign registers `random seed: <name>` and `random seed: generated` in the same way. The determinism test becomes one test per history: `determinism: fixed history: <name>`, `determinism: constructed history: <name>`, and `determinism: generated 0` to `2`. The units of a seed run exactly the histories that the campaign ran. Each unit decides its own first failure, shrinks within its own timeout, and prints its own reach line, `<id> <unit> reach {...}`. A campaign's reach is the sum of its units' lines. A unit that fails or times out prints no reach line. An enumerated source keeps its one `determinism` test.
+
+Test names change. A `-t "fixed seed"` filter still selects the fixed-seed units, and `-t "random seed"` the random units. A filter that names the old `determinism` test of a generated source now selects all its determinism units. The random units of a property share one seed, which the first random unit that runs selects and logs in the one `ORACLE_SEED` line.
+
+A path replay now names its unit in the new `${prefix}_UNIT` variable, such as `ORACLE_UNIT='random seed: generated'`. `UNIT` requires `PROPERTY`. During a path replay, only the random unit of that name runs. A path replay of a generated source without `UNIT` fails when the property registers, with "`ORACLE_PATH` of a generated source requires `ORACLE_UNIT`". A `UNIT` that names no random unit of the replayed property also fails its registration. A failing unit prints its replay command, with the seed, the path, the property, the unit, and its `-t` pattern, under the configured prefix. A failing fixed-seed unit also prints `Rerun: -t '...'`, which reruns it. The retired-variable message names `${prefix}_UNIT`.
+
+To migrate a recorded path, rebase it onto its unit. In the old campaign, the first index of a path selected the history: an index k below n, the number of constructed histories, was constructed history k, and an index k of n or more was generated history k − n. With `PATH=k:<rest>`:
+
+- for k < n, set `${prefix}_UNIT='random seed: <name of constructed history k>'` and `${prefix}_PATH=0:<rest>`;
+- for k ≥ n, set `${prefix}_UNIT='random seed: generated'` and `${prefix}_PATH=(k − n):<rest>`.
+
+A path recorded from a fixed-seed campaign replays the same way, with that fixed seed as `${prefix}_SEED`. A new failure prints its replay command, so only paths recorded before this change need rebasing.
+
+A generated source fails to load when two of its tests would share a name: a constructed history named `generated`, two constructed histories of one name, or two fixed histories of one name.
+
+In `@tanstack/eslint-plugin-oracles`, `oracle/no-disabled` reads `${envPrefix}_UNIT` as a shared replay variable, and reports a `process.env` read of any other variable that ends in `_UNIT` as a private replay variable, as it does for `_SEED`, `_PATH`, `_PROPERTY`, and `_RUNS`.

packages/eslint-plugin-oracles/src/registration.js

@@ -455,6 +455,7 @@ export function oracleSettings(context) {
       seed: `${envPrefix}_SEED`,
       path: `${envPrefix}_PATH`,
       property: `${envPrefix}_PROPERTY`,
+      unit: `${envPrefix}_UNIT`,
       runsMultiplier: `${envPrefix}_RUNS_MULTIPLIER`,
     },
     implementationModules: moduleList(configured, `implementationModules`),

packages/eslint-plugin-oracles/src/rules/no-disabled.js

@@ -41,8 +41,9 @@
  * - A literal run count above 1 that has no fixed history. It does not scale with
  *   the runs-multiplier variable (`${envPrefix}_RUNS_MULTIPLIER`), so `oracleRuns` must wrap it.
  * - A private replay variable: a `process.env` read whose name ends in
- *   `_SEED`, `_PATH`, `_PROPERTY`, or `_RUNS`, other than the framework's
- *   shared `${envPrefix}_SEED`, `${envPrefix}_PATH`, and `${envPrefix}_PROPERTY`. The replay manifest, the guarded replay
+ *   `_SEED`, `_PATH`, `_PROPERTY`, `_UNIT`, or `_RUNS`, other than the framework's
+ *   shared `${envPrefix}_SEED`, `${envPrefix}_PATH`, `${envPrefix}_PROPERTY`, and
+ *   `${envPrefix}_UNIT`, the random unit a path replay runs. The replay manifest, the guarded replay
  *   runner, and the retired-variable check see only the shared variables.
  *
  * The rule has no inline exception. A known counterexample is kept failing with
@@ -76,7 +77,7 @@ const replayConditions = new Set([`oracleReplayRequested`, `oracleReplaySkips`])
 // A replay-selection module (`implementationModules`) may read the replay
 // variables and select a replay from configuration.
 // Any variable whose name ends in one of these suffixes is a replay variable.
-const replayVariableName = /_(PATH|PROPERTY|RUNS|SEED)$/
+const replayVariableName = /_(PATH|PROPERTY|RUNS|SEED|UNIT)$/
 const runHelpers = new Set([
   `oraclePropertyOptions`,
   `oracleRandomParameters`,
@@ -415,7 +416,12 @@ function environmentKey(node) {
 }
 
 function isPrivateReplayVariable(name, envNames) {
-  const shared = [envNames.seed, envNames.path, envNames.property]
+  const shared = [
+    envNames.seed,
+    envNames.path,
+    envNames.property,
+    envNames.unit,
+  ]
   return replayVariableName.test(name) && !shared.includes(name)
 }
 
@@ -729,7 +735,7 @@ export default {
       aliasTest: `\`{{name}}\` is passed around as a value, so the rule cannot see what it registers or switches off. Call it directly, or select a replay with \`oracleReplayTest({{name}}, property)\` from \`@tanstack/oracles\`.`,
       guarded: `This test registers only under a condition, so it can stop running in ordinary runs. Register it unconditionally. A replay selection uses \`oracleReplayRequested()\` or \`oracleReplaySkips(property)\` from \`@tanstack/oracles\`, or \`oracleReplayTest(test, property)\`.`,
       singleRun: `A generated property with one run that has no fixed history checks one random history. Pass a base run count to oracleRuns or oraclePropertyOptions, which also selects a replay. A witness passes a recorded \`path\` or a seed that is a number literal or a constant.`,
-      replayVariable: `\`{{name}}\` selects a replay or a run count outside the shared variables, so the replay manifest, the guarded replay runner, and the retired-variable check cannot see it. Take the options from \`oraclePropertyOptions\` or \`oracleRandomParameters\` in \`@tanstack/oracles\`, which read {{seed}}, {{path}}, and {{property}}, and scale run counts with \`oracleRuns\`.`,
+      replayVariable: `\`{{name}}\` selects a replay or a run count outside the shared variables, so the replay manifest, the guarded replay runner, and the retired-variable check cannot see it. Take the options from \`oraclePropertyOptions\` or \`oracleRandomParameters\` in \`@tanstack/oracles\`, which read {{seed}}, {{path}}, {{property}}, and {{unit}}, and scale run counts with \`oracleRuns\`.`,
       literalRuns: `A literal run count does not scale with {{runsMultiplier}}. Wrap it in oracleRuns, or pass it to oraclePropertyOptions.`,
     },
   },

packages/eslint-plugin-oracles/tests/no-disabled.test.js

@@ -15,6 +15,7 @@ const dbEnvNames = {
   seed: 'TANSTACK_DB_ORACLE_SEED',
   path: 'TANSTACK_DB_ORACLE_PATH',
   property: 'TANSTACK_DB_ORACLE_PROPERTY',
+  unit: 'TANSTACK_DB_ORACLE_UNIT',
   runsMultiplier: 'TANSTACK_DB_ORACLE_RUNS_MULTIPLIER',
 }
 // The fixtures name the TanStack DB project's files and variables, so the

packages/eslint-plugin-oracles/tests/package-specifiers.test.js

@@ -66,6 +66,7 @@ noDisabledTester.run(`oracle/no-disabled`, noDisabled, {
             seed: `ORACLE_SEED`,
             path: `ORACLE_PATH`,
             property: `ORACLE_PROPERTY`,
+            unit: `ORACLE_UNIT`,
             runsMultiplier: `ORACLE_RUNS_MULTIPLIER`,
           },
         },
@@ -100,6 +101,11 @@ projectTester.run(`oracle/no-disabled with a project's settings`, noDisabled, {
       code: `const seed = process.env.TANSTACK_DB_ORACLE_SEED`,
       filename: `/repo/tests/checks-oracle.test.ts`,
     },
+    // The unit a path replay runs is a shared replay variable.
+    {
+      code: `const unit = process.env.TANSTACK_DB_ORACLE_UNIT`,
+      filename: `/repo/tests/checks-oracle.test.ts`,
+    },
     // The configured replay module, named by its path, may read a private
     // replay variable.
     {
@@ -119,5 +125,17 @@ projectTester.run(`oracle/no-disabled with a project's settings`, noDisabled, {
       filename: `/repo/tests/checks-oracle.test.ts`,
       errors: [{ messageId: `replayVariable` }],
     },
+    // A unit under another prefix selects a replay the shared variables do
+    // not see.
+    {
+      code: `const unit = process.env.ORACLE_UNIT`,
+      filename: `/repo/tests/checks-oracle.test.ts`,
+      errors: [{ messageId: `replayVariable` }],
+    },
+    {
+      code: `const { UTILS_ORACLE_UNIT } = process.env`,
+      filename: `/repo/tests/checks-oracle.test.ts`,
+      errors: [{ messageId: `replayVariable` }],
+    },
   ],
 })

packages/oracles/scripts/e2e-shards.mjs

@@ -21,35 +21,133 @@ import { fileURLToPath } from 'node:url'
 
 const suite = `oracle-gaps.report`
 
-// The shards balance the costs measured in one full local run on main
-// (84cd34a, 2026-11-11, an idle 18-core Mac, one Vitest worker), in local
-// seconds: determinism about 345 (all 30 histories, twice each), the random
-// seed 160 to 600 by its seed, the fixed seed 157, the refused-run
-// calibration 102, the two daemon calibrations 56 and 53, and each other
-// calibration 38 to 46. CI took 2.31 times as long. The property registers
-// each campaign, each calibration, and the determinism test as one test, so
-// a shard cannot be smaller than its largest test, and the determinism test
-// alone sets the slowest shard. CI keeps its one fixed-seed job until the
-// property registers its parts as units that a shard can split.
+// The shards balance the costs measured in one full run of every unit on
+// feat/unit-replay (2026-11-11, an 18-core Mac under other agents' load, one
+// Vitest worker): 2,610 s in all, 370 s for the fixed-seed units, 605 s for
+// the random units (their cost varies with the drawn seed), 917 s for the
+// calibrations, and 718 s for the determinism units. The fixed-seed units
+// took 2.36 times the 157 s of an idle run (84cd34a), and CI took 2.31 times
+// an idle run, so a shard of about 522 s here is about 221 s idle and 5.5 to
+// 8.5 minutes in CI, before job setup. The property registers each campaign
+// part, each calibration, and each determinism history as its own test
+// (units), so a shard can hold any of them; the slowest unit, the
+// determinism of the edit sequence, is 236 s here. The shards are a
+// longest-first packing of those costs, listed by kind. A random unit's
+// cost follows its seed, so a shard's time varies from run to run. CI keeps
+// its one fixed-seed job until the random units' tail is bounded (option M
+// of the e2e speed plan): a random unit that draws a long edit sequence over
+// the witness module can pass 10 minutes in CI.
 export const shards = {
-  'e2e-1': [`determinism`],
-  'e2e-2': [`random seed`],
-  'e2e-3': [
-    `fixed seed 20261105`,
+  // 522 s in the measured run, about 221 s idle.
+  'e2e-1': [
+    `fixed seed 20261105: a .cjs config that calls require`,
+    `fixed seed 20261105: a declaration fault: missingSource`,
+    `fixed seed 20261105: a declaration fault: noTypescript`,
+    `fixed seed 20261105: a declaration fault: outsideSource`,
+    `fixed seed 20261105: a declaration fault: projects`,
+    `random seed: a config that --config names`,
+    `random seed: a declaration fault: badEntry`,
+    `random seed: a failing owner under a config that --config names`,
+    `random seed: a load failure beside a hang`,
+    `calibration: a key drops its statement text`,
+    `calibration: a mutant is dropped from the enumeration`,
+    `calibration: an edit moves a key`,
+    `determinism: constructed history: a .ts config in a package with no module type that calls require`,
+    `determinism: constructed history: a config that --config names`,
+    `determinism: constructed history: a declaration fault: badEntry`,
+    `determinism: constructed history: a declaration fault: missingSource`,
+    `determinism: constructed history: a declaration fault: projects`,
+    `determinism: constructed history: a package with no config`,
+    `determinism: constructed history: an edit sequence through the daemon`,
+  ],
+  // 522 s in the measured run, about 221 s idle.
+  'e2e-2': [
+    `fixed seed 20261105: a declaration fault: setupThrows`,
+    `fixed seed 20261105: a package with no config`,
+    `fixed seed 20261105: an ambiguous authored mutant`,
+    `fixed seed 20261105: generated`,
+    `random seed: a config fault in a run that owns nothing`,
+    `random seed: a declaration fault: apiLessTypescript`,
+    `calibration: a crash is reported as a kill`,
+    `calibration: a daemon is left after --stop-daemon`,
     `calibration: a refused run is reported as complete`,
+    `determinism: constructed history: a .cjs config that calls require`,
+    `determinism: constructed history: a declaration fault: configNoDefault`,
+    `determinism: constructed history: a declaration fault: outsideSource`,
+    `determinism: constructed history: a declaration fault: setupThrows`,
+    `determinism: constructed history: a load failure beside a hang`,
+    `determinism: constructed history: a stale authored mutant and a reformat`,
+    `determinism: constructed history: an awaiting config in a package with no module type`,
+    `determinism: constructed history: ordinary tests left out by default`,
+    `determinism: generated 0`,
+    `determinism: generated 1`,
   ],
+  // 522 s in the measured run, about 221 s idle.
+  'e2e-3': [
+    `fixed seed 20261105: a config fault in a run that owns nothing`,
+    `fixed seed 20261105: a declaration fault: brokenOwner`,
+    `fixed seed 20261105: a declaration fault: configThrows`,
+    `fixed seed 20261105: a run with eleven contexts`,
+    `fixed seed 20261105: a stale authored mutant and a reformat`,
+    `fixed seed 20261105: an awaiting config in a package with no module type`,
+    `fixed seed 20261105: every outcome class`,
+    `random seed: a .ts config in a package with no module type that calls require`,
+    `random seed: a declaration fault: configThrows`,
+    `random seed: a declaration fault: noTypescript`,
+    `random seed: a declaration fault: outsideSource`,
+    `random seed: a stale authored mutant and a reformat`,
+    `random seed: an edit sequence through the daemon`,
+    `random seed: every outcome class`,
+    `random seed: generated`,
+    `random seed: two authored mutants`,
+    `determinism: constructed history: a declaration fault: brokenOwner`,
+    `determinism: constructed history: a declaration fault: configThrows`,
+    `determinism: constructed history: a declaration fault: nonLiteral`,
+  ],
+  // 522 s in the measured run, about 221 s idle.
   'e2e-4': [
+    `fixed seed 20261105: a .ts config in a package with no module type that calls require`,
+    `fixed seed 20261105: a config that --config names`,
+    `fixed seed 20261105: a declaration fault: apiLessTypescript`,
+    `fixed seed 20261105: a declaration fault: badEntry`,
+    `fixed seed 20261105: a declaration fault: brokenVitest`,
+    `fixed seed 20261105: a declaration fault: configNoDefault`,
+    `fixed seed 20261105: a declaration fault: nonLiteral`,
+    `fixed seed 20261105: a failing owner under a config that --config names`,
+    `fixed seed 20261105: a load failure beside a hang`,
+    `random seed: a .cjs config that calls require`,
+    `random seed: a declaration fault: configNoDefault`,
+    `random seed: an awaiting config in a package with no module type`,
+    `calibration: production code is reported as probe code`,
     `calibration: the daemon reports another outcome than the cold run`,
-    `calibration: a daemon is left after --stop-daemon`,
-    `calibration: a mutant is dropped from the enumeration`,
-    `calibration: an owned mutant is reported as unowned`,
+    `calibration: the summary omits a gap`,
+    `determinism: constructed history: a run with eleven contexts`,
+    `determinism: constructed history: an ambiguous authored mutant`,
+    `determinism: constructed history: every outcome class`,
+    `determinism: constructed history: two authored mutants`,
   ],
+  // 522 s in the measured run, about 221 s idle.
   'e2e-5': [
-    `calibration: a crash is reported as a kill`,
-    `calibration: a key drops its statement text`,
-    `calibration: the summary omits a gap`,
-    `calibration: an edit moves a key`,
-    `calibration: production code is reported as probe code`,
+    `fixed seed 20261105: an edit sequence through the daemon`,
+    `fixed seed 20261105: ordinary tests left out by default`,
+    `fixed seed 20261105: two authored mutants`,
+    `random seed: a declaration fault: brokenOwner`,
+    `random seed: a declaration fault: brokenVitest`,
+    `random seed: a declaration fault: missingSource`,
+    `random seed: a declaration fault: nonLiteral`,
+    `random seed: a declaration fault: projects`,
+    `random seed: a declaration fault: setupThrows`,
+    `random seed: a package with no config`,
+    `random seed: a run with eleven contexts`,
+    `random seed: an ambiguous authored mutant`,
+    `random seed: ordinary tests left out by default`,
+    `calibration: an owned mutant is reported as unowned`,
+    `determinism: constructed history: a config fault in a run that owns nothing`,
+    `determinism: constructed history: a declaration fault: apiLessTypescript`,
+    `determinism: constructed history: a declaration fault: brokenVitest`,
+    `determinism: constructed history: a declaration fault: noTypescript`,
+    `determinism: constructed history: a failing owner under a config that --config names`,
+    `determinism: generated 2`,
   ],
 }
 

packages/oracles/src/oracle-config.ts

@@ -30,7 +30,7 @@ export type { OracleRoute } from './oracle-route.js'
 export interface OracleConfigurationOptions {
   /**
    * The prefix of every oracle environment variable: `${envPrefix}_SEED`,
-   * `${envPrefix}_PATH`, `${envPrefix}_PROPERTY`,
+   * `${envPrefix}_PATH`, `${envPrefix}_PROPERTY`, `${envPrefix}_UNIT`,
    * `${envPrefix}_RUNS_MULTIPLIER`, `${envPrefix}_CALIBRATION`,
    * `${envPrefix}_REPLAY_WITNESS`, and `${envPrefix}_REPLAY_WITNESS_MODE`. A project with its own prefix keeps its
    * variable names by passing it here.
@@ -132,7 +132,7 @@ export function assertRegisteredOracleProperty(property: string): void {
 export function readOracleRunConfig(
   environment?: OracleEnvironment,
 ): Readonly<OracleReplayConfig & { multiplier: number }> {
-  const { multiplier, campaignSeed, replayPath, replayProperty } =
+  const { multiplier, campaignSeed, replayPath, replayProperty, replayUnit } =
     environment === undefined
       ? replayConfig()
       : readRunConfig(environment, oracleSettings())
@@ -141,6 +141,7 @@ export function readOracleRunConfig(
     campaignSeed,
     replayPath,
     replayProperty,
+    replayUnit,
     // A seed the process's replay environment selected is declared fixed,
     // so options built from this configuration name it `fixed seed`.
     ...(environment === undefined && campaignSeed !== undefined