diff --git a/.changeset/calibration-history-bindings.md b/.changeset/calibration-history-bindings.md new file mode 100644 index 0000000..7f1490d --- /dev/null +++ b/.changeset/calibration-history-bindings.md @@ -0,0 +1,5 @@ +--- +'@tanstack/oracles': minor +--- + +Add direct `history` bindings for calibrations and `probeHistories` for automatic controls, so bound calibrations validate one selected premise instead of searching. Preserve fixed and enumerated history identity, including fixed declarations with unrelated `build` metadata, rebuild constructed histories from stable seeds, and resolve once per driver execution. Apply ordinary verdict ranking and optional difference requirements to bound mutants, validate clean declared reach, and retain selected-history error context. Retain search during migration with one warning per property outside collect-only runs, before search removal ahead of the first npm release. diff --git a/README.md b/README.md index 6e44a81..3a8abc6 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,79 @@ preset's, so merge them: } ``` +## Bind calibrations to histories + +A calibration's `history` selects an existing input directly: + +```ts +calibrations: [ + { + name: 'drops the committed row', + law: 'durable publication', + history: { constructed: 'a committed write' }, + corrupt: dropCommittedRow, + }, +] +``` + +Generated sources accept `{ constructed: name }` and `{ fixed: name }`. +Enumerated sources accept `{ enumerated: index }` (zero-based) or +`{ enumerated: name }`, using `label` or the framework's default data label. +Unknown or duplicate names, invalid indices, and incompatible source kinds fail +registration before a driver runs. + +The selected clean run must pass and actually enter every state the history +declares. When the property declares states, every state the clean run enters +must also be declared. The mutation must apply, and the check must reject it +under exactly the named law. A failed clean run, missing reach, inapplicable corruption, +passing mutant, wrong-law rejection, or driver setup error fails the calibration +directly. It never searches another history. + +Mutant drivers use the same verdict rules as ordinary histories. A violation at +a recorded checkpoint decides before a later driver error. A controlled clock +that exceeds its timer bound reports `bounded clock`, unless an earlier recorded +violation already decides. + +An optional `difference: string | RegExp` also requires the violation's +difference to contain the text or match the pattern. The law is checked first. +A bound mismatch fails directly. An unbound mutant driver can still try a later +candidate, as it can for a wrong law. + +A selected construction uses one deterministic seed derived from the property +id and calibration name. Inserting or reordering other calibrations leaves this +seed unchanged. Each driver execution builds from that seed afresh and resolves +its plan once. This preserves builder-created metadata, including WeakMap +associations, and avoids reusing consumed plans. Fixed and enumerated selectors +preserve the existing history's identity and semantics. Use `resolve` to create a +fresh mutable execution plan from those histories for each driver run. Each +calibration remains on the `calibration` route, so a failed self-check is a +framework error in the gap report. Campaign units now support `UNIT` replay. +Bound calibrations still use their stable name-derived seed; fresh-seed +calibrations and their replay support remain follow-up work. + +For automatic forced-probe controls, opt a state into direct selection: + +```ts +probeHistories: { + 'committed write': { + notEntered: { fixed: 'no writes' }, + // Optional: entered: { constructed: 'a committed write' } + }, +} +``` + +The clean-entered history detects the probe forced not entered. If `entered` +is omitted, exactly one construction must declare the state in `enters`. +The clean-not-entered history detects the probe forced entered and always needs +an explicit selection. Neither control may silently disappear. + +During consumer migration, an unbound authored calibration or entirely unbound +probe state retains candidate search. Registration warns once per property per +process, naming its unbound calibrations. Collect-only runs (`ORACLE_MANIFEST`) +stay silent and do not consume that warning. Bind both probe controls together with +`probeHistories`. Search will be removed after consumers migrate, before the +first npm release. + ## The `oracle/no-unit-test` rule An oracle file registers its tests only through `oracleProperty`. Its diff --git a/packages/oracles/src/oracle-property.ts b/packages/oracles/src/oracle-property.ts index ca6b59e..88cf884 100644 --- a/packages/oracles/src/oracle-property.ts +++ b/packages/oracles/src/oracle-property.ts @@ -198,15 +198,26 @@ * - **Calibration.** A `calibration` names a wrong answer and the law that * must reject it: a `corrupt` function that alters a recorded trace, as an * observation mutant does, or a mutant `driver`, such as production set up - * wrongly. The calibration test finds a history where the corruption - * applies and the clean trace passes (it skips a candidate whose clean trace - * fails), and requires the check to report that law for the corrupted one - * (rubric, ORC-006). A calibration can also name a `difference`, text that + * wrongly. A `history` binding selects one constructed, fixed, or enumerated + * history at registration. The clean run must pass and prove its declared + * reach; the wrong answer must apply and fail under exactly the named law. + * A failed premise fails directly, without candidate search (maintainer + * decision, 2026-11-11; rubric, ORC-006). Constructed calibrations use a stable + * seed derived from property id and calibration name. Each driver execution + * builds afresh from that seed and resolves once, preserving builder metadata + * and isolating consumed plans. Fixed/enumerated selectors preserve the + * existing history's identity; use resolve to create a fresh mutable plan per + * run. During migration only, unbound calibrations warn once per property + * per process outside collect-only registration and retain candidate search. + * `probeHistories` opts a state into direct forced controls: its clean-entered history detects forced-not-entered + * (inferred from the unique construction unless selected explicitly), and its + * explicitly selected clean-not-entered history detects forced-entered. + * A calibration can also name a `difference`, text that * the violation's difference must contain or a pattern it must match, so * the check must reject the wrong answer for the reason the calibration * names, not only under its law (maintainer decision, 2026-11-11). The law - * is judged first. A mutant driver's candidate that reports another law or - * another difference is passed over for the next candidate. A pattern + * is judged first. An unbound mutant driver's candidate that reports another + * law or difference is passed over for the next candidate. A pattern * matches anywhere in the difference, and a sticky (`y`) one only at its * start. A required difference that the empty difference holds, the empty * text or a pattern that matches the empty string, fails registration, and @@ -557,6 +568,18 @@ export interface ConstructedHistory { */ export type DeclaredState = string | { name: string; probe: string } +/** Select one existing history; a binding never searches for another. */ +export type CalibrationHistory = + { constructed: string } | { fixed: string } | { enumerated: number | string } + +/** Clean premises for the two automatic forced-probe controls. */ +export interface ProbeHistories { + /** Inferred from the unique construction entering this state when omitted. */ + entered?: CalibrationHistory + /** There is no unique default for a history that does not enter a state. */ + notEntered: CalibrationHistory +} + export interface GeneratedHistories { arbitrary: fc.Arbitrary /** @@ -607,12 +630,14 @@ export type Calibration = { /** The law the check must report. */ law: string /** - * Text that the violation's difference must contain, or a pattern that it - * must match anywhere in it; a sticky (`y`) pattern matches only at its - * start. Without it, any difference under `law` rejects the wrong answer. - * Text or a pattern that the empty difference holds fails registration. + * Text that the violation's difference must contain, or a pattern it must + * match anywhere; a sticky (`y`) pattern matches only at its start. + * Without it, any difference under `law` rejects the wrong answer. + * A requirement that the empty difference holds fails registration. */ difference?: string | RegExp + /** Run this history directly, with a stable name-derived construction seed. */ + history?: CalibrationHistory } & ( | { /** A corrupted copy of the trace, or undefined when nothing applies. */ @@ -657,6 +682,11 @@ export interface OraclePropertyOptions { expected: TExpected, ) => Verdict calibrations?: ReadonlyArray> + /** + * Opt each probe-bound state into direct forced controls. Entirely unbound + * states retain search with warnings during the consumer migration. + */ + probeHistories?: Readonly> /** * Whether the history enters each probe-bound state, from the plan alone: * a prediction never sees the trace, so it cannot follow production. @@ -964,7 +994,7 @@ function enumeratedTests( const tests = [ ...historyTests, ...(declared ? [reachTest] : []), - ...calibrationTests(options, () => named), + ...calibrationTests(options, () => named, named), determinismTest(options, `determinism`, () => named.filter((_, index) => index % spacing === 0), ), @@ -1700,23 +1730,336 @@ type Violation = Extract const lawOf = (verdict: Violation) => verdict.law ?? verdict.checkpoint +/** A seeded build preserves arbitrary context and the builder's identity metadata. */ +function buildConstruction( + id: string, + arbitrary: fc.Arbitrary, + construction: ConstructedHistory, + seed: number, + registering = false, +): THistory { + let history: THistory + try { + const [generated] = fc.sample(arbitrary, { seed, numRuns: 1 }) + history = construction.build(generated!) + } catch (error) { + if (!registering) throw error + throw new Error( + `oracleProperty ${id}: constructed history "${construction.name}" threw while it was built for registration: ${error instanceof Error ? error.message : reportText(error)}`, + { cause: error }, + ) + } + if (!arbitrary.canShrinkWithoutContext(history)) + throw untakeable(id, `constructed history "${construction.name}" built`) + return history +} + +/** Stable across list insertion and reordering; no process-random replay route. */ +function calibrationSeed(id: string, name: string): number { + let seed = 2166136261 + for (const char of `${id}\0${name}`) + seed = Math.imul(seed ^ char.codePointAt(0)!, 16777619) + return seed | 0 +} + +interface SelectedHistory { + name: string + enters: ReadonlyArray + build: () => THistory +} + +/** Resolve every selector at registration, through one diagnostic boundary. */ +function selectCalibrationHistory( + options: BoundOptions, + calibration: { name: string }, + selector: unknown, + enumerated?: ReadonlyArray<{ name: string; history: THistory }>, +): SelectedHistory { + const { id } = options + const invalid = (reason: string): never => { + throw new Error( + `${id}: calibration history selector for "${calibration.name}": ${reason}.`, + ) + } + if ( + !selector || + typeof selector !== `object` || + Object.keys(selector).length !== 1 || + !Object.keys(selector).every((key) => + [`constructed`, `fixed`, `enumerated`].includes(key), + ) + ) + return invalid(`expected exactly one of constructed, fixed, or enumerated`) + const source = options.histories as GeneratedHistories + if (`enumerated` in selector) { + if (!enumerated) return invalid(`enumerated requires an enumerated source`) + const key = selector.enumerated + const matches = + typeof key === `number` + ? Number.isInteger(key) && key >= 0 && key < enumerated.length + ? [enumerated[key]!] + : [] + : typeof key === `string` + ? enumerated.filter(({ name }) => name === key) + : [] + if (matches.length !== 1) + return invalid( + `enumerated ${reportText(key)} must identify exactly one history (found ${matches.length})`, + ) + const selected = matches[0]! + return { + name: `enumerated history: ${selected.name}`, + enters: [], + build: () => selected.history, + } + } + const kind = `constructed` in selector ? `constructed` : `fixed` + if (enumerated) return invalid(`${kind} requires a generated source`) + const key = + kind === `constructed` + ? (selector as { constructed: string }).constructed + : (selector as { fixed: string }).fixed + const select = ( + declarations: ReadonlyArray, + ) => { + const matches = declarations.filter(({ name }) => name === key) + if (typeof key !== `string` || matches.length !== 1) + return invalid( + `${kind} ${reportText(key)} must identify exactly one history (found ${matches.length})`, + ) + return matches[0]! + } + if (kind === `constructed`) { + const selected = select(source.constructed ?? []) + const seed = calibrationSeed(id, calibration.name) + return { + name: `constructed history: ${selected.name} (seed ${seed})`, + enters: selected.enters, + build: () => buildConstruction(id, source.arbitrary, selected, seed), + } + } + const selected = select(source.fixed ?? []) + return { + name: `fixed history: ${selected.name}`, + enters: selected.enters ?? [], + build: () => selected.history, + } +} + +interface ForcedCalibration { + name: string + law: string + state: string + forced: ForcedProbe +} + +/** The same designated rejection applies to bound and legacy calibrations. */ +function calibrationRejection( + id: string, + calibration: { name: string; law: string; difference?: string | RegExp }, +) { + const required = + `difference` in calibration ? calibration.difference : undefined + const owed = `law "${calibration.law}"${required === undefined ? `` : ` with a difference that ${typeof required === `string` ? `contains ${JSON.stringify(required)}` : `matches ${String(required)}`}`}` + // Whether a violation is the one owed: the law, then the difference. + // `search` ignores a pattern's `lastIndex`. + const rejects = (wrong: Extract) => { + if (lawOf(wrong) !== calibration.law || required === undefined) + return lawOf(wrong) === calibration.law + // A check in JavaScript can give any value. + const difference: unknown = wrong.difference + if (typeof difference !== `string`) + throw new Error( + `${id}: calibration "${calibration.name}" requires a difference under law "${calibration.law}", and the check gave a difference that is not text: ${reportText(difference)}. A violation's difference is text.`, + ) + return typeof required === `string` + ? difference.includes(required) + : difference.search(required) !== -1 + } + // What a violation reported, where it differs from what is owed. + const reported = (wrong: Extract) => + lawOf(wrong) === calibration.law + ? `difference ${JSON.stringify(wrong.difference)} under law "${calibration.law}"` + : `law "${lawOf(wrong)}"` + return { owed, rejects, reported } +} + +/** Execute the selected history without falling back to candidate search. */ +async function runBoundCalibration( + options: BoundOptions, + calibration: Calibration | ForcedCalibration, + selected: SelectedHistory, +): Promise { + const { id } = options + const context = `${id}: calibration "${calibration.name}" on ${selected.name}` + const failed = (reason: string, cause?: unknown): never => { + throw new Error( + `${context}: ${reason}${cause === undefined ? `` : `: ${cause instanceof Error ? cause.message : reportText(cause)}`}`, + { cause }, + ) + } + const freshPlan = (phase: string) => { + try { + return planOf(options, selected.build()) + } catch (error) { + return failed( + `selected ${phase} run failed while building or resolving`, + error, + ) + } + } + const plan = freshPlan(`clean`) + let clean: Awaited>> + let actual: ReturnType> + try { + clean = await recordGuarded(options.driver, plan) + } catch (error) { + return failed(`selected clean run failed`, error) + } + if (clean.driverError || clean.spin) + return failed( + `selected clean run failed`, + clean.driverError?.error ?? clean.spin, + ) + try { + actual = judge(options, plan, clean.observations, undefined, clean.probes) + assertDeclared( + id, + Array.isArray(options.histories) + ? undefined + : declaredOf( + options.histories as + GeneratedHistories | EnumeratedHistories, + ), + actual.reached, + ) + } catch (error) { + return failed(`selected clean run failed`, error) + } + if (actual.verdict.kind !== `pass`) + return failed( + `selected clean run failed: reported law "${lawOf(actual.verdict)}"`, + ) + const missing = selected.enters.filter((state) => !actual.reached.has(state)) + if (missing.length) + return failed(`selected history did not enter ${missing.join(`, `)}`) + if ( + `forced` in calibration && + actual.reached.has(calibration.state) === calibration.forced.entered + ) + return failed( + `selected history did not enter the required opposite clean probe premise`, + ) + let wrong: Verdict + if (`corrupt` in calibration) { + let corrupted: ReadonlyArray | undefined + try { + corrupted = calibration.corrupt( + snapshotOf(clean.observations) as ReadonlyArray, + plan, + ) + } catch (error) { + return failed(`selected corruption failed`, error) + } + if (corrupted === undefined) + return failed(`selected corruption did not apply`) + try { + wrong = judge(options, plan, corrupted, undefined, clean.probes).verdict + } catch (error) { + return failed(`selected mutant judgment failed`, error) + } + } else { + // Rebuild once and let runOne resolve once and rank the recorded verdict. + // A later driver error or clock spin must not hide an earlier violation. + try { + wrong = ( + await runOne( + options, + selected.build(), + `forced` in calibration ? options.driver : calibration.driver, + `forced` in calibration ? calibration.forced : undefined, + ) + ).verdict + } catch (error) { + return failed(`selected mutant run failed`, error) + } + } + const { owed, rejects, reported } = calibrationRejection(id, calibration) + try { + if (wrong.kind === `violation` && rejects(wrong)) return + } catch (error) { + return failed(`selected mutant judgment failed`, error) + } + return failed( + `but the check ${wrong.kind === `pass` ? `passed it` : `reported ${reported(wrong)}`}. It must report ${owed}.`, + ) +} + +const warnedCalibrationProperties = new Set() + /** - * One test per calibration. It tries candidate histories in order until the - * wrong answer applies, and requires the check to report the named law. + * A binding is a premise, never a search hint: require the selected clean run + * and actual reach, then exactly the designated law's rejection. No failed + * premise can be hidden by another history. Legacy search remains during the + * pinned consumers' migration and warns once per property outside collection. */ function calibrationTests( options: BoundOptions, candidates: () => ReadonlyArray<{ name: string; history: THistory }>, + enumerated?: ReadonlyArray<{ name: string; history: THistory }>, ): Array { const { id } = options + for (const state of Object.keys(options.probeHistories ?? {})) + if (!options.probes?.has(state)) + throw new Error( + `${id}: calibration history selector for probe state "${state}": the state is not probe-bound.`, + ) // Each probe-bound state also gets two forced-entry calibrations. const forcedCalibrations = [...(options.probes ?? [])].flatMap( - ([state, probe]) => - [true, false].map((entered) => ({ + ([state, probe]) => { + const hasBinding = + options.probeHistories !== undefined && + Object.hasOwn(options.probeHistories, state) + const suppliedBinding: unknown = hasBinding + ? options.probeHistories?.[state] + : undefined + if ( + hasBinding && + (suppliedBinding === null || + typeof suppliedBinding !== `object` || + Array.isArray(suppliedBinding)) + ) + throw new Error( + `${id}: calibration history selector for probe state "${state}": expected a probe history binding object.`, + ) + const binding = suppliedBinding as ProbeHistories | undefined + let enteredHistory = binding?.entered + const notEntered: unknown = binding?.notEntered + if (binding && notEntered === undefined) + throw new Error( + `${id}: calibration history selector for probe state "${state}": supply notEntered explicitly.`, + ) + if (binding && enteredHistory === undefined) { + const constructions = + (options.histories as GeneratedHistories).constructed ?? [] + const matching = constructions.filter(({ enters }) => + enters.includes(state), + ) + if (matching.length !== 1) + throw new Error( + `${id}: calibration history selector for probe state "${state}": expected one construction entering it (found ${matching.length}); add one or select entered explicitly.`, + ) + enteredHistory = { constructed: matching[0]!.name } + } + return [true, false].map((entered) => ({ name: `probe ${probe} forced ${entered ? `entered` : `not entered`} (state "${state}")`, law: agreementLaw, forced: { probe, entered }, - })), + state, + // The clean premise has the opposite polarity to the forced value. + history: entered ? binding?.notEntered : enteredHistory, + })) + }, ) // A required difference that the empty difference holds rejects any // difference under its law, so it would pass silently. @@ -1730,35 +2073,38 @@ function calibrationTests( throw new Error( `${id}: calibration "${name}" requires difference ${typeof difference === `string` ? `""` : String(difference)}, which the empty difference holds, so it rejects any difference. Require text or a pattern that only the owed difference holds.`, ) - return [...(options.calibrations ?? []), ...forcedCalibrations].map( - (calibration) => ({ + const calibrations = [...(options.calibrations ?? []), ...forcedCalibrations] + const unbound = calibrations.filter(({ history }) => history === undefined) + if ( + process.env.ORACLE_MANIFEST === undefined && + unbound.length && + !warnedCalibrationProperties.has(id) + ) { + warnedCalibrationProperties.add(id) + console.warn( + `${id}: calibrations ${unbound.map(({ name }) => `"${name}"`).join(`, `)} have no history binding; using transitional candidate search. Bind their histories before search is removed.`, + ) + } + return calibrations.map((calibration) => { + const selected = + calibration.history === undefined + ? undefined + : selectCalibrationHistory( + options, + calibration, + calibration.history, + enumerated, + ) + return { name: `calibration: ${calibration.name}`, route: `calibration`, replays: [], run: async () => { - const required = - `difference` in calibration ? calibration.difference : undefined - const owed = `law "${calibration.law}"${required === undefined ? `` : ` with a difference that ${typeof required === `string` ? `contains ${JSON.stringify(required)}` : `matches ${String(required)}`}`}` - // Whether a violation is the one owed: the law, then the difference. - // `search` ignores a pattern's `lastIndex`. - const rejects = (wrong: Extract) => { - if (lawOf(wrong) !== calibration.law || required === undefined) - return lawOf(wrong) === calibration.law - // A check in JavaScript can give any value. - const difference: unknown = wrong.difference - if (typeof difference !== `string`) - throw new Error( - `${id}: calibration "${calibration.name}" requires a difference under law "${calibration.law}", and the check gave a difference that is not text: ${reportText(difference)}. A violation's difference is text.`, - ) - return typeof required === `string` - ? difference.includes(required) - : difference.search(required) !== -1 - } - // What a violation reported, where it differs from what is owed. - const reported = (wrong: Extract) => - lawOf(wrong) === calibration.law - ? `difference ${JSON.stringify(wrong.difference)} under law "${calibration.law}"` - : `law "${lawOf(wrong)}"` + if (selected) return runBoundCalibration(options, calibration, selected) + const { owed, rejects, reported } = calibrationRejection( + id, + calibration, + ) const seen: Array = [] for (const { name, history } of candidates()) { const plan = planOf(options, history) @@ -1816,8 +2162,8 @@ function calibrationTests( `${id}: calibration "${calibration.name}" found no history whose clean trace passes and where the check reports ${owed}${seen.length ? `; ${seen.slice(0, 3).join(`, `)}` : ``}. Add a passing fixed or constructed history that reaches the state its wrong answer changes.`, ) }, - }), - ) + } + }) } /** Reports a law violation to the gap report's hook and builds its error. */ @@ -2018,29 +2364,16 @@ function generatedTests( throw error } } - // Constructed history `index`, built from the history its campaign's seed - // draws at seed + index. At registration, an error from sampling or - // building names its constructed history: it would otherwise fail the - // file's load unnamed. + // Build only the named unit's construction. The shared builder preserves + // registration diagnostics and identity metadata for campaigns and calibrations. const constructedAt = (seed: number, index: number, registering = false) => { - const construction = constructions[index]! - let history: THistory - try { - const [generated] = fc.sample(source.arbitrary, { - seed: seed + index, - numRuns: 1, - }) - history = construction.build(generated!) - } catch (error) { - if (!registering) throw error - throw new Error( - `oracleProperty ${id}: constructed history "${construction.name}" threw while it was built for registration: ${error instanceof Error ? error.message : reportText(error)}`, - { cause: error }, - ) - } - if (!source.arbitrary.canShrinkWithoutContext(history)) - throw untakeable(id, `constructed history "${construction.name}" built`) - return history + return buildConstruction( + id, + source.arbitrary, + constructions[index]!, + seed + index, + registering, + ) } // Registration checks every fixed history, then the constructed histories // built from the first fixed seed (or seed 0 when there is none), so a @@ -2196,6 +2529,7 @@ function generatedTests( replays: [], run: async (task) => { const { verdict, reached } = await runOne(options, fixed.history) + assertDeclared(id, declared, reached) if (verdict.kind === `violation`) throw failure(options, verdict, fixed.history, rerunOf(task)) const missing = (fixed.enters ?? []).filter( diff --git a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts index 7a34ebd..78d4b18 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts +++ b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts @@ -17,7 +17,9 @@ * A constructed history builds its history as its prefix followed by a generated history, * cut to three items. The driver appends every history it runs, with the running test's name, to * the file named by SELF_ORACLE_LOG, so the self-oracle knows which - * histories each campaign ran. A `nondeterministic` spec also records, last, a + * histories each campaign ran. The logical owning name comes from the runner's + * public aroundEach boundary, retained across late continuations; a slow raw log + * separately retains the global runner name for timeout diagnostics. A `nondeterministic` spec also records, last, a * counter that changes on every run. A spec reports reach through the * property's `reached` part or, as the framework's default, through the * driver's `observe("reach", state)`. A `slow` spec's driver takes about @@ -123,17 +125,48 @@ * `/^item \d is 4$/` as a pattern, `item 2 is 3` as text or `/^item 2 /` as a * pattern (which only a three-item candidate's `driver-last` difference * holds), or the empty text or the empty pattern. Only containment matches - * the text `is 3`. + * the text `is 3`. A `throws` driver records that wrong observation before + * throwing, so ordinary verdict ranking still decides its calibration. A + * `clock` driver leaves its item observations unchanged and spins a controlled + * clock; only the mutant execution spins. A `driver-spin` mutant first records + * its wrong item and then spins, and `driver-spin-throws` also throws after the + * spin. Those ordered events distinguish an earlier violation from a later + * clock failure without making the selected clean history fail. + * `corrupt-throws`, `judge-throws`, + * and `judge-driver-throws` exercise exceptions from corruption and from the + * two mutant judgment routes without supplying any expected result. + * + * Histories and execution plans carry an input-side origin association (fixed + * name, construction name, or enumerated index), reported with each driver + * input. A separate build log records the builder's generated input and output, + * so the model can judge selection and the whole construction without sharing + * production's selector or seed hash. `extraReach` reports synthetic-extra + * outside every declaration, either globally or only in authored calibrations. + * The phase driver registers the same property under distinct suites to + * exercise repeated registration and collect-only reporting in one process. + * Each authored calibration can select its own history, or explicitly remain + * unbound beside bound peers. A fixed declaration can carry unrelated own or + * inherited `build` metadata. That method is never the declaration's source + * kind; if called, its copied result and build log expose the wrong dispatch. * * The synthetic ids come from a fixed pool, which this fixture registers with * `configureOracles`. The framework's own id, `self-oracle.framework`, is * registered beside them, so the self-oracle's registration law can reject an * id outside the registry. */ +import { AsyncLocalStorage } from 'node:async_hooks' import { appendFileSync } from 'node:fs' import { runInNewContext } from 'node:vm' import fc from 'fast-check' -import { afterAll, beforeEach, describe, expect, vi } from 'vitest' +import { + afterAll, + aroundEach, + beforeAll, + beforeEach, + describe, + expect, + vi, +} from 'vitest' import { configureOracles, differs, @@ -151,17 +184,17 @@ export const syntheticIds = [ `self-oracle.synthetic-3`, ] as const -// A child run that draws its seed draws it through Math.random, which is a -// fixed-seed generator here (mulberry32), so it draws the same seeds in -// every run of a file and the self-oracle's own determinism test can compare -// the traces. Each draw is still the framework's own: the replay environment -// declares no seed. -let randomState = 20251111 -Math.random = () => { - randomState = (randomState + 0x6d2b79f5) | 0 - let mixed = Math.imul(randomState ^ (randomState >>> 15), 1 | randomState) - mixed = (mixed + Math.imul(mixed ^ (mixed >>> 7), 61 | mixed)) ^ mixed - return ((mixed ^ (mixed >>> 14)) >>> 0) / 4294967296 +// Only a real receiving child controls its host RNG. Importing this fixture +// in the outer owner preserves that process's Math.random and fresh campaign seed. +// Marked children use mulberry32 so their ordinary-route traces repeat. +if (process.env.SELF_ORACLE_LOG !== undefined) { + let randomState = 20251111 + Math.random = () => { + randomState = (randomState + 0x6d2b79f5) | 0 + let mixed = Math.imul(randomState ^ (randomState >>> 15), 1 | randomState) + mixed = (mixed + Math.imul(mixed ^ (mixed >>> 7), 61 | mixed)) ^ mixed + return ((mixed ^ (mixed >>> 14)) >>> 0) / 4294967296 + } } // Every fixture and self-test property must be registered; `self-oracle.unregistered` @@ -235,6 +268,8 @@ export type SyntheticSpec = { name: string history: SyntheticHistory enters?: ReadonlyArray + /** Unrelated declaration metadata, not a construction. */ + buildMetadata?: `own` | `inherited` }> /** * A `.map` the arbitrary applies, with or without an unmapper, to the whole @@ -253,10 +288,48 @@ export type SyntheticSpec = { nondeterministic: boolean calibrations: ReadonlyArray<{ name: string - kind: `right` | `wrong-law` | `no-op` | `driver` | `driver-last` + kind: + | `right` + | `wrong-law` + | `no-op` + | `driver` + | `driver-last` + | `driver-spin` + | `driver-spin-throws` + | `inapplicable` + | `throws` + | `clock` + | `corrupt-throws` + | `judge-throws` + | `judge-driver-throws` /** The difference the calibration requires, as text or a pattern, matching or not; absent, none. */ difference?: CalibrationDifference + /** Absent inherits the property default; null explicitly stays unbound. */ + history?: SyntheticSpec[`calibrationHistory`] | null }> + /** The proposed binding, passed structurally so the unchanged framework can be judged. */ + calibrationHistory?: + | { constructed: string } + | { fixed: string } + | { enumerated: number | string } + /** Preserve generated variation while testing builder identity and consumed plans. */ + isolatedPlans?: boolean + identityHistories?: boolean + /** Public alpha-renaming of logical saw-0; the production probe id stays fixed. */ + probeName?: string + probeBinding?: + | `empty` + | `inferred` + | `explicit` + | `no-absence` + | `null-entered` + | `null-binding` + | `false-binding` + | `zero-binding` + /** Independent public selectors for the two clean opposite premises. */ + probeEntered?: { fixed: string } | { constructed: string } + probeNotEntered?: { fixed: string } | { constructed: string } + duplicateHistoryNames?: boolean /** `observe`: the driver reports reach itself, the framework's default. */ reachVia: `reached` | `observe` /** Each history takes about 300 ms; with `timeout`, tests outlive it. */ @@ -272,6 +345,11 @@ export type SyntheticSpec = { probe?: `agree` | `always` | `never` | `unreached` | `late` /** Reports `saw-0` as reached in every history. */ overReport?: boolean + /** Reports a state outside the declaration, without changing the item language. */ + extraReach?: boolean | `calibration` + /** Registers the same property repeatedly in named enclosing suites. */ + warningProbe?: boolean + calibrationNames?: `indexed` | `varied` | `astral` /** Calls an unbound probe on every run, entering it on every other one, so only determinism sees it. */ noisyProbe?: boolean /** `missing` omits `predict` for a bound probe; `extra` passes it unbound. */ @@ -374,6 +452,11 @@ class SyntheticFault extends Error { } /** A class instance that holds a history's items and an Error. */ class SyntheticBox { + #valid = true + readItems(): SyntheticHistory { + if (!this.#valid) throw new Error(`invalid synthetic box`) + return this.items + } constructor( readonly items: SyntheticHistory, readonly failure: Error, @@ -395,10 +478,29 @@ const itemOf = (failure: unknown): number => { return code ?? 0 } let noise = 0 +// The public aroundEach boundary supplies an immutable owning test context. +// Continuations retain it after their timeout, although Vitest's global +// currentTestName may then name the next test. This is input provenance only: +// the fixture does not change cancellation, deadlines, or execution order. +const owningTest = new AsyncLocalStorage<{ name: string; route?: string }>() +aroundEach((runTest, { task }) => { + const names = [task.name] + for ( + let parent = task.suite; + parent && parent !== task.file; + parent = parent.suite + ) + names.unshift(parent.name) + return owningTest.run({ name: names.join(` > `) }, runTest) +}) +const owningTestName = () => + owningTest.getStore()?.name ?? expect.getState().currentTestName // The route registered for the running test, as the gap report reads it. let route: string | undefined globalThis.__oracleRoute = (registered) => { - route = registered + const owner = owningTest.getStore() + if (owner) owner.route = registered + else route = registered } beforeEach(() => { route = undefined @@ -477,7 +579,7 @@ const itemsOf = return (history as ReadonlyArray).map(itemOf) } if (spec.form === `boxed`) { - if (history instanceof SyntheticBox) return history.items + if (history instanceof SyntheticBox) return history.readItems() throw new Error(`not a boxed history`) } if (spec.form === `trap`) { @@ -498,10 +600,18 @@ const itemsOf = const enteredStates = (history: SyntheticHistory) => allStates.filter((state) => history.includes(Number(state.slice(4)))) -const reachedStates = (spec: SyntheticSpec) => (history: SyntheticHistory) => - spec.overReport +const reachedStates = (spec: SyntheticSpec) => (history: SyntheticHistory) => [ + ...(spec.overReport ? [...new Set([...enteredStates(history), `saw-0`])] - : enteredStates(history) + : enteredStates(history)), + ...(spec.extraReach === true || + (spec.extraReach === `calibration` && + spec.calibrations.some(({ name }) => + owningTestName()?.endsWith(` > calibration: ${name}`), + )) + ? [`synthetic-extra`] + : []), +] /** * The condition the synthetic driver passes to the probe for this history, or @@ -689,17 +799,30 @@ function lateProbe() { * The synthetic driver. `mutant` records a 3 in place of the first item, the * wrong answer a driver calibration supplies. */ -function syntheticDriver(spec: SyntheticSpec, mutant?: `first` | `last`) { +function syntheticDriver( + spec: SyntheticSpec, + mutant?: `first` | `last` | `clock` | `first-spin` | `first-spin-throws`, + consumed = new WeakSet(), + origins = new WeakMap(), +) { return async ( formed: Formed, observe: (checkpoint: string, value: unknown) => void, ) => { const history = itemsOf(spec)(formed) + if ( + (spec.isolatedPlans || spec.identityHistories) && + owningTestName()?.includes(`calibration:`) + ) { + if (consumed.has(formed as object)) + throw new Error(`driver received a consumed plan`) + consumed.add(formed as object) + } const log = process.env.SELF_ORACLE_LOG if (log) appendFileSync( log, - `${JSON.stringify({ id: spec.id, test: expect.getState().currentTestName ?? ``, history, route: route ?? null })}\n`, + `${JSON.stringify({ id: spec.id, test: owningTestName() ?? ``, ...(spec.slow ? { runnerTest: expect.getState().currentTestName ?? `` } : {}), history, origin: origins.get(formed as object), route: owningTest.getStore()?.route ?? route ?? null })}\n`, ) if (spec.odd) observe(`odd`, { reason: oddValue(spec.odd) }) if (spec.slow) await new Promise((resolve) => setTimeout(resolve, 300)) @@ -707,7 +830,7 @@ function syntheticDriver(spec: SyntheticSpec, mutant?: `first` | `last`) { spec.slowShrink && history.includes(4) && / > (?:fixed seed -?\d+|random seed)(?:: .*)?$/.test( - expect.getState().currentTestName ?? ``, + owningTestName() ?? ``, ) ) await new Promise((resolve) => setTimeout(resolve, 2000)) @@ -726,12 +849,19 @@ function syntheticDriver(spec: SyntheticSpec, mutant?: `first` | `last`) { for (const state of reachedStates(spec)(history)) observe(`reach`, state) // A real timer first: a clock that an earlier history released must have // given the real timers back, or this history never goes on. - if (spec.clock) await new Promise((resolve) => setTimeout(resolve, 1)) - const clock = spec.clock ? controlledClock() : undefined + const mutantSpins = + mutant === `clock` || + mutant === `first-spin` || + mutant === `first-spin-throws` + if (spec.clock || mutantSpins) + await new Promise((resolve) => setTimeout(resolve, 1)) + const clock = spec.clock || mutantSpins ? controlledClock() : undefined try { for (const [index, item] of history.entries()) { const mutated = - mutant === `first` + mutant === `first` || + mutant === `first-spin` || + mutant === `first-spin-throws` ? index === 0 : mutant === `last` && index === history.length - 1 observe(`item ${index}`, mutated ? 3 : item) @@ -757,6 +887,15 @@ function syntheticDriver(spec: SyntheticSpec, mutant?: `first` | `last`) { ) } } + if (mutantSpins) { + const spin = () => { + setTimeout(spin, 0) + } + spin() + await vi.advanceTimersByTimeAsync(1000) + if (mutant === `first-spin-throws`) + throw new Error(`mutant setup failed after the spin`) + } } finally { clock?.release() // Production work the driver does not await, calling its probe after @@ -805,6 +944,9 @@ function syntheticCheck( history: SyntheticHistory, trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, ) { + // The marker is a mutant input to the check, never an expected verdict. + if (trace.some(({ checkpoint }) => checkpoint === `synthetic-judgment-fault`)) + throw new Error(`synthetic judgment failed`) // `history` is the items of a history in any form. for (const { checkpoint, value } of trace) { // Only items carry the law; a nonce is not an item. @@ -852,38 +994,107 @@ const differences: Record = { 'empty pattern': /(?:)/, } +/** The authored selector passed to production, including explicit migration opt-out. */ +const calibrationHistory = ( + spec: SyntheticSpec, + calibration: SyntheticSpec[`calibrations`][number], +) => + calibration.history === undefined + ? spec.calibrationHistory + : (calibration.history ?? undefined) + function syntheticCalibrations( spec: SyntheticSpec, + consumed: WeakSet, + origins: WeakMap, ): Array> { - return spec.calibrations.map( - ({ name, kind, difference }): Calibration => { - const required = - difference === undefined ? {} : { difference: differences[difference] } - // A mutant driver records a 3 in place of the first or the last item. - if (kind === `driver` || kind === `driver-last`) - return { - name, - law: `three`, - ...required, - driver: syntheticDriver(spec, kind === `driver` ? `first` : `last`), - } + return spec.calibrations.map((calibration): Calibration => { + const { name, kind, difference } = calibration + const history = calibrationHistory(spec, calibration) + const required = + difference === undefined ? {} : { difference: differences[difference] } + if (kind === `clock`) return { name, - law: kind === `wrong-law` ? `four` : `three`, + history, + law: `bounded clock`, ...required, - corrupt: (trace) => { - const at = trace.findIndex(({ checkpoint }) => - checkpoint.startsWith(`item`), - ) - if (at < 0) return undefined - if (kind === `no-op`) return [...trace] - return trace.map((entry, index) => - index === at ? { ...entry, value: 3 } : entry, - ) + driver: syntheticDriver(spec, `clock`, consumed, origins), + } + if (kind === `driver-spin` || kind === `driver-spin-throws`) + return { + name, + history, + law: `three`, + ...required, + driver: syntheticDriver( + spec, + kind === `driver-spin` ? `first-spin` : `first-spin-throws`, + consumed, + origins, + ), + } + if (kind === `judge-driver-throws`) + return { + name, + history, + law: `three`, + ...required, + driver: async (plan, observe) => { + await syntheticDriver(spec, `first`, consumed, origins)(plan, observe) + observe(`synthetic-judgment-fault`, true) }, } - }, - ) + // A mutant driver records a 3 in place of the first or the last item. + if (kind === `driver` || kind === `driver-last` || kind === `throws`) + return { + name, + history, + law: `three`, + ...required, + driver: + kind === `throws` + ? async (plan, observe) => { + await syntheticDriver( + spec, + `first`, + consumed, + origins, + )(plan, observe) + throw new Error(`mutant setup failed`) + } + : syntheticDriver( + spec, + kind === `driver-last` ? `last` : `first`, + consumed, + origins, + ), + } + return { + name, + history, + law: kind === `wrong-law` ? `four` : `three`, + ...required, + corrupt: (trace) => { + if (kind === `corrupt-throws`) + throw new Error(`synthetic corruption failed`) + if (kind === `judge-throws`) + return [ + ...trace, + { checkpoint: `synthetic-judgment-fault`, value: true }, + ] + if (kind === `inapplicable`) return undefined + const at = trace.findIndex(({ checkpoint }) => + checkpoint.startsWith(`item`), + ) + if (at < 0) return undefined + if (kind === `no-op`) return [...trace] + return trace.map((entry, index) => + index === at ? { ...entry, value: 3 } : entry, + ) + }, + } + }) } // `owns`, the literal list the gap report reads, in the shape the spec names. @@ -932,10 +1143,125 @@ function declaredEnumerated( } export function syntheticProperty(spec: SyntheticSpec): void { + // An ordinary child still selects its seed without ORACLE_SEED. Control + // the host draw so observing its inputs/status does not make the outer + // determinism check compare two different random environments. + if ( + process.env.SELF_ORACLE_LOG !== undefined && + process.env.SELF_ORACLE_RANDOM_DRAW !== undefined + ) { + const random = Math.random + const draw = Number(process.env.SELF_ORACLE_RANDOM_DRAW) + beforeAll(() => { + Math.random = () => draw + }) + afterAll(() => { + Math.random = random + }) + } const id = spec.unregistered ? `self-oracle.unregistered` : spec.id - const form = formOf(spec) + const makeForm = formOf(spec) const items = itemsOf(spec) + const identities = new WeakMap() + const origins = new WeakMap() + const markOrigin = (history: Formed, origin: string): Formed => { + origins.set(history as object, origin) + return history + } + const remember = (history: Formed): Formed => { + if (!spec.identityHistories) return history + identities.set(history as object, history) + return history + } + const form = (history: SyntheticHistory): Formed => + remember(makeForm(history)) + // Only the public state name changes: the synthetic model still reasons + // about item zero and the production probe id remains synthetic#saw-0. + const publicState = (state: string) => + spec.source.kind === `generated` && state === `saw-0` + ? (spec.probeName ?? state) + : state + // A builder may attach metadata by object identity. A plan may own consumed + // resources. Neither cloning a built history nor resolving it twice preserves + // that contract. These sets record those premises, not expected observations. + const built = new WeakSet() + const resolved = new WeakSet() + const resolve = + spec.isolatedPlans || spec.identityHistories + ? (history: Formed) => { + // Count invocations separately from driver calls: even a discarded + // resolved plan can have acquired resources. This records input-side + // work only; the independent checker derives the allowed budget. + const log = process.env.SELF_ORACLE_LOG + if (log) + appendFileSync( + `${log}.resolutions`, + `${JSON.stringify({ + id: spec.id, + test: owningTestName() ?? ``, + })}\n`, + ) + if (spec.identityHistories && !identities.has(history as object)) + throw new Error(`selected history lost identity metadata`) + if ( + spec.isolatedPlans && + (spec.calibrations.some((calibration) => { + const binding = calibrationHistory(spec, calibration) + return ( + binding !== undefined && + `constructed` in binding && + owningTestName()?.endsWith( + ` > calibration: ${calibration.name}`, + ) + ) + }) || + (spec.probeBinding && + spec.probeBinding !== `empty` && + ((owningTestName()?.includes(`forced not entered`) && + (spec.probeBinding !== `explicit` || + `constructed` in + (spec.probeEntered ?? { + constructed: `constructed-0`, + }))) || + (owningTestName()?.includes(`forced entered`) && + `constructed` in + (spec.probeNotEntered ?? { fixed: `fixed-0` }))))) + ) { + if ( + !built.has(history as object) || + resolved.has(history as object) + ) + throw new Error( + `calibration lost builder metadata or reused a built history`, + ) + resolved.add(history as object) + } + const plan = makeForm([ + ...(spec.identityHistories + ? items(identities.get(history as object)) + : items(history)), + ]) + const origin = origins.get(history as object) + return origin === undefined ? plan : markOrigin(plan, origin) + } + : undefined + // Clean and mutant driver instances share the consumption boundary. A + // resolved plan belongs to one execution even when different drivers use it. + const consumed = new WeakSet() + const runDriver = spec.slow + ? trackedSlow(syntheticDriver(spec, undefined, consumed, origins)) + : syntheticDriver(spec, undefined, consumed, origins) + const driver: typeof runDriver = (history, observe) => + runDriver(history, (checkpoint, value) => + observe( + checkpoint, + checkpoint === `reach` && typeof value === `string` + ? publicState(value) + : value, + ), + ) const parts = { + ...(resolve ? { resolve } : {}), model: spec.strictModel ? (history: Formed, trace: ReadonlyArray<{ checkpoint: string }>) => { const recorded = trace.filter(({ checkpoint }) => @@ -964,20 +1290,53 @@ export function syntheticProperty(spec: SyntheticSpec): void { : () => undefined, // Without `reached`, the framework counts the driver's `reach` records. ...(spec.reachVia === `reached` - ? { reached: (history: Formed) => reachedStates(spec)(items(history)) } + ? { + reached: (history: Formed) => + reachedStates(spec)(items(history)).map(publicState), + } : {}), ...((spec.probe && spec.predictMode !== `missing`) || spec.predictMode === `extra` ? { predict: (history: Formed) => ({ - 'saw-0': items(history).includes(0), + [publicState(`saw-0`)]: items(history).includes(0), }), } : {}), - driver: spec.slow - ? trackedSlow(syntheticDriver(spec)) - : syntheticDriver(spec), - calibrations: syntheticCalibrations(spec), + ...(spec.probeBinding + ? { + probeHistories: + spec.probeBinding === `empty` + ? {} + : { + [publicState(`saw-0`)]: + spec.probeBinding === `null-binding` + ? (null as never) + : spec.probeBinding === `false-binding` + ? (false as never) + : spec.probeBinding === `zero-binding` + ? (0 as never) + : spec.probeBinding === `no-absence` + ? ({} as never) + : { + notEntered: spec.probeNotEntered ?? { + fixed: `fixed-0`, + }, + ...(spec.probeBinding === `explicit` + ? { + entered: spec.probeEntered ?? { + constructed: `constructed-0`, + }, + } + : spec.probeBinding === `null-entered` + ? { entered: null as never } + : {}), + }, + }, + } + : {}), + driver, + calibrations: syntheticCalibrations(spec, consumed, origins), check: ( history: Formed, trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, @@ -1012,10 +1371,12 @@ export function syntheticProperty(spec: SyntheticSpec): void { // The last history's late probe call comes after its test; the file waits // for it, so the run still reports it. if (spec.probe === `late`) - afterAll(() => new Promise((resolve) => setTimeout(resolve, 50))) + afterAll(() => new Promise((settle) => setTimeout(settle, 50))) const register = () => { if (spec.source.kind === `enumerated`) { - const enumerated = spec.source.histories.map(form) + const enumerated = spec.source.histories.map((history, index) => + markOrigin(form(history), `enumerated:${index}`), + ) const first = JSON.stringify(spec.source.histories[0]) oracleProperty({ id, @@ -1028,10 +1389,20 @@ export function syntheticProperty(spec: SyntheticSpec): void { ? {} : { label: (history: Formed) => - spec.labelStyle?.startsWith(`reserved-`) && - JSON.stringify(items(history)) === first - ? spec.labelStyle.slice(`reserved-`.length) - : `items ${items(history).join(` | `)}${spec.labelStyle === `shell` ? ` ${shellText}` : ``}`, + spec.duplicateHistoryNames && + spec.calibrations.some((calibration) => { + const selected = calibrationHistory(spec, calibration) + return ( + selected !== undefined && + `enumerated` in selected && + typeof selected.enumerated === `string` + ) + }) + ? `items ${spec.source.kind === `enumerated` ? spec.source.histories[0]!.join(` | `) : ``}` + : spec.labelStyle?.startsWith(`reserved-`) && + JSON.stringify(items(history)) === first + ? spec.labelStyle.slice(`reserved-`.length) + : `items ${items(history).join(` | `)}${spec.labelStyle === `shell` ? ` ${shellText}` : ``}`, }), ...parts, }) @@ -1040,9 +1411,12 @@ export function syntheticProperty(spec: SyntheticSpec): void { // With a probe, `saw-0` is a probe-bound state wherever it is declared. const declare = (state: string) => spec.probe && state === `saw-0` - ? { name: state, probe: syntheticProbe } - : state - const arbitrary = arbitraryOf(spec) + ? { name: publicState(state), probe: syntheticProbe } + : publicState(state) + const baseArbitrary = arbitraryOf(spec) + const arbitrary = spec.identityHistories + ? baseArbitrary.map(remember, (value) => value) + : baseArbitrary oracleProperty({ id, law, @@ -1059,19 +1433,64 @@ export function syntheticProperty(spec: SyntheticSpec): void { ], constructed: spec.constructed.map(({ name, prefix, enters }, at) => ({ name, - enters, + enters: enters.map(publicState), // A constructed history extends a generated history, so its histories // depend on the campaign's seed, as real constructed histories do. build: (generated: Formed) => { if (at === 0 && spec.throwingBuild) throw new Error(`synthetic build failed`) - return form([...prefix, ...items(generated)].slice(0, 3)) + const tail = spec.isolatedPlans + ? items(generated).map((item) => item % 3) + : items(generated) + const history = markOrigin( + form([...prefix, ...tail].slice(0, 3)), + `constructed:${name}`, + ) + const log = process.env.SELF_ORACLE_LOG + if (log) + appendFileSync( + `${log}.builds`, + `${JSON.stringify({ + id: spec.id, + test: owningTestName() ?? ``, + name, + generated: items(generated), + history: items(history), + })}\n`, + ) + built.add(history as object) + return history }, })), - fixed: spec.fixed.map((fixed) => ({ - ...fixed, - history: form(fixed.history), - })), + fixed: spec.fixed.map((fixed) => { + const declaration = { + ...fixed, + ...(fixed.enters ? { enters: fixed.enters.map(publicState) } : {}), + history: markOrigin(form(fixed.history), `fixed:${fixed.name}`), + } + // Fixed declarations are structurally open metadata wrappers. An + // incidental build method must never run or replace their original. + const build = (generated: Formed) => { + const history = makeForm([...fixed.history]) + const log = process.env.SELF_ORACLE_LOG + if (log) + appendFileSync( + `${log}.builds`, + `${JSON.stringify({ + id: spec.id, + test: owningTestName() ?? ``, + name: `fixed-metadata:${fixed.name}`, + generated: items(generated), + history: items(history), + })}\n`, + ) + return history + } + if (fixed.buildMetadata === `own`) return { ...declaration, build } + if (fixed.buildMetadata === `inherited`) + Object.setPrototypeOf(declaration, { build }) + return declaration + }), }, ...parts, }) diff --git a/packages/oracles/tests/oracle-property-self-oracle-reporter.mjs b/packages/oracles/tests/oracle-property-self-oracle-reporter.mjs index 38d2fe0..560aa63 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-reporter.mjs +++ b/packages/oracles/tests/oracle-property-self-oracle-reporter.mjs @@ -20,9 +20,13 @@ const serialize = (error) => { export default class SelfOracleReporter { /** @type {Map} console output by task id */ console = new Map() + /** @type {string[]} registration-time warnings */ + warnings = [] /** @param {{ taskId?: string, content: unknown }} log */ onUserConsoleLog(log) { + if (String(log.content).includes(`transitional candidate search`)) + this.warnings.push(String(log.content)) if (log.taskId === undefined) return this.console.set( log.taskId, @@ -56,6 +60,7 @@ export default class SelfOracleReporter { process.env.SELF_ORACLE_REPORT ?? `self-oracle-report.json`, JSON.stringify({ files, + warnings: this.warnings, unhandled: (unhandledErrors ?? []).map(serialize), }), ) diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 6eb8f26..d266401 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -90,25 +90,64 @@ * history does (maintainer decision, 2026-11-10, F23: every enumerated history runs, so a * required state needs no constructed history). Run alone, under a `-t` pattern that selects * only it, the reach test judges every history itself and owes the same outcome and counts. - * - `calibration`: a calibration uses the first candidate whose clean trace - * passes, and passes only when its wrong answer (a corrupted trace or a - * mutant driver) fails with the named law and, when the calibration names a - * `difference`, with a violation whose difference contains that text or - * matches that pattern. The law is judged first: a wrong law fails the - * calibration as a wrong law whatever its difference. A mutant driver's - * candidate that reports another difference is passed over as one that - * reports another law is, so a later candidate whose difference matches - * passes it (maintainer decision, 2026-11-11, F27). A required difference - * that the empty difference holds, the empty text or a pattern that - * matches the empty string, fails the property's load, naming the - * calibration. A check that gives a difference that is not text, where a - * calibration requires one, fails that calibration naming it and the law. - * - `determinism`: a generated source's determinism units run every fixed and - * constructed history and three generated ones, each unit its one history - * twice (an enumerated source's one test runs its spaced histories), and fail - * for a driver whose trace changes, + * - `calibration`: a bound calibration runs only its selected history. Its clean + * run must pass and enter the history's declared states, then its wrong answer + * must fail under exactly the named law. An observation mutant uses one driver + * call; a driver mutant uses two fresh builds and plans from the same seed. + * Invalid selectors fail registration before any driver runs. These are the + * maintainer-approved history-binding contract (2026-11-11). During migration, + * unbound calibrations retain candidate search. Rejection ranks recorded law + * violations and bounded-clock violations identically on bound/unbound paths. + * Required difference text is contained; a pattern matches the difference, + * after the designated law. Empty requirements fail registration. + * Named construction inputs are stable under reorder and vary across the + * finite ordinary/astral name families declared here; no universal collision + * freedom is claimed. Forced premises are judged from declared input origins + * and author-built callback inputs, never the framework's selected driver log. + * Executed bound callback errors retain the calibration name, the selected + * authored history's kind and name, and the original cause in every active + * receiving child. Public original-history name/origin associations identify + * opaque default enumeration labels; callback text cannot authorize selection. + * Seed annotations, punctuation, stacks and name hashing are not laws. The + * receiving grammar varies quoted/bracketed calibration names, optional + * parenthesized/bracketed seed suffixes, and independently positive whitespace at structural + * separators, including incoming raw context variants before presentation. The + * generated separator sample includes one/two/three spaces, tabs and mixed gaps. + * Opaque callback reason prose varies independently, including empty text and + * words used by other diagnostics; actual exception cause fixes its kind. + * The outer diagnostic wrapper name varies independently and does not change + * callback kind; original cause name/object identity is not judged. + * A whole context frame must decode to one available authored identity; + * this finite renderer language is not arbitrary human-language parsing. + * Both forced + * premises independently select fixed or constructed originals; only omitted + * entered selectors infer a unique construction. Equal data never permits + * substituting another named original. Registration warns once per + * property/process without dropping tests, and collect-only stays silent. + * Every receiving child preserves that warning and each unbound name once; + * neither property order nor name order changes that promise. + * Every real child must finish with a complete report and no unmodeled + * unhandled errors, including replay and registration-only children. + * Each authored calibration selects independently, including bound/unbound + * mixtures. Extra own or inherited metadata on a fixed declaration cannot + * change its selected source kind. A mutant may record a wrong value, then + * spin and optionally throw: that earlier violation still decides. + * - `determinism`: every fixed, constructed, and sampled generated history + * has its own determinism unit, which runs it twice and fails if its trace changes, * compared by their data encoding: `-0` is not `0`, and key order and * Map and Set order count, which a check that iterates them can see. + * - `campaign budget`: each passing constructed unit runs one example, and + * each passing generated unit runs its authored sample count. A wholly + * passing lane retains their sum: runs plus constructions, independently + * in ordinary/seedless and fixed/random lanes. Children discard parent replay + * settings under both prefixes, so their multiplier is one. Path replay runs + * one history; failures, shrinking, and timeouts owe no fixed call count. + * - `host RNG scope`: importing the synthetic fixture in the outer owner + * preserves the supplied host draw, which actual seed selection uses. + * Marked receiving children separately repeat under the same host input. + * Forwarding wrappers can consume that draw more than once. This law + * promises no function identity, call count, seed arithmetic or entropy distribution. + * The forwarding state requires the actual wrapper to run during seed selection. * - `agreement`: for a probe-bound state, a history whose `predict` says it * enters the state while the history does not enter it, or does not enter * it while the history does, violates law `reach agreement` at checkpoint @@ -216,14 +255,27 @@ * property. The model learns an unlabelled enumerated history's test name * from the log, so it checks that names are distinct, not their text. A slow * spec's late probe calls and routes are not counted exactly: its driver runs - * on after its test times out. That shrinking keeps a model error apart + * on after its test times out. Every child projects away slow invocation + * runner attribution and multiplicity, while retaining logical owning tests, + * each owner's first input, raw log/report diagnostics and per-started-call + * build/resolve/input relations. Determinism resolves one plan for both drives. + * Slow+late processes observe the deficit/excess against their raw started-call + * bounds; fast-only processes retain exact late multiplicity. The check rejects + * either nonzero relation and every unrelated error. Status, registration, + * process completeness and first build/resolve inputs remain compared independently. + * That shrinking keeps a model error apart * from a setup error has no witness: no shrink in this language turns one * into the other. The gap report's observation hook, * `forwardCut`, and `differs` have no law here; `withinTurns` has its own * oracle. A counterexample of the `errors` form prints its Errors without * the items they hold, so the shrink and replay checks cannot read it back * and skip it. Only generated sources bind a probe, and only - * `saw-0`; an enumerated source declares its states by name. A late probe is judged by whether the run reports it, not by which + * logical `saw-0`, alpha-renamed to saw-0, toString, constructor, or __proto__ + * at the fixture's public state boundary. The model uses logical integer reach + * and translates only printed names; the production probe id is unchanged. + * Resolution may require history identity, and generated boxed histories have + * private slots; each resolve returns a fresh plan. A separate invocation log + * checks exactly one resolution per driver execution, including discarded plans. An enumerated source declares its states by name. A late probe is judged by whether the run reports it, not by which * test Vitest blames. A slow-shrink spec is slow only in its campaigns and only on histories * that hold a 4, so one slow history at most comes before a campaign's first failure. Whether a * campaign's shrinking was cut is not observed. Under a timeout, how far a campaign shrinks is @@ -244,7 +296,9 @@ * four patterns, including the empty ones, judged against the difference a wrong answer * reports, `item k is 3`, where k is the first item or a `driver-last` candidate's last item. A * pattern with the global or sticky flag is not drawn. A difference that is not text is drawn - * only as a number. A calibration's failure message is judged by its kind, not by its text. + * only as a number. Calibration failure kinds are compared semantically. Callback identity + * and original cause are checked separately; exact diagnostic punctuation and seed decoration + * are not promised beyond the bounded presentation grammar exercised below. */ import { spawnSync } from 'node:child_process' import { @@ -255,6 +309,7 @@ import { writeFileSync, } from 'node:fs' import { createRequire } from 'node:module' +import { tmpdir } from 'node:os' import { dirname, join } from 'node:path' import { fileURLToPath } from 'node:url' import fc from 'fast-check' @@ -266,7 +321,6 @@ import { syntheticArbitrary, syntheticIds, } from './oracle-property-self-oracle-fixture.js' -import type { Verdict } from '../src/index.js' import type { CalibrationDifference, OddKind, @@ -292,6 +346,14 @@ const calibrationKinds = [ `no-op`, `driver`, `driver-last`, + `driver-spin`, + `driver-spin-throws`, + `clock`, + `corrupt-throws`, + `judge-throws`, + `judge-driver-throws`, + `inapplicable`, + `throws`, ] as const const calibrationDifferences = [ `text match`, @@ -301,10 +363,11 @@ const calibrationDifferences = [ `later text`, `later pattern`, ] as const -/** A calibration's kind and the difference it requires; `none` requires none. */ type CalibrationDraft = { kind: (typeof calibrationKinds)[number] difference: CalibrationDifference | `none` + /** Absent inherits the property default; null explicitly retains search. */ + history?: SyntheticSpec[`calibrationHistory`] | null } type Draft = { source: @@ -313,7 +376,12 @@ type Draft = { /** `exact` declares the constructed histories' states; the others break registration. */ statesMode: `exact` | `extra` | `missing` constructed: Array<{ prefix: SyntheticHistory; enters: Array }> - fixed: Array<{ history: SyntheticHistory; enters: Array | null }> + fixed: Array<{ + history: SyntheticHistory + enters: Array | null + /** Unrelated declaration metadata, never a history construction. */ + buildMetadata?: `own` | `inherited` + }> /** * How the constructed histories are named: by index; with names that * extend each other (`c`, `c.1`), which a `-t` pattern without its `$` @@ -349,9 +417,30 @@ type Draft = { timing: `normal` | `slow` | `slow shrink` /** A normal-timing spec's `timeout`: the default, 0, or -1 (both no timeout). */ timeoutMode: `default` | `zero` | `negative` - calibrations: Array - /** The check reports a number as its difference. */ + calibrations: Array numberDifference: boolean + extraReach: boolean | `calibration` + calibrationNames: `indexed` | `varied` | `astral` + warningProbe: boolean + calibrationHistory: NonNullable | null + isolatedPlans: boolean + /** Resolution requires the original history identity and returns a fresh plan. */ + identityHistories: boolean + /** Alpha-renames logical saw-0 at the fixture's public boundary. */ + probeName: `saw-0` | `toString` | `constructor` | `__proto__` + probeBinding: + | `unbound` + | `empty` + | `inferred` + | `explicit` + | `no-absence` + | `null-entered` + | `null-binding` + | `false-binding` + | `zero-binding` + probeEntered: NonNullable + probeNotEntered: NonNullable + duplicateHistoryNames: boolean /** An id no oracle configuration registers. */ unregistered: boolean /** How the driver calls the probe bound to `saw-0`; `none` binds no probe. */ @@ -386,15 +475,45 @@ type Draft = { strictModel: boolean } // `seedless`: one more child run, without `ORACLE_SEED`, records the routes -// of each random unit as an ordinary run draws its seed, and the seed it logged. -// `prefix`: every spec of the suite runs under this environment prefix. -// `environment`: one more child run under a drawn replay environment (law -// `replay-config`), whose path is `0` and whose property is the first spec's. +// of each random campaign as an ordinary run draws its seed. +type CallbackGap = 1 | 2 | 3 | `\t` | ` \t ` +type CallbackPresentation = { + calibration: `quoted` | `bracketed` + seed: `parenthesized` | `bracketed` | `omitted` + spacing: + | `plain` + | `padded` + | [ + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + ] +} type Suite = { + /** Equivalent bounded context renderings; alternate varies between two per drive. */ + callbackPresentation?: CallbackPresentation | `alternate` + callbackIncomingPresentation?: CallbackPresentation + callbackReason?: string + callbackWrapperName?: string drafts: Array seedless: boolean - prefix: Prefix - environment: ReplayEnvironment | null + /** A controlled host draw, independent of the framework's seed selection. */ + seedlessDraw?: number + /** Exercise equivalent warning-name presentations across repeated drives. */ + permuteWarnings?: boolean + /** Alternate legal post-timeout call attribution and order across drives. */ + timeoutPresentation?: boolean + /** A controlled outer host draw, checked independently from receiving children. */ + hostRngDraw?: 0.25 | 0.75 + /** Equivalent outer-host wrappers; alternate one/two forwards across drives. */ + hostRngForwarding?: 1 | 2 | `alternate` + /** Omission preserves the default prefix of existing fixed histories. */ + prefix?: Prefix + /** A separately received replay environment; omission means ordinary execution. */ + environment?: ReplayEnvironment | null } type Prefix = `ORACLE` | `SYNTH_ORACLE` type ReplayEnvironment = { @@ -422,7 +541,77 @@ const insideOrOutside = (insideWeight: number) => { arbitrary: syntheticArbitrary, weight: insideWeight }, { arbitrary: outside, weight: 1 }, ) +const probeSelector = fc.oneof( + fc.record({ fixed: fc.constantFrom(`fixed-0`, `fixed-1`, `missing`) }), + fc.record({ + constructed: fc.constantFrom(`constructed-0`, `constructed-1`, `missing`), + }), +) +// Selection belongs to each calibration. The property default remains useful +// for migrations; an explicit null chooses search beside selected neighbors. +const calibrationSelector = fc.oneof( + fc.record({ + fixed: fc.constantFrom(`fixed-0`, `fixed-1`, `unknown`, `missing`), + }), + fc.record({ + constructed: fc.constantFrom( + `constructed-0`, + `constructed-1`, + `c`, + `c.1`, + `unknown`, + `missing`, + ), + }), + fc.record({ + enumerated: fc.oneof( + fc.integer({ min: -1, max: 4 }), + fc.constant(1.5), + fc.constantFrom(`items 0 | 1`, `items 0 | 1 ${shellText}`, `[0,1]`), + ), + }), +) const draft: fc.Arbitrary = fc.record({ + probeEntered: probeSelector, + probeNotEntered: probeSelector, + calibrationHistory: fc.option( + fc.oneof( + fc.record({ + fixed: fc.constantFrom(`fixed-0`, `fixed-1`, `unknown`, `missing`), + }), + fc.record({ + constructed: fc.constantFrom(`constructed-0`, `unknown`, `missing`), + }), + fc.record({ + enumerated: fc.oneof( + fc.integer({ min: -1, max: 4 }), + fc.constant(1.5), + fc.constantFrom(`items 0 | 1`, `items 0 | 1 ${shellText}`, `[0,1]`), + ), + }), + ), + { freq: 20 }, + ), + isolatedPlans: fc.boolean(), + identityHistories: fc.boolean(), + probeName: fc.constantFrom(`saw-0`, `toString`, `constructor`, `__proto__`), + probeBinding: fc.oneof( + { weight: 20, arbitrary: fc.constant(`unbound` as const) }, + { + weight: 1, + arbitrary: fc.constantFrom( + `empty` as const, + `inferred` as const, + `explicit` as const, + `no-absence` as const, + `null-entered` as const, + `null-binding` as const, + `false-binding` as const, + `zero-binding` as const, + ), + }, + ), + duplicateHistoryNames: fc.boolean(), source: fc.oneof( fc.record({ kind: fc.constant(`generated` as const), @@ -450,10 +639,17 @@ const draft: fc.Arbitrary = fc.record({ { maxLength: 2 }, ), fixed: fc.array( - fc.record({ - history: insideOrOutside(4), - enters: fc.option(fc.subarray(allStates), { freq: 2 }), - }), + fc.record( + { + history: insideOrOutside(4), + enters: fc.option(fc.subarray(allStates), { freq: 2 }), + buildMetadata: fc.option( + fc.constantFrom(`own` as const, `inherited` as const), + { nil: undefined, freq: 8 }, + ), + }, + { requiredKeys: [`history`, `enters`] }, + ), { maxLength: 2 }, ), // Names that extend each other are common; names that break registration @@ -533,23 +729,46 @@ const draft: fc.Arbitrary = fc.record({ ), // A calibration may require its violation's difference, matching or not. calibrations: fc.array( - fc.record({ - kind: fc.constantFrom(...calibrationKinds), - difference: fc.oneof( - // Rare: an empty difference fails the whole property's load. - { weight: 9, arbitrary: fc.constant(`none` as const) }, - { weight: 9, arbitrary: fc.constantFrom(...calibrationDifferences) }, + fc.oneof( + fc.constantFrom(...calibrationKinds), + fc.record( { - weight: 1, - arbitrary: fc.constantFrom( - `empty text` as const, - `empty pattern` as const, + kind: fc.constantFrom(...calibrationKinds), + history: fc.option(fc.option(calibrationSelector, { nil: null }), { + nil: undefined, + freq: 4, + }), + difference: fc.oneof( + // Rare: an empty difference fails the whole property's load. + { weight: 9, arbitrary: fc.constant(`none` as const) }, + { + weight: 9, + arbitrary: fc.constantFrom(...calibrationDifferences), + }, + { + weight: 1, + arbitrary: fc.constantFrom( + `empty text` as const, + `empty pattern` as const, + ), + }, ), }, + { requiredKeys: [`kind`, `difference`] }, ), - }), + ), { maxLength: 3 }, ), + extraReach: fc.oneof( + { weight: 30, arbitrary: fc.constant(false) }, + { weight: 1, arbitrary: fc.constant(true) }, + { weight: 1, arbitrary: fc.constant(`calibration` as const) }, + ), + calibrationNames: fc.constantFrom(`indexed`, `varied`, `astral`), + warningProbe: fc.oneof( + { weight: 30, arbitrary: fc.constant(false) }, + { weight: 1, arbitrary: fc.constant(true) }, + ), numberDifference: fc.oneof( { weight: 5, arbitrary: fc.constant(false) }, { weight: 1, arbitrary: fc.constant(true) }, @@ -737,22 +956,92 @@ const replayEnvironment: fc.Arbitrary = fc.record({ `missing` as const, ), }) -const suite: fc.Arbitrary = fc.record({ - drafts: fc.array(draft, { minLength: 1, maxLength: 2 }), - seedless: fc.oneof( - { weight: 5, arbitrary: fc.constant(false) }, - { weight: 1, arbitrary: fc.constant(true) }, - ), - prefix: fc.oneof( - { weight: 4, arbitrary: fc.constant(`ORACLE` as const) }, - { weight: 1, arbitrary: fc.constant(`SYNTH_ORACLE` as const) }, +const cleanHostDraw = 0.5000000016298145 +const callbackPresentationArbitrary = fc.record({ + calibration: fc.constantFrom(`quoted` as const, `bracketed` as const), + seed: fc.constantFrom( + `parenthesized` as const, + `bracketed` as const, + `omitted` as const, ), - // Rare: one more child run under a replay environment. - environment: fc.oneof( - { weight: 4, arbitrary: fc.constant(null) }, - { weight: 1, arbitrary: replayEnvironment }, + spacing: fc.oneof( + fc.constantFrom(`plain` as const, `padded` as const), + fc.tuple( + ...Array.from({ length: 6 }, () => + fc.constantFrom( + 1 as const, + 2 as const, + 3 as const, + `\t` as const, + ` \t ` as const, + ), + ), + ) as fc.Arbitrary< + [ + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + ] + >, ), }) +const genericSuite: fc.Arbitrary = fc.record( + { + drafts: fc.array(draft, { minLength: 1, maxLength: 2 }), + callbackWrapperName: fc.option(fc.string({ maxLength: 64 }), { + nil: undefined, + freq: 12, + }), + callbackReason: fc.option(fc.string({ maxLength: 192 }), { + nil: undefined, + freq: 12, + }), + callbackIncomingPresentation: fc.option(callbackPresentationArbitrary, { + nil: undefined, + freq: 12, + }), + callbackPresentation: fc.option( + fc.oneof( + fc.constant(`alternate` as const), + callbackPresentationArbitrary, + ), + { nil: undefined, freq: 12 }, + ), + seedless: fc.oneof( + { weight: 5, arbitrary: fc.constant(false) }, + { weight: 1, arbitrary: fc.constant(true) }, + ), + seedlessDraw: fc.option(fc.constantFrom(0.5, cleanHostDraw), { + nil: undefined, + freq: 8, + }), + hostRngDraw: fc.option(fc.constantFrom(0.25 as const, 0.75 as const), { + nil: undefined, + freq: 24, + }), + hostRngForwarding: fc.option( + fc.constantFrom(1 as const, 2 as const, `alternate` as const), + { + nil: undefined, + freq: 24, + }, + ), + prefix: fc.oneof( + { weight: 3, arbitrary: fc.constant(`ORACLE` as const) }, + { weight: 1, arbitrary: fc.constant(`SYNTH_ORACLE` as const) }, + ), + environment: fc.oneof( + { weight: 10, arbitrary: fc.constant(null) }, + { weight: 1, arbitrary: replayEnvironment }, + ), + permuteWarnings: fc.option(fc.boolean(), { nil: undefined, freq: 12 }), + timeoutPresentation: fc.option(fc.boolean(), { nil: undefined, freq: 12 }), + }, + { requiredKeys: [`drafts`, `seedless`] }, +) // --------------------------------------------------------------------------- // Resolution: ids, names, and declared states @@ -765,8 +1054,18 @@ const suite: fc.Arbitrary = fc.record({ // constructed or fixed histories. type Plan = { + callbackPresentation?: CallbackPresentation | `alternate` + callbackIncomingPresentation?: CallbackPresentation + callbackReason?: string + callbackWrapperName?: string specs: Array seedless: boolean + seedlessDraw?: number + permuteWarnings?: boolean + /** Alternate legal post-timeout call attribution and order across drives. */ + timeoutPresentation?: boolean + hostRngDraw?: 0.25 | 0.75 + hostRngForwarding?: 1 | 2 | `alternate` prefix: Prefix environment: ReplayEnvironment | null } @@ -783,16 +1082,53 @@ const constructedName = (names: Draft[`constructedNames`], at: number) => ? `constructed` : `constructed-${at}` -function resolveSuite({ drafts, seedless, prefix, environment }: Suite): Plan { +function resolveSuite({ + drafts, + callbackPresentation, + callbackIncomingPresentation, + callbackReason, + callbackWrapperName, + seedless, + seedlessDraw, + permuteWarnings, + timeoutPresentation, + hostRngDraw, + hostRngForwarding, + prefix = `ORACLE`, + environment = null, +}: Suite): Plan { return { + ...(callbackPresentation ? { callbackPresentation } : {}), + ...(callbackIncomingPresentation ? { callbackIncomingPresentation } : {}), + ...(callbackReason === undefined ? {} : { callbackReason }), + ...(callbackWrapperName === undefined ? {} : { callbackWrapperName }), seedless, + ...(seedlessDraw === undefined ? {} : { seedlessDraw }), + ...(permuteWarnings ? { permuteWarnings: true } : {}), + ...(timeoutPresentation ? { timeoutPresentation: true } : {}), + ...(hostRngDraw === undefined ? {} : { hostRngDraw }), + ...(hostRngForwarding === undefined ? {} : { hostRngForwarding }), prefix, environment, specs: drafts.map((d, index) => { const generated = d.source.kind === `generated` + const hasBinding = + (d.probe !== `none` && + d.probeBinding !== `unbound` && + d.probeBinding !== `empty`) || + d.calibrations.some((calibration) => + typeof calibration === `string` + ? d.calibrationHistory !== null + : calibration.history === undefined + ? d.calibrationHistory !== null + : calibration.history !== null, + ) const constructed = generated ? d.constructed.map((construction, at) => ({ - name: constructedName(d.constructedNames, at), + name: + d.duplicateHistoryNames && hasBinding + ? `constructed-0` + : constructedName(d.constructedNames, at), ...construction, })) : [] @@ -809,14 +1145,22 @@ function resolveSuite({ drafts, seedless, prefix, environment }: Suite): Plan { return { id: syntheticIds[index]!, source: d.source, + ...(generated && d.secondSeed ? { secondSeed: true } : {}), states, constructed, - ...(generated && d.secondSeed ? { secondSeed: true } : {}), fixed: generated ? d.fixed.map((fixedHistory, at) => ({ - name: d.fixedNames === `duplicate` ? `fixed` : `fixed-${at}`, + name: + d.duplicateHistoryNames && hasBinding + ? `fixed-0` + : d.fixedNames === `duplicate` + ? `fixed` + : `fixed-${at}`, history: fixedHistory.history, ...(fixedHistory.enters ? { enters: fixedHistory.enters } : {}), + ...(fixedHistory.buildMetadata + ? { buildMetadata: fixedHistory.buildMetadata } + : {}), })) : [], ...(generated && d.arbitraryMap !== `none` @@ -916,11 +1260,47 @@ function resolveSuite({ drafts, seedless, prefix, environment }: Suite): Plan { : {}), ...(d.nested ? { nested: true } : {}), ...(d.strictModel ? { strictModel: true } : {}), - calibrations: d.calibrations.map(({ kind, difference }, at) => ({ - name: `cal-${at}`, - kind, - ...(difference === `none` ? {} : { difference }), - })), + ...(d.extraReach && + (generated || + (d.enumeratedStates !== null && d.extraReach === `calibration`)) + ? { extraReach: d.extraReach } + : {}), + ...(d.warningProbe ? { warningProbe: true } : {}), + calibrationNames: d.calibrationNames, + ...(d.calibrationHistory + ? { calibrationHistory: d.calibrationHistory } + : {}), + ...(d.isolatedPlans ? { isolatedPlans: true } : {}), + ...(d.identityHistories ? { identityHistories: true } : {}), + probeName: d.probeName, + ...(d.probeBinding !== `unbound` + ? { probeBinding: d.probeBinding } + : {}), + probeEntered: d.probeEntered, + probeNotEntered: d.probeNotEntered, + ...(d.duplicateHistoryNames ? { duplicateHistoryNames: true } : {}), + calibrations: (d.calibrationNames === `indexed` + ? d.calibrations + : Array.from({ length: 16 }, () => d.calibrations[0]).filter( + (item) => item !== undefined, + ) + ).map((entry, at) => { + const { kind, difference, history } = + typeof entry === `string` + ? { kind: entry, difference: `none` as const, history: undefined } + : entry + return { + name: + d.calibrationNames === `astral` + ? `drops ${String.fromCodePoint(0x1f600 + at)}` + : d.calibrationNames === `varied` + ? `named calibration ${at}` + : `cal-${at}`, + kind, + ...(difference === `none` ? {} : { difference }), + ...(history === undefined ? {} : { history }), + } + }), } }), } @@ -952,6 +1332,9 @@ type Kind = | `calibration-wrong-difference` | `calibration-odd-difference` | `calibration-none` + | `calibration-clean` + | `calibration-reach` + | `calibration-inapplicable` | `timeout` | `model-not-total` | `data` @@ -961,6 +1344,8 @@ type Outcome = { law?: `three` | `four` | `reach agreement` | `bounded clock` /** A violation whose driver threw afterwards carries that error. */ cause?: `driver` + /** The exact finite driver error a violation carries. */ + causeMessage?: string /** The checkpoint a framework-made violation names. */ checkpoint?: string /** For a predicted entry that did not happen: whether the probe was called. */ @@ -977,6 +1362,9 @@ type LogEntry = { history: SyntheticHistory /** The route registered for the test when the history ran. */ route: string | null + origin?: string + /** Raw diagnostic attribution; logical test ownership is `test`. */ + runnerTest?: string } const allowed = (item: number) => [0, 1, 2, 3, 4, 6, 7].includes(item) @@ -1025,7 +1413,21 @@ function verdictOf(history: SyntheticHistory, spec: SyntheticSpec): Outcome { (history[stopAt] === 6 || spec.throwOnSpin === true || history.slice(stopAt + 1).includes(6)) - const carried = driverThrows ? { cause: `driver` as const } : {} + const driverThrowAt = history.findIndex( + (item) => item === 6 || (clock && spec.throwOnSpin === true && item === 7), + ) + const carried = driverThrows + ? { + cause: `driver` as const, + causeMessage: + history[driverThrowAt] === 7 + ? `synthetic driver failed after the spin at item ${driverThrowAt}` + : `synthetic driver failed at item ${driverThrowAt}`, + } + : {} + // A model-not-total error wraps the judgment error, while retaining the + // driver error separately. Its inner cause does not use violation shape. + const modelCarried = driverThrows ? { cause: `driver` as const } : {} const judgedItems = stopAt >= 0 ? history.slice(0, stopAt + 1) : history const modelThrows = spec.strictModel ? stopAt >= 0 && stopAt < history.length - 1 @@ -1044,7 +1446,7 @@ function verdictOf(history: SyntheticHistory, spec: SyntheticSpec): Outcome { stopAt >= 0 && (stopAt < history.length - 1 || trailing || driverThrows) ) - return { kind: `model-not-total`, ...carried } + return { kind: `model-not-total`, ...modelCarried } return { kind: `setup` } } // A check that compares data past the encoding limit fails as that limit, @@ -1056,6 +1458,7 @@ function verdictOf(history: SyntheticHistory, spec: SyntheticSpec): Outcome { return { kind: `violation`, law: item === 3 ? `three` : `four`, + checkpoint: `item ${at}`, ...(history.slice(at + 1).some(stops) ? carried : {}), } if (item === 6) return { kind: `setup` } @@ -1156,6 +1559,12 @@ function judged( forced?: boolean, ): Outcome { const verdict = verdictOf(history, spec) + if ( + spec.extraReach === true && + (verdict.kind === `pass` || verdict.kind === `violation`) && + (spec.source.kind === `generated` || spec.enumeratedStates) + ) + return { kind: `undeclared-state` } if (verdict.kind !== `pass` || !spec.probe) return verdict const predicts = predictsZero(history) if (enteredFor(spec, history, forced) === predicts) return verdict @@ -1189,7 +1598,7 @@ const forcedCalibrationNames = (spec: SyntheticSpec) => spec.probe ? [true, false].map( (entered) => - `calibration: probe ${syntheticProbe} ${entered ? `forced entered` : `forced not entered`} (state "saw-0")`, + `calibration: probe ${syntheticProbe} ${entered ? `forced entered` : `forced not entered`} (state "${spec.probeName ?? `saw-0`}")`, ) : [] // An enumerated history's test name. An unlabelled one is named by the @@ -1229,20 +1638,30 @@ const calibrationCandidates = ( return [ ...spec.fixed.map(({ history }) => history), ...spec.constructed.map(({ prefix }, at) => - constructedAt(prefix, seed + at), + [ + ...prefix, + ...fc + .sample(syntheticArbitrary, { seed: seed + at, numRuns: 1 })[0]! + .map((item) => (spec.isolatedPlans ? item % 3 : item)), + ].slice(0, 3), ), ...fc.sample(syntheticArbitrary, { seed, numRuns: 100 }), ] } - // Model law `units`. A synthetic constructed history is its prefix followed by // a generated history, cut to three items; the generated history is the // arbitrary's sample at the seed it is built from. -const constructedAt = (prefix: SyntheticHistory, seed: number) => - [...prefix, ...fc.sample(syntheticArbitrary, { seed, numRuns: 1 })[0]!].slice( - 0, - 3, - ) +const constructedAt = ( + spec: SyntheticSpec, + prefix: SyntheticHistory, + seed: number, +) => + [ + ...prefix, + ...fc + .sample(syntheticArbitrary, { seed, numRuns: 1 })[0]! + .map((item) => (spec.isolatedPlans ? item % 3 : item)), + ].slice(0, 3) // The seed every seeded child run gives the random units (`${prefix}_SEED`). const childSeed = 20251103 const fixedSeedsOf = (spec: SyntheticSpec): Array => @@ -1284,7 +1703,8 @@ function unitStream( const at = spec.constructed.findIndex( (construction) => construction.name === name, ) - if (at >= 0) return [constructedAt(spec.constructed[at]!.prefix, seed + at)] + if (at >= 0) + return [constructedAt(spec, spec.constructed[at]!.prefix, seed + at)] const weighted = spec.constructed.length === 0 ? syntheticArbitrary @@ -1292,7 +1712,10 @@ function unitStream( { arbitrary: syntheticArbitrary, weight: spec.constructed.length }, ...spec.constructed.map(({ prefix }) => ({ arbitrary: syntheticArbitrary.map((items) => - [...prefix, ...items].slice(0, 3), + [ + ...prefix, + ...items.map((item) => (spec.isolatedPlans ? item % 3 : item)), + ].slice(0, 3), ), weight: 1, })), @@ -1314,7 +1737,7 @@ function determinismUnits(spec: SyntheticSpec): Map { ({ name, prefix }, at) => [ `determinism: constructed history: ${name}`, - constructedAt(prefix, seed + at), + constructedAt(spec, prefix, seed + at), ] as const, ), ...fc @@ -1345,6 +1768,211 @@ const mutatedDifference = ( ) => `item ${kind === `driver-last` ? history.length - 1 : 0} is 3` const emptyDifference = (difference: CalibrationDifference | undefined) => difference === `empty text` || difference === `empty pattern` + +const driverCalibration = ( + kind: SyntheticSpec[`calibrations`][number][`kind`], +) => + [ + `driver`, + `driver-last`, + `driver-spin`, + `driver-spin-throws`, + `throws`, + `clock`, + `judge-driver-throws`, + ].includes(kind) +// Binding is authored input, not an observation of the framework's selection. +// Explicit unbound entries remain distinct from inheritance of the default. +const bindingOf = ( + spec: SyntheticSpec, + calibration: SyntheticSpec[`calibrations`][number], +) => + calibration.history === undefined + ? spec.calibrationHistory + : (calibration.history ?? undefined) +const bindingsOf = (spec: SyntheticSpec) => + spec.calibrations + .map((calibration) => bindingOf(spec, calibration)) + .filter((binding) => binding !== undefined) +const unboundCalibrationNames = (spec: SyntheticSpec) => [ + ...spec.calibrations + .filter((calibration) => !bindingOf(spec, calibration)) + .map(({ name }) => name), + ...(!spec.probeBinding || spec.probeBinding === `empty` + ? forcedCalibrationNames(spec).map((name) => + name.slice(`calibration: `.length), + ) + : []), +] +const registeredSpec = (spec: SyntheticSpec, context: ChildContext) => + context.kind === `binding-reordered` + ? { + ...spec, + calibrations: [ + { name: `unrelated calibration`, kind: `right` as const }, + ...[...spec.calibrations].reverse(), + ], + } + : spec +const propertyWarnings = (warnings: ReadonlyArray, id: string) => + warnings.filter((warning) => warning.startsWith(`${id}:`)) +const unattributedWarnings = ( + warnings: ReadonlyArray, + ids: ReadonlyArray, +) => + warnings.filter((warning) => !ids.some((id) => warning.startsWith(`${id}:`))) +// Decode the actual rendered list, without consulting expected names. The +// finite name grammar includes inner quotes in forced-control state labels, +// but no name contains the list delimiter `", "`. Malformed lists stay raw. +const warningParts = (warning: string) => { + const match = + /^(.*?: calibrations )"(.*)"( have no history binding;.*)$/s.exec(warning) + return match + ? { prefix: match[1]!, names: match[2]!.split(`", "`), suffix: match[3]! } + : undefined +} +const renderWarning = ( + parts: NonNullable>, + names: ReadonlyArray, +) => + `${parts.prefix}${names.map((name) => `"${name}"`).join(`, `)}${parts.suffix}` +const semanticWarning = (warning: string) => { + const parts = warningParts(warning) + return parts ? renderWarning(parts, [...parts.names].sort()) : warning +} +const reversedWarning = (warning: string) => { + const parts = warningParts(warning) + return parts ? renderWarning(parts, [...parts.names].reverse()) : warning +} +const warningMismatch = ( + spec: SyntheticSpec, + warnings: ReadonlyArray, + collectOnly: boolean, +) => { + const unbound = collectOnly ? [] : unboundCalibrationNames(spec) + const own = propertyWarnings(warnings, spec.id) + if (own.length !== (unbound.length ? 1 : 0)) return true + if (own.length === 0) return false + const actual = warningParts(own[0]!)?.names + return ( + actual === undefined || + JSON.stringify([...actual].sort()) !== JSON.stringify([...unbound].sort()) + ) +} +const boundTest = (spec: SyntheticSpec, name: string) => { + const calibration = spec.calibrations.find( + (entry) => name === `calibration: ${entry.name}`, + ) + return calibration + ? bindingOf(spec, calibration) !== undefined + : Boolean( + spec.probeBinding && + spec.probeBinding !== `empty` && + forcedCalibrationNames(spec).includes(name), + ) +} +type BuildEntry = { + id: string + test: string + name: string + generated: SyntheticHistory + history: SyntheticHistory +} +// Named enumeration selectors refer to the label the author supplied, not +// merely the underlying items. The selector string domain below includes +// authored items labels (plain/shell) and one plain-JSON default label. +// Other forms' encoded default labels are outside that string domain. +const authoredEnumerationLabel = ( + spec: SyntheticSpec, + history: SyntheticHistory, +): string | undefined => { + if (spec.labelStyle === `default`) + return spec.form === undefined ? JSON.stringify(history) : undefined + if ( + spec.duplicateHistoryNames && + bindingsOf(spec).some( + (binding) => + `enumerated` in binding && typeof binding.enumerated === `string`, + ) && + spec.source.kind === `enumerated` + ) + return `items ${spec.source.histories[0]!.join(` | `)}` + if ( + spec.labelStyle?.startsWith(`reserved-`) && + spec.source.kind === `enumerated` && + JSON.stringify(history) === JSON.stringify(spec.source.histories[0]) + ) + return spec.labelStyle.slice(`reserved-`.length) + return `items ${history.join(` | `)}${spec.labelStyle === `shell` ? ` ${shellText}` : ``}` +} +const authoredInput = ( + spec: SyntheticSpec, + test: string, + binding: NonNullable, + builds: ReadonlyArray, +): SyntheticHistory | undefined => { + if (`fixed` in binding) + return spec.fixed.find(({ name }) => name === binding.fixed)?.history + if (`enumerated` in binding && spec.source.kind === `enumerated`) + return spec.source.histories.find((history, index) => + typeof binding.enumerated === `number` + ? index === binding.enumerated + : authoredEnumerationLabel(spec, history) === binding.enumerated, + ) + if (`constructed` in binding) { + const construction = spec.constructed.find( + ({ name }) => name === binding.constructed, + ) + const input = builds.find( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test}` && + entry.name === binding.constructed, + ) + if (construction && input) + return [ + ...construction.prefix, + ...input.generated.map((item) => + spec.isolatedPlans ? item % 3 : item, + ), + ].slice(0, 3) + } + return undefined +} +// Resolve from authored declarations, never from the driver-selected input. +// A unique state declaration is insufficient if another construction shares +// its name: every selected name must identify exactly one original. +const selectedControlBinding = (spec: SyntheticSpec, forced: boolean) => + forced + ? (spec.probeNotEntered ?? { fixed: `fixed-0` }) + : spec.probeBinding === `explicit` + ? (spec.probeEntered ?? { constructed: `constructed-0` }) + : { + constructed: + spec.constructed.find(({ enters }) => enters.includes(`saw-0`)) + ?.name ?? `missing`, + } +const selectedControls = (spec: SyntheticSpec, forced: boolean) => { + const binding = selectedControlBinding(spec, forced) + return `fixed` in binding + ? spec.fixed.filter(({ name }) => name === binding.fixed) + : spec.constructed.filter(({ name }) => name === binding.constructed) +} +const authoredOrigin = ( + spec: SyntheticSpec, + binding: NonNullable, +): string | undefined => { + if (`fixed` in binding) return `fixed:${binding.fixed}` + if (`constructed` in binding) return `constructed:${binding.constructed}` + if (spec.source.kind !== `enumerated`) return undefined + const index = spec.source.histories.findIndex((history, at) => + typeof binding.enumerated === `number` + ? at === binding.enumerated + : authoredEnumerationLabel(spec, history) === binding.enumerated, + ) + return index < 0 ? undefined : `enumerated:${index}` +} + const isCampaign = (name: string) => unitOf(name) !== undefined // Model law for an enumerated source's declared states: every history runs, @@ -1382,15 +2010,13 @@ const enumeratedReach = ( judgedHistory: (history: SyntheticHistory) => boolean, ): { outcome: Outcome; counts: Record } => { const counts = Object.fromEntries( - [...declaredOf(spec)!] - .sort() - .map((state) => [ - state, - histories.filter( - (history) => - judgedHistory(history) && reachedOf(spec, history).includes(state), - ).length, - ]), + [...declaredOf(spec)!].map((state) => [ + state, + histories.filter( + (history) => + judgedHistory(history) && reachedOf(spec, history).includes(state), + ).length, + ]), ) const missed = spec.enumeratedStates!.states.some( (state) => counts[state] === 0, @@ -1466,7 +2092,7 @@ type SpecExpectation = /** A label that names a framework test. */ reservedLabel?: boolean dataError?: boolean - /** Two tests would share a name. */ + /** A generated declaration would register two tests under one name. */ nameError?: boolean /** Which history failed law `membership`. */ membership?: `fixed` | `constructed` @@ -1476,21 +2102,29 @@ type SpecExpectation = | { loads: true tests: Map - /** The history each determinism unit must run, twice. */ + /** Histories a passing determinism test must run, each twice. */ + determinismRuns?: number + /** The input of each generated determinism unit, repeated twice. */ determinism: Map - /** The histories each campaign unit draws in this run (law `units`). */ + /** Unit streams derived independently from authored histories and a seed. */ streams: Map> - /** For each passing unit, the histories entering each state. */ + /** The first failing fixed-seed unit whose exact rerun is owed. */ + unitRerun?: string + /** For each passing campaign, the histories entering each state. */ reach: Map> - /** Units owed a replay probe, and the tests owed a rerun probe. */ + /** Campaigns owed a replay probe, and the test owed a rerun probe. */ replays: Array rerun?: string - unitRerun?: string } function predictSpec( spec: SyntheticSpec, logged: ReadonlyArray, + builds: ReadonlyArray = [], + // The full ordinary run supplies only history/name association. Filtered + // children's own logs still determine every invocation and outcome. + nameLog: ReadonlyArray = logged, + randomSeed = childSeed, ): SpecExpectation { const ran = (test: string) => logged @@ -1613,8 +2247,21 @@ function predictSpec( } } } - // A required difference that the empty difference holds rejects any - // difference, so the property fails to load, naming the calibration. + // Calibration declaration checks precede requirement checks; resolving a + // selected history comes last. This diagnostic order is analyst-chosen, + // like the earlier shape checks, rather than a new product guarantee. + if ( + spec.probeBinding && + spec.probeBinding !== `empty` && + (!spec.probe || + [`no-absence`, `null-binding`, `false-binding`, `zero-binding`].includes( + spec.probeBinding, + ) || + (spec.probeBinding === `inferred` && + spec.constructed.filter(({ enters }) => enters.includes(`saw-0`)) + .length !== 1)) + ) + return { loads: false, loadError: `calibration history selector` } const empty = spec.calibrations.find(({ difference }) => emptyDifference(difference), ) @@ -1624,6 +2271,58 @@ function predictSpec( loadError: `${spec.id}: calibration "${empty.name}" requires difference`, emptyDifference: true, } + // The synthetic namespace is deliberately small: names identify the fixed + // or constructed index; enumerated names state their items. This model judges + // selection independently of the framework's resolver. + const selectedDeclaration = ( + binding: NonNullable, + ) => + `fixed` in binding + ? spec.fixed.find(({ name }) => name === binding.fixed) + : `constructed` in binding + ? spec.constructed.find(({ name }) => name === binding.constructed) + : spec.source.kind === `enumerated` + ? spec.source.histories.find((history, index) => + typeof binding.enumerated === `number` + ? index === binding.enumerated + : authoredEnumerationLabel(spec, history) === + binding.enumerated, + ) + : undefined + const duplicateSelection = ( + binding: NonNullable, + ) => + `fixed` in binding + ? spec.fixed.filter(({ name }) => name === binding.fixed).length > 1 + : `constructed` in binding + ? spec.constructed.filter(({ name }) => name === binding.constructed) + .length > 1 + : typeof binding.enumerated === `string` && + spec.source.kind === `enumerated` + ? spec.source.histories.filter( + (history) => + authoredEnumerationLabel(spec, history) === binding.enumerated, + ).length > 1 + : false + if ( + bindingsOf(spec).some( + (binding) => + !selectedDeclaration(binding) || + duplicateSelection(binding) || + (spec.source.kind === `enumerated` + ? !(`enumerated` in binding) + : `enumerated` in binding), + ) + ) + return { loads: false, loadError: `calibration history selector` } + if ( + spec.probeBinding && + spec.probeBinding !== `empty` && + (spec.probeBinding === `null-entered` || + selectedControls(spec, true).length !== 1 || + selectedControls(spec, false).length !== 1) + ) + return { loads: false, loadError: `calibration history selector` } // Each campaign unit's histories in this run: a fixed seed's units at that // seed, and the random units at the child run's seed. const streams = new Map>([ @@ -1636,7 +2335,7 @@ function predictSpec( ...(generated ? unitNames(spec).map( (name) => - [randomUnit(name), unitStream(spec, childSeed, name)] as const, + [randomUnit(name), unitStream(spec, randomSeed, name)] as const, ) : []), ]) @@ -1644,7 +2343,7 @@ function predictSpec( const names = spec.source.kind === `enumerated` ? spec.source.histories.map((history) => - enumeratedName(spec, history, logged), + enumeratedName(spec, history, nameLog), ) : [ ...streams.keys(), @@ -1668,7 +2367,11 @@ function predictSpec( // Every history fails before a verdict, so an enumerated source's reach // test counts none. if (data && spec.source.kind === `enumerated` && declaredOf(spec)) { - for (const name of all) tests.set(name, data) + for (const name of all) + tests.set( + name, + boundTest(spec, name) ? { kind: `calibration-clean` } : data, + ) const { outcome, counts } = enumeratedReach( spec, spec.source.histories, @@ -1684,17 +2387,24 @@ function predictSpec( } } if (data) { - for (const name of all) tests.set(name, data) - const failing = names.filter(isCampaign) + for (const name of all) + tests.set( + name, + boundTest(spec, name) ? { kind: `calibration-clean` } : data, + ) return { loads: true, tests, ...loaded, reach: new Map(), - replays: failing, - ...(failing.find((name) => unitOf(name)?.seed !== undefined) + replays: names.filter(isCampaign), + ...(names.find( + (name) => isCampaign(name) && unitOf(name)?.seed !== undefined, + ) ? { - unitRerun: failing.find((name) => unitOf(name)?.seed !== undefined), + unitRerun: names.find( + (name) => isCampaign(name) && unitOf(name)?.seed !== undefined, + ), } : {}), } @@ -1710,33 +2420,51 @@ function predictSpec( ) return { loads: true, tests, ...loaded, reach: new Map(), replays: [] } } - // A calibration needs a candidate whose clean trace passes; it skips the - // others (maintainer decision, 2026-11-03). Every history has an item, so - // the corruption applies to every candidate. A wrong answer violates law - // `three`, and its difference names the item it changed: the first item - // for a corruption or a `driver` mutant, the candidate's last item for a - // `driver-last` mutant. The law is judged first. A corruption decides at - // its first clean candidate; a mutant driver passes over a candidate whose - // difference does not hold, so it passes when any clean candidate it tried - // holds it. A difference that is not text fails at the first clean - // candidate. The candidates come from the contract's rule, not from the - // log, which stops where the framework stopped. + // The source changes how the premise is chosen, never how a mutant verdict ranks. const calibrate = ( - name: string, - kind: (typeof calibrationKinds)[number], - difference?: CalibrationDifference, + calibration: SyntheticSpec[`calibrations`][number], ): Outcome => { - const clean = calibrationCandidates(spec).filter( - (history) => judged(spec, history).kind === `pass`, + const { name, kind, difference } = calibration + const binding = bindingOf(spec, calibration) + const selected = binding && selectedDeclaration(binding) + const chosen = binding + ? authoredInput(spec, `calibration: ${name}`, binding, builds) + : undefined + const clean = binding + ? chosen + ? [chosen] + : [] + : calibrationCandidates(spec).filter( + (history) => + judged({ ...spec, extraReach: false }, history).kind === `pass`, + ) + if (binding) { + if (!chosen || judged(spec, chosen).kind !== `pass` || spec.extraReach) + return { kind: `calibration-clean` } + const enters = + selected && !Array.isArray(selected) && `enters` in selected + ? (selected.enters ?? []) + : [] + if (!reaches(spec, chosen, enters)) return { kind: `calibration-reach` } + } + if (!clean.length || kind === `inapplicable`) + return { kind: binding ? `calibration-inapplicable` : `calibration-none` } + if ( + [`corrupt-throws`, `judge-throws`, `judge-driver-throws`].includes(kind) ) - if (clean.length === 0) return { kind: `calibration-none` } + return { kind: `setup` } if (kind === `no-op`) return { kind: `calibration-passed` } if (kind === `wrong-law`) return { kind: `calibration-wrong-law` } if (difference === undefined) return { kind: `pass` } - if (spec.numberDifference) return { kind: `calibration-odd-difference` } + if (spec.numberDifference && kind !== `clock`) + return { kind: `calibration-odd-difference` } const holds = (history: SyntheticHistory) => - differenceHolds[difference](mutatedDifference(kind, history)) - if (kind === `right`) + differenceHolds[difference]( + kind === `clock` + ? `clock passed its timer bound` + : mutatedDifference(kind, history), + ) + if (binding || !driverCalibration(kind)) return holds(clean[0]!) ? { kind: `pass` } : { kind: `calibration-wrong-difference` } @@ -1766,19 +2494,23 @@ function predictSpec( if (throwsIn(history)) return { kind: `setup` } if (varies) return { kind: `determinism` } } + if (varies && generated) return { kind: `determinism` } return { kind: `pass` } } // A driver error decides a history first, then a law violation, then a // constructed history that misses a state it names (maintainer decision, - // 2026-11-03). Each unit decides its own first failure, over the stream it - // draws (law `units`): a constructed unit's one history, or the generated - // unit's stream. + // 2026-11-03). Each campaign unit decides its own first failure. const campaign = (test: string): Outcome => { const construction = spec.constructed.find( ({ name }) => name === unitOf(test)!.name, ) for (const history of streams.get(test)!) { - const { cause: _shrinkable, ...verdict } = judged(spec, history) + const { + cause: _shrinkableCause, + causeMessage: _shrinkableCauseMessage, + checkpoint: _shrinkableCheckpoint, + ...verdict + } = judged(spec, history) // Without a budget, a constructed history's failure shrinks to the // end. No smaller history is a constructed one, so a missed state // keeps the constructed history itself. @@ -1801,7 +2533,7 @@ function predictSpec( for (const history of spec.source.histories) { const verdict = verdictOf(history, spec) tests.set( - enumeratedName(spec, history, logged), + enumeratedName(spec, history, nameLog), declared && judgedKind(verdict) && reachedOf(spec, history).some((state) => !declared.includes(state)) @@ -1830,18 +2562,39 @@ function predictSpec( ) } } - for (const { name, kind, difference } of spec.calibrations) - tests.set(`calibration: ${name}`, calibrate(name, kind, difference)) + for (const calibration of spec.calibrations) + tests.set(`calibration: ${calibration.name}`, calibrate(calibration)) // A forced-probe calibration needs a candidate whose clean trace passes and // whose prediction differs from the forced probe; the check must then report // reach agreement. for (const [at, name] of forcedCalibrationNames(spec).entries()) { const forced = at === 0 + if (spec.probeBinding && spec.probeBinding !== `empty`) { + const clean = authoredInput( + spec, + name, + selectedControlBinding(spec, forced), + builds, + ) + // The clean opposite-polarity premise also owes every state declared by + // its selected fixed case or construction, not just the probe's state. + const selectedControl = selectedControls(spec, forced)[0] + tests.set( + name, + !clean || judged(spec, clean).kind !== `pass` + ? { kind: `calibration-clean` } + : !reaches(spec, clean, selectedControl?.enters ?? []) || + predictsZero(clean) === forced + ? { kind: `calibration-reach` } + : { kind: `pass` }, + ) + continue + } tests.set( name, ran(name).some( (history) => - judged(spec, history).kind === `pass` && + judged({ ...spec, extraReach: false }, history).kind === `pass` && predictsZero(history) !== forced, ) ? { kind: `pass` } @@ -1866,23 +2619,20 @@ function predictSpec( judgedKind(verdictOf(history, spec)), ).counts, ) - // Each passing unit prints its reach: the histories of its stream that - // entered each state. for (const [name, histories] of streams) if (tests.get(name)!.kind === `pass`) reach.set( name, Object.fromEntries( allStates.map((state) => [ - state, + state === `saw-0` ? (spec.probeName ?? state) : state, histories.filter((history) => reachedOf(spec, history).includes(state), ).length, ]), ), ) - // A unit that fails inside fast-check reports a seed and path to replay, - // and a fixed-seed unit a `-t` pattern to rerun. + // A campaign that fails inside fast-check reports a seed and path to replay. const replays = names.filter( (name) => isCampaign(name) && @@ -1904,8 +2654,11 @@ function predictSpec( ...loaded, reach, replays, - ...(rerun ? { rerun } : {}), ...(unitRerun ? { unitRerun } : {}), + ...(rerun ? { rerun } : {}), + ...(generated + ? { determinismRuns: spec.fixed.length + spec.constructed.length + 3 } + : {}), } } @@ -1916,15 +2669,35 @@ function suiteModel( const run = trace.find(({ checkpoint }) => checkpoint === `run`)?.value as SuiteRun | undefined const logged = run?.log ?? [] - return new Map(plan.specs.map((spec) => [spec.id, predictSpec(spec, logged)])) + return Object.assign( + new Map( + plan.specs.map((spec) => [ + spec.id, + predictSpec(spec, logged, run?.builds ?? []), + ]), + ), + { + hostScope: + plan.hostRngDraw === undefined + ? undefined + : { + draw: plan.hostRngDraw, + outerHostUsed: true as const, + outerForwarded: plan.hostRngForwarding !== undefined, + }, + }, + ) } // A late spec calls its probe again, unawaited, after every run of a history // holding a 0, also when its driver threw, and each such call throws: one // unhandled error per run the log records. A slow spec's driver runs on -// after its test timed out, so its calls may land after the run ended, and -// the driver records no log for it: its calls are not bounded above. -const lateCalls = (plan: Plan, logged: ReadonlyArray) => { +// after its test timed out, so its calls may land after the run ended; they +// count at most once each. +const lateCalls = ( + plan: Plan, + logged: ReadonlyArray>, +) => { let min = 0 let max = 0 for (const spec of plan.specs) { @@ -1932,15 +2705,15 @@ const lateCalls = (plan: Plan, logged: ReadonlyArray) => { const calls = logged.filter( (entry) => entry.id === spec.id && entry.history.includes(0), ).length - max += spec.slow ? Infinity : calls + max += calls if (!spec.slow) min += calls } return { min, max } } -// The route of the campaign each test runs; a unit keeps its campaign's. Every -// child run sets a seed-only replay (`${prefix}_SEED`), which reseeds the -// random units; a path replay of one runs one recorded history. +// The route of the campaign each test runs. Every child run sets a +// seed-only replay (ORACLE_SEED), which reseeds the random campaign; a path +// replay of it runs one recorded history. function owedRoute(name: string, replay = false): string { const unit = unitOf(name) if (unit?.seed !== undefined) return `fixed seed` @@ -1958,7 +2731,110 @@ function owedRoute(name: string, replay = false): string { // The self-oracle's own reach: states of the framework that its laws are // about. Each fixed history names the state it witnesses, so every one is reached on // every run; campaigns count them. +const warningReceivingKinds = [ + `run`, + `seedless`, + `replay`, + `rerun`, + `reach-alone`, + `binding-reordered`, + `registration-warnings`, + `environment`, +] as const +const passingBudgetLanes = [ + { context: `run`, lane: `fixed` }, + { context: `run`, lane: `random` }, + { context: `seedless`, lane: `fixed` }, + { context: `seedless`, lane: `random` }, +] as const +const passingUnitStates = passingBudgetLanes.flatMap(({ context, lane }) => + ([`constructed`, `generated`] as const).map( + (unit) => `passing-unit-${context}-${lane}-${unit}` as const, + ), +) +const callbackReceivingKinds = [ + `run`, + `binding-reordered`, + `seedless`, + `environment`, +] as const +const callbackReceivingStates = callbackReceivingKinds.map( + (kind) => `callback-failure-${kind}` as const, +) +const callbackRepresentationStates = callbackReceivingKinds.map( + (kind) => `callback-representation-${kind}` as const, +) +const round4FrameworkStates = [ + ...callbackRepresentationStates, + ...callbackReceivingStates, + `host-rng-scope`, + `host-rng-forwarding`, + `receiving-host-scope`, + ...passingUnitStates, + `warning-order-equivalence`, + ...warningReceivingKinds.map((kind) => `warning-receiving-${kind}` as const), + ...passingBudgetLanes.map( + ({ context, lane }) => `passing-budget-${context}-${lane}` as const, + ), + `passing-budget-constructed`, + `campaign-budget-failure-neighbor`, +] as const +const round3FrameworkStates = [ + ...round4FrameworkStates, + `bound-fixed-own-metadata-authored`, + `bound-fixed-own-metadata-forced-entered`, + `bound-fixed-own-metadata-forced-not-entered`, + `bound-fixed-inherited-metadata-authored`, + `bound-fixed-inherited-metadata-forced-entered`, + `bound-fixed-inherited-metadata-forced-not-entered`, + `bound-authored-distinct-selectors`, + `bound-authored-search-neighbor`, + `bound-mutant-violation-before-spin`, + `legacy-mutant-violation-before-spin`, + `bound-enumerated-label-rejected`, + `enumerated-reach-key-order-neighbor`, + `bound-registration-warning-silence`, + `receiving-run`, + `receiving-seedless`, + `receiving-replay`, + `receiving-rerun`, + `receiving-reach-alone`, + `receiving-binding-reordered`, + `receiving-registration-warnings`, + `receiving-environment`, +] as const +const timeoutReceivingKinds = [ + `run`, + `binding-reordered`, + `seedless`, + `environment`, +] as const +const timeoutReceivingStates = timeoutReceivingKinds.map( + (kind) => `timeout-input-${kind}` as const, +) const frameworkStates = [ + ...timeoutReceivingStates, + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ...round3FrameworkStates, + `auxiliary-late-error`, + `bound-enumerated-label-selector`, + `bound-control-fixed-entered`, + `bound-control-constructed-absent`, + `bound-control-inferred-name-gate`, + `bound-control-inferred-ambiguous`, + `bound-control-inferred-zero`, + `bound-control-explicit-name-gate`, + `bound-equal-fixed-origin`, + + `bound-ranked-mutants`, + `bound-independent-premise`, + `bound-name-seed`, + `bound-extra-state`, + `bound-error-context`, + `bound-null-selector`, + `registration-warning-policy`, + `bound-difference`, `registration-error`, `campaign-violation`, `campaign-setup-failure`, @@ -1978,12 +2854,20 @@ const frameworkStates = [ `constructed-build-load-error`, `calibration-applied`, `calibration-rejected`, - `calibration-skipped-candidate`, `calibration-difference-applied`, `calibration-difference-rejected`, `calibration-difference-later`, `calibration-empty-difference-error`, `calibration-odd-difference`, + `calibration-skipped-candidate`, + `bound-fixed-identity`, + `bound-enumerated-identity`, + `bound-combined-controls`, + `unbound-prototype-probe`, + `bound-control-full-reach`, + `bound-calibration`, + `bound-constructed-driver`, + `bound-calibration-registration-error`, `determinism-caught`, `observed-reach`, `timeout`, @@ -2055,19 +2939,304 @@ function frameworkReach( expected: ReturnType, ): Iterable { const reached = new Set() + const host = trace.find(({ checkpoint }) => checkpoint === `host-scope`) + ?.value as HostScopeRun | undefined + const hostChildren = trace + .filter( + ({ checkpoint, value }) => + checkpoint === `child` && + (value as ChildRun).context.kind === `host-scope`, + ) + .map(({ value }) => value as ChildRun) + if ( + host && + host.outer !== null && + host.outer.draw === expected.hostScope?.draw && + host.outer.childMarker === false && + host.outer.hostUsed === true && + host.outer.forwarded === expected.hostScope.outerForwarded && + Number.isInteger(host.outer.seed) && + host.children.length === 2 && + host.children.every( + (child) => child?.childMarker === true && Number.isInteger(child.seed), + ) && + host.children[0]?.seed === host.children[1]?.seed && + hostChildren.length === 3 && + hostChildren.every( + (child) => + child.tests.length === 1 && + child.tests[0]?.state === `passed` && + child.loads.every(({ errors }) => errors.length === 0), + ) + ) { + reached.add(`host-rng-scope`) + if (host.outer.forwarded) reached.add(`host-rng-forwarding`) + } + if ( + trace.some( + ({ checkpoint, value }) => + checkpoint === `warning-permutation` && + (value as { lists: number; stable: boolean }).lists > 0 && + (value as { stable: boolean }).stable, + ) + ) + reached.add(`warning-order-equivalence`) + // Count the real host receiving cut, not a plan flag or a derived summary. + // Registration-only children owe a loaded test inventory; every other cut + // additionally owes a test that actually ran in that child. + for (const { checkpoint, value } of trace) { + if (checkpoint !== `child`) continue + const child = value as ChildRun + const timeoutKind = timeoutReceivingKinds.find( + (kind) => kind === child.context.kind, + ) + if ( + timeoutKind && + timeoutInputsMatch(plan, child) && + child.timeoutInputs?.some((owner) => + child.log.some( + (entry) => entry.id === owner.id && entry.test === owner.test, + ), + ) + ) + reached.add(`timeout-input-${timeoutKind}`) + if ( + child.loads.some(({ errors }) => errors.length === 0) && + child.tests.length > 0 && + (child.context.kind === `registration-warnings` || + child.tests.some(({ state }) => state !== `skipped`)) + ) + reached.add(`receiving-${child.context.kind}`) + for (const original of plan.specs) { + if ( + `id` in child.context && + child.context.id !== original.id && + child.context.kind !== `replay` + ) + continue + const spec = registeredSpec(original, child.context) + if (!predictSpec(spec, child.log, child.builds).loads) continue + if ( + unboundCalibrationNames(spec).length > 0 && + !( + child.context.kind === `registration-warnings` && + child.context.phases.length === 0 + ) && + !warningMismatch(spec, child.warnings, false) + ) + reached.add(`warning-receiving-${child.context.kind}`) + if ( + callbackReceivingKinds.some((kind) => kind === child.context.kind) && + child.loads.some( + (load) => load.id === spec.id && load.errors.length === 0, + ) + ) { + const prediction = predictSpec(spec, child.log, child.builds) + const nameLog = + ( + trace.find((entry) => entry.checkpoint === `run`)?.value as + SuiteRun | undefined + )?.log ?? [] + if ( + prediction.loads && + child.tests.some((test) => { + if ( + test.id !== spec.id || + test.state !== `failed` || + test.errors.length !== 1 || + test.errors[0]?.kind !== `setup` + ) + return false + const outcome = prediction.tests.get(test.name) + const diagnostic = + outcome && callbackExpectation(spec, test.name, outcome, nameLog) + const calibration = spec.calibrations.find( + (entry) => test.name === `calibration: ${entry.name}`, + ) + if ( + !diagnostic || + !calibration || + !callbackDiagnosticMatches(diagnostic, test.errors[0]) + ) + return false + const calls = child.log.filter( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + ) + return ( + calls.length === (driverCalibration(calibration.kind) ? 2 : 1) + ) + }) + ) { + reached.add(`callback-failure-${child.context.kind}`) + if (child.callbackPresented === true) + reached.add(`callback-representation-${child.context.kind}`) + } + } + if (spec.source.kind !== `generated`) continue + const units = child.tests.filter( + ({ id, name }) => id === spec.id && isCampaign(name), + ) + for (const test of units) { + const calls = child.log.filter( + ({ id, test: name }) => + id === spec.id && name === `${spec.id} > ${test.name}`, + ) + if (test.state === `failed` && calls.length > 0) + reached.add(`campaign-budget-failure-neighbor`) + } + if (child.context.kind !== `run` && child.context.kind !== `seedless`) + continue + for (const seed of [...fixedSeedsOf(spec), undefined]) { + const lane = seed === undefined ? `random` : `fixed` + const names = unitNames(spec).map((unit) => + seed === undefined ? randomUnit(unit) : `fixed seed ${seed}: ${unit}`, + ) + let total = 0 + const complete = names.every((name) => { + const test = units.find((candidate) => candidate.name === name) + const calls = child.log.filter( + ({ id, test: actual }) => + id === spec.id && actual === `${spec.id} > ${name}`, + ) + const generated = unitOf(name)?.name === `generated` + const count = generated + ? spec.source.kind === `generated` + ? spec.source.runs + : 0 + : 1 + if (test?.state !== `passed` || calls.length !== count) return false + total += calls.length + reached.add( + `passing-unit-${child.context.kind}-${lane}-${generated ? `generated` : `constructed`}`, + ) + return true + }) + if (complete && total === spec.source.runs + spec.constructed.length) { + reached.add(`passing-budget-${child.context.kind}-${lane}`) + if (spec.constructed.length > 0) + reached.add(`passing-budget-constructed`) + } + } + } + } + if ( + trace + .filter(({ checkpoint }) => checkpoint === `child`) + .slice(1) + .some(({ value }) => { + const child = value as { + log: Array> + unhandled: Array + } + return ( + child.log.some( + ({ id, history }) => + history.includes(0) && + plan.specs.some( + (spec) => spec.id === id && spec.probe === `late`, + ), + ) && + child.unhandled.some((message) => + message.includes(`was called after its history ended`), + ) + ) + }) + ) + reached.add(`auxiliary-late-error`) const run = trace.find(({ checkpoint }) => checkpoint === `run`)?.value as SuiteRun | undefined for (const spec of plan.specs) { const owed = expected.get(spec.id)! - if (!owed.loads) { - reached.add(`registration-error`) - if (owed.shapeError) reached.add(`enumerated-shape-error`) + const reachOnly = trace.find( + ({ checkpoint, value }) => + checkpoint === `reach-alone` && (value as ReachAlone).id === spec.id, + )?.value as ReachAlone | undefined + if ( + spec.enumeratedStates && + declaredOf(spec)!.join(`,`) !== [...declaredOf(spec)!].sort().join(`,`) && + reachOnly?.state === `passed` + ) + reached.add(`enumerated-reach-key-order-neighbor`) + const file = run?.files[spec.id] + const ranCalibration = (name: string) => + file?.loaded && + file.tests.some( + (test) => test.name === name && test.state !== `skipped`, + ) && + run?.log.some( + (entry) => + entry.id === spec.id && entry.test === `${spec.id} > ${name}`, + ) + if ( + file && + spec.source.kind === `enumerated` && + bindingsOf(spec).some( + (binding) => + `enumerated` in binding && typeof binding.enumerated === `string`, + ) && + [`shell`, `default`].includes(spec.labelStyle ?? ``) + ) { + if ( + owed.loads && + spec.calibrations.some(({ name }) => + ranCalibration(`calibration: ${name}`), + ) + ) + reached.add(`bound-enumerated-label-selector`) + else if (!owed.loads && file.loaded === false) + reached.add(`bound-enumerated-label-rejected`) + } + if (!owed.loads) { + if ( + file?.loaded === false && + owed.nameError && + (spec.probeBinding === `explicit` || spec.probeBinding === `inferred`) + ) + reached.add(`bound-control-${spec.probeBinding}-name-gate`) + if ( + file?.loaded === false && + !owed.nameError && + owed.loadError === `calibration history selector` && + spec.probeBinding === `inferred` + ) { + if ( + spec.constructed.filter(({ enters }) => enters.includes(`saw-0`)) + .length > 1 + ) + reached.add(`bound-control-inferred-ambiguous`) + if ( + spec.constructed.filter(({ enters }) => enters.includes(`saw-0`)) + .length === 0 + ) + reached.add(`bound-control-inferred-zero`) + } + if ( + [ + `null-entered`, + `null-binding`, + `false-binding`, + `zero-binding`, + ].includes(spec.probeBinding ?? ``) + ) + reached.add(`bound-null-selector`) + reached.add(`registration-error`) + if (bindingsOf(spec).length) + reached.add(`bound-calibration-registration-error`) + if (owed.shapeError) reached.add(`enumerated-shape-error`) if (owed.reservedLabel) reached.add(`reserved-label-error`) if (owed.dataError) reached.add(`history-out-of-domain`) if (owed.buildError) reached.add(`constructed-build-load-error`) if (owed.emptyDifference) reached.add(`calibration-empty-difference-error`) - if (owed.nameError) reached.add(`unit-name-load-error`) + if ( + owed.nameError && + owed.loadError && + file?.loaded === false && + file.loadErrors.some((message) => message.includes(owed.loadError!)) + ) + reached.add(`unit-name-load-error`) if (owed.membership === `fixed`) reached.add(`fixed-membership-failure`) if (owed.membership === `constructed`) reached.add(`constructed-membership-failure`) @@ -2080,23 +3249,217 @@ function frameworkReach( } continue } - if (spec.secondSeed) reached.add(`second-fixed-seed`) - // A constructed unit and the generated unit of one seed that fail under - // different laws: each unit decided its own first failure. + for (const declaration of spec.fixed) { + if (!declaration.buildMetadata) continue + const prefix = `bound-fixed-${declaration.buildMetadata}-metadata` + if ( + spec.calibrations.some((calibration) => { + const binding = bindingOf(spec, calibration) + return ( + binding && + `fixed` in binding && + binding.fixed === declaration.name && + ranCalibration(`calibration: ${calibration.name}`) + ) + }) + ) + reached.add(`${prefix}-authored`) + if (spec.probeBinding === `explicit`) + for (const [at, name] of forcedCalibrationNames(spec).entries()) { + const binding = selectedControlBinding(spec, at === 0) + if ( + `fixed` in binding && + binding.fixed === declaration.name && + ranCalibration(name) + ) + reached.add( + `${prefix}-${at === 0 ? `forced-entered` : `forced-not-entered`}`, + ) + } + } + const authored = spec.calibrations.filter(({ name }) => + ranCalibration(`calibration: ${name}`), + ) + if ( + new Set( + authored + .map((calibration) => bindingOf(spec, calibration)) + .filter(Boolean) + .map((binding) => JSON.stringify(binding)), + ).size > 1 + ) + reached.add(`bound-authored-distinct-selectors`) + if ( + authored.some((calibration) => bindingOf(spec, calibration)) && + authored.some((calibration) => !bindingOf(spec, calibration)) + ) + reached.add(`bound-authored-search-neighbor`) + for (const calibration of authored) + if ([`driver-spin`, `driver-spin-throws`].includes(calibration.kind)) + reached.add( + bindingOf(spec, calibration) + ? `bound-mutant-violation-before-spin` + : `legacy-mutant-violation-before-spin`, + ) + if ( + run?.files[spec.id]?.loaded && + spec.probeBinding === `explicit` && + `fixed` in selectedControlBinding(spec, false) + ) + reached.add(`bound-control-fixed-entered`) + if ( + run?.files[spec.id]?.loaded && + spec.probeBinding && + spec.probeBinding !== `empty` && + `constructed` in selectedControlBinding(spec, true) + ) + reached.add(`bound-control-constructed-absent`) + if ( + run?.files[spec.id]?.loaded && + spec.identityHistories && + bindingsOf(spec).some( + (binding) => + `fixed` in binding && + spec.fixed.filter( + ({ history }) => + JSON.stringify(history) === + JSON.stringify(authoredInput(spec, ``, binding, [])), + ).length > 1, + ) + ) + reached.add(`bound-equal-fixed-origin`) + if ( + bindingsOf(spec).length > 0 && + spec.calibrations.some( + ({ kind }) => kind === `throws` || kind === `clock`, + ) + ) + reached.add(`bound-ranked-mutants`) + if ( + spec.probeBinding === `explicit` && + spec.constructed.length > 1 && + spec.fixed.length > 1 + ) + reached.add(`bound-independent-premise`) + if (spec.calibrationNames !== `indexed` && spec.calibrations.length > 1) + reached.add(`bound-name-seed`) + if (bindingsOf(spec).length && spec.extraReach) + reached.add(`bound-extra-state`) + if ( + bindingsOf(spec).length > 0 && + spec.calibrations.some(({ kind }) => + [`corrupt-throws`, `judge-throws`, `judge-driver-throws`].includes( + kind, + ), + ) + ) + reached.add(`bound-error-context`) + if (spec.warningProbe) { + reached.add(`registration-warning-policy`) + if ( + unboundCalibrationNames(spec).length === 0 && + trace.some( + ({ checkpoint, value }) => + checkpoint === `registration-warnings` && + (value as { id: string; loaded: Record }).id === + spec.id && + Object.values( + (value as { loaded: Record }).loaded, + ).every(Boolean), + ) + ) + reached.add(`bound-registration-warning-silence`) + } + if ( + bindingsOf(spec).length > 0 && + spec.calibrations.some(({ difference }) => difference !== undefined) + ) + reached.add(`bound-difference`) + if (spec.identityHistories) { + if (bindingsOf(spec).some((binding) => `fixed` in binding)) + reached.add(`bound-fixed-identity`) + if (bindingsOf(spec).some((binding) => `enumerated` in binding)) + reached.add(`bound-enumerated-identity`) + } + if ( + bindingsOf(spec).length > 0 && + spec.probeBinding && + spec.probeBinding !== `empty` + ) + reached.add(`bound-combined-controls`) + if ( + spec.probe && + spec.probeBinding === `empty` && + spec.probeName !== `saw-0` + ) + reached.add(`unbound-prototype-probe`) + if ( + spec.probeBinding && + spec.probeBinding !== `empty` && + [...owed.tests.values()].some(({ kind }) => kind === `calibration-reach`) + ) + reached.add(`bound-control-full-reach`) + const ranUnit = (name: string) => + file?.loaded === true && + file.tests.some( + (test) => + test.name === name && + (test.state === `passed` || test.state === `failed`), + ) && + run!.log.some( + ({ id, test }) => id === spec.id && test === `${spec.id} > ${name}`, + ) + const secondSeed = fixedSeedsOf(spec)[1] + if ( + spec.secondSeed && + secondSeed !== undefined && + unitNames(spec).every((unit) => ranUnit(fixedUnit(secondSeed, unit))) + ) + reached.add(`second-fixed-seed`) + // The real test results and calls must exhibit the model's two laws. for (const seed of fixedSeedsOf(spec)) { - const laws = unitNames(spec).map( - (unit) => owed.tests.get(fixedUnit(seed, unit))?.law, + const generatedName = fixedUnit(seed, `generated`) + const generated = file?.tests.find(({ name }) => name === generatedName) + const generatedLaw = owed.tests.get(generatedName)?.law + if ( + generatedLaw === undefined || + generated?.state !== `failed` || + generated.failure?.law !== generatedLaw || + !ranUnit(generatedName) ) - const generatedLaw = laws.at(-1) + continue if ( - generatedLaw !== undefined && - laws - .slice(0, -1) - .some((law) => law !== undefined && law !== generatedLaw) + spec.constructed.some(({ name }) => { + const unit = fixedUnit(seed, name) + const law = owed.tests.get(unit)?.law + const observed = file?.tests.find((test) => test.name === unit) + return ( + law !== undefined && + law !== generatedLaw && + observed?.state === `failed` && + observed.failure?.law === law && + ranUnit(unit) + ) + }) ) reached.add(`unit-split-verdict`) } const named = spec.fixed.length + spec.constructed.length > 0 + if ( + bindingsOf(spec).length > 0 || + (spec.probeBinding && spec.probeBinding !== `empty`) + ) + reached.add(`bound-calibration`) + if ( + spec.isolatedPlans && + spec.calibrations.some((calibration) => { + const binding = bindingOf(spec, calibration) + return ( + binding && `constructed` in binding && calibration.kind === `driver` + ) + }) + ) + reached.add(`bound-constructed-driver`) if (spec.arbitraryMap?.unmapper === true && named) reached.add(`unmapper-history-loads`) if (spec.arbitraryMap?.unmapper === false && !named) @@ -2285,26 +3648,72 @@ function frameworkReach( if (checkpoint === `rerun` && spec?.nested) reached.add(`nested-rerun`) if (checkpoint === `replay` && (probe.routes?.length ?? 0) > 0) reached.add(`replayed-route`) - if (checkpoint === `replay` && probe.failure) { + if (checkpoint !== `replay` && checkpoint !== `rerun`) continue + const unit = unitOf(probe.target) + if ( + !unit || + !spec || + !probe.failure || + probe.failure.kind !== probe.original.kind || + probe.failure.law !== probe.original.law + ) + continue + const received = trace.find(({ checkpoint: cut, value: raw }) => { + if (cut !== `child`) return false + const child = raw as ChildRun + const context = child.context + return ( + (context.kind === `replay` || context.kind === `rerun`) && + context.kind === checkpoint && + context.id === probe.id && + context.target === probe.target && + context.probed === probe.probed + ) + })?.value as ChildRun | undefined + const executed = + received?.tests.filter( + ({ state }) => state === `passed` || state === `failed`, + ) ?? [] + const isolated = + executed.length === 1 && + executed[0]!.id === probe.id && + executed[0]!.name === probe.probed && + executed[0]!.state === `failed` && + received!.log.some( + ({ id, test }) => + id === probe.id && test === `${probe.id} > ${probe.probed}`, + ) + if (!isolated) continue + if ( + checkpoint === `replay` && + probe.probed === randomUnit(unit.name) && + (probe.ran?.length ?? 0) > 0 && + probe.routes?.every((route) => route === `fixed history`) + ) { reached.add( - unitOf(probe.target)?.name === `generated` + unit.name === `generated` ? `unit-replay-generated` : `unit-replay-constructed`, ) - if (plan.prefix !== `ORACLE`) reached.add(`unit-replay-custom-prefix`) + if ( + plan.prefix !== `ORACLE` && + probe.variables?.[`${plan.prefix}_UNIT`] === probe.probed && + probe.variables[`${plan.prefix}_PROPERTY`] === spec.id && + probe.variables[`${plan.prefix}_SEED`] !== undefined && + probe.variables[`${plan.prefix}_PATH`] !== undefined + ) + reached.add(`unit-replay-custom-prefix`) } - if (checkpoint === `rerun` && spec && isCampaign(probe.target)) { + if (checkpoint === `rerun` && probe.probed === probe.target) { reached.add(`unit-rerun`) - // A sibling unit whose name extends the target's, which a pattern - // without its `$` anchor would also select. + // The extended sibling was actually registered and stayed skipped. if ( - expected.get(spec.id)!.loads && - [ - ...( - expected.get(spec.id) as { tests: Map } - ).tests.keys(), - ].some( - (other) => other !== probe.target && other.startsWith(probe.target), + probe.states.some( + ([id, name, state]) => + id === spec.id && + name !== probe.target && + name.startsWith(probe.target) && + state === `skipped`, ) ) reached.add(`unit-rerun-beside-extension`) @@ -2343,10 +3752,30 @@ function frameworkReach( reached.add(`seedless-route`) if ( seedless && - plan.specs.some( - (spec) => - !spec.slow && Object.keys(seedless[spec.id]?.ran ?? {}).length > 0, - ) + plan.specs.some((spec) => { + const seen = seedless[spec.id] + if ( + !expected.get(spec.id)!.loads || + spec.source.kind !== `generated` || + spec.slow || + seen?.seeds.length !== 1 || + JSON.stringify(seen.routes) !== JSON.stringify([`random seed`]) + ) + return false + return unitNames(spec) + .map(randomUnit) + .every((unit) => { + const { ran, owedRun } = comparedRun( + spec, + unit, + unitStream(spec, seen.seeds[0]!, unitOf(unit)!.name), + seen.ran[unit] ?? [], + ) + return ( + ran.length > 0 && JSON.stringify(ran) === JSON.stringify(owedRun) + ) + }) + }) ) reached.add(`seedless-units-drawn`) // Law `replay-config`: the environment's first error, or the selection a @@ -2380,6 +3809,23 @@ function frameworkReach( } if (trace.some(({ checkpoint }) => checkpoint === `rerun`)) reached.add(`rerun-probe`) + if ( + trace.some( + ({ checkpoint, value }) => + checkpoint === `timeout-presentation` && + (value as { exercised: boolean }).exercised, + ) + ) + reached.add(`timeout-presentation-equivalence`) + if ( + trace.some( + ({ checkpoint, value }) => + checkpoint === `timeout-presentation` && + (value as { cutExercised: boolean; stable: boolean }).cutExercised && + (value as { stable: boolean }).stable, + ) + ) + reached.add(`timeout-continuation-cut`) return reached } @@ -2392,20 +3838,18 @@ function frameworkReach( // records each file's load errors, each test's state and failure (with the // cause fast-check attaches), each passing campaign's printed reach counts, and // the synthetic driver's log. Then it probes: -// - replay: for every failing campaign unit that printed a replay command, -// /bin/sh runs that command as printed, with this child's Vitest in place -// of `vitest run`, over all the files and with its own log, which shows the -// history the replay started from; -// - rerun: for each property's first history test that printed a rerun -// command, and its first fixed-seed unit that did, Vitest runs that file -// with the command's `-t` pattern; -// - environment: for a suite that draws a replay environment, Vitest runs -// all the files once under it; +// - replay: for every failing campaign with a reported seed and path, Vitest runs +// all the files again with that replay selected and its own log, which +// shows the history the replay started from; +// - rerun: for each property's first failure that printed a rerun command, +// Vitest runs that file with the command's `-t` pattern; // - reach alone: for each loaded enumerated source that declares states, // Vitest runs that file with a `-t` pattern that selects only its reach // test. // Child runs fix the random campaign's seed (a seed without a path reseeds the -// campaign and selects no replay), so a child run records the same facts every time. +// campaign and selects no replay). The ordinary-route child instead controls +// Math.random's draw while leaving ORACLE_SEED absent. This fixes the host +// input without changing the route, so every recorded fact can repeat. type ReportedError = { name?: string; message?: string; cause?: ReportedError } type ReportedTest = { @@ -2422,9 +3866,17 @@ type ReportedFile = { type Report = { files: Array stdout: string + warnings: Array + warningPermutation?: { lists: number; stable: boolean } unhandled: Array + exit: number | null + signal: string | null } type Failure = { + message?: string + causeMessage?: string + /** Semantic bound-callback identity; seed annotations and stacks are excluded. */ + callbackContext?: { calibration: string; kind: string; name: string } kind: Kind | `other` dataPath?: string dataKind?: string @@ -2433,9 +3885,10 @@ type Failure = { diagnosis?: `never-reached` | `false-condition` cause?: `driver` counterexample?: string + /** The concrete reported candidate, when its finite data form shows its items. */ + counterexampleHistory?: SyntheticHistory seed?: number path?: string - /** The `-t` part of a rerun command, from the violation or from the unit's failure. */ rerun?: string /** A violation's whole rerun line, as printed. */ violationRerun?: string @@ -2466,8 +3919,56 @@ type SuiteRun = { } > log: Array + resolutions: Array> + builds: Array + stdout: string + warnings: Array /** The run's unhandled errors, by message. */ unhandled: Array + lateAgreement?: LateAgreement + timeoutInputs?: Array +} +// Every real child has a named receiving cut. The context is supplied by the +// driver, independently of the test states Vitest reports, so absent tests or +// an unexpected first error cannot authorize their own outcome. +type ChildContext = + | { kind: `run` | `seedless` | `environment` } + | { kind: `binding-reordered`; id: string } + | { kind: `registration-warnings`; id: string; phases: Array } + | { kind: `replay`; id: string; target: string; probed: string } + | { kind: `rerun`; id: string; target: string; probed: string } + | { kind: `reach-alone`; id: string } + | { + kind: `host-scope` + mode: `outer` | `child` + ordinal: number + draw: 0.25 | 0.75 + } +type ChildRun = { + context: ChildContext + exit: number | null + signal: string | null + files: Array + expectedFiles: Array + loads: Array<{ id: string; errors: Array; causes: Array }> + tests: Array<{ + id: string + name: string + state: string + errors: Array + seedLogs: Array + }> + log: Array + builds: Array + resolutions: SuiteRun[`resolutions`] + /** Registration warnings retained for every process, including filtered runs. */ + warnings: Array + secondaryErrors: Array<{ test: string; message: string }> + unhandled: Array + lateAgreement?: LateAgreement + timeoutInputs?: Array + failed: boolean + callbackPresented?: boolean } type ProbeRun = { id: string @@ -2509,6 +4010,23 @@ type EnvironmentRun = { log: Array } +// The ordinary-host receiving witness reads its selected seed after importing +// the fixture. Marked children separately show that controlled inputs repeat. +type HostSeed = { + seed: number + childMarker: boolean + draw: number + /** The supplied host counter advanced during actual seed selection. */ + hostUsed: boolean + /** The actual configured forwarding wrapper ran during seed selection. */ + forwarded: boolean +} +type HostScopeRun = { + draw: 0.25 | 0.75 + outer: HostSeed | null + children: Array +} + /** A reach test run alone, under a `-t` pattern that selects only it. */ type ReachAlone = { id: string @@ -2527,8 +4045,302 @@ const vitest = join( `vitest.mjs`, ) let runCount = 0 +let warningPresentationCount = 0 +let timeoutPresentationCount = 0 +let hostForwardingCount = 0 +let callbackPresentationCount = 0 + +function classifyKind( + inner: ReportedError, + text: string, + outer: string, +): Kind | `other` { + if (callbackExceptionCause(inner)) return `setup` + if (text.includes(`selected clean run failed`)) return `calibration-clean` + if (text.includes(`selected history did not enter`)) + return `calibration-reach` + if (text.includes(`selected corruption did not apply`)) + return `calibration-inapplicable` + // A wrapped runtime exception is still setup failure. Context and cause are + // checked separately, so binding does not change the semantic verdict kind. + if (text.includes(`selected mutant run failed`)) return `setup` + if (inner.name === `OracleDataError`) return `data` + if (text.includes(`the framework's encoding limit`)) return `limit` + if (inner.name === `OracleModelError`) return `model-not-total` + if (inner.name === `OracleViolation`) return `violation` + if (/timed out in \d+ms/.test(text) || /timed out in \d+ms/.test(outer)) + return `timeout` + if (text.includes(`did not enter`) && text.includes(`: fixed history "`)) + return `fixed-reach` + if ( + text.includes(`did not enter`) && + text.includes(`: constructed history "`) + ) + return `constructed-reach` + if (text.includes(`no enumerated history entered`)) return `enumerated-reach` + if (text.includes(`which the property does not declare`)) + return `undeclared-state` + if (text.includes(`two runs of`)) return `determinism` + if (text.includes(`a difference that is not text`)) + return `calibration-odd-difference` + if (text.includes(`but the check reported difference`)) + return `calibration-wrong-difference` + if (text.includes(`but the check passed it`)) return `calibration-passed` + if (text.includes(`but the check reported law`)) + return `calibration-wrong-law` + if (text.includes(`found no history`)) return `calibration-none` + if ( + text.includes(`synthetic driver failed`) || + text.includes(`cannot read a 7`) || + text.includes(`mutant setup failed`) || + text.includes(`synthetic corruption failed`) || + text.includes(`synthetic judgment failed`) + ) + return `setup` + return `other` +} + +// This input transformation changes opaque reason prose after the actual primary +// callback frame, retaining the rendered identity and original finite cause. It +// does not consult the expected selected binding or restrict the reason's words. +const callbackExceptionCause = (error: ReportedError) => + [`synthetic corruption failed`, `synthetic judgment failed`].includes( + error.cause?.message ?? ``, + ) +// Outer wrapper names are presentation input. Preserve the actual original +// cause and all identity/message bytes while changing only the wrapper's name. +const incomingCallbackWrapper = ( + error: ReportedError | undefined, + name: string | undefined, +) => + error && + name !== undefined && + [`synthetic corruption failed`, `synthetic judgment failed`].includes( + error.cause?.message ?? ``, + ) + ? { ...error, name } + : error +const incomingCallbackReason = ( + error: ReportedError | undefined, + reason: string | undefined, + candidates: ReadonlyArray, +) => { + if (!error?.message || reason === undefined || !callbackExceptionCause(error)) + return error + const observed = observedCallbackFrame(error.message, candidates) + if (!observed) return error + const separator = /^:\s+/.exec(error.message.slice(observed.end)) + const suffix = error.message.lastIndexOf(`: ${error.cause!.message}`) + if (!separator || suffix < observed.end) return error + return { + ...error, + message: + error.message.slice(0, observed.end + separator[0].length) + + reason + + error.message.slice(suffix), + } +} + +const incomingCallbackError = ( + error: ReportedError | undefined, + presentation: CallbackPresentation | undefined, + candidates: ReadonlyArray, +) => { + if (!error?.message || !presentation) return error + const observed = observedCallbackFrame(error.message, candidates) + if (!observed) return error + const spaces = + presentation.spacing === `plain` + ? [1, 1, 1, 1, 1, 1] + : presentation.spacing === `padded` + ? [3, 3, 3, 3, 3, 3] + : presentation.spacing + const gaps = spaces.map((count) => + typeof count === `number` ? ` `.repeat(count) : count, + ) + const { calibration, kind, name } = observed.context + const label = + presentation.calibration === `quoted` + ? `"${calibration}"` + : `[${calibration}]` + const seed = + observed.seed === undefined || presentation.seed === `omitted` + ? `` + : gaps[5] + + (presentation.seed === `bracketed` + ? `[seed ${observed.seed}]` + : `(seed ${observed.seed})`) + const frame = `calibration${gaps[0]}${label}${gaps[1]}on${gaps[2]}${kind}${gaps[3]}history:${gaps[4]}${name}${seed}` + return { + ...error, + message: + error.message.slice(0, observed.start) + + frame + + error.message.slice(observed.end), + } +} + +// Presentation consumes the actual finite raw frame independently of its selected +// binding. Raw source reports and original causes remain unchanged. Incoming raw +// variants separately challenge this receiver before it renders a requested form. +const renderCallbackContext = ( + context: { calibration: string; kind: string; name: string }, + presentation: CallbackPresentation, + seed?: string, +) => { + const calibration = + presentation.calibration === `quoted` + ? `"${context.calibration}"` + : `[${context.calibration}]` + const spaces = + presentation.spacing === `plain` + ? [1, 1, 1, 1, 1, 1] + : presentation.spacing === `padded` + ? [3, 3, 3, 3, 3, 3] + : presentation.spacing + const gaps = spaces.map((count) => + typeof count === `number` ? ` `.repeat(count) : count, + ) + const annotation = + context.kind !== `constructed` || + seed === undefined || + presentation.seed === `omitted` + ? `` + : presentation.seed === `parenthesized` + ? `${gaps[5]}(seed ${seed})` + : `${gaps[5]}[seed ${seed}]` + return `calibration${gaps[0]}${calibration}${gaps[1]}on${gaps[2]}${context.kind}${gaps[3]}history:${gaps[4]}${context.name}${annotation}` +} +const presentCallbackError = ( + error: ReportedError | undefined, + presentation: CallbackPresentation | undefined, + candidates: ReadonlyArray, +) => { + if (!error?.message || !presentation) return { error, applied: false } + const observed = observedCallbackFrame(error.message, candidates) + if (!observed) return { error, applied: false } + // The consumed identity is the actual complete observed frame, never the + // expected selected binding. Successful rendering is an executed operation + // even when the source already uses this rendering and bytes stay equal. + const message = + error.message.slice(0, observed.start) + + renderCallbackContext(observed.context, presentation, observed.seed) + + error.message.slice(observed.end) + return { error: { ...error, message }, applied: true } +} + +// Decode a whole context frame against the available authored name universe, +// not the selected binding. Exact literal names remain exact inside their frame. +// The bounded rendering grammar varies wrappers, optional seed decoration, and +// positive structural whitespace. Distinct matching identities, unknown text, or multiple frames remain +// unrecognized; the shared comparison then rejects them. This is not an arbitrary +// English-message parser or a product promise about the frozen source's template. +type CallbackCandidate = NonNullable +const callbackPresentations: Array = [ + `quoted`, + `bracketed`, +].flatMap((calibration) => + [`parenthesized`, `bracketed`, `omitted`].flatMap((seed) => + [`plain`, `padded`].map( + (spacing) => ({ calibration, seed, spacing }) as CallbackPresentation, + ), + ), +) +const callbackCandidateUniverse = ( + plan: Plan, + nameLog: ReadonlyArray, +): Array => { + const candidates: Array = [] + for (const spec of plan.specs) { + const calibrations = [ + ...spec.calibrations.map(({ name }) => name), + ...forcedCalibrationNames(spec).map((name) => + name.slice(`calibration: `.length), + ), + ] + const histories = [ + ...spec.fixed.map(({ name }) => ({ kind: `fixed`, name })), + ...spec.constructed.map(({ name }) => ({ kind: `constructed`, name })), + ...(spec.source.kind !== `enumerated` + ? [] + : spec.source.histories.flatMap((history, index) => { + const literal = authoredEnumerationLabel(spec, history) + const names = + literal === undefined + ? [ + ...new Set( + nameLog + .filter( + (entry) => + entry.id === spec.id && + entry.origin === `enumerated:${index}` && + !frameworkTest( + entry.test.slice(spec.id.length + 3), + ), + ) + .map((entry) => entry.test.slice(spec.id.length + 3)), + ), + ] + : [literal] + return names.map((name) => ({ kind: `enumerated`, name })) + })), + ] + for (const calibration of calibrations) + for (const history of histories) + candidates.push({ calibration, ...history }) + } + return [ + ...new Map( + candidates.map((candidate) => [JSON.stringify(candidate), candidate]), + ).values(), + ] +} +const escapeCallbackPattern = (text: string) => + text.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`) +const observedCallbackFrame = ( + message: string, + candidates: ReadonlyArray, +) => { + // The reason/cause region is opaque. It cannot provide missing identity names. + const prefix = /^.*?: /s.exec(message) + if (!prefix) return undefined + const start = prefix[0].length + const frame = message.slice(start) + // Only structural separators vary. Literal spaces/punctuation inside authored + // names are escaped intact, including a name's numeric or seed-like suffix. + const gap = `\\s+` + const matches = candidates.flatMap((candidate) => { + const calibration = escapeCallbackPattern(candidate.calibration) + const label = `(?:"${calibration}"|\\[${calibration}\\])` + const suffix = + candidate.kind === `constructed` + ? `(?:${gap}(?:\\(seed (-?\\d+)\\)|\\[seed (-?\\d+)\\]))?` + : `` + const pattern = `^calibration${gap}${label}${gap}on${gap}${escapeCallbackPattern(candidate.kind)}${gap}history:${gap}${escapeCallbackPattern(candidate.name)}${suffix}(?=:\\s)` + const match = new RegExp(pattern).exec(frame) + return match + ? [ + { + context: candidate, + seed: match[1] ?? match[2], + start, + end: start + match[0].length, + }, + ] + : [] + }) + return matches.length === 1 ? matches[0] : undefined +} +const reportedCallbackContext = ( + message: string, + candidates: ReadonlyArray, +): Failure[`callbackContext`] => + observedCallbackFrame(message, candidates)?.context -function classify(error: ReportedError | undefined): Failure { +function classify( + error: ReportedError | undefined, + candidates: ReadonlyArray = [], +): Failure { if (!error) return { kind: `other` } const outer = error.message ?? `` // Only fast-check's wrapper carries the framework's failure as its cause; @@ -2539,8 +4351,7 @@ function classify(error: ReportedError | undefined): Failure { const seed = /seed: (-?\d+)/.exec(outer)?.[1] const path = /path: "([\d:]+)"/.exec(outer)?.[1] const counterexample = /Counterexample: (.*)/.exec(outer)?.[1] - // The rerun command as printed, to be pasted into a shell: the violation's, - // or, for a campaign unit's failure, the one its message prints. + // The violation's command, or the failing unit's command on its wrapper. const rerun = /^Rerun: (-t .*)$/m.exec(text)?.[1] ?? /^Rerun: (-t .*)$/m.exec(outer)?.[1] const violationRerun = @@ -2569,48 +4380,15 @@ function classify(error: ReportedError | undefined): Failure { : undefined // A data error names the value's path, then its kind. const data = /^(.*): (.*) is not (?:observation|history) data$/.exec(text) - const kind: Kind | `other` = - inner.name === `OracleDataError` - ? `data` - : text.includes(`the framework's encoding limit`) - ? `limit` - : inner.name === `OracleModelError` - ? `model-not-total` - : inner.name === `OracleViolation` - ? `violation` - : /timed out in \d+ms/.test(text) || - /timed out in \d+ms/.test(outer) - ? `timeout` - : text.includes(`did not enter`) && - text.includes(`: fixed history "`) - ? `fixed-reach` - : text.includes(`did not enter`) && - text.includes(`: constructed history "`) - ? `constructed-reach` - : text.includes(`no enumerated history entered`) - ? `enumerated-reach` - : text.includes(`which the property does not declare`) - ? `undeclared-state` - : text.includes(`two runs of`) - ? `determinism` - : text.includes(`a difference that is not text`) - ? `calibration-odd-difference` - : text.includes(`but the check passed it`) - ? `calibration-passed` - : text.includes(`but the check reported law`) - ? `calibration-wrong-law` - : text.includes( - `but the check reported difference`, - ) - ? `calibration-wrong-difference` - : text.includes(`found no history`) - ? `calibration-none` - : text.includes(`synthetic driver failed`) || - text.includes(`cannot read a 7`) - ? `setup` - : `other` + const kind = classifyKind(inner, text, outer) + const callbackContext = reportedCallbackContext(text, candidates) return { kind, + // Context rendering is not a deterministic difference; keep its decoded + // actual identity while raw reports/sidecars retain the original text. + message: callbackContext ? JSON.stringify(callbackContext) : text, + ...(callbackContext ? { callbackContext } : {}), + ...(inner.cause?.message ? { causeMessage: inner.cause.message } : {}), // A model error that quotes a machine path, such as an inspected stack. ...(inner.name === `OracleModelError` && text.includes(packageDirectory) ? { machinePath: true } @@ -2652,6 +4430,7 @@ function runVitest( shellTail, seeded = true, prefix, + warningPresentation, }: { env?: Record // Shell text before the command (variable assignments) and after it (a @@ -2662,6 +4441,8 @@ function runVitest( seeded?: boolean // The environment prefix every spec of the suite configures. prefix: Prefix + // The representation control alternates across repeated outer drives. + warningPresentation?: boolean }, ): Report { const report = join(directory, `report-${runCount++}.json`) @@ -2680,6 +4461,7 @@ function runVitest( `run`, `--coverage.enabled=false`, `--typecheck.enabled=false`, + `--maxWorkers=2`, `--reporter=${join(here, `oracle-property-self-oracle-reporter.mjs`)}`, // Each child scans only its own generated directory. The generated // directory sits outside `tests/`, so the package's default run never @@ -2717,10 +4499,52 @@ function runVitest( ) // The framework prints reach counts straight to stdout, past Vitest's // console capture, so they arrive in the child's own output. - const { files: reported, unhandled } = JSON.parse( - readFileSync(report, `utf8`), - ) as { files: Array; unhandled?: Array } - return { files: reported, stdout: run.stdout, unhandled: unhandled ?? [] } + const { + files: reported, + unhandled, + warnings, + } = JSON.parse(readFileSync(report, `utf8`)) as { + files: Array + unhandled?: Array + warnings?: Array + } + const presentedWarnings = (warnings ?? []).map((warning) => + warningPresentation === true ? reversedWarning(warning) : warning, + ) + const projectedWarnings = presentedWarnings.map(semanticWarning) + const warningPermutation = + warningPresentation === undefined + ? undefined + : { + lists: presentedWarnings.filter( + (warning) => reversedWarning(warning) !== warning, + ).length, + stable: + JSON.stringify([...projectedWarnings].sort()) === + JSON.stringify( + presentedWarnings + .map(reversedWarning) + .map(semanticWarning) + .sort(), + ), + } + if (warningPresentation !== undefined) + writeFileSync( + `${report}.presented-warnings.json`, + JSON.stringify({ + reversed: warningPresentation, + warnings: presentedWarnings, + }), + ) + return { + files: reported, + stdout: run.stdout, + warnings: projectedWarnings, + ...(warningPermutation ? { warningPermutation } : {}), + unhandled: unhandled ?? [], + exit: run.status, + signal: run.signal, + } } // A test's name within its property, without a nested spec's outer suite. @@ -2728,16 +4552,222 @@ const unnested = (name: string) => name.startsWith(`${outerSuite} > `) ? name.slice(outerSuite.length + 3) : name const short = (id: string, name: string) => unnested(name).replace(`${id} > `, ``) -// A slow spec's driver runs on after its test timed out and logs under -// whichever test runs then, which the machine's speed decides, so the driver -// records no log entry of a slow spec; the model judges none. +// A slow spec can start calls after its test times out. Attribution, order +// and count are then host-dependent. The deterministic observation keeps its +// late-error interval relation; the full raw log/report remain diagnostics. const settledLog = (plan: Plan, log: Array) => log.filter( ({ id }) => !plan.specs.some((spec) => spec.slow && spec.id === id), ) -// Each property's entries in the order its file ran them. The child runs -// spec files in parallel, so entries of two files interleave differently -// from run to run; a stable sort by id keeps each file's own order. +// Model-derived sufficient statistics of a raw process report. This relation +// combines actual started-call input evidence with actual late-error output. +// It imports no production policy: fast calls owe one error each, while slow +// calls may still be running when the reporter closes. Only the checker +// decides whether the observed deficit/excess violates the law. +type LateAgreement = { belowMinimum: number; aboveMaximum: number } +const lateMessage = `probe synthetic#saw-0 was called after its history ended. The driver must await the work it starts, so every probe call belongs to one history.` +const usesLateRelation = (plan: Plan) => + plan.specs.some((spec) => spec.slow && spec.probe === `late`) +// Timed-out tests owe no completed call count. For each logical owner, retain +// its first input and the relations of every actually started execution. A +// determinant resolves one shared plan; other executions resolve individually. +// A selected constructed calibration builds once for each actual execution. +// Thus an optional late second call changes no observation, while extra or +// missing resolution/build work, or changed selected input, remains visible. +type TimeoutInputAgreement = { + id: string + test: string + resolutionDifference: number + buildDifference: number + changedInput: boolean + changedBuild: boolean + excessDeterminismCalls: number + excessCalibrationCalls: number +} +const invocationObservation = ( + plan: Plan, + log: Array, + unhandled: Array, + builds: Array = [], + resolutions: SuiteRun[`resolutions`] = [], +): Pick< + ChildRun, + | `log` + | `unhandled` + | `lateAgreement` + | `builds` + | `resolutions` + | `timeoutInputs` +> => { + const slow = (id: string) => + plan.specs.some((spec) => spec.id === id && spec.slow) + const firstInputs = ( + entries: Array, + ) => { + const seen = new Set() + return entries + .filter((entry) => { + if (!slow(entry.id) || entry.test === ``) return true + const key = JSON.stringify([entry.id, entry.test, entry.name]) + if (seen.has(key)) return false + seen.add(key) + return true + }) + .sort( + (a, b) => + a.id.localeCompare(b.id) || + (slow(a.id) ? a.test.localeCompare(b.test) : 0), + ) + } + const timeoutInputs: Array = [] + for (const spec of plan.specs.filter((candidate) => candidate.slow)) { + const tests = new Set( + [...log, ...builds, ...resolutions] + .filter(({ id, test }) => id === spec.id && test !== ``) + .map(({ test }) => test), + ) + for (const test of [...tests].sort()) { + const name = short(spec.id, test) + const calls = log.filter( + (entry) => entry.id === spec.id && entry.test === test, + ) + const constructed = builds.filter( + (entry) => entry.id === spec.id && entry.test === test, + ) + const resolved = resolutions.filter( + (entry) => entry.id === spec.id && entry.test === test, + ) + const forced = forcedCalibrationNames(spec).indexOf(name) + const calibration = spec.calibrations.find( + (entry) => `calibration: ${entry.name}` === name, + ) + const binding = + forced >= 0 && spec.probeBinding && spec.probeBinding !== `empty` + ? selectedControlBinding(spec, forced === 0) + : calibration + ? bindingOf(spec, calibration) + : undefined + const deterministic = name.startsWith(`determinism`) + const same = (a: unknown, b: unknown) => + JSON.stringify(a) === JSON.stringify(b) + timeoutInputs.push({ + id: spec.id, + test, + resolutionDifference: + spec.isolatedPlans || spec.identityHistories + ? resolved.length - + (deterministic ? Number(calls.length > 0) : calls.length) + : 0, + buildDifference: binding + ? constructed.length - (`constructed` in binding ? calls.length : 0) + : 0, + changedInput: Boolean( + (binding || deterministic) && + calls.some( + (entry) => + !same( + [entry.history, entry.origin], + [calls[0]!.history, calls[0]!.origin], + ), + ), + ), + changedBuild: Boolean( + binding && + constructed.some( + (entry) => + !same( + [entry.name, entry.generated, entry.history], + [ + constructed[0]!.name, + constructed[0]!.generated, + constructed[0]!.history, + ], + ), + ), + ), + excessDeterminismCalls: deterministic + ? Math.max(0, calls.length - 2) + : 0, + excessCalibrationCalls: binding + ? Math.max( + 0, + calls.length - + (forced >= 0 || + (calibration && driverCalibration(calibration.kind)) + ? 2 + : 1), + ) + : 0, + }) + } + } + const owed = lateCalls(plan, log) + const actual = unhandled.filter((message) => message === lateMessage).length + return { + log: firstInputs(log).map((entry) => { + const projected = { ...entry } + delete projected.runnerTest + return projected + }), + builds: firstInputs(builds), + resolutions: firstInputs(resolutions), + ...(plan.specs.some((spec) => spec.slow) ? { timeoutInputs } : {}), + // Only this known late-probe multiplicity has a timing-dependent bound. + // Unrelated errors remain exact, including similar but nonidentical text. + unhandled: usesLateRelation(plan) + ? unhandled.filter((message) => message !== lateMessage) + : unhandled, + ...(usesLateRelation(plan) + ? { + lateAgreement: { + belowMinimum: Math.max(0, owed.min - actual), + aboveMaximum: Math.max(0, actual - owed.max), + }, + } + : {}), + } +} +const timeoutInputsMatch = ( + plan: Plan, + seen: Pick, +) => { + if (!plan.specs.some((spec) => spec.slow)) return true + const owners = [ + ...new Set( + [...seen.log, ...seen.builds, ...seen.resolutions] + .filter( + ({ id, test }) => + test !== `` && + plan.specs.some((spec) => spec.id === id && spec.slow), + ) + .map(({ id, test }) => JSON.stringify([id, test])), + ), + ].sort() + return ( + seen.timeoutInputs !== undefined && + JSON.stringify( + seen.timeoutInputs + .map(({ id, test }) => JSON.stringify([id, test])) + .sort(), + ) === JSON.stringify(owners) && + seen.timeoutInputs.every( + (entry) => + entry.resolutionDifference === 0 && + entry.buildDifference === 0 && + !entry.changedInput && + !entry.changedBuild && + entry.excessDeterminismCalls === 0 && + entry.excessCalibrationCalls === 0, + ) + ) +} +const lateAgreementMatches = ( + plan: Plan, + seen: { lateAgreement?: LateAgreement }, +) => + !usesLateRelation(plan) || + (seen.lateAgreement?.belowMinimum === 0 && + seen.lateAgreement.aboveMaximum === 0) const readLog = (log: string): Array => existsSync(log) ? readFileSync(log, `utf8`) @@ -2748,7 +4778,37 @@ const readLog = (log: string): Array => const entry = JSON.parse(line) as LogEntry return { ...entry, test: unnested(entry.test) } }) - .sort((a, b) => (a.id < b.id ? -1 : a.id > b.id ? 1 : 0)) + // Vitest may run files concurrently. Each property's call order is + // observable; interleaving independent properties is not a framework + // promise. Stable grouping retains every call and each local order. + .sort((a, b) => a.id.localeCompare(b.id)) + : [] +// A wrong-answer control changes every ordinary-run representation of its +// invocation. Summary and full child observations are separate data objects; +// either valid projection must retain the same deliberately wrong origin. +const ordinaryInvocationLogs = ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, +): Array> => + trace.flatMap(({ checkpoint, value }) => + checkpoint === `run` + ? [(value as SuiteRun).log] + : checkpoint === `child` && (value as ChildRun).context.kind === `run` + ? [(value as ChildRun).log] + : [], + ) +// Resolve callbacks may create resources even when no driver receives the +// result. Their separate input-side log preserves extra and omitted calls. +const readResolutions = (log: string): SuiteRun[`resolutions`] => + existsSync(log) + ? readFileSync(log, `utf8`) + .trim() + .split(`\n`) + .filter(Boolean) + .map((line) => { + const entry = JSON.parse(line) as Pick + return { ...entry, test: unnested(entry.test) } + }) + .sort((a, b) => a.id.localeCompare(b.id)) : [] // The items of a counterexample that fast-check printed, in any form: the // first bracketed list of integers, where the lossy form's `Number.NaN` is a @@ -2775,6 +4835,50 @@ const statesIn = (report: Report, plan: Plan, files: Array) => ), ) +const readBuilds = (log: string): Array => + existsSync(`${log}.builds`) + ? readFileSync(`${log}.builds`, `utf8`) + .trim() + .split(`\n`) + .filter(Boolean) + .map((line) => { + const entry = JSON.parse(line) as BuildEntry + return { ...entry, test: unnested(entry.test) } + }) + // Independent files may finish in either order. Keep every build and + // each property's local order, as the invocation log does. + .sort((a, b) => a.id.localeCompare(b.id)) + : [] + +// Only semantic diagnostic facts enter the outer observation. Host stacks, +// temporary paths, elapsed times and a campaign wrapper's random seed are not +// promised failure differences. Error multiplicity remains observable. +const diagnosticFacts = ({ + kind, + law, + checkpoint, + diagnosis, + cause, + causeMessage, + callbackContext, + counterexample, + dataPath, + dataKind, +}: Failure): Failure => ({ + kind, + ...(law === undefined ? {} : { law }), + ...(checkpoint === undefined ? {} : { checkpoint }), + ...(diagnosis === undefined ? {} : { diagnosis }), + ...(cause === undefined ? {} : { cause }), + ...(causeMessage === undefined ? {} : { causeMessage }), + ...(callbackContext === undefined ? {} : { callbackContext }), + ...(counterexample !== undefined && counterexampleItems(counterexample) + ? { counterexampleHistory: counterexampleItems(counterexample)! } + : {}), + ...(dataPath === undefined ? {} : { dataPath }), + ...(dataKind === undefined ? {} : { dataKind }), +}) + // The variables a printed replay command sets, by name: each `NAME=value` // word before `vitest run`, a value single-quoted as a POSIX shell quotes it. const replayParts = ( @@ -2844,34 +4948,329 @@ function environmentOf(plan: Plan): Record { ) } -async function driveSuite( +function driveSuite( plan: Plan, observe: (checkpoint: string, value: unknown) => void, ) { - const directory = join( - packageDirectory, - `.self-oracle-${process.pid}-${runCount++}`, - ) - const { prefix } = plan - mkdirSync(directory, { recursive: true }) - try { - const log = join(directory, `log.jsonl`) - const files = plan.specs.map((spec, index) => { - const file = join(directory, `spec-${index}-oracle.test.ts`) - writeFileSync( - file, - `import { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\nsyntheticProperty(${JSON.stringify(spec)})\n`, - ) - return file - }) - const report = runVitest(directory, files, { + const timeoutPresentation = plan.timeoutPresentation + ? timeoutPresentationCount++ % 2 === 0 + : false + const presentation = + plan.callbackPresentation === `alternate` + ? callbackPresentationCount++ % 2 === 0 + ? { + calibration: `bracketed` as const, + seed: `parenthesized` as const, + spacing: `padded` as const, + } + : { + calibration: `quoted` as const, + seed: `bracketed` as const, + spacing: `plain` as const, + } + : plan.callbackPresentation + const presentedDiagnostics: Array<{ before: string; after: string }> = [] + let reasonInputCount = 0 + let wrapperInputCount = 0 + const classifyForSuite = ( + error: ReportedError | undefined, + nameLog?: ReadonlyArray, + ): Failure => { + const candidates = callbackCandidateUniverse( + plan, + nameLog ?? readLog(join(directory, `log.jsonl`)), + ) + const reasoned = incomingCallbackReason( + error, + plan.callbackReason, + candidates, + ) + if ( + reasoned?.message && + error?.message && + reasoned.message !== error.message + ) { + writeFileSync( + join(directory, `callback-reason-${reasonInputCount++}.json`), + JSON.stringify( + { before: error.message, after: reasoned.message }, + null, + 2, + ), + ) + } + const wrapped = incomingCallbackWrapper(reasoned, plan.callbackWrapperName) + if (wrapped?.name !== reasoned?.name) + writeFileSync( + join(directory, `callback-wrapper-${wrapperInputCount++}.json`), + JSON.stringify( + { + before: reasoned?.name, + after: wrapped?.name, + message: wrapped?.message, + cause: wrapped?.cause, + }, + null, + 2, + ), + ) + const incoming = incomingCallbackError( + wrapped, + plan.callbackIncomingPresentation, + candidates, + ) + const { error: presented, applied } = presentCallbackError( + incoming, + presentation, + candidates, + ) + if (applied && incoming?.message && presented?.message) { + presentedDiagnostics.push({ + before: incoming.message, + after: presented.message, + }) + writeFileSync( + join(directory, `callback-presentations.json`), + JSON.stringify(presentedDiagnostics, null, 2), + ) + } + return classify(presented, candidates) + } + const warningPresentation = plan.permuteWarnings + ? warningPresentationCount++ % 2 === 0 + : undefined + const directory = join( + packageDirectory, + `.self-oracle-${process.pid}-${runCount++}`, + ) + const { prefix } = plan + mkdirSync(directory, { recursive: true }) + try { + const receive = ( + context: ChildContext, + ...args: Parameters + ): Report => { + const options = args[2] + const childLog = + options.env?.SELF_ORACLE_LOG ?? + join(directory, `child-${runCount}.jsonl`) + const report = runVitest(args[0], args[1], { + ...options, + env: { ...options.env, SELF_ORACLE_LOG: childLog }, + warningPresentation, + }) + if (report.warningPermutation) + observe(`warning-permutation`, { + context, + ...report.warningPermutation, + }) + const rawLog = readLog(childLog) + // These calls have already outlived their owning test. Reordering them + // and changing the current runner name leaves every timeout obligation + // unchanged; the receiving report and late-error capacity stay intact. + const slowCalls = rawLog.filter(({ id }) => + plan.specs.some((spec) => spec.id === id && spec.slow), + ) + const presentedLog = timeoutPresentation + ? [ + ...settledLog(plan, rawLog), + ...[...new Set(slowCalls.map((entry) => entry.test))] + .reverse() + .flatMap((test) => + slowCalls + .filter((entry) => entry.test === test) + .map((entry) => ({ + ...entry, + runnerTest: `${entry.id} > post-timeout continuation`, + })), + ), + ] + : rawLog + const rawUnhandled = report.unhandled.map(({ message }) => message ?? ``) + const rawBuilds = readBuilds(childLog) + const rawResolutions = readResolutions(`${childLog}.resolutions`) + const cutLog = [...presentedLog] + const cutBuilds = [...rawBuilds] + const cutResolutions = [...rawResolutions] + let cutExercised = false + for (const spec of plan.specs.filter((candidate) => candidate.slow)) { + for (const test of new Set( + slowCalls + .filter(({ id }) => id === spec.id) + .map((entry) => entry.test), + )) { + const name = short(spec.id, test) + const calibration = spec.calibrations.find( + (entry) => `calibration: ${entry.name}` === name, + ) + const forced = Boolean( + spec.probeBinding && + spec.probeBinding !== `empty` && + forcedCalibrationNames(spec).includes(name), + ) + const bound = forced || (calibration && bindingOf(spec, calibration)) + const calls = cutLog.filter( + (entry) => entry.id === spec.id && entry.test === test, + ) + if (calls.length < 2 || (!name.startsWith(`determinism`) && !bound)) + continue + // Only the optional continuation after the first slow call is cut. + // Its paired input work is omitted at the same legal reporting cut. + cutLog.splice(cutLog.indexOf(calls.at(-1)!), 1) + if (!name.startsWith(`determinism`)) { + const resolved = cutResolutions.lastIndexOf( + [...cutResolutions] + .reverse() + .find((entry) => entry.id === spec.id && entry.test === test)!, + ) + if (resolved >= 0) cutResolutions.splice(resolved, 1) + const built = cutBuilds.lastIndexOf( + [...cutBuilds] + .reverse() + .find((entry) => entry.id === spec.id && entry.test === test)!, + ) + if (built >= 0) cutBuilds.splice(built, 1) + } + cutExercised = true + } + } + const cutBounds = lateCalls(plan, cutLog) + const actualLate = rawUnhandled.filter( + (message) => message === lateMessage, + ).length + let keptLate = 0 + const cutLate = Math.max( + cutBounds.min, + Math.min(cutBounds.max, actualLate), + ) + const cutUnhandled = rawUnhandled.filter( + (message) => message !== lateMessage || ++keptLate <= cutLate, + ) + const rawObservation = invocationObservation( + plan, + rawLog, + rawUnhandled, + rawBuilds, + rawResolutions, + ) + const cutObservation = invocationObservation( + plan, + cutLog, + cutUnhandled, + cutBuilds, + cutResolutions, + ) + const observation = timeoutPresentation ? cutObservation : rawObservation + // A diagnostic is deliberately outside the deterministic trace; no raw + // evidence is destroyed by its semantic projection. + if (slowCalls.length || usesLateRelation(plan)) { + const diagnostics = join( + tmpdir(), + `oracle-property-timeout-${process.pid}`, + ) + mkdirSync(diagnostics, { recursive: true }) + writeFileSync( + join(diagnostics, `${runCount}-${context.kind}.json`), + JSON.stringify({ + context, + log: rawLog, + presentedLog, + builds: rawBuilds, + resolutions: rawResolutions, + cutLog, + cutBuilds, + cutResolutions, + cutUnhandled, + report, + }), + ) + } + if (plan.timeoutPresentation && context.kind === `run`) + observe(`timeout-presentation`, { + exercised: slowCalls.length > 0, + cutExercised, + stable: + JSON.stringify(rawObservation) === JSON.stringify(cutObservation), + }) + const presentationsBeforeChild = presentedDiagnostics.length + observe(`child`, { + context, + ...observation, + // Independent property files may register in either order. + warnings: [...report.warnings].sort(), + tests: report.files + .flatMap(({ file, tests }) => { + const id = plan.specs[files.indexOf(file)]?.id ?? `unknown` + return tests.map((test) => ({ + id, + name: short(id, test.name), + state: test.state, + errors: test.errors.map((error) => + diagnosticFacts(classifyForSuite(error, readLog(childLog))), + ), + seedLogs: seedLogsOf(test, id), + })) + }) + .sort((a, b) => a.id.localeCompare(b.id)), + loads: report.files.map(({ file, errors }) => ({ + id: plan.specs[files.indexOf(file)]?.id ?? `unknown`, + errors: errors.map(({ message }) => message ?? ``), + causes: errors.map(({ cause }) => cause?.message ?? ``), + })), + ...(presentation + ? { + callbackPresented: + presentedDiagnostics.length > presentationsBeforeChild, + } + : {}), + exit: report.exit, + signal: report.signal, + files: report.files.map(({ file }) => + file.replace(`${directory}/`, ``), + ), + expectedFiles: args[1].map((file) => file.replace(`${directory}/`, ``)), + // Synthetic tests owe one primary failure; causes retain its nested + // diagnostic chain. Separate teardown/hook failures must stay visible. + secondaryErrors: report.files.flatMap(({ tests }) => + tests.flatMap(({ name, errors }) => + errors + .slice(1) + .map(({ message }) => ({ test: name, message: message ?? `` })), + ), + ), + failed: report.files.some( + (file) => + file.errors.length > 0 || + file.tests.some(({ state }) => state === `failed`), + ), + } satisfies ChildRun) + return report + } + const log = join(directory, `log.jsonl`) + const files = plan.specs.map((spec, index) => { + const file = join(directory, `spec-${index}-oracle.test.ts`) + writeFileSync( + file, + `import { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\nsyntheticProperty(${JSON.stringify(spec)})\n`, + ) + return file + }) + const report = receive({ kind: `run` }, directory, files, { env: { SELF_ORACLE_LOG: log }, prefix, }) const run: SuiteRun = { files: {}, - log: settledLog(plan, readLog(log)), - unhandled: report.unhandled.map((error) => error.message ?? ``), + ...invocationObservation( + plan, + readLog(log), + report.unhandled.map((error) => error.message ?? ``), + readBuilds(log), + readResolutions(`${log}.resolutions`), + ), + // Independent files may finish in either order; every line remains. + stdout: report.stdout.split(`\n`).sort().join(`\n`), + warnings: [...report.warnings].sort(), } for (const [index, spec] of plan.specs.entries()) { const reported = report.files.find((file) => file.file === files[index]) @@ -2894,7 +5293,7 @@ async function driveSuite( name: short(spec.id, test.name), state: test.state, ...(test.state === `failed` - ? { failure: classify(test.errors[0]) } + ? { failure: classifyForSuite(test.errors[0]) } : {}), seedLogs: seedLogsOf(test, spec.id), })), @@ -2902,10 +5301,159 @@ async function driveSuite( } } observe(`run`, run) + // Adding a calibration ahead of another must not change the latter's seed + // or built history. This is a metamorphic comparison of driver inputs, + // independent of the framework's particular seed hash. + for (const [index, spec] of plan.specs.entries()) { + if ( + !spec.calibrations.some((calibration) => { + const binding = bindingOf(spec, calibration) + return binding && `constructed` in binding + }) || + !run.files[spec.id]!.loaded + ) + continue + const file = files[index]! + const original = readFileSync(file, `utf8`) + const reorderedLog = join(directory, `reordered-${index}.jsonl`) + try { + const reordered = { + ...spec, + calibrations: [ + { name: `unrelated calibration`, kind: `right` }, + ...[...spec.calibrations].reverse(), + ], + } + writeFileSync( + file, + `import { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\nsyntheticProperty(${JSON.stringify(reordered)})\n`, + ) + receive({ kind: `binding-reordered`, id: spec.id }, directory, [file], { + env: { SELF_ORACLE_LOG: reorderedLog }, + shellTail: `-t 'calibration:'`, + prefix, + }) + observe(`binding-reordered`, { + id: spec.id, + log: invocationObservation( + plan, + readLog(reorderedLog), + [], + readBuilds(reorderedLog), + readResolutions(`${reorderedLog}.resolutions`), + ).log, + }) + } finally { + writeFileSync(file, original) + } + } + for (const [index, spec] of plan.specs.entries()) { + if (!spec.warningProbe || !run.files[spec.id]!.loaded) continue + const file = files[index]!, + original = readFileSync(file, `utf8`) + // These probes supply the enclosing suites themselves. Register the + // property synchronously inside each phase's callback so its environment + // cannot change before a deferred nested describe runs. + const call = `syntheticProperty(${JSON.stringify({ ...spec, nested: false })})` + try { + // Repeated ordinary registrations share one process/id, as do a collector + // followed by ordinary registration. Tests are collected but not run. + writeFileSync( + file, + `import { describe } from 'vitest'\nimport { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\ndescribe('first',()=>{${call}})\ndescribe('second',()=>{${call}})\n`, + ) + const repeated = receive( + { + kind: `registration-warnings`, + id: spec.id, + phases: [`first`, `second`], + }, + directory, + [file], + { shellTail: `-t '__registration_only__'`, prefix }, + ) + const manifest = join(directory, `manifest-${index}.jsonl`) + writeFileSync( + file, + `import { describe } from 'vitest'\nimport { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\ndescribe('collect',()=>{process.env[${JSON.stringify(`ORACLE_MANIFEST`)}]=${JSON.stringify(manifest)};try{${call}}finally{delete process.env[${JSON.stringify(`ORACLE_MANIFEST`)}]}})\ndescribe('ordinary',()=>{${call}})\n`, + ) + const collectThenRun = receive( + { + kind: `registration-warnings`, + id: spec.id, + phases: [`collect`, `ordinary`], + }, + directory, + [file], + { shellTail: `-t '__registration_only__'`, prefix }, + ) + const collectedManifest = join( + directory, + `collected-manifest-${index}.jsonl`, + ) + writeFileSync( + file, + `import { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\n${call}\n`, + ) + const collected = receive( + { kind: `registration-warnings`, id: spec.id, phases: [] }, + directory, + [file], + { + env: { ORACLE_MANIFEST: collectedManifest }, + shellTail: `-t '__registration_only__'`, + prefix, + }, + ) + const loaded = (phase: Report) => + phase.files.length === 1 && + phase.files[0]!.file === file && + phase.files[0]!.errors.length === 0 && + phase.unhandled.length === 0 + const registered = (path: string) => + existsSync(path) && + readFileSync(path, `utf8`) + .trim() + .split(`\n`) + .filter(Boolean) + .map((line) => (JSON.parse(line) as { id: string }).id) + observe(`registration-warnings`, { + manifests: { + collectThenRun: registered(manifest), + collected: registered(collectedManifest), + }, + tests: { + repeated: repeated.files + .flatMap(({ tests }) => tests.map(({ name }) => name)) + .sort(), + collectThenRun: collectThenRun.files + .flatMap(({ tests }) => tests.map(({ name }) => name)) + .sort(), + collected: collected.files + .flatMap(({ tests }) => tests.map(({ name }) => name)) + .sort(), + }, + loaded: { + repeated: loaded(repeated), + collectThenRun: loaded(collectThenRun), + collected: loaded(collected), + }, + id: spec.id, + repeated: repeated.warnings, + collectThenRun: collectThenRun.warnings, + collected: collected.warnings, + }) + } finally { + writeFileSync(file, original) + } + } if (plan.seedless) { const seedlessLog = join(directory, `seedless-${runCount}.jsonl`) - const seedless = runVitest(directory, files, { - env: { SELF_ORACLE_LOG: seedlessLog }, + const seedless = receive({ kind: `seedless` }, directory, files, { + env: { + SELF_ORACLE_LOG: seedlessLog, + SELF_ORACLE_RANDOM_DRAW: String(plan.seedlessDraw ?? 0.5), + }, seeded: false, prefix, }) @@ -2954,13 +5502,18 @@ async function driveSuite( const probed = randomUnit(unitOf(unit.name)!.name) const replayLog = join(directory, `replay-${runCount}.jsonl`) const replay = parts - ? runVitest(directory, files, { - env: { SELF_ORACLE_LOG: replayLog }, - shellHead: parts.head, - shellTail: parts.tail, - seeded: false, - prefix, - }) + ? receive( + { kind: `replay`, id: spec.id, target: unit.name, probed }, + directory, + files, + { + env: { SELF_ORACLE_LOG: replayLog }, + shellHead: parts.head, + shellTail: parts.tail, + seeded: false, + prefix, + }, + ) : undefined const index = plan.specs.indexOf(spec) const random = replay?.files @@ -2977,7 +5530,7 @@ async function driveSuite( probed, states: replay ? statesIn(replay, plan, files) : [], ...(random?.state === `failed` - ? { failure: classify(random.errors[0]) } + ? { failure: classifyForSuite(random.errors[0]) } : {}), ran: replayed.map((entry) => entry.history), routes: replayed.map((entry) => entry.route), @@ -2998,10 +5551,20 @@ async function driveSuite( for (const target of targets) { if (!target) continue const index = plan.specs.indexOf(spec) - const rerun = runVitest(directory, [files[index]!], { - shellTail: target.failure!.rerun, - prefix, - }) + const rerun = receive( + { + kind: `rerun`, + id: spec.id, + target: target.name, + probed: target.name, + }, + directory, + [files[index]!], + { + shellTail: target.failure!.rerun, + prefix, + }, + ) const ran = rerun.files[0]?.tests.find( (test) => test.state === `failed`, ) @@ -3011,14 +5574,14 @@ async function driveSuite( original: target.failure!, probed: target.name, states: statesIn(rerun, plan, files), - ...(ran ? { failure: classify(ran.errors[0]) } : {}), + ...(ran ? { failure: classifyForSuite(ran.errors[0]) } : {}), } satisfies ProbeRun) } } // environment: every file once under the drawn replay environment. if (plan.environment) { const environmentLog = join(directory, `environment-${runCount}.jsonl`) - const under = runVitest(directory, files, { + const under = receive({ kind: `environment` }, directory, files, { env: { ...environmentOf(plan), SELF_ORACLE_LOG: environmentLog }, seeded: false, prefix, @@ -3040,7 +5603,7 @@ async function driveSuite( name: short(spec.id, test.name), state: test.state, ...(test.state === `failed` - ? { failure: classify(test.errors[0]) } + ? { failure: classifyForSuite(test.errors[0]) } : {}), })), }, @@ -3050,6 +5613,75 @@ async function driveSuite( log: settledLog(plan, readLog(environmentLog)), } satisfies EnvironmentRun) } + if (plan.hostRngDraw !== undefined) { + const draw = plan.hostRngDraw + const forwarding = + plan.hostRngForwarding === `alternate` + ? hostForwardingCount++ % 2 === 0 + ? 1 + : 2 + : (plan.hostRngForwarding ?? 0) + const hostSeed = ( + mode: `outer` | `child`, + ordinal: number, + ): HostSeed | null => { + const file = join( + directory, + `host-seed-${mode}-${ordinal}-diagnostic.test.ts`, + ) + const output = join(directory, `host-seed-${mode}-${ordinal}.json`) + // Dynamic import gives the host draw and marker their intended cut: + // before fixture evaluation, rather than after its side effects. + writeFileSync( + file, + `import { test } from 'vitest' +import { writeFileSync } from 'node:fs' +let hostCalls=0 +const hostRandom=()=>{hostCalls++;return ${draw}} +Math.random=hostRandom +${mode === `outer` ? `delete process.env.SELF_ORACLE_LOG\n` : ``}await import(${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}) +const forwardingCalls=${mode === `outer` ? forwarding : 0} +let forwardingInvocations=0 +if(forwardingCalls>0){const retainedRandom=Math.random;Math.random=()=>{forwardingInvocations++;const value=retainedRandom();if(forwardingCalls===2)retainedRandom();return value}} +const postImportDraw=Math.random() +const { oraclePropertyOptions }=await import(${JSON.stringify(join(packageDirectory, `src/oracle-config.ts`))}) +test('host seed diagnostic',()=>{ + const hostCallsBeforeSeed=hostCalls + const forwardingBeforeSeed=forwardingInvocations + const {seed}=oraclePropertyOptions(1,'self-oracle.framework') + const hostCallsAfterSeed=hostCalls + const forwardingAfterSeed=forwardingInvocations + writeFileSync(${JSON.stringify(output)},JSON.stringify({seed,childMarker:process.env.SELF_ORACLE_LOG!==undefined,draw:postImportDraw,hostUsed:hostCallsAfterSeed>hostCallsBeforeSeed,forwarded:forwardingAfterSeed>forwardingBeforeSeed,hostCallsBeforeSeed,hostCallsAfterSeed,forwardingCalls,forwardingBeforeSeed,forwardingAfterSeed})) +}) +`, + ) + receive( + { kind: `host-scope`, mode, ordinal, draw }, + directory, + [file], + { + seeded: false, + prefix, + }, + ) + if (!existsSync(output)) return null + const actual = JSON.parse(readFileSync(output, `utf8`)) as HostSeed + // Raw diagnostic JSON retains counter magnitudes and wrapper details. + // Only public draw/use/seed facts enter the deterministic observation. + return { + seed: actual.seed, + childMarker: actual.childMarker, + draw: actual.draw, + hostUsed: actual.hostUsed, + forwarded: actual.forwarded, + } + } + observe(`host-scope`, { + draw, + outer: hostSeed(`outer`, 0), + children: [hostSeed(`child`, 0), hostSeed(`child`, 1)], + } satisfies HostScopeRun) + } // reach: for each loaded enumerated source that declares states, Vitest // runs that file again with a `-t` pattern that selects only its reach // test, so the reach test judges histories that no test of the run @@ -3060,10 +5692,15 @@ async function driveSuite( const escape = (text: string) => text.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`) const pattern = `^${[...(spec.nested ? [outerSuite] : []), spec.id, `reach`].map(escape).join(`(?: | > )`)}$` - const alone = runVitest(directory, [files[index]!], { - shellTail: `-t ${shellWord(pattern)}`, - prefix, - }) + const alone = receive( + { kind: `reach-alone`, id: spec.id }, + directory, + [files[index]!], + { + shellTail: `-t ${shellWord(pattern)}`, + prefix, + }, + ) const test = alone.files[0]?.tests.find( (candidate) => short(spec.id, candidate.name) === `reach`, ) @@ -3075,7 +5712,7 @@ async function driveSuite( id: spec.id, state: test?.state ?? `missing`, ...(test?.state === `failed` - ? { failure: classify(test.errors[0]) } + ? { failure: classifyForSuite(test.errors[0]) } : {}), ...(printed ? { reach: JSON.parse(printed) as unknown } : {}), others: statesIn(alone, plan, files).filter( @@ -3095,21 +5732,17 @@ async function driveSuite( // // For each property: it loads exactly when the model says so; it registers // exactly the owed tests; each test passes, or fails with the owed kind, law, -// and cause; a unit's counterexample still violates the owed law; each unit -// ran the stream the model draws (law `units`); a passing determinism unit -// ran its history twice; and each passing unit printed the owed reach counts. -// Then the probes: the replays and reruns the model owes all happened; a -// replay command names the seed, path, property, and unit under the suite's -// prefix; in each, every other test stood aside; the probed test failed again -// the same way; and a replay started from the counterexample itself, as a -// direct replay must, and reported the same one. Last, the seedless run's -// random units ran the seed they logged, and a run under a drawn replay -// environment failed or selected as law `replay-config` owes. +// and cause; a campaign's counterexample still violates the owed law; a passing +// determinism test ran every history it owes; and each passing campaign printed +// the owed reach counts. Then the probes: the replays and reruns the model +// owes all happened; in each, every other test stood aside; the probed test +// failed again the same way; and a replay started from the counterexample +// itself, as a direct replay must, and reported the same one. const lawOfTest = (name: string) => name.startsWith(`calibration:`) ? `calibration` - : name.startsWith(`determinism`) + : name === `determinism` || name.startsWith(`determinism:`) ? `determinism` : name === `reach` ? `reach` @@ -3127,6 +5760,137 @@ const lawOfKind = (kind: string, name: string) => ? `reach` : lawOfTest(name) +const matchesOutcome = ( + outcome: Outcome, + failure: Failure | undefined, + kind: string, + campaign: boolean, +) => + kind === outcome.kind && + (outcome.law === undefined || failure?.law === outcome.law) && + (outcome.checkpoint === undefined || + failure?.checkpoint === outcome.checkpoint) && + failure?.diagnosis === outcome.diagnosis && + failure?.dataPath === outcome.dataPath && + failure?.dataKind === outcome.dataKind && + (campaign || failure?.cause === outcome.cause) && + (campaign || + (outcome.kind === `violation` + ? failure?.causeMessage === outcome.causeMessage + : outcome.causeMessage === undefined || + failure?.causeMessage === outcome.causeMessage)) + +// Bound callback errors owe the calibration and selected history names (original +// authorized review item 6), plus the callback's original cause. Authored bindings +// decide identity; reported callback text never decides which input was selected. +// Opaque default enumeration labels use the original direct history test's public +// name associated with its authored index/origin. Missing or ambiguous associations +// fail this relation. This does not prescribe the framework's data-label codec. +const callbackExpectation = ( + spec: SyntheticSpec, + test: string, + outcome: Outcome, + nameLog: ReadonlyArray, +) => { + const calibration = spec.calibrations.find( + ({ name }) => test === `calibration: ${name}`, + ) + const binding = calibration && bindingOf(spec, calibration) + const invalidData = dataOutcome(spec) + if ( + !calibration || + !binding || + outcome.kind === `pass` || + !( + (outcome.kind === `setup` && + [`corrupt-throws`, `judge-throws`, `judge-driver-throws`].includes( + calibration.kind, + )) || + (spec.odd && invalidData) + ) + ) + return undefined + const kind = + `constructed` in binding + ? `constructed` + : `fixed` in binding + ? `fixed` + : `enumerated` + let selectedName: string | undefined + if (`constructed` in binding) selectedName = binding.constructed + else if (`fixed` in binding) selectedName = binding.fixed + else if (spec.source.kind === `enumerated`) { + const index = spec.source.histories.findIndex((history, at) => + typeof binding.enumerated === `number` + ? at === binding.enumerated + : authoredEnumerationLabel(spec, history) === binding.enumerated, + ) + const history = spec.source.histories[index] + if (history) { + selectedName = authoredEnumerationLabel(spec, history) + if (selectedName === undefined) { + const originalNames = [ + ...new Set( + nameLog + .filter( + (entry) => + entry.id === spec.id && + entry.origin === `enumerated:${index}` && + !frameworkTest(entry.test.slice(spec.id.length + 3)), + ) + .map((entry) => entry.test.slice(spec.id.length + 3)), + ), + ] + if (originalNames.length === 1) selectedName = originalNames[0] + } + } + } + return { + context: { calibration: calibration.name, kind, name: selectedName }, + causeMessage: + calibration.kind === `corrupt-throws` + ? `synthetic corruption failed` + : [`judge-throws`, `judge-driver-throws`].includes(calibration.kind) + ? `synthetic judgment failed` + : undefined, + causeIncludes: + invalidData?.kind === `data` + ? [invalidData.dataPath!, invalidData.dataKind!] + : invalidData?.kind === `limit` + ? [`the framework's encoding limit`] + : [], + } +} +const callbackDiagnosticMatches = ( + diagnostic: ReturnType, + failure: Failure | undefined, +) => + !diagnostic || + (diagnostic.context.name !== undefined && + failure?.callbackContext?.calibration === diagnostic.context.calibration && + failure.callbackContext.kind === diagnostic.context.kind && + failure.callbackContext.name === diagnostic.context.name && + (diagnostic.causeMessage !== undefined + ? failure.causeMessage === diagnostic.causeMessage + : diagnostic.causeIncludes.length > 0 && + diagnostic.causeIncludes.every((part) => + failure.causeMessage?.includes(part), + ))) + +// A concrete campaign candidate owes its own diagnostics, not those of the +// first failing history before shrinking. Missing a construction's state is +// separate from the candidate's row-value law and can have a passing trace. +const campaignCandidateOutcome = ( + spec: SyntheticSpec, + concrete: SyntheticHistory, + campaign: Outcome, +): Outcome => { + const precise = dataOutcome(spec) ?? judged(spec, concrete) + return campaign.kind === `constructed-reach` && precise.kind === `pass` + ? campaign + : precise +} + // Whether a unit's history fails it: a data error, a failing verdict, or, in // a constructed unit, a state its construction names and does not enter. const failsUnit = ( @@ -3189,199 +5953,1108 @@ function environmentError(plan: Plan): string | undefined { return undefined } -function checkSuite( +// A replay configuration is checked before a declaration's UNIT selection. +// A declaration still owes its own first error before a valid environment can +// select its generated unit. This account is shared by every receiving child. +function environmentSelection(plan: Plan, spec: SyntheticSpec) { + const error = environmentError(plan) + const drawn = plan.environment! + const replay = drawn.path === `valid` && error === undefined + const selected = replay && plan.specs[0]!.id === spec.id && !spec.unregistered + const unit = environmentOf(plan)[`${plan.prefix}_UNIT`] + const replayError = !selected + ? undefined + : spec.source.kind === `enumerated` + ? unit === undefined + ? undefined + : `${spec.id}: ${plan.prefix}_UNIT names "${unit}"` + : unit === undefined + ? `${spec.id}: ${plan.prefix}_PATH of a generated source requires ${plan.prefix}_UNIT` + : unitNames(spec).map(randomUnit).includes(unit) + ? undefined + : `${spec.id}: ${plan.prefix}_UNIT names "${unit}"` + return { error, replay, selected, unit, replayError } +} + +function environmentUnitOutcome(spec: SyntheticSpec, unit: string): Outcome { + const history = unitStream(spec, 5, unitOf(unit)!.name)[0]! + const ordinary = + dataOutcome(spec) ?? + (spec.slow ? { kind: `timeout` as const } : judged(spec, history)) + return ordinary.kind === `pass` && failsUnit(spec, unit, history) + ? { kind: `constructed-reach` } + : ordinary +} + +// The same independent synthetic model judges every receiving child. The +// context determines selection, never the reported state. Full registration +// remains owed when a filter or path replay makes most tests stand aside. +function childExpectation( plan: Plan, - trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + child: ChildRun, expected: ReturnType, + nameLog: ReadonlyArray, ) { - const run = trace.find(({ checkpoint }) => checkpoint === `run`)?.value as - SuiteRun | undefined - if (!run) return violation(`run`, `the driver recorded no run`, `trace`) - const probes = trace - .filter( - ({ checkpoint }) => checkpoint === `replay` || checkpoint === `rerun`, + const context = child.context + if (context.kind === `host-scope`) { + const diagnostic = child.tests[0] + if ( + child.tests.length !== 1 || + diagnostic?.name !== `host seed diagnostic` || + diagnostic.state !== `passed` || + diagnostic.errors.length !== 0 || + child.warnings.length !== 0 ) - .map(({ checkpoint, value }) => ({ - checkpoint, - probe: value as ProbeRun, - })) - const name = (setting: string) => `${plan.prefix}_${setting}` - for (const spec of plan.specs) { - const owed = expected.get(spec.id)! - const seen = run.files[spec.id]! - if (!owed.loads) { - const law = owed.membership ? `membership` : `registration` - if (seen.loaded) - return violation( - `run`, - `${spec.id} loaded, but its id is unregistered, a required state has no constructed history, a constructed history names an undeclared state, two of its tests would share a name, a fixed or constructed history is one the arbitrary cannot take back, \`owns\` is not a non-empty literal list of strings, or a history is not data`, - law, + return { + failure: violation( + `host-scope`, + `host seed diagnostic did not complete once: ${JSON.stringify(child.tests)}`, + `host RNG scope`, + ), + } + return { failed: false } + } + const selectedSpecs = + `id` in context + ? plan.specs.filter( + ({ id }) => id === context.id || context.kind === `replay`, ) - const owedText = [owed.loadError, owed.loadFix].filter( - (text) => text !== undefined, - ) + : plan.specs + if ( + unattributedWarnings( + child.warnings, + selectedSpecs.map(({ id }) => id), + ).length + ) + return { + failure: violation( + `child`, + `registration reported a warning for no declared property: ${JSON.stringify(child.warnings)}`, + `calibration`, + ), + } + if ( + child.tests.some(({ id }) => !selectedSpecs.some((spec) => spec.id === id)) + ) + return { + failure: violation( + `child`, + `child registered tests outside its declared property files`, + `registration`, + ), + } + let failed = false + for (const original of selectedSpecs) { + const spec = registeredSpec(original, context) + // The selected seed is host input. Read its explicit seed line, never + // infer a seed or a stream from the driver's observed histories. + const randomSeed = + child.tests + .filter(({ id }) => id === spec.id) + .flatMap(({ seedLogs }) => seedLogs)[0] ?? childSeed + const prediction = predictSpec( + spec, + child.log, + child.builds, + nameLog, + randomSeed, + ) + const environment = + context.kind === `environment` + ? environmentSelection(plan, spec) + : undefined + const rejected = + !prediction.loads || + environment?.error !== undefined || + environment?.replayError !== undefined + if (rejected) failed = true + const names = !rejected ? [...prediction.tests.keys()] : [] + const owedNames = + context.kind === `registration-warnings` && context.phases.length + ? context.phases.flatMap((phase) => + names.map((name) => `${phase} > ${name}`), + ) + : names + const seen = child.tests.filter(({ id }) => id === spec.id) + const actualNames = seen.map(({ name }) => name).sort() + if (JSON.stringify(actualNames) !== JSON.stringify([...owedNames].sort())) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} omitted or duplicated registrations`, + `registration`, + { actual: actualNames, expected: [...owedNames].sort() }, + ), + } + if (rejected) continue + const collectOnly = + context.kind === `registration-warnings` && context.phases.length === 0 + if (warningMismatch(spec, child.warnings, collectOnly)) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} lost its once-per-process unbound calibration warning or names: ${JSON.stringify(propertyWarnings(child.warnings, spec.id))}`, + `calibration`, + ), + } + for (const test of seen) { + const selected = + context.kind === `registration-warnings` + ? false + : context.kind === `binding-reordered` + ? test.name.startsWith(`calibration:`) + : context.kind === `replay` + ? spec.id === context.id && test.name === context.probed + : environment?.replay + ? environment.selected && test.name === environment.unit + : context.kind === `rerun` + ? test.name === context.target + : context.kind === `reach-alone` + ? test.name === `reach` + : true + if (!selected) { + if (test.state !== `skipped` || test.errors.length !== 0) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} unselected test ${JSON.stringify(test.name)} did not stand aside`, + `registration`, + ), + } + continue + } + const originalPrediction = expected.get(spec.id)! + if (!originalPrediction.loads) + return { + failure: violation( + `child`, + `${spec.id} ran despite its rejected declaration`, + `registration`, + ), + } + const ordinary = + context.kind === `replay` + ? originalPrediction.tests.get(context.target)! + : prediction.tests.get(test.name)! + const outcome = + environment?.replay && environment.selected && environment.unit + ? environmentUnitOutcome(spec, environment.unit) + : context.kind === `reach-alone` + ? reachAloneOutcome(spec, ordinary) + : ordinary + if (outcome.kind !== `pass`) failed = true + const failure = test.errors[0] + const kind = test.state === `passed` ? `pass` : (failure?.kind ?? `other`) + const owesFailure = outcome.kind !== `pass` if ( - owedText.length > 0 && - !seen.loadErrors.some( - (message, at) => - owedText.every((text) => message.includes(text)) && - (owed.loadCause === undefined || - seen.loadCauses[at] === owed.loadCause), + test.state !== (owesFailure ? `failed` : `passed`) || + test.errors.length !== (owesFailure ? 1 : 0) || + !matchesOutcome(outcome, failure, kind, isCampaign(test.name)) || + !callbackDiagnosticMatches( + callbackExpectation(spec, test.name, outcome, nameLog), + failure, ) ) - return violation( - `run`, - `${spec.id} failed to load with ${JSON.stringify(seen.loadErrors)}, owed ${JSON.stringify(owedText)}`, - owed.membership || owed.nameError ? law : `data`, - { actual: seen.loadErrors, expected: owedText }, + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} ${JSON.stringify(test.name)} ${test.state} as ${JSON.stringify(test.errors)}, owed ${JSON.stringify(outcome)}`, + lawOfKind(outcome.kind === `pass` ? kind : outcome.kind, test.name), + { actual: test.errors, expected: outcome }, + ), + } + if ( + spec.source.kind === `generated` && + isCampaign(test.name) && + outcome.kind === `pass` + ) { + const actualCalls = child.log.filter( + ({ id, test: name }) => + id === spec.id && name === `${spec.id} > ${test.name}`, + ).length + const owedCalls = + context.kind === `replay` || environment?.replay + ? 1 + : unitOf(test.name)!.name === `generated` + ? spec.source.runs + : 1 + if (actualCalls !== owedCalls) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} ${JSON.stringify(test.name)} passed after ${actualCalls} driver calls, owed ${owedCalls} from its declared campaign budget`, + `campaign budget`, + { actual: actualCalls, expected: owedCalls }, + ), + } + } + if (isCampaign(test.name) && failure) { + const calls = child.log.filter( + ({ id, test: name }) => + id === spec.id && name === `${spec.id} > ${test.name}`, ) - continue + // A path replay runs its one requested concrete history directly. + // The input-side log also supplies its items when an Error history's + // inspection cannot be read back as an array of integers. + if (context.kind === `replay` && calls.length !== 1) + return { + failure: violation( + `child`, + `${spec.id} replay ran ${calls.length} histories, owed exactly one`, + `replay`, + ), + } + const concrete = + context.kind === `replay` + ? calls[0]!.history + : failure.counterexampleHistory + if ( + context.kind === `replay` && + failure.counterexampleHistory && + JSON.stringify(failure.counterexampleHistory) !== + JSON.stringify(concrete) + ) + return { + failure: violation( + `child`, + `${spec.id} replay reported a different concrete candidate than it ran`, + `replay`, + ), + } + if (concrete) { + if ( + !calls.some( + ({ history }) => + JSON.stringify(history) === JSON.stringify(concrete), + ) + ) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} reported a candidate the driver never ran`, + `shrink`, + ), + } + const diagnostic = campaignCandidateOutcome(spec, concrete, outcome) + if (!matchesOutcome(diagnostic, failure, kind, false)) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} campaign lost its concrete candidate's diagnostic facts: ${JSON.stringify(failure)}, owed ${JSON.stringify(diagnostic)} for ${JSON.stringify(concrete)}`, + context.kind === `replay` ? `replay` : `shrink`, + { actual: failure, expected: diagnostic }, + ), + } + } + } } - if (!seen.loaded) - return violation(`run`, `${spec.id} failed to load`, `registration`) - const placed = seen.tests.find((test) => - test.name.includes(packageDirectory), - ) - if (placed) - return violation( - `run`, - `${spec.id} named a test with this machine's path: ${JSON.stringify(placed.name.slice(0, 300))}`, - `names`, - ) - const names = [...owed.tests.keys()].sort() - const seenNames = seen.tests.map((test) => test.name).sort() - if (JSON.stringify(names) !== JSON.stringify(seenNames)) + } + return { failed } +} + +function checkSuite( + plan: Plan, + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + expected: ReturnType, +) { + const run = trace.find(({ checkpoint }) => checkpoint === `run`)?.value as + SuiteRun | undefined + if (!run) return violation(`run`, `the driver recorded no run`, `trace`) + if ( + unattributedWarnings( + run.warnings, + plan.specs.map(({ id }) => id), + ).length + ) + return violation( + `run`, + `ordinary warnings name an undeclared property: ${JSON.stringify(run.warnings)}`, + `calibration`, + ) + for (const { checkpoint, value } of trace) + if ( + checkpoint === `warning-permutation` && + !(value as { stable: boolean }).stable + ) return violation( - `run`, - `${spec.id} registered ${JSON.stringify(seenNames)}, owed ${JSON.stringify(names)}`, - `registration`, - { actual: seenNames, expected: names }, + `child`, + `permuting actual warning names changed their semantic observation`, + `calibration`, ) - // The property's one logged seed, which its first random unit logs. - const firstRandom = - spec.source.kind === `generated` - ? randomUnit(unitNames(spec)[0]!) - : undefined - const loggedSeed = seen.tests.find((test) => test.name === firstRandom) - ?.seedLogs[0] - for (const test of seen.tests) { - const outcome = owed.tests.get(test.name)! - const failure = test.failure - const observedKind = - test.state === `passed` ? `pass` : (failure?.kind ?? `other`) + if ( + plan.timeoutPresentation && + plan.specs.some((spec) => spec.slow) && + !trace.some( + ({ checkpoint, value }) => + checkpoint === `timeout-presentation` && + (value as { exercised: boolean }).exercised, + ) + ) + return violation( + `child`, + `timeout presentation was not exercised`, + `timeout`, + ) + for (const { checkpoint, value } of trace) + if ( + checkpoint === `timeout-presentation` && + !(value as { stable: boolean }).stable + ) + return violation( + `child`, + `legal timeout presentation changed its semantic observation`, + `timeout`, + ) + const children = trace.filter(({ checkpoint }) => checkpoint === `child`) + const owedChildren = + 1 + + trace.reduce( + (count, { checkpoint }) => + count + + (checkpoint === `registration-warnings` || checkpoint === `host-scope` + ? 3 + : [ + `binding-reordered`, + `seedless`, + `replay`, + `rerun`, + `reach-alone`, + `environment`, + ].includes(checkpoint) + ? 1 + : 0), + 0, + ) + if (children.length !== owedChildren) + return violation( + `child`, + `recorded ${children.length} child reports, owed ${owedChildren}`, + `trace`, + ) + // Expected synthetic failures exit 1; unexpected process failures do not + // disappear merely because a reporter managed to write a partial result. + const ordinaryLog = + children + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `run`)?.log ?? run.log + for (const { value } of children) { + const child = value as ChildRun + if (child.secondaryErrors.length) + return violation( + `child`, + `unexpected secondary test errors: ${JSON.stringify(child.secondaryErrors)}`, + `trace`, + ) + if (child.context.kind === `host-scope`) { if ( - observedKind !== outcome.kind || - (outcome.law !== undefined && failure?.law !== outcome.law) || - (outcome.checkpoint !== undefined && - failure?.checkpoint !== outcome.checkpoint) || - failure?.diagnosis !== outcome.diagnosis || - failure?.dataPath !== outcome.dataPath || - failure?.dataKind !== outcome.dataKind || - (!isCampaign(test.name) && failure?.cause !== outcome.cause) + child.loads.length !== 1 || + child.loads.some(({ errors, causes }) => errors.length || causes.length) + ) + return violation( + `host-scope`, + `host seed diagnostic reported load errors: ${JSON.stringify(child.loads)}`, + `host RNG scope`, + ) + } else { + for (const file of child.loads) { + const spec = plan.specs.find(({ id }) => id === file.id) + const declaration = + spec && + predictSpec( + registeredSpec(spec, child.context), + child.log, + child.builds, + ordinaryLog, + ) + const environment = + spec && child.context.kind === `environment` + ? environmentSelection(plan, spec) + : undefined + const loadError = + environment?.error ?? + (declaration && !declaration.loads + ? declaration.loadError + : environment?.replayError) + const rejects = + !declaration?.loads || + environment?.error !== undefined || + environment?.replayError !== undefined + if ( + !declaration || + (rejects + ? file.errors.length !== 1 || + file.causes.length !== 1 || + (loadError !== undefined && + !file.errors[0]!.includes(loadError)) || + (!environment?.error && + !declaration.loads && + ((declaration.loadFix !== undefined && + !file.errors[0]!.includes(declaration.loadFix)) || + (declaration.loadCause !== undefined && + file.causes[0] !== declaration.loadCause))) + : file.errors.length !== 0 || file.causes.length !== 0) + ) + return violation( + `child`, + `unexpected child load/suite errors: ${JSON.stringify(file)}`, + `registration`, + ) + } + } + const modeled = childExpectation(plan, child, expected, ordinaryLog) + if (modeled.failure) return modeled.failure + if (!timeoutInputsMatch(plan, child)) + return violation( + `child`, + `timed-out input work disagrees: ${JSON.stringify(child.timeoutInputs ?? null)}`, + `calibration`, + ) + if (!lateAgreementMatches(plan, child)) + return violation( + `child`, + `late probe interval disagrees: ${JSON.stringify(child.lateAgreement ?? null)}`, + `agreement`, + ) + const owedLate = lateCalls(plan, child.log) + const actualLate = child.unhandled.filter( + (message) => message === lateMessage, + ).length + if ( + !usesLateRelation(plan) && + (actualLate < owedLate.min || actualLate > owedLate.max) + ) + return violation( + `child`, + `child reported ${actualLate} late probe calls, owed ${owedLate.min} to ${owedLate.max}`, + `agreement`, + ) + const unexpected = child.unhandled.filter( + (message) => + message !== lateMessage || + !plan.specs.some(({ probe }) => probe === `late`), + ) + if (unexpected.length) + return violation( + `child`, + `unexpected unhandled errors: ${JSON.stringify(unexpected)}`, + `trace`, + ) + if ( + child.signal !== null || + child.exit !== + (modeled.failed || + child.unhandled.length || + (usesLateRelation(plan) && owedLate.min > 0) + ? 1 + : 0) || + JSON.stringify([...child.files].sort()) !== + JSON.stringify([...child.expectedFiles].sort()) + ) + return violation( + `child`, + `incomplete or abnormal child process: ${JSON.stringify(child)}`, + `trace`, + ) + } + const probes = trace + .filter( + ({ checkpoint }) => checkpoint === `replay` || checkpoint === `rerun`, + ) + .map(({ checkpoint, value }) => ({ + checkpoint, + probe: value as ProbeRun, + })) + const envName = (setting: string) => `${plan.prefix}_${setting}` + if (expected.hostScope) { + const scope = trace.find(({ checkpoint }) => checkpoint === `host-scope`) + ?.value as HostScopeRun | undefined + if ( + !scope || + scope.draw !== expected.hostScope.draw || + scope.outer === null || + scope.outer.draw !== expected.hostScope.draw || + scope.outer.hostUsed !== expected.hostScope.outerHostUsed || + scope.outer.forwarded !== expected.hostScope.outerForwarded || + !Number.isInteger(scope.outer.seed) || + scope.outer.childMarker !== false || + scope.children.length !== 2 || + scope.children.some( + (child) => + child === null || + child.childMarker !== true || + !Number.isFinite(child.draw) || + typeof child.hostUsed !== `boolean` || + child.forwarded !== false || + !Number.isInteger(child.seed), + ) || + scope.children[0]?.seed !== scope.children[1]?.seed + ) + return violation( + `host-scope`, + `fixture import changed host seed selection or receiving-child repeatability: ${JSON.stringify(scope ?? null)}, owed ${JSON.stringify(expected.hostScope)}`, + `host RNG scope`, + ) + } + for (const spec of plan.specs) { + const owed = expected.get(spec.id)! + const seen = run.files[spec.id]! + if (!owed.loads) { + if ( + (bindingsOf(spec).length || + (spec.probeBinding && spec.probeBinding !== `empty`)) && + ordinaryLog.some((entry) => entry.id === spec.id) ) return violation( `run`, - `${spec.id} "${test.name}" ${test.state} as ${JSON.stringify(failure ?? null)}, owed ${JSON.stringify(outcome)}`, - // A unit that owes a failure and times out shrank too long. - isCampaign(test.name) && - observedKind === `timeout` && - ![`pass`, `timeout`].includes(outcome.kind) - ? `shrink budget` - : lawOfKind( - outcome.kind === `pass` ? observedKind : outcome.kind, - test.name, - ), - { actual: failure ?? { kind: `pass` }, expected: outcome }, + `${spec.id} ran a driver despite an invalid calibration selector`, + `calibration`, ) - // A model error quotes the driver's error without this machine's paths. - if (failure?.machinePath) + const law = owed.membership ? `membership` : `registration` + if (seen.loaded) return violation( `run`, - `${spec.id} "${test.name}" reported a model error that quotes this machine's paths`, - `verdict`, + `${spec.id} loaded, but its id is unregistered, a required state has no constructed history, a constructed history names an undeclared state, two of its tests would share a name, a fixed or constructed history is one the arbitrary cannot take back, \`owns\` is not a non-empty literal list of strings, or a history is not data`, + law, ) - // A law's failure prints the history it ran, also a generated one that - // is not data. + if (seen.loadErrors.length !== 1 || seen.loadCauses.length !== 1) + return violation( + `run`, + `${spec.id} must report its first load error alone`, + `registration`, + { actual: seen.loadErrors, expected: `one load error` }, + ) + const owedText = [owed.loadError, owed.loadFix].filter( + (text) => text !== undefined, + ) if ( - spec.form === `boxed` && - failure?.kind === `violation` && - failure.showsHistory !== true + owedText.length > 0 && + !seen.loadErrors.some( + (message, at) => + owedText.every((text) => message.includes(text)) && + (owed.loadCause === undefined || + seen.loadCauses[at] === owed.loadCause), + ) ) return violation( `run`, - `${spec.id} "${test.name}" printed a History line that does not show the history`, - `verdict`, + `${spec.id} failed to load with ${JSON.stringify(seen.loadErrors)}, owed ${JSON.stringify(owedText)}`, + owed.membership || owed.nameError ? law : `data`, + { actual: seen.loadErrors, expected: owedText }, ) - // A property selects its seed once, in its first random unit, and - // logs that selection once; every random unit runs it. - const owedSeedLogs = test.name === firstRandom ? 1 : 0 - if (test.seedLogs.length !== owedSeedLogs) + continue + } + if (!seen.loaded) + return violation(`run`, `${spec.id} failed to load`, `registration`) + const reordered = trace.find( + ({ checkpoint, value }) => + checkpoint === `binding-reordered` && + (value as { id: string }).id === spec.id, + )?.value as { log: Array } | undefined + if (reordered) { + const tests = new Set( + spec.calibrations.map( + ({ name }) => `${spec.id} > calibration: ${name}`, + ), + ) + const selected = (log: ReadonlyArray) => + log + .filter(({ test }) => tests.has(test)) + .map(({ test, history }) => ({ test, history })) + .sort((a, b) => a.test.localeCompare(b.test)) + if ( + JSON.stringify(selected(ordinaryLog)) !== + JSON.stringify(selected(reordered.log)) + ) return violation( `run`, - `${spec.id} "${test.name}" logged ${test.seedLogs.length} ORACLE_SEED lines, owed ${owedSeedLogs}`, - `seed-log`, - { actual: test.seedLogs, expected: owedSeedLogs }, + `${spec.id} changed bound histories after calibration insertion/reordering`, + `calibration`, ) + } + if ( + spec.calibrationNames !== `indexed` && + spec.calibrations.length > 1 && + spec.calibrations.every((calibration) => { + const binding = bindingOf(spec, calibration) + return binding && `constructed` in binding + }) + ) { + const inputs = spec.calibrations.map( + ({ name }) => + run.builds.find( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > calibration: ${name}`, + )?.generated, + ) if ( - failure?.seed !== undefined && - unitOf(test.name) !== undefined && - unitOf(test.name)!.seed === undefined && - loggedSeed !== failure.seed + inputs.some((input) => !input) || + new Set(inputs.map((input) => JSON.stringify(input))).size < 2 ) return violation( `run`, - `${spec.id} "${test.name}" ran seed ${failure.seed}, but the property logged seed ${loggedSeed}`, - `seed-log`, - { actual: failure.seed, expected: loggedSeed }, + `${spec.id} distinct named calibrations did not vary their full construction inputs`, + `calibration`, ) - // A failing unit prints its replay command; a failing fixed-seed unit - // also prints its own `-t` pattern; a failing random unit's violation - // prints the replay command as its rerun line. - const unit = unitOf(test.name) - if (unit && failure?.seed !== undefined) { - if (failure.replay === undefined) - return violation( - `run`, - `${spec.id} "${test.name}" failed with seed ${failure.seed} and printed no replay command`, - `replay`, - ) - if (unit.seed !== undefined && failure.rerun === undefined) - return violation( - `run`, - `${spec.id} "${test.name}" failed and printed no \`-t\` rerun of itself`, - `rerun`, + } + const warningProbe = trace.find( + ({ checkpoint, value }) => + checkpoint === `registration-warnings` && + (value as { id: string }).id === spec.id, + )?.value as + | { + tests: { + repeated: Array + collectThenRun: Array + collected: Array + } + repeated: Array + collectThenRun: Array + collected: Array + manifests: { + collectThenRun: Array | false + collected: Array | false + } + loaded: { + repeated: boolean + collectThenRun: boolean + collected: boolean + } + } + | undefined + if (warningProbe) { + // Each ordinary registration owes the full model-derived test set. + // Preserve suite scope and multiplicity: warning deduplication grants no + // permission to drop the second property's tests or either forced control. + const names = [...owed.tests.keys()] + const phaseNames = (phases: Array) => + phases + .flatMap((phase) => + names.map((name) => `${phase} > ${spec.id} > ${name}`), ) - const owedLine = - unit.seed === undefined ? failure.replay : failure.rerun + .sort() + for (const [phase, expectedNames] of [ + [`repeated`, phaseNames([`first`, `second`])], + [`collectThenRun`, phaseNames([`collect`, `ordinary`])], + [`collected`, names.map((name) => `${spec.id} > ${name}`).sort()], + ] as const) { if ( - failure.violationRerun !== undefined && - failure.violationRerun !== owedLine + JSON.stringify(warningProbe.tests[phase]) !== + JSON.stringify(expectedNames) ) return violation( `run`, - `${spec.id} "${test.name}" printed the rerun line ${JSON.stringify(failure.violationRerun)} in its violation, owed ${JSON.stringify(owedLine)}`, - unit.seed === undefined ? `replay` : `rerun`, - { actual: failure.violationRerun, expected: owedLine }, + `${spec.id} ${phase} omitted or duplicated registrations`, + `registration`, + { actual: warningProbe.tests[phase], expected: expectedNames }, ) } - // A passing determinism unit ran its one history twice. - const twice = owed.determinism.get(test.name) - if (twice && test.state === `passed`) { - const runs = run.log - .filter( + if ( + [ + warningProbe.manifests.collectThenRun, + warningProbe.manifests.collected, + ].some((ids) => JSON.stringify(ids) !== JSON.stringify([spec.id])) + ) + return violation( + `run`, + `${spec.id} did not complete its collect-only registration in each phase`, + `calibration`, + ) + if ( + !warningProbe.loaded.repeated || + !warningProbe.loaded.collectThenRun || + !warningProbe.loaded.collected || + [ + warningProbe.repeated, + warningProbe.collectThenRun, + warningProbe.collected, + ].some( + (warnings) => unattributedWarnings(warnings, [spec.id]).length > 0, + ) || + warningMismatch(spec, warningProbe.repeated, false) || + warningMismatch(spec, warningProbe.collectThenRun, false) || + warningMismatch(spec, warningProbe.collected, true) + ) + return violation( + `run`, + `${spec.id} repeated or collect-only registration violated the one-warning boundary`, + `calibration`, + ) + } + // The compatibility route must be visible before any selected test runs, + // including automatic controls; bound calibrations must never warn search. + const warnings = propertyWarnings(run.warnings, spec.id) + if (warningMismatch(spec, run.warnings, false)) + return violation( + `run`, + `${spec.id} did not report unbound calibrations once per property: ${JSON.stringify(warnings)}`, + `calibration`, + ) + const placed = seen.tests.find((test) => + test.name.includes(packageDirectory), + ) + if (placed) + return violation( + `run`, + `${spec.id} named a test with this machine's path: ${JSON.stringify(placed.name.slice(0, 300))}`, + `names`, + ) + const names = [...owed.tests.keys()].sort() + const seenNames = seen.tests.map((test) => test.name).sort() + if (JSON.stringify(names) !== JSON.stringify(seenNames)) + return violation( + `run`, + `${spec.id} registered ${JSON.stringify(seenNames)}, owed ${JSON.stringify(names)}`, + `registration`, + { actual: seenNames, expected: names }, + ) + const firstRandom = + spec.source.kind === `generated` + ? randomUnit(unitNames(spec)[0]!) + : undefined + const loggedSeed = seen.tests.find(({ name }) => name === firstRandom) + ?.seedLogs[0] + for (const test of seen.tests) { + const outcome = owed.tests.get(test.name)! + const failure = test.failure + if ( + spec.probeBinding && + spec.probeBinding !== `empty` && + forcedCalibrationNames(spec).includes(test.name) && + (!spec.slow || + ordinaryLog.some( (entry) => entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, - ) - .map((entry) => entry.history) - if (JSON.stringify(runs) !== JSON.stringify([twice, twice])) - return violation( + )) + ) { + const calls = ordinaryLog.filter( + (entry) => + entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, + ) + const expectedBinding = selectedControlBinding( + spec, + test.name === forcedCalibrationNames(spec)[0], + ) + const expectedHistory = authoredInput( + spec, + test.name, + expectedBinding, + run.builds, + ) + const expectedOrigin = authoredOrigin(spec, expectedBinding) + if ( + calls.some( + (entry) => + entry.origin !== expectedOrigin || + JSON.stringify(entry.history) !== JSON.stringify(expectedHistory), + ) + ) + return violation( + `run`, + `${spec.id} forced control ran a different declared premise`, + `calibration`, + ) + const expectedCalls = spec.slow + ? 1 + : outcome.kind === `calibration-clean` || + outcome.kind === `calibration-reach` + ? 1 + : 2 + const expectedBuilds = + `constructed` in expectedBinding ? expectedCalls : 0 + const builds = run.builds.filter( + (entry) => + entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, + ) + if ( + builds.length !== expectedBuilds || + (`constructed` in expectedBinding && + builds.some((entry) => entry.name !== expectedBinding.constructed)) + ) + return violation( + `run`, + `${spec.id} "${test.name}" used ${builds.length} construction builds; selected control owes ${expectedBuilds}`, + `calibration`, + ) + if ( + (spec.isolatedPlans || spec.identityHistories) && + run.resolutions.filter( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + ).length !== expectedCalls + ) + return violation( + `run`, + `${spec.id} "${test.name}" did not resolve once per driver execution (owed ${expectedCalls})`, + `calibration`, + ) + if ( + calls.length !== expectedCalls || + calls.some( + ({ history }) => + JSON.stringify(history) !== JSON.stringify(calls[0]!.history), + ) + ) + return violation( + `run`, + `${spec.id} forced control did not run ${expectedCalls} equivalent selected histories`, + `calibration`, + ) + } + const calibration = spec.calibrations.find( + ({ name }) => test.name === `calibration: ${name}`, + ) + const calibrationBinding = calibration && bindingOf(spec, calibration) + if ( + calibrationBinding && + (!spec.slow || + ordinaryLog.some( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + )) + ) { + const calls = ordinaryLog.filter( + (entry) => + entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, + ) + const expectedCalls = spec.slow + ? 1 + : driverCalibration(calibration.kind) && + outcome.kind !== `calibration-clean` && + outcome.kind !== `calibration-reach` + ? 2 + : 1 + const expectedConstruction = + `constructed` in calibrationBinding + ? calibrationBinding.constructed + : undefined + const expectedBuilds = + expectedConstruction === undefined ? 0 : expectedCalls + const builds = run.builds.filter( + (entry) => + entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, + ) + if ( + builds.length !== expectedBuilds || + (expectedConstruction !== undefined && + builds.some((entry) => entry.name !== expectedConstruction)) + ) + return violation( + `run`, + `${spec.id} "${test.name}" used ${builds.length} construction builds; binding owes ${expectedBuilds}`, + `calibration`, + ) + if ( + (spec.isolatedPlans || spec.identityHistories) && + run.resolutions.filter( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + ).length !== expectedCalls + ) + return violation( + `run`, + `${spec.id} "${test.name}" did not resolve once per driver execution (owed ${expectedCalls})`, + `calibration`, + ) + if (calls.length !== expectedCalls) + return violation( + `run`, + `${spec.id} "${test.name}" used ${calls.length} driver calls; binding owes ${expectedCalls}`, + `calibration`, + ) + const binding = calibrationBinding + const first = calls[0]!.history + const fullInput = authoredInput(spec, test.name, binding, run.builds) + if ( + fullInput && + calls.some( + ({ history, origin }) => + origin !== authoredOrigin(spec, binding) || + JSON.stringify(history) !== JSON.stringify(fullInput), + ) + ) + return violation( + `run`, + `${spec.id} selected calibration did not run its full authored input and origin`, + `calibration`, + ) + const selected = + `fixed` in binding + ? spec.fixed.find(({ name }) => name === binding.fixed)?.history + : `enumerated` in binding && spec.source.kind === `enumerated` + ? spec.source.histories.find((history, index) => + typeof binding.enumerated === `number` + ? index === binding.enumerated + : authoredEnumerationLabel(spec, history) === + binding.enumerated, + ) + : undefined + const prefix = + `constructed` in binding + ? spec.constructed.find(({ name }) => name === binding.constructed)! + .prefix + : undefined + if ( + (selected && JSON.stringify(first) !== JSON.stringify(selected)) || + (prefix && + JSON.stringify(first.slice(0, prefix.length)) !== + JSON.stringify(prefix)) || + calls.some( + ({ history }) => JSON.stringify(history) !== JSON.stringify(first), + ) + ) + return violation( + `run`, + `${spec.id} "${test.name}" did not run equivalent selected histories`, + `calibration`, + ) + } + if ( + !callbackDiagnosticMatches( + callbackExpectation(spec, test.name, outcome, ordinaryLog), + failure, + ) + ) + return violation( + `run`, + `${spec.id} callback failure lost its authored calibration/history identity or cause`, + `calibration`, + ) + const observedKind = + test.state === `passed` ? `pass` : (failure?.kind ?? `other`) + if ( + observedKind !== outcome.kind || + (outcome.law !== undefined && failure?.law !== outcome.law) || + (outcome.checkpoint !== undefined && + failure?.checkpoint !== outcome.checkpoint) || + failure?.diagnosis !== outcome.diagnosis || + failure?.dataPath !== outcome.dataPath || + failure?.dataKind !== outcome.dataKind || + (!isCampaign(test.name) && failure?.cause !== outcome.cause) + ) + return violation( + `run`, + `${spec.id} "${test.name}" ${test.state} as ${JSON.stringify(failure ?? null)}, owed ${JSON.stringify(outcome)}`, + // A campaign that owes a failure and times out shrank too long. + isCampaign(test.name) && + observedKind === `timeout` && + ![`pass`, `timeout`].includes(outcome.kind) + ? `shrink budget` + : lawOfKind( + outcome.kind === `pass` ? observedKind : outcome.kind, + test.name, + ), + { actual: failure ?? { kind: `pass` }, expected: outcome }, + ) + // A model error quotes the driver's error without this machine's paths. + if (failure?.machinePath) + return violation( + `run`, + `${spec.id} "${test.name}" reported a model error that quotes this machine's paths`, + `verdict`, + ) + // A law's failure prints the history it ran, also a generated one that + // is not data. + if ( + spec.form === `boxed` && + failure?.kind === `violation` && + failure.showsHistory !== true + ) + return violation( + `run`, + `${spec.id} "${test.name}" printed a History line that does not show the history`, + `verdict`, + ) + // A property selects its seed once, in its first random unit, and + // logs that selection once; every random unit runs it. + const owedSeedLogs = test.name === firstRandom ? 1 : 0 + if (test.seedLogs.length !== owedSeedLogs) + return violation( + `run`, + `${spec.id} "${test.name}" logged ${test.seedLogs.length} ORACLE_SEED lines, owed ${owedSeedLogs}`, + `seed-log`, + { actual: test.seedLogs, expected: owedSeedLogs }, + ) + if ( + failure?.seed !== undefined && + unitOf(test.name) !== undefined && + unitOf(test.name)!.seed === undefined && + loggedSeed !== failure.seed + ) + return violation( + `run`, + `${spec.id} "${test.name}" ran seed ${failure.seed}, but the property logged seed ${loggedSeed}`, + `seed-log`, + { actual: failure.seed, expected: loggedSeed }, + ) + // A failing unit prints its replay command; a failing fixed-seed unit + // also prints its own `-t` pattern; a failing random unit's violation + // prints the replay command as its rerun line. + const unit = unitOf(test.name) + if (unit && failure?.seed !== undefined) { + if (failure.replay === undefined) + return violation( + `run`, + `${spec.id} "${test.name}" failed with seed ${failure.seed} and printed no replay command`, + `replay`, + ) + if (unit.seed !== undefined && failure.rerun === undefined) + return violation( + `run`, + `${spec.id} "${test.name}" failed and printed no \`-t\` rerun of itself`, + `rerun`, + ) + const owedLine = + unit.seed === undefined ? failure.replay : failure.rerun + if ( + failure.violationRerun !== undefined && + failure.violationRerun !== owedLine + ) + return violation( + `run`, + `${spec.id} "${test.name}" printed the rerun line ${JSON.stringify(failure.violationRerun)} in its violation, owed ${JSON.stringify(owedLine)}`, + unit.seed === undefined ? `replay` : `rerun`, + { actual: failure.violationRerun, expected: owedLine }, + ) + } + // A passing determinism unit ran its one history twice. + const twice = owed.determinism.get(test.name) + if (twice && test.state === `passed`) { + const runs = run.log + .filter( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + ) + .map((entry) => entry.history) + if (JSON.stringify(runs) !== JSON.stringify([twice, twice])) + return violation( `run`, `${spec.id} "${test.name}" ran ${JSON.stringify(runs)}, owed ${JSON.stringify(twice)} twice`, `determinism`, { actual: runs, expected: [twice, twice] }, ) } + if ( + test.name === `determinism` && + test.state === `passed` && + owed.determinismRuns !== undefined + ) { + const runs = run.log.filter( + ({ id, test: name }) => + id === spec.id && name === `${spec.id} > determinism`, + ).length + if (runs !== 2 * owed.determinismRuns) + return violation( + `run`, + `${spec.id} determinism ran ${runs / 2} histories, owed ${owed.determinismRuns}`, + `determinism`, + ) + } // Without a timeout, a campaign shrinks without a budget. if (outcome.counterexample) { const shrunk = @@ -3427,8 +7100,29 @@ function checkSuite( { actual: ran, expected: owedRun }, ) } + if ( + (bindingsOf(spec).length || + (spec.probeBinding && spec.probeBinding !== `empty`)) && + run.stdout + .split(`\n`) + .some( + (line) => + line.startsWith(`${spec.id}: calibration "`) && + line.includes(` on `), + ) + ) + return violation( + `run`, + `${spec.id} printed a per-calibration selected-history announcement`, + `calibration`, + ) for (const [campaign, counts] of owed.reach) - if (JSON.stringify(seen.reach[campaign]) !== JSON.stringify(counts)) + // Reach is a dictionary of counts, not an ordered observation. Renaming + // a state may reorder its printed keys without changing any count. + if ( + JSON.stringify(Object.entries(seen.reach[campaign] ?? {}).sort()) !== + JSON.stringify(Object.entries(counts).sort()) + ) return violation( `run`, `${spec.id} "${campaign}" printed reach ${JSON.stringify(seen.reach[campaign] ?? null)}, owed ${JSON.stringify(counts)}`, @@ -3559,7 +7253,12 @@ function checkSuite( { actual: alone ?? null, expected: outcome }, ) const counts = outcome.kind === `pass` && owed.reach.get(`enumerated`) - if (counts && JSON.stringify(alone.reach) !== JSON.stringify(counts)) + if ( + counts && + JSON.stringify( + Object.entries((alone.reach ?? {}) as Record).sort(), + ) !== JSON.stringify(Object.entries(counts).sort()) + ) return violation( `reach-alone`, `${spec.id}: its reach test run alone printed ${JSON.stringify(alone.reach ?? null)}, owed ${JSON.stringify(counts)}`, @@ -3571,8 +7270,20 @@ function checkSuite( const late = run.unhandled.filter((message) => message.includes(`was called after its history ended`), ).length - const owedLate = lateCalls(plan, run.log) - if (late < owedLate.min || late > owedLate.max) + const owedLate = lateCalls(plan, ordinaryLog) + if (!timeoutInputsMatch(plan, run)) + return violation( + `run`, + `timed-out input work disagrees: ${JSON.stringify(run.timeoutInputs ?? null)}`, + `calibration`, + ) + if (!lateAgreementMatches(plan, run)) + return violation( + `run`, + `late probe interval disagrees: ${JSON.stringify(run.lateAgreement ?? null)}`, + `agreement`, + ) + if (!usesLateRelation(plan) && (late < owedLate.min || late > owedLate.max)) return violation( `run`, `the run reported ${late} late probe calls, owed ${owedLate.min === owedLate.max ? owedLate.min : `${owedLate.min} to ${owedLate.max}`}`, @@ -3583,39 +7294,38 @@ function checkSuite( for (const spec of plan.specs) { if (!expected.get(spec.id)!.loads || spec.slow) continue for (const entry of run.log.filter(({ id }) => id === spec.id)) { - const test = entry.test.slice(spec.id.length + 3) - if (entry.route !== owedRoute(test)) + const name = entry.test.slice(spec.id.length + 3) + if (entry.route !== owedRoute(name)) return violation( `run`, - `${spec.id} "${test}" ran ${JSON.stringify(entry.history)} under route ${JSON.stringify(entry.route)}, owed ${JSON.stringify(owedRoute(test))}`, + `${spec.id} "${name}" ran ${JSON.stringify(entry.history)} under route ${JSON.stringify(entry.route)}, owed ${JSON.stringify(owedRoute(name))}`, `route`, - { actual: entry.route, expected: owedRoute(test) }, + { actual: entry.route, expected: owedRoute(name) }, ) } } for (const { checkpoint, probe } of probes) { + const probed = probe.probed if (checkpoint === `replay`) { - // The printed command names the seed, the path, the property, and the - // random unit of the target's name, under the suite's prefix. const owedVariables = { - [name(`SEED`)]: String(probe.original.seed), - [name(`PATH`)]: probe.original.path, - [name(`PROPERTY`)]: probe.id, - [name(`UNIT`)]: probe.probed, + [envName(`SEED`)]: String(probe.original.seed), + [envName(`PATH`)]: probe.original.path, + [envName(`PROPERTY`)]: probe.id, + [envName(`UNIT`)]: probe.probed, } const printed = Object.fromEntries( Object.keys(owedVariables).map((key) => [key, probe.variables?.[key]]), ) if (JSON.stringify(printed) !== JSON.stringify(owedVariables)) return violation( - checkpoint, - `${probe.id} "${probe.target}" printed a replay command with ${JSON.stringify(probe.variables)}, owed ${JSON.stringify(owedVariables)}`, + `replay`, + `${probe.id} "${probe.target}" printed replay variables ${JSON.stringify(probe.variables)}, owed ${JSON.stringify(owedVariables)}`, `replay`, { actual: printed, expected: owedVariables }, ) } const others = probe.states.filter( - ([id, test]) => !(id === probe.id && test === probe.probed), + ([id, name]) => !(id === probe.id && name === probed), ) if (others.some(([, , state]) => state !== `skipped`)) return violation( @@ -3626,7 +7336,7 @@ function checkSuite( if (!probe.failure) return violation( checkpoint, - `${probe.id} ${checkpoint} of "${probe.target}": "${probe.probed}" did not fail again`, + `${probe.id} ${checkpoint} of "${probe.target}": "${probed}" did not fail again`, checkpoint, ) if ( @@ -3638,7 +7348,77 @@ function checkSuite( `${probe.id}: reproduced ${JSON.stringify(probe.failure)}, first failure ${JSON.stringify(probe.original)}`, checkpoint, ) - if (checkpoint !== `replay`) continue + if (checkpoint === `rerun`) { + const prediction = expected.get(probe.id as SyntheticSpec[`id`])! + if (!prediction.loads) + return violation( + checkpoint, + `${probe.id}: rerun of a rejected declaration`, + `registration`, + ) + const ordinary = prediction.tests.get(probe.target)! + const spec = plan.specs.find(({ id }) => id === probe.id)! + const concrete = + probe.failure.counterexampleHistory ?? + (probe.failure.counterexample + ? counterexampleItems(probe.failure.counterexample) + : undefined) + const outcome = + isCampaign(probe.target) && concrete + ? campaignCandidateOutcome(spec, concrete, ordinary) + : ordinary + if (!matchesOutcome(outcome, probe.failure, probe.failure.kind, false)) + return violation( + checkpoint, + `${probe.id}: rerun lost its modeled diagnostic facts: ${JSON.stringify(probe.failure)}, owed ${JSON.stringify(outcome)}`, + `rerun`, + { actual: diagnosticFacts(probe.failure), expected: outcome }, + ) + continue + } + const spec = plan.specs.find(({ id }) => id === probe.id)! + const prediction = expected.get(spec.id)! + if (!prediction.loads) + return violation( + checkpoint, + `${probe.id}: replay of a rejected declaration`, + `registration`, + ) + const receiving = children + .map(({ value }) => value as ChildRun) + .find( + ({ context }) => + context.kind === `replay` && + context.id === probe.id && + context.target === probe.target, + ) + const concreteInputs = receiving?.log + .filter( + ({ id, test }) => + id === spec.id && test === `${spec.id} > ${probe.probed}`, + ) + .map(({ history }) => history) + if ( + concreteInputs?.length !== 1 || + JSON.stringify(probe.ran) !== JSON.stringify(concreteInputs) + ) + return violation( + checkpoint, + `${probe.id}: replay summary omitted or changed its one received concrete history`, + `replay`, + ) + const diagnostic = campaignCandidateOutcome( + spec, + concreteInputs[0]!, + prediction.tests.get(probe.target)!, + ) + if (!matchesOutcome(diagnostic, probe.failure, probe.failure.kind, false)) + return violation( + checkpoint, + `${probe.id}: replay summary lost its concrete candidate's diagnostic facts: ${JSON.stringify(probe.failure)}, owed ${JSON.stringify(diagnostic)}`, + `replay`, + { actual: diagnosticFacts(probe.failure), expected: diagnostic }, + ) const replayRoute = owedRoute(probe.probed, true) if (probe.routes?.some((route) => route !== replayRoute)) return violation( @@ -3680,7 +7460,7 @@ function checkEnvironment( plan: Plan, trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, expected: ReturnType, -): Verdict | undefined { +): ReturnType | undefined { const under = trace.find(({ checkpoint }) => checkpoint === `environment`) ?.value as EnvironmentRun | undefined if (!plan.environment || !under) return undefined @@ -3722,6 +7502,25 @@ function checkEnvironment( return failed( `${spec.id} failed to load with ${JSON.stringify(seen.loadErrors)} under ${JSON.stringify(environmentOf(plan))}, owed ${JSON.stringify(owedError)}`, ) + if (seen.loaded && owed.loads && !replays) { + const ordinary = trace.find(({ checkpoint }) => checkpoint === `run`) + ?.value as SuiteRun | undefined + for (const test of seen.tests) { + const outcome = owed.tests.get(test.name) + if ( + outcome && + !callbackDiagnosticMatches( + callbackExpectation(spec, test.name, outcome, ordinary?.log ?? []), + test.failure, + ) + ) + return violation( + `environment`, + `${spec.id} callback failure lost its authored calibration/history identity or cause`, + `calibration`, + ) + } + } if (!seen.loaded || !replays) continue // A path replay: only the selected unit runs, if any. const running = seen.tests.filter((test) => test.state !== `skipped`) @@ -3791,8 +7590,19 @@ const generatedDraft = (overrides: Partial): Draft => ({ timing: `normal`, timeoutMode: `default`, calibrations: [], + calibrationHistory: null, + isolatedPlans: false, + identityHistories: false, + probeName: `saw-0`, + probeBinding: `unbound`, + probeEntered: { constructed: `constructed-0` }, + probeNotEntered: { fixed: `fixed-0` }, + duplicateHistoryNames: false, unregistered: false, numberDifference: false, + extraReach: false, + calibrationNames: `indexed`, + warningProbe: false, probe: `none`, overReport: false, noisyProbe: false, @@ -3869,25 +7679,2161 @@ const replayOf = ( ...environment, }) const at = (history: SyntheticHistory) => ({ history, enters: null }) -const cal = ( - kind: CalibrationDraft[`kind`], - difference: CalibrationDraft[`difference`] = `none`, -): CalibrationDraft => ({ kind, difference }) - -oracleProperty({ - id: `self-oracle.framework`, - law: { +// This legal grammar branch crosses callback kind with selected source kind. +// Distinct same-kind names make truthful selected identity observable. A clean +// constructed-bound neighbor admits the reordered receiver for fixed selections. +// Enumerated declarations have no constructed binding, so that receiver is absent. +const callbackKinds = [ + `corrupt-throws`, + `judge-throws`, + `judge-driver-throws`, +] as const +const callbackSelections = [`constructed`, `fixed`, `enumerated`] as const +const callbackDiagnosticSuite = ( + kind: (typeof callbackKinds)[number], + selection: (typeof callbackSelections)[number], + tail: 1 | 2 = 1, + prefix: Prefix = `ORACLE`, +): Suite => ({ + seedless: true, + prefix, + environment: replayOf({ path: `absent`, property: `absent`, unit: `absent` }), + drafts: [ + selection === `enumerated` + ? generatedDraft({ + source: { + kind: `enumerated`, + histories: [ + [0, tail], + [2, 2], + ], + }, + enumeratedStates: [`measured`, `measured`, `measured`], + calibrations: [ + { kind, difference: `none`, history: { enumerated: 0 } }, + ], + identityHistories: true, + warningProbe: true, + }) + : generatedDraft({ + fixed: [at([0, tail]), at([2, 2])], + constructed: [ + { prefix: [0, tail], enters: [`saw-0`] }, + { prefix: [2, 2], enters: [`saw-2`] }, + ], + calibrations: [ + { + kind, + difference: `none`, + history: + selection === `fixed` + ? { fixed: `fixed-0` } + : { constructed: `constructed-0` }, + }, + { + kind: `right`, + difference: `none`, + history: { constructed: `constructed-1` }, + }, + ], + isolatedPlans: true, + identityHistories: true, + warningProbe: true, + }), + ], +}) +// The same callback grammar supplies alternate public representations. Single +// originals distinguish name presence from choosing among siblings; prefixed +// names distinguish complete identity from an incidental substring. Default +// enumeration labels keep numeric data beside construction-only seed metadata. +const callbackRepresentationSuite = ( + kind: (typeof callbackKinds)[number], + selection: (typeof callbackSelections)[number], + siblings: boolean, + presentation: Suite[`callbackPresentation`], + tail: 1 | 2 = 1, + prefix: Prefix = `ORACLE`, +): Suite => { + const value = callbackDiagnosticSuite(kind, selection, tail, prefix) + value.callbackPresentation = presentation + const d = value.drafts[0]! + if (!siblings) { + d.calibrations = d.calibrations.slice(0, 1) + d.fixed = d.fixed.slice(0, 1) + d.constructed = selection === `constructed` ? d.constructed.slice(0, 1) : [] + if (d.source.kind === `enumerated`) + d.source.histories = d.source.histories.slice(0, 1) + } + if (selection === `enumerated`) d.labelStyle = `default` + else { + d.constructedNames = `prefixed` + d.calibrations = d.calibrations.map((entry) => + typeof entry === `string` || + !entry.history || + !(`constructed` in entry.history) + ? entry + : { + ...entry, + history: { + constructed: + entry.history.constructed === `constructed-0` ? `c` : `c.1`, + }, + }, + ) + } + return value +} +const suite: fc.Arbitrary = fc.oneof( + { weight: 8, arbitrary: genericSuite }, + { + weight: 1, + arbitrary: fc + .record({ + kind: fc.constantFrom(...callbackKinds), + selection: fc.constantFrom(...callbackSelections), + tail: fc.constantFrom(1 as const, 2 as const), + prefix: fc.constantFrom(`ORACLE` as const, `SYNTH_ORACLE` as const), + siblings: fc.boolean(), + presentation: fc.oneof( + fc.constantFrom(...callbackPresentations), + callbackPresentationArbitrary, + ), + incoming: fc.option(callbackPresentationArbitrary, { nil: undefined }), + reason: fc.option(fc.string({ maxLength: 192 }), { nil: undefined }), + wrapperName: fc.option(fc.string({ maxLength: 64 }), { + nil: undefined, + }), + }) + .map( + ({ + kind, + selection, + tail, + prefix, + siblings, + presentation, + incoming, + reason, + wrapperName, + }) => ({ + ...callbackRepresentationSuite( + kind, + selection, + siblings, + presentation, + tail, + prefix, + ), + callbackIncomingPresentation: incoming, + callbackReason: reason, + callbackWrapperName: wrapperName, + }), + ), + }, +) +const receivingCutStates = [ + `receiving-run`, + `receiving-seedless`, + `receiving-replay`, + `receiving-rerun`, + `receiving-reach-alone`, + `receiving-binding-reordered`, + `receiving-registration-warnings`, + `receiving-environment`, +] as const +const receivingCutDrafts = (tail: 1 | 2 = 1): Array => [ + generatedDraft({ + fixed: [at([3, 6])], + constructed: [ + { prefix: [0, tail], enters: [`saw-0`] }, + { prefix: [3, 6], enters: [] }, + ], + calibrationHistory: { constructed: `constructed-0` }, + calibrations: [ + `right`, + { kind: `driver`, difference: `none`, history: null }, + { kind: `right`, difference: `none`, history: null }, + ], + isolatedPlans: true, + warningProbe: true, + }), + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [tail]] }, + enumeratedStates: [`measured`, `measured`, `measured`], + calibrations: [`right`, `driver`], + }), +] +const campaignBudgetSuite = ( + runs: 2 | 3 = 2, + constructions: 0 | 1 | 2 = 1, + tail: 1 | 2 = 1, +): Suite => ({ + seedless: true, + seedlessDraw: cleanHostDraw, + drafts: [ + generatedDraft({ + source: { kind: `generated`, runs, seed: 7 }, + constructed: [ + { prefix: [0, tail, 1], enters: [`saw-0`] }, + { prefix: [tail, 0, tail], enters: [`saw-0`] }, + ].slice(0, constructions), + }), + generatedDraft({ + constructed: [{ prefix: [3, 6], enters: [] }], + }), + ], +}) +const timeoutRepresentationSuite = (tail: 1 | 2 = 1): Suite => ({ + seedless: false, + timeoutPresentation: true, + drafts: [ + generatedDraft({ timing: `slow`, probe: `late`, fixed: [at([0, tail])] }), + generatedDraft({ probe: `late`, fixed: [at([0, tail])] }), + ], +}) +const timeoutInputSuite = (bound: boolean, tail: 1 | 2 = 1): Suite => ({ + seedless: false, + timeoutPresentation: true, + drafts: [ + generatedDraft({ + timing: `slow`, + probe: `late`, + probeBinding: `explicit`, + isolatedPlans: true, + fixed: [at([1, tail])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + ...(bound + ? { + calibrations: [`driver`], + calibrationHistory: { constructed: `constructed-0` }, + } + : {}), + }), + ], +}) +const timeoutReceivingSuite = (tail: 1 | 2 = 1): Suite => { + const base = timeoutInputSuite(true, tail) + return { + ...base, + seedless: true, + prefix: tail === 1 ? `ORACLE` : `SYNTH_ORACLE`, + environment: replayOf({ + path: `absent`, + property: `absent`, + unit: `absent`, + }), + drafts: base.drafts.map((entry) => ({ ...entry, warningProbe: true })), + } +} +const warningRepresentationSuite = (): Suite => ({ + drafts: [ + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [1]] }, + calibrations: [`right`, `driver`], + }), + ], + seedless: false, + permuteWarnings: true, +}) +const cal = ( + kind: CalibrationDraft[`kind`], + difference: CalibrationDraft[`difference`] = `none`, + history?: CalibrationDraft[`history`], +): CalibrationDraft => ({ + kind, + difference, + ...(history === undefined ? {} : { history }), +}) + +// One wrapper dimension crosses all three public selection receivers. Its +// metadata must never execute, regardless of the selected clean polarity. +const metadataDraft = ( + buildMetadata: `own` | `inherited` | undefined, + tail = 1, + receiver: + `all` | `authored` | `forced-entered` | `forced-not-entered` = `all`, +) => + generatedDraft({ + fixed: [ + { + history: [0, tail], + enters: null, + ...(buildMetadata && receiver !== `forced-entered` + ? { buildMetadata } + : {}), + }, + { + history: [tail], + enters: null, + ...(buildMetadata && + (receiver === `all` || receiver === `forced-entered`) + ? { buildMetadata } + : {}), + }, + ], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: + receiver === `all` || receiver === `authored` ? [`driver`] : [], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: receiver === `authored` ? `none` : `agree`, + probeBinding: receiver === `authored` ? `unbound` : `explicit`, + probeEntered: { fixed: `fixed-0` }, + probeNotEntered: { fixed: `fixed-1` }, + identityHistories: true, + }) +const mixedBindingsDraft = (second: `fixed` | `constructed`) => + generatedDraft({ + fixed: [at([0]), at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + // The last calibration explicitly opts out of this property default. + calibrationHistory: { fixed: `fixed-1` }, + calibrations: [ + cal(`driver`, `none`, { fixed: `fixed-0` }), + cal( + `driver`, + `none`, + second === `fixed` + ? { fixed: `fixed-1` } + : { constructed: `constructed-0` }, + ), + cal(`driver`, `none`, null), + ], + isolatedPlans: true, + }) +const spinDraft = ( + kind: `driver-spin` | `driver-spin-throws`, + bound: boolean, + tail = 1, +) => + generatedDraft({ + fixed: [at([0, tail])], + calibrations: [ + cal(kind, `text match`, bound ? { fixed: `fixed-0` } : null), + ], + identityHistories: true, + }) + +// Add to histories.states with ...mainUnitRequiredStates, remove these names +// from histories.measured, and append ...mainUnitRequiredConstructed to the +// existing histories.constructed. Keep every branch required state/builder. +const mainUnitRequiredStates = [ + `unit-split-verdict`, + `unit-name-load-error`, + `second-fixed-seed`, + `unit-replay-constructed`, + `unit-replay-generated`, + `unit-replay-custom-prefix`, + `unit-rerun`, + `unit-rerun-beside-extension`, + `seedless-units-drawn`, + `replay-environment-error`, + `unit-requires-property`, +] as const +const mainUnitPositiveStates = [ + `unit-split-verdict`, + `second-fixed-seed`, + `unit-replay-constructed`, + `unit-replay-generated`, + `unit-replay-custom-prefix`, + `unit-rerun`, + `unit-rerun-beside-extension`, +] as const +const unitWitnessSuite = ( + drafts: Array, + options: Partial> = {}, +): Suite => ({ + drafts, + seedless: false, + prefix: `ORACLE`, + environment: null, + ...options, +}) +const mainUnitRequiredConstructed: Array<{ + name: string + enters: ReadonlyArray<(typeof mainUnitRequiredStates)[number]> + build: (generated: Suite) => Suite +}> = [ + { + name: `generated UNIT split, named rerun, and custom-prefix replay`, + enters: mainUnitPositiveStates, + build: (generated) => { + const tail = generated.drafts[0]!.clock ? 1 : 2 + return unitWitnessSuite( + [ + generatedDraft({ + source: { kind: `generated`, runs: 2, seed: 5 }, + secondSeed: true, + constructedNames: `prefixed`, + constructed: [ + { prefix: [3, tail], enters: [] }, + { prefix: [0, tail, tail], enters: [`saw-0`] }, + ], + }), + ], + { prefix: `SYNTH_ORACLE` }, + ) + }, + }, + { + name: `generated UNIT history-name gate`, + enters: [`unit-name-load-error`], + build: (generated) => { + const tail = generated.drafts[0]!.clock ? 1 : 2 + const naming = generated.drafts[0]!.constructedNames + return unitWitnessSuite([ + generatedDraft( + naming === `indexed` + ? { + constructedNames: `generated`, + constructed: [{ prefix: [0, tail], enters: [`saw-0`] }], + } + : naming === `prefixed` + ? { + constructedNames: `duplicate`, + constructed: [ + { prefix: [0, tail], enters: [`saw-0`] }, + { prefix: [1, tail], enters: [`saw-1`] }, + ], + } + : { + fixedNames: `duplicate`, + fixed: [at([0, tail]), at([1, tail])], + }, + ), + ]) + }, + }, + { + name: `generated UNITs share their drawn property seed`, + enters: [`seedless-units-drawn`], + build: (generated) => { + const tail = generated.drafts[0]!.clock ? 1 : 2 + return unitWitnessSuite( + [ + generatedDraft({ + constructed: [ + { prefix: [0, tail, tail], enters: [`saw-0`] }, + { prefix: [1, tail, tail], enters: [`saw-1`] }, + ], + }), + ], + { + seedless: true, + seedlessDraw: generated.drafts[0]!.clock ? 0.5 : cleanHostDraw, + }, + ) + }, + }, + { + name: `generated UNIT requires its replay property`, + enters: [`replay-environment-error`, `unit-requires-property`], + build: (generated) => { + const tail = generated.drafts[0]!.clock ? 1 : 2 + return unitWitnessSuite( + [ + generatedDraft({ + constructed: [{ prefix: [0, tail, tail], enters: [`saw-0`] }], + }), + ], + { + prefix: generated.drafts[0]!.clock ? `ORACLE` : `SYNTH_ORACLE`, + environment: replayOf({ + seed: generated.drafts[0]!.clock ? `valid` : `absent`, + path: `absent`, + property: `absent`, + }), + }, + ) + }, + }, +] + +oracleProperty({ + id: `self-oracle.framework`, + law: { statement: `oracleProperty registers, fails, shrinks, replays, reruns, calibrates, times out, and checks determinism, membership, and reach as its contract states.`, authority: `packages/oracles/src/oracle-property.ts header; maintainer decisions of 2026-11-03; oracle rubric ORC-006, ORC-007, ORC-015`, limits: `Synthetic properties with histories of one to three integers, one or two per run; real Vitest child processes; one rerun probe per property.`, }, histories: { arbitrary: suite, - states: [], - measured: frameworkStates, + states: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ...timeoutReceivingStates, + ...mainUnitRequiredStates, + ...callbackReceivingStates, + ...callbackRepresentationStates, + ...round3FrameworkStates, + `auxiliary-late-error`, + `bound-enumerated-label-selector`, + `bound-control-fixed-entered`, + `bound-control-constructed-absent`, + `bound-control-inferred-name-gate`, + `bound-control-inferred-ambiguous`, + `bound-control-inferred-zero`, + `bound-control-explicit-name-gate`, + `bound-equal-fixed-origin`, + `bound-ranked-mutants`, + `bound-independent-premise`, + `bound-name-seed`, + `bound-extra-state`, + `bound-error-context`, + `bound-null-selector`, + `registration-warning-policy`, + `bound-difference`, + `bound-fixed-identity`, + `bound-enumerated-identity`, + `bound-combined-controls`, + `unbound-prototype-probe`, + `bound-control-full-reach`, + `bound-calibration`, + `bound-calibration-registration-error`, + `bound-constructed-driver`, + ], + measured: frameworkStates.filter( + (state) => + state !== `timeout-presentation-equivalence` && + state !== `timeout-continuation-cut` && + !timeoutReceivingStates.some((required) => required === state) && + state !== `auxiliary-late-error` && + !mainUnitRequiredStates.some((required) => required === state) && + !callbackReceivingStates.some((required) => required === state) && + !callbackRepresentationStates.some((required) => required === state) && + !round3FrameworkStates.some((required) => required === state) && + !state.startsWith(`bound-`) && + state !== `unbound-prototype-probe` && + state !== `registration-warning-policy`, + ), runs: 3, fixedSeeds: [1], + constructed: [ + { + name: `timeout every receiving cut`, + enters: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ...timeoutReceivingStates, + ], + build: (generated: Suite) => + timeoutReceivingSuite(generated.drafts[0]!.clock ? 1 : 2), + }, + ...([false, true] as const).map((bound) => ({ + name: `timeout ${bound ? `bound` : `isolated`} construction`, + enters: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ] as const, + build: (generated: Suite) => + timeoutInputSuite(bound, generated.drafts[0]!.clock ? 1 : 2), + })), + { + name: `timeout continuation presentation`, + enters: [`timeout-presentation-equivalence`], + build: (generated: Suite) => + timeoutRepresentationSuite(generated.drafts[0]!.clock ? 1 : 2), + }, + ...mainUnitRequiredConstructed, + { + name: `round8 callback presentation freedom`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + build: (generated: Suite) => + callbackRepresentationSuite( + generated.drafts[0]!.clock + ? `corrupt-throws` + : `judge-driver-throws`, + `constructed`, + generated.drafts[0]!.isolatedPlans, + `alternate`, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + }, + { + name: `round11 outer callback wrapper`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + build: (generated: Suite) => ({ + ...callbackRepresentationSuite( + generated.drafts[0]!.clock + ? `corrupt-throws` + : `judge-driver-throws`, + `constructed`, + generated.drafts[0]!.isolatedPlans, + `alternate`, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + callbackIncomingPresentation: + generated.callbackIncomingPresentation ?? { + calibration: `bracketed`, + seed: `parenthesized`, + spacing: [2, `\t`, 3, ` \t `, 2, 1], + }, + callbackReason: generated.callbackReason ?? `but the check passed it`, + callbackWrapperName: + generated.callbackWrapperName ?? `CalibrationFailure`, + }), + }, + { + name: `round10 opaque callback reason`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + build: (generated: Suite) => ({ + ...callbackRepresentationSuite( + generated.drafts[0]!.clock + ? `corrupt-throws` + : `judge-driver-throws`, + `constructed`, + generated.drafts[0]!.isolatedPlans, + `alternate`, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + callbackIncomingPresentation: + generated.callbackIncomingPresentation ?? { + calibration: `bracketed`, + seed: `parenthesized`, + spacing: [2, `\t`, 3, ` \t `, 2, 1], + }, + callbackReason: + generated.callbackReason ?? + (generated.drafts[0]!.clock + ? `but the check passed it` + : `calibration [unrelated] on fixed history: c.1: selected another context`), + }), + }, + { + name: `round9 incoming callback context`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + build: (generated: Suite) => ({ + ...callbackRepresentationSuite( + generated.drafts[0]!.clock + ? `corrupt-throws` + : `judge-driver-throws`, + `constructed`, + generated.drafts[0]!.isolatedPlans, + `alternate`, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + callbackIncomingPresentation: + generated.callbackIncomingPresentation ?? { + calibration: `bracketed`, + seed: `parenthesized`, + spacing: [3, 3, 3, 3, 3, 1], + }, + }), + }, + ...callbackKinds.map((kind, ordinal) => ({ + name: `round7 callback diagnostics ${kind}`, + enters: + ordinal === 2 + ? callbackReceivingStates.filter( + (state) => state !== `callback-failure-binding-reordered`, + ) + : [...callbackReceivingStates], + build: (generated: Suite) => + callbackDiagnosticSuite( + kind, + callbackSelections[ordinal]!, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + })), + ...([0, 2] as const).map((entering) => ({ + name: `inferred control with ${entering} distinct entering constructions`, + enters: [ + entering === 0 + ? (`bound-control-inferred-zero` as const) + : (`bound-control-inferred-ambiguous` as const), + ], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: entering === 2 ? [`saw-0`] : [] }, + { + prefix: [0, generated.drafts[0]!.clock ? 1 : 2], + enters: entering === 2 ? [`saw-0`] : [], + }, + ], + probe: `agree`, + probeBinding: `inferred`, + }), + ], + }), + })), + { + name: `fixture preserves outer host RNG`, + enters: [ + `host-rng-scope`, + `receiving-host-scope`, + `host-rng-forwarding`, + ], + build: (generated: Suite) => ({ + ...generated, + hostRngDraw: generated.drafts[0]!.clock + ? (0.25 as const) + : (0.75 as const), + hostRngForwarding: `alternate` as const, + drafts: [generatedDraft({})], + }), + }, + { + name: `round3 every actual receiving cut`, + enters: [ + ...receivingCutStates, + `warning-order-equivalence`, + ...warningReceivingKinds.map( + (kind) => `warning-receiving-${kind}` as const, + ), + ], + build: (generated: Suite) => ({ + ...generated, + seedless: true, + permuteWarnings: true, + prefix: generated.drafts[0]!.clock + ? (`ORACLE` as const) + : (`SYNTH_ORACLE` as const), + environment: replayOf({ + path: `absent`, + property: `absent`, + unit: `absent`, + }), + drafts: receivingCutDrafts(generated.drafts[0]!.clock ? 1 : 2), + }), + }, + { + name: `round4 successful campaign budgets and failing neighbor`, + enters: [ + ...passingUnitStates, + ...passingBudgetLanes.map( + ({ context, lane }) => `passing-budget-${context}-${lane}` as const, + ), + `passing-budget-constructed`, + `campaign-budget-failure-neighbor`, + ], + build: (generated: Suite) => + campaignBudgetSuite( + 2, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? 1 : 2, + ), + }, + { + name: `round3 replayed clock failure retains driver cause`, + enters: [`receiving-replay`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + clock: true, + throwOnSpin: true, + constructed: [{ prefix: [7], enters: [] }], + }), + ], + }), + }, + { + name: `round3 named reach counts permit key reorder`, + enters: [`enumerated-reach-key-order-neighbor`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [0, 1], [2]] }, + enumeratedStates: generated.drafts[0]!.clock + ? [`measured`, `measured`, `required`] + : [`measured`, `required`, `measured`], + }), + ], + }), + }, + ...([`own`, `inherited`] as const).flatMap((buildMetadata) => + ([`authored`, `forced-entered`, `forced-not-entered`] as const).map( + (receiver) => ({ + name: `round3 ${buildMetadata} fixed metadata ${receiver}`, + enters: [`bound-fixed-${buildMetadata}-metadata-${receiver}`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + metadataDraft( + buildMetadata, + generated.drafts[0]!.clock ? 1 : 2, + receiver, + ), + ], + }), + }), + ), + ), + { + name: `round3 independently selected authored calibrations`, + enters: [ + `bound-authored-distinct-selectors`, + `bound-authored-search-neighbor`, + ], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + mixedBindingsDraft( + generated.drafts[0]!.clock ? `fixed` : `constructed`, + ), + ], + }), + }, + { + name: `round3 recorded violation before mutant clock spin`, + enters: [ + `bound-mutant-violation-before-spin`, + `legacy-mutant-violation-before-spin`, + ], + build: (generated: Suite) => { + const kind = generated.drafts[0]!.clock + ? `driver-spin` + : `driver-spin-throws` + return { + ...generated, + drafts: [spinDraft(kind, true), spinDraft(kind, false)], + } + }, + }, + { + name: `review auxiliary late errors`, + enters: [`auxiliary-late-error`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + probe: `late`, + clock: true, + strictModel: true, + constructed: [{ prefix: [7, 0], enters: [] }], + }), + ], + }), + }, + { + name: `review enumerated selector labels`, + enters: [`bound-enumerated-label-selector`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + source: { kind: `enumerated`, histories: [[0, 1]] }, + labelStyle: generated.drafts[0]!.clock ? `shell` : `default`, + calibrationHistory: { + enumerated: generated.drafts[0]!.clock + ? `items 0 | 1 ${shellText}` + : `[0,1]`, + }, + calibrations: [`right`], + }), + ], + }), + }, + { + name: `round3 rejected named enumeration selector`, + enters: [`bound-enumerated-label-rejected`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + source: { kind: `enumerated`, histories: [[0, 1]] }, + labelStyle: generated.drafts[0]!.clock ? `shell` : `default`, + calibrationHistory: { enumerated: `items 0 | 1` }, + calibrations: [`right`], + }), + ], + }), + }, + { + name: `review fixed entered control`, + enters: [`bound-control-fixed-entered`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1]), at([0, generated.drafts[0]!.clock ? 1 : 2])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `explicit`, + probeEntered: { fixed: `fixed-1` }, + identityHistories: true, + }), + ], + }), + }, + { + name: `review constructed absent control`, + enters: [`bound-control-constructed-absent`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { + prefix: [1, generated.drafts[0]!.clock ? 1 : 2, 2], + enters: [], + }, + ], + probe: `agree`, + probeBinding: generated.drafts[0]!.clock + ? `explicit` + : `inferred`, + probeNotEntered: { constructed: `constructed-1` }, + isolatedPlans: true, + }), + ], + }), + }, + ...([`inferred`, `explicit`] as const).map((probeBinding) => ({ + name: `review ${probeBinding} ambiguous control`, + enters: [`bound-control-${probeBinding}-name-gate` as const], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [generated.drafts[0]!.clock ? 1 : 2], enters: [] }, + ], + probe: `agree`, + probeBinding, + duplicateHistoryNames: true, + calibrationHistory: { fixed: `fixed-0` }, + }), + ], + }), + })), + { + name: `review equal fixed originals`, + enters: [`bound-equal-fixed-origin`], + build: (generated: Suite) => { + const history = [0, generated.drafts[0]!.clock ? 1 : 2] + return { + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([...history]), at([...history])], + calibrationHistory: { fixed: `fixed-1` }, + calibrations: [`driver`], + identityHistories: true, + }), + ], + } + }, + }, + { + name: `round2 ranked mutant verdicts`, + enters: [`bound-ranked-mutants`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [generated.drafts[0]!.clock ? `clock` : `throws`], + }), + ], + }), + }, + { + name: `round2 independent forced premises`, + enters: [`bound-independent-premise`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1]), at([0])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [0, 2], enters: [`saw-0`] }, + ], + probe: `agree`, + probeBinding: `explicit`, + isolatedPlans: true, + }), + ], + }), + }, + { + name: `round2 named seed variation`, + enters: [`bound-name-seed`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + constructed: [{ prefix: [0], enters: [`saw-0`] }], + calibrationHistory: { constructed: `constructed-0` }, + calibrations: [`right`], + calibrationNames: generated.drafts[0]!.clock + ? `astral` + : `varied`, + isolatedPlans: true, + }), + ], + }), + }, + { + name: `round2 declared clean reach`, + enters: [`bound-extra-state`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`right`], + extraReach: `calibration`, + }), + ], + }), + }, + { + name: `round2 contextual callback errors`, + enters: [`bound-error-context`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [ + generated.drafts[0]!.clock + ? `judge-driver-throws` + : `corrupt-throws`, + ], + }), + ], + }), + }, + { + name: `round2 invalid explicit probe selectors`, + enters: [`bound-null-selector`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: generated.drafts[0]!.clock + ? `null-entered` + : `null-binding`, + }), + ], + }), + }, + { + name: `round2 registration warning phases`, + enters: [ + `registration-warning-policy`, + `bound-registration-warning-silence`, + ], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + warningProbe: true, + calibrations: [`right`, `driver`], + }), + generatedDraft({ + fixed: [at([0, generated.drafts[0]!.clock ? 1 : 2])], + calibrationHistory: { fixed: `fixed-0` }, + warningProbe: true, + calibrations: [`right`, `driver`], + }), + ], + }), + }, + { + name: `round2 selected difference criterion`, + enters: [`bound-difference`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([0]), at([1, 1, 0])], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [ + cal( + `driver-last`, + generated.drafts[0]!.clock ? `later pattern` : `later text`, + ), + ], + }), + ], + }), + }, + ...([`fixed`, `enumerated`] as const).map((source) => ({ + name: `selected ${source} preserves identity`, + enters: [ + source === `fixed` + ? `bound-fixed-identity` + : `bound-enumerated-identity`, + ] as const, + build: (generated: Suite) => { + const varied = generated.drafts[0]! + const history = [ + 0, + varied.source.kind === `generated` ? varied.source.seed % 3 : 1, + ] + return { + ...generated, + drafts: [ + generatedDraft({ + ...(source === `fixed` + ? { + fixed: [at(history)], + calibrationHistory: { fixed: `fixed-0` }, + } + : { + source: { kind: `enumerated`, histories: [history] }, + calibrationHistory: { enumerated: 0 }, + }), + form: source === `fixed` && varied.clock ? `boxed` : `plain`, + identityHistories: true, + isolatedPlans: varied.isolatedPlans, + calibrations: [varied.clock ? `driver` : `right`], + }), + ], + } + }, + })), + { + name: `authored bindings coexist with automatic controls`, + enters: [`bound-combined-controls`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + calibrationHistory: { constructed: `constructed-0` }, + calibrations: [generated.drafts[0]!.clock ? `right` : `driver`], + isolatedPlans: generated.drafts[0]!.isolatedPlans, + probe: `agree`, + probeName: generated.drafts[0]!.probeName, + probeBinding: generated.drafts[0]!.clock + ? `inferred` + : `explicit`, + }), + ], + }), + }, + { + name: `empty tables leave arbitrary probe names unbound`, + enters: [`unbound-prototype-probe`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `empty`, + probeName: + generated.drafts[0]!.probeName === `saw-0` + ? `constructor` + : generated.drafts[0]!.probeName, + }), + ], + }), + }, + { + name: `automatic controls require all selected states`, + enters: [`bound-control-full-reach`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [ + { + history: [1], + enters: generated.drafts[0]!.clock ? [`saw-2`] : null, + }, + ], + constructed: [ + { + prefix: [0, 0, 0], + enters: generated.drafts[0]!.clock + ? [`saw-0`] + : [`saw-0`, `saw-1`], + }, + ], + probe: `agree`, + probeBinding: generated.drafts[0]!.isolatedPlans + ? `explicit` + : `inferred`, + }), + ], + }), + }, + { + name: `bound constructed drivers rebuild seeded plans`, + enters: [`bound-constructed-driver`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([6])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + calibrations: [`driver`], + calibrationHistory: { constructed: `constructed-0` }, + }), + ], + }), + }, + { + name: `bound calibrations bypass unrelated histories`, + enters: [`bound-calibration`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + // The decoy fails if scanned. The selected construction keeps a + // generated tail; builder identity and fresh resolution are checked. + fixed: [at([6])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + calibrations: [generated.drafts[0]!.clock ? `driver` : `right`], + calibrationHistory: { constructed: `constructed-0` }, + }), + ], + }), + }, + { + name: `invalid bindings fail before execution`, + enters: [`bound-calibration-registration-error`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([0])], + calibrations: [`right`], + calibrationHistory: { fixed: `unknown` }, + }), + ], + }), + }, + { + name: `bound forced controls retain both clean premises`, + enters: [`bound-calibration`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + probe: `agree`, + probeBinding: generated.drafts[0]!.clock + ? `inferred` + : `explicit`, + }), + ], + }), + }, + ], fixed: [ + { + name: `timeout every receiving cut`, + enters: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ...timeoutReceivingStates, + ], + history: timeoutReceivingSuite(), + }, + ...([false, true] as const).map((bound) => ({ + name: `timeout ${bound ? `bound` : `isolated`} construction`, + enters: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ] as const, + history: timeoutInputSuite(bound), + })), + { + name: `timeout continuation presentation`, + enters: [`timeout-presentation-equivalence`], + history: timeoutRepresentationSuite(), + }, + ...([`wrapper`, `reason`, `combined`] as const).map((axis) => ({ + name: `round11 outer wrapper ${axis}`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + ...(axis === `reason` + ? {} + : { callbackWrapperName: `CalibrationFailure` }), + ...(axis === `wrapper` + ? {} + : { callbackReason: `but the check passed it` }), + }, + })), + ...( + [`but the check passed it`, `but the check reported law`] as const + ).map((reason) => ({ + name: `round10 opaque reason ${reason}`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + callbackReason: reason, + }, + })), + ...([false, true] as const).map((identityWords) => ({ + name: identityWords + ? `round10 opaque reason contains identity words` + : `round10 opaque callback reason`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + callbackIncomingPresentation: { + calibration: `bracketed` as const, + seed: `parenthesized` as const, + spacing: [2, `\t`, 3, ` \t `, 2, 1] as [ + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + ], + }, + callbackReason: identityWords + ? `calibration [unrelated] on fixed history: c.1: selected another context` + : `corruption callback failed`, + }, + })), + { + name: `round9 incoming positive whitespace`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + callbackIncomingPresentation: { + calibration: `bracketed`, + seed: `parenthesized`, + spacing: [2, `\t`, 3, ` \t `, 2, 1], + }, + }, + }, + ...([false, true] as const).map((mixed) => ({ + name: mixed + ? `round9 incoming mixed separators` + : `round9 incoming bracketed context`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + callbackIncomingPresentation: { + calibration: `bracketed` as const, + seed: `parenthesized` as const, + spacing: mixed + ? ([3, 3, 3, 3, 3, 1] as [ + 1 | 3, + 1 | 3, + 1 | 3, + 1 | 3, + 1 | 3, + 1 | 3, + ]) + : (`plain` as const), + }, + }, + })), + { + name: `round8 callback presentation freedom`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + }, + ...callbackKinds.map((kind, ordinal) => ({ + name: `round7 callback diagnostics ${kind}`, + enters: + ordinal === 2 + ? callbackReceivingStates.filter( + (state) => state !== `callback-failure-binding-reordered`, + ) + : [...callbackReceivingStates], + history: callbackDiagnosticSuite(kind, callbackSelections[ordinal]!), + })), + { + name: `custom prefix preserves every receiving cut`, + enters: [ + ...receivingCutStates, + ...warningReceivingKinds.map( + (kind) => `warning-receiving-${kind}` as const, + ), + ], + history: { + drafts: receivingCutDrafts(), + seedless: true, + permuteWarnings: true, + prefix: `SYNTH_ORACLE` as const, + environment: replayOf({ + path: `absent`, + property: `absent`, + unit: `absent`, + }), + }, + }, + ...([0, 2] as const).map((entering) => + fixedHistory( + `inferred control with ${entering} distinct entering constructions`, + [ + entering === 0 + ? `bound-control-inferred-zero` + : `bound-control-inferred-ambiguous`, + ], + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: entering === 2 ? [`saw-0`] : [] }, + { prefix: [0, 1], enters: entering === 2 ? [`saw-0`] : [] }, + ], + probe: `agree`, + probeBinding: `inferred`, + }), + ), + ), + ...([0.25, 0.75] as const).map((hostRngDraw) => ({ + name: `fixture preserves outer host RNG ${hostRngDraw}`, + enters: [`host-rng-scope` as const, `receiving-host-scope` as const], + history: { drafts: [generatedDraft({})], seedless: false, hostRngDraw }, + })), + ...([1, 2, `alternate`] as const).map((hostRngForwarding) => ({ + name: `fixture preserves forwarded outer host RNG ${hostRngForwarding}`, + enters: [ + `host-rng-scope` as const, + `receiving-host-scope` as const, + `host-rng-forwarding` as const, + ], + history: { + drafts: [generatedDraft({})], + seedless: false, + hostRngDraw: 0.25 as const, + hostRngForwarding, + }, + })), + { + name: `round3 every actual receiving cut`, + enters: [ + ...receivingCutStates, + `warning-order-equivalence`, + ...warningReceivingKinds.map( + (kind) => `warning-receiving-${kind}` as const, + ), + ], + history: { + drafts: receivingCutDrafts(), + seedless: true, + permuteWarnings: true, + environment: replayOf({ + path: `absent`, + property: `absent`, + unit: `absent`, + }), + }, + }, + { + name: `round5 warning-order equivalence`, + enters: [`warning-order-equivalence`], + history: warningRepresentationSuite(), + }, + { + name: `round4 successful campaign budgets and failing neighbor`, + enters: [ + ...passingUnitStates, + ...passingBudgetLanes.map( + ({ context, lane }) => `passing-budget-${context}-${lane}` as const, + ), + `passing-budget-constructed`, + `campaign-budget-failure-neighbor`, + ], + history: campaignBudgetSuite(), + }, + { + name: `round4 successful campaign budget without constructions`, + enters: passingBudgetLanes.map( + ({ context, lane }) => `passing-budget-${context}-${lane}` as const, + ), + history: campaignBudgetSuite(2, 0), + }, + { + name: `round4 larger successful fixed and seedless budgets`, + enters: [ + `passing-budget-run-fixed`, + `passing-budget-seedless-fixed`, + `passing-budget-seedless-random`, + ], + history: campaignBudgetSuite(3, 2), + }, + fixedHistory( + `round3 replayed clock failure retains driver cause`, + [`receiving-replay`], + generatedDraft({ + clock: true, + throwOnSpin: true, + constructed: [{ prefix: [7], enters: [] }], + }), + ), + fixedHistory( + `round3 named reach count key order neighbor`, + [`enumerated-reach-key-order-neighbor`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [0, 1], [2]] }, + enumeratedStates: [`measured`, `measured`, `required`], + }), + ), + fixedHistory( + `round3 fully bound warning phases stay silent`, + [`bound-registration-warning-silence`], + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + warningProbe: true, + calibrations: [`right`, `driver`], + }), + ), + fixedHistory( + `round3 plain fixed declaration metadata neighbor`, + [], + metadataDraft(undefined), + ), + ...([`own`, `inherited`] as const).flatMap((buildMetadata) => + ([`authored`, `forced-entered`, `forced-not-entered`] as const).map( + (receiver) => + fixedHistory( + `round3 ${buildMetadata} fixed metadata ${receiver} receiver`, + [`bound-fixed-${buildMetadata}-metadata-${receiver}`], + metadataDraft(buildMetadata, 1, receiver), + ), + ), + ), + ...([`fixed`, `constructed`] as const).map((second) => + fixedHistory( + `round3 distinct ${second} and unbound authored selectors`, + [ + `bound-authored-distinct-selectors`, + `bound-authored-search-neighbor`, + ], + mixedBindingsDraft(second), + ), + ), + ...([`driver-spin`, `driver-spin-throws`] as const).flatMap((kind) => + [true, false].map((bound) => + fixedHistory( + `round3 ${kind} after recorded violation ${bound ? `bound` : `legacy`} receiver`, + [ + bound + ? `bound-mutant-violation-before-spin` + : `legacy-mutant-violation-before-spin`, + ], + spinDraft(kind, bound), + ), + ), + ), + fixedHistory( + `review auxiliary late errors preserve counts`, + [`auxiliary-late-error`], + generatedDraft({ + probe: `late`, + clock: true, + strictModel: true, + constructed: [{ prefix: [7, 0], enters: [] }], + }), + ), + ...([`shell`, `default`] as const).flatMap((labelStyle) => + [true, false].map((valid) => + fixedHistory( + `review ${labelStyle} enumerated selector ${valid ? `exact` : `missing`}`, + [ + valid + ? `bound-enumerated-label-selector` + : `bound-enumerated-label-rejected`, + ], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0, 1]] }, + labelStyle, + calibrationHistory: { + enumerated: valid + ? labelStyle === `shell` + ? `items 0 | 1 ${shellText}` + : `[0,1]` + : `items 0 | 1`, + }, + calibrations: [`right`], + identityHistories: true, + }), + ), + ), + ), + ...([`inferred`, `explicit`] as const).map((probeBinding) => + fixedHistory( + `review duplicate selected ${probeBinding} names`, + [`bound-control-${probeBinding}-name-gate`], + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [1], enters: [] }, + ], + probe: `agree`, + probeBinding, + duplicateHistoryNames: true, + calibrationHistory: { fixed: `fixed-0` }, + }), + ), + ), + fixedHistory( + `review fixed entered selector`, + [`bound-control-fixed-entered`], + generatedDraft({ + fixed: [at([1]), at([0, 1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `explicit`, + probeEntered: { fixed: `fixed-1` }, + identityHistories: true, + }), + ), + fixedHistory( + `review constructed absent selector`, + [`bound-control-constructed-absent`], + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [1, 2, 2], enters: [] }, + ], + probe: `agree`, + probeBinding: `explicit`, + probeNotEntered: { constructed: `constructed-1` }, + isolatedPlans: true, + }), + ), + fixedHistory( + `review equal fixed input origins`, + [`bound-equal-fixed-origin`], + generatedDraft({ + fixed: [at([0]), at([0])], + calibrationHistory: { fixed: `fixed-1` }, + calibrations: [`driver`], + identityHistories: true, + }), + ), + ...([`throws`, `clock`] as const).map((kind) => + fixedHistory( + `round2 ranked ${kind} bound and legacy`, + [`bound-ranked-mutants`], + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [kind], + identityHistories: true, + }), + generatedDraft({ fixed: [at([0])], calibrations: [kind] }), + ), + ), + fixedHistory( + `round2 forced premise origins`, + [`bound-independent-premise`], + generatedDraft({ + fixed: [at([1]), at([0])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [0, 2], enters: [`saw-0`] }, + ], + probe: `agree`, + probeBinding: `explicit`, + isolatedPlans: true, + }), + ), + ...([`varied`, `astral`] as const).map((calibrationNames) => + fixedHistory( + `round2 ${calibrationNames} calibration seeds`, + [`bound-name-seed`], + generatedDraft({ + constructed: [{ prefix: [0], enters: [`saw-0`] }], + calibrationHistory: { constructed: `constructed-0` }, + calibrations: [`right`], + calibrationNames, + isolatedPlans: true, + }), + ), + ), + fixedHistory( + `round2 bound calibrations keep stdout quiet`, + [], + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`right`], + }), + ), + fixedHistory( + `round2 undeclared clean source reach`, + [`bound-extra-state`], + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`right`], + extraReach: `calibration`, + }), + ), + fixedHistory( + `round2 declared enumerated clean source reach`, + [`bound-extra-state`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [1]] }, + enumeratedStates: [`required`, `measured`, `measured`], + calibrationHistory: { enumerated: 0 }, + calibrations: [`right`], + extraReach: `calibration`, + }), + ), + fixedHistory( + `round2 generated fixed history rejects undeclared reach`, + [], + generatedDraft({ fixed: [at([0])], extraReach: true }), + ), + ...( + [`corrupt-throws`, `judge-throws`, `judge-driver-throws`] as const + ).map((kind) => + fixedHistory( + `round2 context ${kind}`, + [`bound-error-context`], + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [kind], + }), + ), + ), + fixedHistory( + `round2 context clean invalid observation`, + [], + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`right`], + odd: `class`, + }), + ), + ...( + [ + `null-entered`, + `null-binding`, + `false-binding`, + `zero-binding`, + ] as const + ).map((probeBinding) => + fixedHistory( + `round2 invalid ${probeBinding}`, + [`bound-null-selector`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding, + }), + ), + ), + ...([`empty text`, `empty pattern`] as const).flatMap((difference) => + ( + [ + `null-binding`, + `false-binding`, + `zero-binding`, + `no-absence`, + `inferred`, + `null-entered`, + `explicit`, + ] as const + ).map((probeBinding) => + fixedHistory( + `registration phases ${probeBinding} with ${difference}`, + [`registration-error`], + generatedDraft({ + fixed: [at([1])], + // Inferred entry lacks a unique construction; explicit entry + // selects a missing one. These fail in different phases. + constructed: + probeBinding === `inferred` || probeBinding === `explicit` + ? [] + : [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding, + calibrations: [cal(`right`, difference)], + }), + ), + ), + ), + fixedHistory( + `round2 warning repeat and manifest phases`, + [`registration-warning-policy`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + warningProbe: true, + calibrations: [`right`, `driver`], + }), + ), + ...([`fixed`, `enumerated`, `constructed`] as const).map((selectedKind) => + fixedHistory( + `round2 selected ${selectedKind} difference never searches`, + [`bound-difference`], + generatedDraft({ + ...(selectedKind === `enumerated` + ? { + source: { + kind: `enumerated`, + histories: [[0], [1, 1, 0]], + }, + calibrationHistory: { enumerated: 0 }, + } + : selectedKind === `fixed` + ? { + fixed: [at([0]), at([1, 1, 0])], + calibrationHistory: { fixed: `fixed-0` }, + } + : { + fixed: [at([1, 1, 0])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + calibrationHistory: { + constructed: `constructed-0`, + }, + isolatedPlans: true, + }), + calibrations: [ + cal(`driver-last`, `later text`), + cal(`right`, `pattern match`), + ], + }), + ), + ), + ...([false, true] as const).map((isolatedPlans) => + fixedHistory( + `review combined controls isolated=${isolatedPlans}`, + [`bound-combined-controls`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans, + probe: `agree`, + probeBinding: `explicit`, + calibrationHistory: { constructed: `constructed-0` }, + calibrations: [`driver`], + }), + ), + ), + fixedHistory( + `review entered control full reach`, + [`bound-control-full-reach`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0, 0, 0], enters: [`saw-0`, `saw-1`] }], + probe: `agree`, + probeBinding: `inferred`, + }), + ), + fixedHistory( + `review absence control full reach`, + [`bound-control-full-reach`], + generatedDraft({ + fixed: [{ history: [1], enters: [`saw-2`] }], + constructed: [{ prefix: [0, 0, 0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `inferred`, + }), + ), + ...([`saw-0`, `toString`, `constructor`, `__proto__`] as const).map( + (probeName) => + fixedHistory( + `review empty binding table ${probeName}`, + [], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `empty`, + probeName, + }), + ), + ), + ...([`toString`, `constructor`, `__proto__`] as const).map((probeName) => + fixedHistory( + `review own binding table ${probeName}`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `explicit`, + probeName, + isolatedPlans: true, + }), + ), + ), + fixedHistory( + `review enumerated alpha naming and invalid data precedence`, + [], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [1]] }, + probeName: `toString`, + identityHistories: true, + enumeratedStates: [`required`, `measured`, `measured`], + }), + generatedDraft({ + source: { kind: `enumerated`, histories: [[0]] }, + identityHistories: true, + form: `trap`, + }), + ), + fixedHistory( + `review fixed private slots without resolve`, + [], + generatedDraft({ + fixed: [at([0])], + form: `boxed`, + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`driver`], + }), + ), + fixedHistory( + `review fixed plain identity metadata`, + [`bound-fixed-identity`], + generatedDraft({ + fixed: [at([0])], + identityHistories: true, + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`driver`], + }), + ), + fixedHistory( + `review fixed identity and private slots`, + [`bound-fixed-identity`], + generatedDraft({ + fixed: [at([0])], + form: `boxed`, + identityHistories: true, + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`driver`], + }), + ), + fixedHistory( + `review enumerated identity metadata`, + [`bound-enumerated-identity`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0]] }, + identityHistories: true, + calibrationHistory: { enumerated: 0 }, + calibrations: [`driver`], + }), + ), + fixedHistory( + `selected clean driver errors cannot search for another history`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([6]), at([0])], + calibrations: [`driver`], + calibrationHistory: { fixed: `fixed-0` }, + }), + ), + fixedHistory( + `bound constructed drivers rebuild seeded plans`, + [`bound-constructed-driver`], + generatedDraft({ + fixed: [at([6])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + calibrations: [`driver`], + calibrationHistory: { constructed: `constructed-0` }, + }), + ), + fixedHistory( + `duplicate enumerated labels fail before execution`, + [`bound-calibration-registration-error`], + generatedDraft({ + source: { + kind: `enumerated`, + histories: [ + [0, 1], + [1, 0], + ], + }, + duplicateHistoryNames: true, + calibrations: [`right`], + calibrationHistory: { enumerated: `items 0 | 1` }, + }), + ), + ...([`inferred`, `explicit`] as const).map((probeBinding) => + fixedHistory( + `bound forced controls use ${probeBinding} entered history`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + probe: `agree`, + probeBinding, + }), + ), + ), + fixedHistory( + `probe inference rejects ambiguous and missing constructions`, + [`registration-error`], + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [0], enters: [`saw-0`] }, + ], + probe: `agree`, + probeBinding: `inferred`, + }), + generatedDraft({ + fixed: [at([1])], + probe: `agree`, + probeBinding: `inferred`, + }), + ), + fixedHistory( + `a bound probe control needs an absent history`, + [`registration-error`], + generatedDraft({ probe: `agree`, probeBinding: `no-absence` }), + ), + fixedHistory( + `fixed and constructed duplicate selectors fail before execution`, + [`bound-calibration-registration-error`], + generatedDraft({ + fixed: [at([0]), at([1])], + duplicateHistoryNames: true, + calibrations: [`right`], + calibrationHistory: { fixed: `fixed-0` }, + }), + generatedDraft({ + constructed: [ + { prefix: [0], enters: [] }, + { prefix: [1], enters: [] }, + ], + duplicateHistoryNames: true, + calibrations: [`right`], + calibrationHistory: { constructed: `constructed-0` }, + }), + ), + fixedHistory( + `bound calibrations use only their selected histories`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([6]), at([0, 1])], + calibrations: [`right`], + calibrationHistory: { fixed: `fixed-1` }, + }), + generatedDraft({ + source: { kind: `enumerated`, histories: [[6], [0, 2]] }, + calibrations: [`driver`], + calibrationHistory: { enumerated: 1 }, + }), + ), + ...( + [`right`, `no-op`, `wrong-law`, `inapplicable`, `throws`] as const + ).map((kind) => + fixedHistory( + `bound ${kind} requires the designated rejection`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([6]), at([0])], + calibrations: [kind], + calibrationHistory: { fixed: `fixed-1` }, + }), + ), + ), + fixedHistory( + `a failed selected clean run cannot search for another`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([3]), at([0])], + calibrations: [`right`], + calibrationHistory: { fixed: `fixed-0` }, + }), + ), + fixedHistory( + `a selected history must prove its declared reach`, + [`bound-calibration`], + generatedDraft({ + fixed: [{ history: [0], enters: [`saw-1`] }], + calibrations: [`right`], + calibrationHistory: { fixed: `fixed-0` }, + }), + ), + ...( + [ + { fixed: `missing` }, + { constructed: `missing` }, + { enumerated: -1 }, + { enumerated: 1.5 }, + { enumerated: 2 }, + { fixed: `fixed-0` }, + ] as const + ).map((binding, index) => + fixedHistory( + `invalid binding ${index} fails at registration`, + [`bound-calibration-registration-error`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [1]] }, + calibrations: [`right`], + calibrationHistory: binding, + }), + ), + ), + fixedHistory( + `enumerated histories can be selected by label`, + [`bound-calibration`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[6], [0, 1]] }, + calibrations: [`right`], + calibrationHistory: { enumerated: `items 0 | 1` }, + }), + ), fixedHistory(`a rerun command escapes a label with |`, [`rerun-probe`], { ...generatedDraft({}), source: { @@ -4835,6 +10781,738 @@ oracleProperty({ reached: frameworkReach, driver: driveSuite, check: checkSuite, + calibrations: [ + ...timeoutReceivingKinds.map((receiver) => ({ + name: `reject timeout ${receiver} input work`, + history: { fixed: `timeout every receiving cut` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find( + ({ checkpoint, value }) => + checkpoint === `child` && + (value as ChildRun).context.kind === receiver, + )!.value as ChildRun + child.timeoutInputs![0]!.resolutionDifference = 1 + return altered + }, + })), + ...( + [ + `resolution`, + `construction`, + `changed input`, + `changed build`, + `determinism calls`, + `calibration calls`, + `missing relation`, + `selected origin`, + ] as const + ).map((fault) => ({ + name: `reject timeout input ${fault}`, + history: { fixed: `timeout bound construction` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find( + ({ checkpoint, value }) => + checkpoint === `child` && + (value as ChildRun).context.kind === `run`, + )!.value as ChildRun + const owner = child.timeoutInputs!.find(({ test }) => + test.endsWith(` > calibration: cal-0`), + )! + if (fault === `resolution`) owner.resolutionDifference = 1 + if (fault === `construction`) owner.buildDifference = 1 + if (fault === `changed input`) owner.changedInput = true + if (fault === `changed build`) owner.changedBuild = true + if (fault === `determinism calls`) owner.excessDeterminismCalls = 1 + if (fault === `calibration calls`) owner.excessCalibrationCalls = 1 + if (fault === `missing relation`) + child.timeoutInputs!.splice(child.timeoutInputs!.indexOf(owner), 1) + if (fault === `selected origin`) + for (const logged of ordinaryInvocationLogs(altered)) { + const entry = logged.find(({ test }) => + test.endsWith(` > calibration: cal-0`), + )! + entry.origin = `constructed:unrelated` + } + return altered + }, + })), + ...( + [ + `missing late`, + `excess late`, + `status`, + `timeout`, + `registration`, + `exit`, + `unhandled`, + `presentation`, + ] as const + ).map((fault) => ({ + name: `reject timeout continuation ${fault}`, + history: { fixed: `timeout continuation presentation` }, + law: + fault === `missing late` || fault === `excess late` + ? `agreement` + : fault === `registration` + ? `registration` + : fault === `exit` || fault === `unhandled` + ? `trace` + : `timeout`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find( + ({ checkpoint, value }) => + checkpoint === `child` && + (value as ChildRun).context.kind === `run`, + )!.value as ChildRun + const timedOut = child.tests.find( + ({ id, state }) => id === syntheticIds[0] && state === `failed`, + )! + if (fault === `missing late`) child.lateAgreement!.belowMinimum = 1 + if (fault === `excess late`) child.lateAgreement!.aboveMaximum = 1 + if (fault === `status`) timedOut.state = `passed` + if (fault === `timeout`) timedOut.errors[0] = { kind: `other` } + if (fault === `registration`) + child.tests.splice(child.tests.indexOf(timedOut), 1) + if (fault === `exit`) child.exit = 0 + if (fault === `unhandled`) + child.unhandled.push(`unrelated timeout teardown error`) + if (fault === `presentation`) + ( + altered.find( + ({ checkpoint }) => checkpoint === `timeout-presentation`, + )!.value as { stable: boolean } + ).stable = false + return altered + }, + })), + ...callbackReceivingKinds.flatMap((receiver) => + ([`context`, `selected name`, `cause`] as const).map((fault) => ({ + name: `reject callback ${receiver} lost ${fault}`, + history: { fixed: `round7 callback diagnostics corrupt-throws` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find( + (entry) => + entry.checkpoint === `child` && + (entry.value as ChildRun).context.kind === receiver, + )?.value as ChildRun | undefined + const failure = child?.tests.find( + (test) => test.name === `calibration: cal-0`, + )?.errors[0] + if (!failure?.callbackContext) return undefined + if (fault === `context`) + failure.callbackContext.calibration = `unrelated calibration` + if (fault === `selected name`) + failure.callbackContext.name = `constructed-1` + if (fault === `cause`) delete failure.causeMessage + return altered + }, + })), + ), + ...( + [`draw`, `host use`, `forwarding`, `malformed`, `child seed`] as const + ).map((fault) => ({ + name: `reject wrong host RNG ${fault}`, + history: { + fixed: `fixture preserves forwarded outer host RNG alternate`, + }, + law: `host RNG scope`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const scope = altered.find( + ({ checkpoint }) => checkpoint === `host-scope`, + )!.value as HostScopeRun + if (fault === `draw`) scope.outer!.draw = 1 - scope.draw + if (fault === `host use`) scope.outer!.hostUsed = false + if (fault === `forwarding`) scope.outer!.forwarded = false + if (fault === `malformed`) + delete (scope.outer as Partial).draw + if (fault === `child seed`) scope.children[1]!.seed++ + return altered + }, + })), + ...([`child`, `run`, `phase`] as const).map((receiver) => ({ + name: `reject unrelated ${receiver} warning property`, + history: { + fixed: + receiver === `phase` + ? `round3 every actual receiving cut` + : `round5 warning-order equivalence`, + }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const unknown = `self-oracle.unrelated: calibrations "cal-0" have no history binding; using transitional candidate search.` + if (receiver === `phase`) { + const phase = altered.find( + ({ checkpoint }) => checkpoint === `registration-warnings`, + )!.value as { repeated: Array } + phase.repeated.push(unknown) + } else { + const observation = altered.find( + ({ checkpoint }) => checkpoint === receiver, + )!.value as { warnings: Array } + observation.warnings.push(unknown) + } + return altered + }, + })), + ...([`suffix`, `replacement`, `malformed`] as const).map((mutation) => ({ + name: `reject ${mutation} actual warning identity`, + history: { fixed: `round5 warning-order equivalence` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find(({ checkpoint }) => checkpoint === `child`)! + .value as ChildRun + child.warnings = child.warnings.map((warning) => { + if (mutation === `malformed`) + return warning.replace(`: calibrations `, `: malformed names `) + const parts = warningParts(warning)! + return renderWarning( + parts, + parts.names.map((name, index) => + mutation === `suffix` + ? `${name}-extra` + : index === 0 + ? `replacement calibration` + : name, + ), + ) + }) + return altered + }, + })), + { + name: `reject suffixed ordinary warning summary identity`, + history: { fixed: `round5 warning-order equivalence` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const run = altered.find(({ checkpoint }) => checkpoint === `run`)! + .value as SuiteRun + run.warnings = run.warnings.map((warning) => { + const parts = warningParts(warning)! + return renderWarning( + parts, + parts.names.map((name) => `${name}-extra`), + ) + }) + return altered + }, + }, + { + name: `reject suffixed registration phase summary identity`, + history: { fixed: `round3 every actual receiving cut` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const phase = altered.find( + ({ checkpoint }) => checkpoint === `registration-warnings`, + )!.value as { repeated: Array } + phase.repeated = phase.repeated.map((warning) => { + const parts = warningParts(warning)! + return renderWarning( + parts, + parts.names.map((name) => `${name}-extra`), + ) + }) + return altered + }, + }, + { + name: `reject lost warning permutation equivalence`, + history: { fixed: `round5 warning-order equivalence` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const permutation = altered.find( + ({ checkpoint }) => checkpoint === `warning-permutation`, + )!.value as { stable: boolean } + permutation.stable = false + return altered + }, + }, + ...warningReceivingKinds.flatMap((context) => + ([`omit`, `duplicate`] as const).map((mutation) => ({ + name: `reject ${mutation} ${context} registration warning`, + history: { fixed: `round3 every actual receiving cut` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find( + ({ context: cut, warnings }) => + cut.kind === context && warnings.length > 0, + )! + if (mutation === `omit`) child.warnings.splice(0, 1) + else child.warnings.push(child.warnings[0]!) + return altered + }, + })), + ), + ...([`omit`, `duplicate`] as const).map((mutation) => ({ + name: `reject ${mutation} unbound warning name`, + history: { fixed: `round3 every actual receiving cut` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `seedless`)! + child.warnings = child.warnings.map((warning) => + warning.replace( + `"cal-1"`, + mutation === `omit` ? `"missing calibration"` : `"cal-1", "cal-1"`, + ), + ) + return altered + }, + })), + { + name: `reject warning from all-bound registration`, + history: { fixed: `round3 fully bound warning phases stay silent` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.find(({ checkpoint }) => checkpoint === `child`)! + .value as ChildRun + child.warnings.push( + `${syntheticIds[0]}: calibrations "cal-0" have no history binding`, + ) + return altered + }, + }, + { + name: `reject warning from collect-only registration`, + history: { fixed: `round3 every actual receiving cut` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const children = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + const collected = children.find( + ({ context }) => + context.kind === `registration-warnings` && + context.phases.length === 0, + )! + collected.warnings.push( + children.find(({ context }) => context.kind === `run`)!.warnings[0]!, + ) + return altered + }, + }, + // Each successful UNIT owes its own budget. Mutating one unit must not + // borrow the whole lane's aggregate runs + constructions allowance. + ...passingBudgetLanes.flatMap(({ context, lane }) => + ([`generated`, `constructed`] as const).flatMap((unitKind) => + ([`few`, `many`] as const).map((mutation) => ({ + name: `reject too ${mutation} successful ${context} ${lane} ${unitKind} unit calls`, + history: { + fixed: `round4 successful campaign budgets and failing neighbor`, + }, + law: `campaign budget`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context: cut }) => cut.kind === context)! + const target = child.tests.find(({ id, name, state }) => { + const unit = unitOf(name) + return ( + id === syntheticIds[0] && + state === `passed` && + unit && + (unitKind === `generated` + ? unit.name === `generated` + : unit.name !== `generated`) && + (lane === `random` + ? unit.seed === undefined + : unit.seed !== undefined) + ) + })!.name + const callAt = child.log.findIndex( + ({ id, test }) => + id === syntheticIds[0] && test === `${id} > ${target}`, + ) + if (mutation === `few`) child.log.splice(callAt, 1) + else child.log.push(structuredClone(child.log[callAt]!)) + return altered + }, + })), + ), + ), + // Moving a construction's call to the generated unit keeps the lane's + // aggregate unchanged. It still omits one owed unit and over-runs another. + ...passingBudgetLanes.map(({ context, lane }) => ({ + name: `reject redistributed successful ${context} ${lane} unit calls`, + history: { + fixed: `round4 successful campaign budgets and failing neighbor`, + }, + law: `campaign budget`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context: cut }) => cut.kind === context)! + const units = child.tests.filter(({ id, name, state }) => { + const unit = unitOf(name) + return ( + id === syntheticIds[0] && + state === `passed` && + unit && + (lane === `random` + ? unit.seed === undefined + : unit.seed !== undefined) + ) + }) + const construction = units.find( + ({ name }) => unitOf(name)!.name !== `generated`, + )!.name + const generated = units.find(({ name }) => { + const unit = unitOf(name)! + return ( + unit.name === `generated` && + unit.seed === unitOf(construction)!.seed + ) + })!.name + const call = child.log.find( + ({ id, test }) => + id === syntheticIds[0] && test === `${id} > ${construction}`, + )! + call.test = `${syntheticIds[0]} > ${generated}` + return altered + }, + })), + { + name: `reject replay summary cause loss`, + history: { fixed: `round3 replayed clock failure retains driver cause` }, + law: `replay`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const probe = altered.find(({ checkpoint }) => checkpoint === `replay`)! + .value as ProbeRun + if (probe.failure?.cause !== `driver`) return undefined + delete probe.failure.cause + delete probe.failure.causeMessage + return altered + }, + }, + { + name: `reject rerun wrong cause ordinal`, + history: { + fixed: `a driver error after a violation is the violation's cause`, + }, + law: `verdict`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `rerun`)! + const test = child.tests.find(({ state }) => state === `failed`)! + test.errors[0]!.causeMessage = `synthetic driver failed at item 2` + return altered + }, + }, + { + name: `reject replay concrete cause loss`, + history: { fixed: `round3 replayed clock failure retains driver cause` }, + law: `replay`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `replay`)! + const test = child.tests.find(({ state }) => state === `failed`)! + if (test.errors[0]?.cause !== `driver`) return undefined + delete test.errors[0].cause + delete test.errors[0].causeMessage + return altered + }, + }, + ...[`omit`, `duplicate`].map((mutation) => ({ + name: `reject ${mutation} auxiliary registration`, + history: { fixed: `a replayed campaign failure reproduces its path` }, + law: `registration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `replay`)! + const omittedAt = child.tests.findIndex( + ({ state }) => state === `skipped`, + ) + if (mutation === `omit`) child.tests.splice(omittedAt, 1) + else child.tests.push(structuredClone(child.tests[omittedAt]!)) + return altered + }, + })), + { + name: `reject seedless first primary error`, + history: { + fixed: `an ordinary run's random campaign runs under random seed`, + }, + law: `determinism`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `seedless`)! + const test = child.tests.find(({ name }) => + name.startsWith(`determinism:`), + )! + test.state = `failed` + test.errors = [{ kind: `other` }] + child.failed = true + child.exit = 1 + return altered + }, + }, + { + name: `reject rerun recorder cause loss`, + history: { + fixed: `a driver error after a violation is the violation's cause`, + }, + law: `verdict`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `rerun`)! + const test = child.tests.find(({ state }) => state === `failed`)! + delete test.errors[0]!.cause + delete test.errors[0]!.causeMessage + return altered + }, + }, + { + name: `reject rerun probe cause loss`, + history: { + fixed: `a driver error after a violation is the violation's cause`, + }, + law: `rerun`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const probe = altered.find(({ checkpoint }) => checkpoint === `rerun`)! + .value as ProbeRun + delete probe.failure!.cause + delete probe.failure!.causeMessage + return altered + }, + }, + { + name: `reject changed reach-only count`, + history: { + fixed: `an enumerated source counts the histories that entered each declared state`, + }, + law: `reach`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const alone = altered.find( + ({ checkpoint }) => checkpoint === `reach-alone`, + )!.value as ReachAlone + const counts = alone.reach as Record + counts[`saw-0`] = (counts[`saw-0`] ?? 0) + 1 + return altered + }, + }, + { + name: `reject missing child observation`, + history: { constructed: `round2 ranked mutant verdicts` }, + law: `trace`, + corrupt: (trace) => { + const first = trace.findIndex( + ({ checkpoint }) => checkpoint === `child`, + ) + return trace.filter((_, index) => index !== first) + }, + }, + { + name: `reject abnormal child exit`, + history: { constructed: `round2 ranked mutant verdicts` }, + law: `trace`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.find(({ checkpoint }) => checkpoint === `child`)! + .value as { exit: number } + child.exit = 2 + return altered + }, + }, + { + name: `reject secondary auxiliary test errors`, + history: { constructed: `review auxiliary late errors` }, + law: `trace`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.filter( + ({ checkpoint }) => checkpoint === `child`, + )[1]!.value as { + secondaryErrors: Array<{ test: string; message: string }> + } + child.secondaryErrors.push({ + test: `random seed`, + message: `unexpected replay teardown failure`, + }) + return altered + }, + }, + { + name: `reject auxiliary teardown errors`, + history: { constructed: `round2 registration warning phases` }, + law: `registration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.filter( + ({ checkpoint }) => checkpoint === `child`, + )[1]!.value as { + loads: Array<{ errors: Array }> + failed: boolean + exit: number + } + child.loads[0]!.errors.push(`unexpected replay teardown failure`) + child.failed = true + child.exit = 1 + return altered + }, + }, + { + name: `reject auxiliary late error duplicates`, + history: { constructed: `review auxiliary late errors` }, + law: `agreement`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .slice(1) + .map(({ value }) => value as { unhandled: Array }) + .find(({ unhandled }) => unhandled.length > 0)! + child.unhandled.push(child.unhandled[0]!) + return altered + }, + }, + { + name: `reject extra registration diagnostics`, + history: { constructed: `review inferred ambiguous control` }, + law: `registration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const run = altered.find(({ checkpoint }) => checkpoint === `run`)! + .value as SuiteRun + const file = Object.values(run.files)[0]! + file.loadErrors.push(file.loadErrors[0]!) + file.loadCauses.push(file.loadCauses[0]!) + return altered + }, + }, + ...[`authored`, `forced`].map((kind) => ({ + name: `reject lost ${kind} repeated registration`, + history: { constructed: `round2 registration warning phases` }, + law: `registration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const phase = altered.find( + ({ checkpoint }) => checkpoint === `registration-warnings`, + )!.value as { + tests: { repeated: Array } + } + const omitted = phase.tests.repeated.findIndex( + (name) => + name.startsWith(`second >`) && + (kind === `forced` + ? name.includes(`forced entered`) + : name.includes(`calibration: cal-0`)), + ) + phase.tests.repeated.splice(omitted, 1) + return altered + }, + })), + { + name: `reject unexpected child error`, + history: { constructed: `round2 ranked mutant verdicts` }, + law: `trace`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.find(({ checkpoint }) => checkpoint === `child`)! + .value as { + exit: number + unhandled: Array + } + child.exit = 1 + child.unhandled.push(`unexpected registration rejection`) + return altered + }, + }, + { + name: `reject equal-data origin substitution`, + history: { constructed: `review equal fixed originals` }, + law: `calibration`, + corrupt: (trace, plan) => { + const altered = structuredClone([...trace]) + const spec = plan.specs[0]! + const calibration = spec.calibrations[0]! + const binding = bindingOf(spec, calibration)! + const selectedOrigin = authoredOrigin(spec, binding) + const wrongOrigin = `fixed:${spec.fixed[0]!.name}` + const test = `${spec.id} > calibration: ${calibration.name}` + const entries = ordinaryInvocationLogs(altered) + .flat() + .filter((entry) => entry.id === spec.id && entry.test === test) + if ( + selectedOrigin === wrongOrigin || + entries.length === 0 || + entries.some((entry) => entry.origin !== selectedOrigin) + ) + return undefined + for (const entry of entries) entry.origin = wrongOrigin + return altered + }, + }, + ], + owns: [`src/oracle-property.ts`], // Each history spawns one to five child Vitest runs. timeout: 600_000, }) diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md new file mode 100644 index 0000000..7673122 --- /dev/null +++ b/reviews/calibration-history-self-oracle.md @@ -0,0 +1,758 @@ +# Calibration history self-oracle review + +Reviewed production and starting oracle: `2dc95d4e838678a03eb345c08f141dbd20a8c4a7`. +This follow-up changes only the primary self-oracle and its fixture. It changes +no framework production behavior. Four independent reviewers evaluated execution +semantics, judgment independence, grammar, and receiving observations; two then +rechecked the repairs. This record distinguishes their evidence from the final +repair validation below. + +## Laws and experiments + +The README's history-binding contract requires a selected name to identify one +existing original. Both automatic clean premises accept fixed or constructed +selectors; only omitted entered selectors infer a construction. Identity matters +even when two originals have equal data. The registration contract owes the same +test list on every registration: suppressing duplicate warnings does not suppress +tests. Real Vitest must report complete results and every unexpected failure. +The self-oracle separately chooses diagnostic precedence and explicitly promises +to check its first registration error alone; that ordering is not a new public +framework policy. + +The primary model now derives both selected premises from authored declarations, +checks cardinality, and compares input origins, item values and exact build, +resolve and driver budgets. Independent selector axes include non-first names, +fixed entry, constructed absence, and inferred entry. Equal-data fixed originals +remain distinguishable by fixture-side identity associations. Named enumeration +selectors match authored labels, including shell suffixes and the plain JSON +label represented in this bounded string grammar. + +The driver uses actual child Vitest processes for normal runs, replays, reruns, +reach-only runs, seedless runs, reordered calibrations and registration phases. +Each child records status, signal, file completeness, suite/load errors, +secondary test errors, unhandled errors and the input histories that determine +late-call obligations. The check permits expected synthetic failed tests to exit +1 while independently rejecting unexpected errors. Relative file names preserve +identity within a run without recording random temporary directory paths. +Repeated ordinary and collecting registrations retain every scoped test name +with multiplicity and compare against the same model-derived registration list. + +## Finding accounting + +| ID | Verified finding | Disposition and durable enforcement | +| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| I-1 | Model falsely permitted ambiguous selected construction names for inferred/explicit automatic controls. Production correctly rejected them. | Repaired cardinality model; both generated required paths and fixed replays. | +| I-2 | Authored refinement accepted the wrong equal-valued original. | Repaired origin comparison; equal-data distinct originals and corruption calibration. | +| G1 | Grammar omitted fixed clean-entry and constructed clean-absence. | Independent selectors and separately required receiving paths; actual-kind budgets. | +| OBS-1 | Repeated-warning phase discarded registrations. | Scoped complete name lists, shared expectations, authored/forced omission calibrations. | +| OBS-2 | Unrelated child unhandled failures were ignored. | Every receiving child checked; unexpected-error and process/report calibrations. | +| LOADERR-CAND | Expected diagnostic plus extra diagnostics was accepted. | Confirmed by a failing calibration before repair; exact cardinality and permanent duplicate-diagnostic calibration. | +| SIB-1 | Named enumeration model ignored custom/default labels. | Confirmed false failures on correct production; authored-label model and valid/missing neighboring receivers. | +| OBS-2a | Replay-only afterAll failure hid behind an expected failing test/exit1. | Per-file suite/load errors independently judged; auxiliary calibration. | +| OBS-2b | Extra real late-probe error during replay escaped main-only counting. | Input-based late-call bounds checked separately for every child; required actual replay receiver and duplicate-error calibration. | +| OBS-2c | Replay-only afterEach error hid behind the first test error. | Secondary test errors retained and rejected for every child; auxiliary calibration. | +| CHRON-1 | Earlier clean declared-enumerated receiving gate was run after production repair. | Historical process violation remains recorded. Current receiver and hostile omission check exist; retrospective evidence cannot make the earlier gate pre-edit. | + +The raw intake has seven canonical items. Rechecks added four sibling items. +Ten review-derived executable gaps are repaired; CHRON-1 remains a historical process finding, +with no claim that today's tests repair the past ordering. No item was deferred, +refuted, waived, or reclassified as a product-law decision. The temporary stdout +announcement note is preserved as chronology, not a final net production defect. +The cross-file warning experiment used different worker processes and did not +reach its premise; its two warnings establish no defect. + +## Sensitivity + +Before repair, the reviewers demonstrated two false model predictions for +ambiguous selectors and four targeted surviving wrong implementations. They +also ran 25 clean execution-semantics controls and rejected incorrect bound +verdict ranking, a narrow throw-only repair (the neighboring clock case failed), +and difference-before-law. These are bounded reviewer results, not a whole-suite +run against each variant. + +After repair, eleven isolated source mutants reject through the primary model +and check at their intended comparison: + +- ignoring an explicit fixed entered selector; +- canonicalizing an equal-data fixed original; +- dropping calibrations on the second registration; +- introducing an unrelated unhandled rejection; +- dropping only a repeated forced control; +- substituting fixed absence for a selected constructed absence; +- adding replay-only afterAll or afterEach failures; +- doubling replay-only late-probe errors; +- accepting an ambiguous selected name; +- ignoring an enumerated label's shell suffix. + +Ten permanent observation-corruption calibrations also challenge registration +multiplicity, selected identity, extra load diagnostics, unknown child errors, +omitted reports, abnormal status, auxiliary suite/test errors and late-call +multiplicity. They all use the same primary law/model/driver/check. + +## Validation and bounds + +At publication, the final full framework/determinism run is still in progress; +no final-head GREEN or merge-readiness claim is made here. The earlier focused +receivers passed 14/14; the independent observation recheck passed 12/12, +including all ten corruption calibrations. All eleven isolated source mutants +are assertion-rejected. Package type checking, changed-file lint/format and the +package build pass. The ordinary core run passed 54 checks; two npm-pack checks +hit the sandbox's read-only default npm cache and are being retried with a +scratch cache. CI and the final local run must validate the published head. +Commands use the installed Node entry point, `--maxWorkers=2`, and no dependency +install workaround or override. A first build command used a nonexistent +package-local Vite entry; rerunning the installed workspace entry succeeded. +These setup failures are not product RED or mutant rejection evidence. + +Required new states are fixed-entry, constructed-absence, inferred ambiguity, +explicit ambiguity, equal-data original selection, enumerated named labels, and +an actual auxiliary late-call receiver. Both normal campaigns must reach each. +The existing registration-warning and bound-state requirements remain in place. +Fixed inputs replay the shared judgment without separate expected answers. + +The independent model imports no production selector or verdict helper. Input +origins and construction callback inputs are fixture-side facts, not answers +learned from production. Selection kind changes build obligations; equal data +with different input identity changes the promised selected original. These +justify the retained distinctions. No concrete missing-source-fact hypothesis +remains in the repaired cuts: the real reporter supplies all the compared facts. +The bounded faults are distinguished by the legal neighbors and mutants above; +no second implementation of production selection was introduced. + +The full local sweep then passed 226 of 227 checks and caught REPAIR-1, an +observation mistake introduced by this repair: the new child recorder included +raw seedless histories and status, so legitimate host randomness broke the +outer determinism comparison. The seedless receiver now controls Math.random's +draw through fixture hooks while leaving ORACLE_SEED absent. It still exercises +ordinary random-route selection and all child error/count assertions. A focused +real-Vitest receiver runs the same model/driver/check for ordinary, late-probe, +and two-property inputs and then the framework's own determinism check: 4/4 +pass. Entropy or seed-distribution quality is not claimed. The complete expanded +suite remains subject to final-head CI; the failed intermediate run is not GREEN. + +The owner retains its explicit finite synthetic-history, name-family, callback, +regex and scheduler limits. It does not establish all possible JavaScript labels +or errors, browser/provider behavior, downstream DB consumer integration, a DB +release-time improvement, or fresh-seed UNIT support. Real Vitest is the actual +receiving host; no controlled-provider extrapolation is claimed. + +ORC-001 through ORC-006, ORC-008 through ORC-010, ORC-013, ORC-015 and ORC-016 +are covered by the authority, independent model, shared generated/fixed inputs, +receiving observations, neighbor controls and explicit mutants above. ORC-007 +uses the owner's unchanged fixed/random budgets and direct replay checks. +ORC-011's named identity/selection faults are challenged by declaration-origin +relations instead of another copy of production selection. ORC-014 needs no +external-provider handoff for this Node framework claim. This versioned record +and the executable owner provide ORC-012 evidence; CHRON-1 remains an explicit +historical exception to a complete pre-edit receiving-gate claim. + +The four reviewers distinguished current production from oracle integrity and +supplied concrete witnesses or useful positive controls. Strong hire for all +four in their reviewed scopes; none establishes whole-PR correctness alone. + +## Reviewed content identity + +Parent commit: `2dc95d4e838678a03eb345c08f141dbd20a8c4a7`. + +- `packages/oracles/tests/oracle-property-self-oracle.test.ts`: SHA256 `c8df3de2569460a80ff1eab23811d21f9ac6fffe6e4d935bdcfdc5ed5f579d0f`. +- `packages/oracles/tests/oracle-property-self-oracle-fixture.ts`: SHA256 `27195bf184b7ae58f9eaea7bab82973aae5876ce292e2b6aa7e6fa02ac20627d`. + +## Fresh review and oracle-first repair from fc4bad8 + +Starting reviewed head: `fc4bad8dc2c3da3a80b5002bcc36281b75f12846`. +The previous full-head CI passed both Vitest versions. A fresh independent +round nevertheless demonstrated one production defect and six oracle defects. +The repair begins with all seven findings together, before changing production. + +The public binding law preserves an explicitly selected original and gives each +authored calibration its own selector. Incidental declaration metadata does not +change a fixed history into a construction. The verdict law gives an earlier +recorded violation precedence over a later controlled-clock spin, including a +supplied mutant driver. The receiving law requires full registration, expected +test outcomes, faithful diagnostic causes, and named reach counts independent +of their print order. README and the production header state these promises. +No product-law change is proposed. + +The primary oracle repair adds declaration-wrapper metadata, independent +per-calibration binding, and ordered mutant-driver events. Required real Vitest +receivers include authored selection and both automatic clean-premise paths, +mixed authored bindings, and bound/legacy wrong-observation-before-spin. Full +child records must carry every registered name and independently judged primary +outcome through normal, seedless, replay, rerun and other auxiliary cuts. A +rerun owes diagnostic facts from its concrete reproduced history; shrinking may +legitimately remove an unrelated throw. Count dictionaries permit key reordering +while retaining exact names and values. + +Pre-edit production identity is SHA256 +`4d4c9ac25a83ebc49e1604519176418f1618c5f70bf9b9231e555bd2b5e8ee39` +for `packages/oracles/src/oracle-property.ts`. Required receiving RED and +neighbor results must be recorded before its source-kind dispatch is repaired. +Hostile variants then challenge each new distinction. This section is the +pre-edit inventory, not a claim of completed validation. The historical CHRON-1 +entry above remains unchanged. + +### Pre-edit gate at fc4bad8 + +All oracle changes preceded the production repair. The final combined command +used installed Node Vitest from `packages/oracles`, framework config, +`--maxWorkers=2`, and selector +`fixed history: round3|calibration: reject (omit|duplicate|seedless first|rerun|replay|changed reach)`. +On unchanged production it reports **six intended RED and twenty GREEN** out of +253 registered checks (227 skipped), in 45.08 seconds. RED separately compares +own/inherited metadata on authored, forced-entered and forced-not-entered +receiving paths. Every failure is a calibration comparison observing lost +original identity; no setup failure supplies this gate. The plain neighbor, +mixed selectors, bound/legacy spin and throw, dictionary order, warning phases, +seven actual receiving cuts, concrete replay causes and nine corruption controls +are GREEN. Package TypeScript checking passes. + +The primary grammar separately declares each metadata receiving path, bound and +legacy wrong-observation-before-spin, mixed authored selectors/search, successful +and rejected named enumeration selectors, legal count dictionary order, silent +fully-bound warning phases, and all seven actual child receiving contexts. Full +fixed/random reach counts remain to be measured after production repair. + +Isolated source/fixture/reporter variants already reject first-selector sharing, +supplied-driver spin ranking, seedless primary errors, missing replay and +reach-only registrations, duplicate reach-only registrations, reordered-child +primary errors, lost rerun/replay causes, wrong cause ordinals and unexpected +causes. Reversing only count-key printing is accepted. These are bounded +assertion results, not full-suite mutation claims. The warning-phase extension +also caught an oracle mistake: fully-bound properties emit no migration warning, +but the old phase model demanded one. Its false RED preceded the independent +per-entry warning model repair and both bound/unbound GREEN neighbors. + +Concrete diagnostics are judged from the history actually reproduced. A +campaign's initial checkpoint or cause may change while shrinking. Replay's +input-side log supplies its one concrete history; an ordinary parseable reported +counterexample must have appeared in that test's invocation log. Exact finite +driver causes and item checkpoints are compared at that concrete cut. Nested +judgment causes and arbitrary AggregateError metadata remain outside this +synthetic diagnostic grammar; no universal error-serialization claim is made. + +### Source-kind repair after the gate + +Oracle-only commit `d5e615a` precedes the production repair. Selection now +uses the selector kind to look up fixed or constructed declarations, preserving +fixed originals even when the declaration has unrelated `build` metadata. The +same gate passes **26/26** after repair (227 skipped), in 49.59 seconds. Package +TypeScript, scoped ESLint and the Vite package/declaration build pass. Full +fixed/random validation and a new independent review round follow this commit; +this focused result alone does not establish PR readiness. + +## Fresh review and receiving repairs from b09a403 + +Four fresh independent reviewers found two P2 primary-oracle gaps and no new +production defect. Auxiliary children discarded registration warnings, and +passing campaign counts were learned from the executions being checked. +README's once-per-process warning promise and the header/config's equal +campaign-budget promise already supply the laws. Both are enforcement repairs. + +The owner now records every child's warning facts and independently judges +names and multiplicity for its effective declaration and collecting phase. +Required warning-positive receivers cover run, seedless, replay, rerun, +reach-alone, reordered bindings and registration phases separately. Passing +campaigns owe the authored run budget plus constructed examples; failed, +shrinking and timed-out campaigns retain their variable execution counts. +Seedless host draws remain controlled inputs, with ORACLE_SEED absent. Required +states separately cover successful fixed/random lanes in seeded and seedless +children, construction examples, and a failing neighbor. The count model never +uses observed reach totals as its owed budget. + +Same-head CI also caught two oracle regressions from the preceding expansion. +A filtered rerun cannot learn skipped default data labels from its own invocation +log. Auxiliary inventories now use the full ordinary input/name association, +consistent with the owner's existing default-label distinctness limit. Each +child's execution and build logs remain separate for outcomes and counts. +The per-entry binding refactor also stopped constructing duplicate declaration +names for bound probe controls with no authored calibrations. Effective probe +bindings now restore that grammar, preserving the required ambiguity states. +Neither issue required a source change. + +On unchanged b09a403 production, six combined warning/budget controls pass, +all five CI regression receivers pass, and 26 permanent corruption calibrations +pass. These share the primary model/driver/check. Real-child missing-warning and +fixed-budget-one source mutants are assertion-rejected; legal warning-name +reversal passes. The default-label repair still rejects omitted skipped +registrations on both Error and bigint receivers. The duplicate-name grammar +still rejects first-match selection on both inferred and explicit controls. +Package TypeScript passes. Raw RED/GREEN, malformed reports, source patches, +controls and setup attempts remain in the task audit. + +The preceding CI Vitest5 framework run reported 7 failures/246 passes: two +campaigns, two fixed cases and one diagnostic calibration share the ambiguity +premise; two default-label reruns share the inventory defect. These are two +oracle defects, not seven source bugs. The old local full sweep was superseded +and interrupted after these same-head RED results; it is not GREEN. A complete +new full sweep and another fresh review round must validate this owner. + +Warning judgment permits name/property ordering. The executed allowed control +uses a fixed reversal; it does not establish arbitrary variation between repeat +executions, whose raw warning spelling also reaches outer determinism. This is +a remaining representation question for the next independent review, not an +asserted full closure of every legal warning representation. Source and fixture +remain byte-identical to b09a403. Historical CHRON-1 remains unchanged. + +## Fresh warning-observation review from 68ed541 + +Four new scoped reviews found two P2 self-oracle defects. Warning-name order is +not promised, but its raw spelling reached outer determinism. Alternating only +valid name order passed the shared judgment and failed outer determinism. The +warning-name matcher also accepted `cal-0-extra` as `cal-0`: both the ordinary +case and outer determinism stayed green under that wrong-name renderer. These +are distinct observation/formulation and enforcement defects under the README's +once-per-property/process, exact-unbound-names promise. + +The owner now decodes the actual finite quoted-name list, retaining duplicates +and unknown names, and sorts its entries before any child, ordinary-run or +registration-phase observation. The authored expected-name multiset remains +independent of the warning renderer. Malformed lists stay observable and fail; +unknown-property warnings fail at each full-list checkpoint. The finite grammar +allows inner state-name quotes but excludes the list delimiter inside a name; +no claim about arbitrary future warning prose or arbitrary JavaScript strings +is made. + +An authored representation dimension alternates valid name presentation across +real executions. The existing seven-context receiver carries multiple unbound +names and measures nontrivial permutation reach. Its ordinary case and actual +outer determinism reuse the same model, driver and check. Raw reports remain +unchanged; sidecars preserve the presented ordering. Permanent corruptions add +wrong suffixed/replaced names, malformed lists and unrelated-property warnings, +while retaining missing/duplicate warnings and names, all-bound silence and +collect-only silence. No framework production or fixture edit is part of this +repair. + +The frozen pre-repair experiment passed forward2/2 and consistent reverse2/2, +but alternating representation failed outer determinism. The wrong suffix +renderer survived2/2 before repair. With the repaired observation it fails the +intended `child` / `calibration` comparison; the actual ordinary/determinism +representation control passes2/2. Four live neighboring receivers pass. Package +TypeScript and scoped lint pass. Final warning-calibration completion is +recorded in the next entry, without claiming a whole-suite GREEN. + +Reviewed owner SHA256: +`18a5c3e091cf7b218f2d08fad99df51aa548ec2daca4ab7df27973d074b776ac`. +Production and fixture hashes remain those of b09a403. Independent grammar, +seed/premise/identity, and execution-semantics reviewers found no new supported +defect in their bounded scopes. The independence review accepted an alternative +stable hash, preserving implementation freedom. Their negative results are not +whole-framework correctness claims. Historical CHRON-1 remains unchanged. + +The semantics review also preserved an optional policy question: a deliberately +wrong driver may enter an undeclared state while its designated visible-result +violation is accepted as a successful calibration. The current contract scopes +declared-reach validation to the clean run and describes shared verdict rules +for recorded violation/clock/error priority. No stronger mutant-reach obligation +has been established or silently added. A maintainer clarification was requested +separately; the executed trace and its authority qualification remain in the +audit. + +The completed frozen warning sweep passes **27/27** in 184.04 seconds: nine +identity/property/permutation controls and eighteen retained warning controls +across seven receiving contexts. Together with the four positive receivers, +two actual ordinary/outer-determinism checks and the rejected suffix renderer, +this completes the two warning findings within the stated finite grammar. An +independent reviewer is challenging this frozen repair before main integration. +The superseded full68ed541 run completed both campaigns, including required +warning/budget reach, then was explicitly interrupted before the incoming +source/fixture integration. It is incomplete, not a full-suite GREEN. + +An independent frozen-repair review found no new supported warning defect. Six +actual source variants were rejected: seedless omission, duplicate name, +malformed list, unknown property, extra unknown name and duplicate warning +event. Three legal presentation inputs passed the actual outer determinism +receiver using this same model, driver and check, including sixteen varied and +astral names across two properties, all receiving contexts and forced state +labels with inner quotes. Thirty real child reports were retained. Vitest +coalesced the duplicate console calls; that variant still failed name identity, +so the result does not claim two separately observed warning records. These +scoped results validate the warning repair; main integration remains separate. + +## Main UNIT integration and host observation repair + +Integration starts from published `f3f39a02d804cc3c74a3b1a9ba9333aad693a34d` +and normally merges main `d38af3812dc07f447a23259673cdecf96a99fb3b`. +Main removes `declareOracle`, splits generated campaigns and determinism into +named units, and supplies `UNIT` replay. The accepted main contract changes the +registration and budget model: each constructed unit owes one history; each +passing generated unit owes its authored runs. Existing binding identity, +fresh plans, stable calibration seeds, forced premises and verdict ranking +remain. Fresh calibration seeds remain follow-up work. + +Before replacing production, the combined primary oracle ran against unchanged +main. Seventeen UNIT neighbors passed, seven binding/control/origin/clean-reach +checks failed their intended comparisons, and six own/inherited fixed-wrapper +metadata receivers failed separately across authored and both forced premises. +Two mixed inputs failed first on missing migration warnings, supplying warning +feature evidence rather than selector proof. Two recorded-violation-before-spin +neighbors passed. After the merged source, the same thirty binding, metadata +and UNIT cuts passed; the four mixed/spin checks and nine receiving, warning and +budget closures passed. Setup failures and initially empty filters are retained +in the integration audit and do not count as behavioral evidence. + +The combined grammar retains every branch required state. Main's global name +uniqueness now rejects old duplicate-name inputs before selector resolution; +those inputs explicitly earn name-gate reach. Zero and two distinctly named +entering constructions retain separate inference coverage. Four generated UNIT +constructions require eleven states, with actual receiving observations. +Per-unit budgets have sixteen omission/extra controls and four redistributions +that preserve aggregate counts. Two clean shared-law enumerated receivers pass; +source mutants duplicating constructed examples, reducing generated runs to one, +and reusing the first bound selector fail the intended campaign-budget or +calibration assertion. The enumeration avoids letting a global mutant fail the +outer generated reach before the inner comparison. + +Integration exposed four owner/fixture siblings before final validation: + +- The synthetic fixture replaced the outer host RNG as well as marked children. + Both supplied host inputs failed the primary HOST comparison on unchanged main. + Scoping fixture control to marked children made both pass before source repair. +- The inference model counted the first name rather than independently authored + entering declarations. It now retains genuine two-construction ambiguity. +- The custom-prefix driver renamed the literal tooling key `ORACLE_MANIFEST`. + It now translates only configured replay variables; the receiving witness passes. +- A new host receiving state was initially undeclared. Its declaration and + required construction now agree, preserving the branch's declared-state guard. + +A subsequent HOST review found an observation overconstraint: function identity +is not promised by the host seed contract. An allowed forwarding wrapper failed +the old primary comparison despite preserving the supplied draw and its use. +The owner now checks the actual post-import draw and a semantic use fact measured +immediately around actual seed selection. Exact counters remain raw diagnostics. +The legal grammar includes one-call, two-call and alternating forwarding forms; +actual outer determinism passes. Four permanent corruptions reject wrong or +malformed draws, lost host use and unequal marked-child seeds. The original +unscoped fixture and a separate seed-source bypass still fail the HOST comparison. +This changes owner observations, not production or the scoped fixture. + +The merged production SHA256 is +`a1c132e8e096228e34aad07ba39c20df88facfe998e8c8e4e155ff75b213e01b`; +fixture SHA256 is +`e2d2034ad7a57067e45c79a85a48970452b61dc9eb02e6e62c9e524da4bf9198`. +An independent source review passed ten focused receivers and rejected wrong +original identity, ignored difference, reversed forced premises and wrong UNIT +seed offsets. An initially insensitive full-prefix seed witness passed the +mutant; three existing short-prefix histories then passed correct source and +rejected it. That filtered survivor is not a full-owner false green. A legal +alternative stable name mixer passed. A cached-builder variant failed on the +synthetic runtime path and is not credited as an inner assertion rejection. + +Local package/root types, package build and scoped lint pass. Core tests pass +113 of 114; the fresh consumer-install check reaches registry HTTP403 responses +for Vitest and fast-check and remains an environment failure. No dependency or +lockfile workaround is committed. Plugin assertions pass 6,438 checks plus two +package-citation checks retried with a writable uppercase npm cache. The actual +96-test e2e inventory partitions across five shards with no partition errors; +this is partition evidence, not a full gap-driver run. Complete merged framework +validation and final owner reviews are still pending at this entry. + +This appendix retains the finite synthetic grammar, real Node/Vitest boundary, +prior raw findings and historical CHRON-1. Current experiments cannot repair the +missed historical pre-edit gate. The optional mutant-only undeclared-reach +question remains a possible stronger policy, not an established contract defect. + +The final HOST challenge exposed one reach gap (I-11): omitting wrapper +installation while retaining the requested plan passed the ordinary, reach and +outer determinism checks, falsely reporting forwarding reach. The owner now +records a semantic `forwarded` fact from actual wrapper invocations immediately +around seed selection. Both its assertion and its reach require that fact; +configured call counts cannot establish it. Exact invocation counts remain raw, +so allowed one/two-call representation changes preserve deterministic traces. +The clean ordinary/reach/determinism neighbor passes all three checks. Final +focused HOST controls and the omitted-wrapper challenge are recorded after they +complete; the pre-repair survivor is preserved rather than counted as a kill. + +The frozen owner SHA256 is +`336ab5a735116bcbaff52caaeb0356422be27e56258e0fdb7d3468728474d9d9`. +The final HOST focused gate passes **15/15**, including actual per-history +outer determinism and five corruption controls. The clean required-forwarding +ordinary/reach/determinism receiver passes **3/3**. Omitting wrapper installation +now fails the HOST comparison and required forwarding reach (zero); its outer +determinism still passes, isolating enforcement from nondeterminism. Package +types, scoped lint and diff checks pass. Production and fixture retain the +hashes above. Full frozen-head validation and three fresh owner reviewers start +next; no whole-PR readiness claim is made by these focused results. + +## Fresh merged-head review: callback diagnostics and calibration instrument + +Three fresh reviewers examined immutable `8b4543aa5fd0e887ddf77ed2765fe43d76eb116c`. +Independence and grammar/reach reviews found no new supported defect in their +bounded scopes. They executed lawful name-mixer/forwarding variants, original +identity/forced-premise attacks, exact UNIT stream/budget attacks and actual +receiving reach. The observation review found two oracle defects; its context +and cause subclaims are retained separately below. Current production preserves +the disputed diagnostics. No production or fixture repair is indicated. + +The law comes from original user-authorized review item 6: failures retain the +calibration and selected history names. The accepted owner also retains the +original cause. The README supplies neighboring selection semantics; it does +not prescribe this callback diagnostic template. Exact punctuation, hash/seed +arithmetic, stack paths and an unbound-callback context promise are not added. + +| ID | Verified gap | Repair and evidence | +| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| R7-OBS-CONTEXT | Reordered, seedless and environment callback errors could lose their selected calibration/history context while the shared judgment passed. | Semantic identity survives every active child/summary projection and is compared with authored selection. Actual reordered source mutant now fails child/calibration. | +| R7-OBS-CAUSE | Those auxiliary errors could lose or replace their original cause. | The same independent diagnostic judgment requires the finite callback cause; separate cause-loss source mutant now fails child/calibration. | +| R7-OBS-SELECTED-NAME | Ordinary and reordered errors could falsely name another declared history of the same kind. | Exact authored selected name is checked; a same-kind source substitution now fails. | +| R7-CI-ORIGIN | Permanent origin corruption changed a stale summary log while the comparison consumed the ordinary child log. | The control now changes the exact semantic invocation in both representations; identity comparison is unchanged. Same calibration passes and a real wrong-origin source still fails run/calibration. | +| R7-SIB-CALLBACK-REPRESENTATION | Initial repair falsely rejected harmless separator whitespace. | An actual legal renderer failed before normalization and passes after it; wrong selected names still fail. | + +All findings were grouped before owner repair. The existing Suite arbitrary now +has a weighted legal callback branch crossing three callback kinds, three +selection kinds, two tails and both prefixes. Fixed inputs and required +constructions use the same model, real driver and comparison. Callbacks actually +execute in ordinary, reordered, seedless and seed-only environment paths; each +has required measured callback-failure reach. Registration phases, path replay, +rerun and reach-alone keep their full registration/skipped checks and do not +claim callback execution. Across nine clean kind/selection neighbors, actual +counts are ordinary 9, reordered 6, seedless 9 and environment 9. Enumerated +sources have no reordered-construction receiver under this driver. + +Three bad-source variants survive the frozen owner and fail the repaired +comparison; clean source passes both. Thirty-six independent wrong diagnostics +are rejected across four child cuts plus ordinary/environment summaries. +Opaque default enumeration names are associated with the public direct-history +name of the authored index/origin, independently of callback text. Five opaque +forms pass; fifteen wrong-name, missing-association and ambiguous-association +controls fail. The codec and name hash are not copied into the expectation. + +The callback scratch candidate passed eighteen focused checks, followed by five +on the final whitespace-normalized bytes, and all thirty-six controls were +rerun. One earlier scratch run overlapped source-mutant experiments; its four +failures are contaminated evidence, superseded by the restored-source clean +gates. No setup/runtime failure is counted as an assertion rejection. + +Exact-head Vitest 5 CI on 8b4543a passed 729 of 730 checks and exposed the stale +origin corruption. The same calibration failed on installed Vitest 4, ruling +out a version-only flake. Captured traces show the summary changed to fixed-0 +while the canonical child retained correctly selected fixed-1; changing the +canonical or both logs triggers the intended identity assertion. The nearby +permanent-control audit found no other log/build/resolve corruption with this +stale target. Ten focused origin and adjacent controls pass after the repair; +a genuine wrong-fixed-original source mutant still fails the primary law. + +The superseded local 8b4543a run completed both campaigns' forty-five named +units, retaining their full reach counts, then was explicitly interrupted with +exit 130 after the confirmed CI/focused failure. It is not a full-suite GREEN. +Core/package, plugin, Vitest 5 core and both gap-oracle jobs on that head passed; +the consumer-install CI path supplies the witness blocked by local registry403. + +Combined live owner SHA256 is +`b6efd19d8dd086ed73d65c36e3b471685cc96789d055dcfcb82d94c972d4b3ad`. +The actual combined focused gate passes **19/19** in 84.90 seconds, including +callback fixed histories/outer determinism, twelve corruptions and the origin +control. Full package TypeScript, scoped lint, formatting and diff checks pass. +Production/fixture/reporter retain the a1c/e2d/1918 identities above. Complete +final-head framework validation and fresh focused re-review start next; this +entry makes no full-framework GREEN or whole-PR readiness claim. + +Loss accounting: two raw observation findings retain three independently +verified obligations, plus one CI instrument finding and one repair-discovered +representation sibling. All have applied repairs and focused enforcement; none +is deferred or resolved by a policy change. Other reviewer scopes retain their +negative evidence and limits. Strong hire for each reviewer in its demonstrated +scope, particularly the observation review's actual source survivors and the +CI diagnosis that distinguished an inert instrument from missing law enforcement. +Historical CHRON-1 remains irreversible; optional mutant-only undeclared-state +policy remains a separate unapproved proposal. The finite synthetic grammar, +Node/Vitest receiving boundary and absence of downstream DB/browser/performance +claims remain unchanged. + +## Diagnostic representation and timeout observation follow-up + +This follow-up starts from `c18a30f`. Production retains SHA256 +`a1c132e8e096228e34aad07ba39c20df88facfe998e8c8e4e155ff75b213e01b`. +These repairs change the self-oracle and its synthetic fixture. They change no +framework behavior or accepted diagnostic promise. Latest main, including #36, +was integrated by normal merge `3a58b96`; its gap CLI files are disjoint. + +The diagnostic law requires truthful authored calibration and selected-history +identity plus the finite original callback cause. It permits representation +changes to punctuation, structural whitespace, optional seed annotations, the +outer wrapper's name and opaque reason prose. Literal characters within names +remain significant. An unknown or ambiguous complete identity cannot authorize +its own expected selection. Original cause object identity, class and metadata +remain outside the established law. + +| ID | Executed defect before repair | General repair | +| ----------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| R8-REPRESENTATION | Lawful bracketed calibration labels and alternative seed annotations falsely fail the callback comparison. | Decode complete actual frames against available literal identities; derive expected selection independently from authored bindings. | +| R9-REP-SEAM | Already-bracketed source output passes semantic checks but reports zero presentation reach. | Count an actual rendering operation even when its output bytes equal its input. | +| R9-REP-RANGE | Mixed spacing, two spaces and tabs preserve identity/cause but fail decoding. | Generate and accept positive structural whitespace while retaining literal name characters. | +| R10-REP-REASON | Replacing the reason's first word loses identity; category-marker prose can change the verdict. | Locate the complete frame separator without inspecting reason words; the actual finite callback cause determines exception kind. | +| R11-DIAG-1 | Custom outer wrapper name plus opaque category-marker reason fails, although both marginals pass. | Remove the arbitrary outer-name guard; generate wrapper names independently of reason and frame presentation. | + +Every group extended the primary Suite grammar/real receiving driver before +changing its observation judgment. No production edit was needed. R10's +unchanged-observation gate rejected four legal fixed histories while their four +outer determinism checks passed. R11's unchanged guard rejected the combined +history while both marginals and all three determinism checks passed: one +intended false RED and five passes. The repaired R11 gate passes six of six. +Required constructions vary callback and selection kinds, siblings, prefixes, +incoming presentation and bounded arbitrary reason/wrapper text. All four +active receivers require actual callback failure and presentation reach: +ordinary, reordered, seedless and environment. Registration-only receivers +retain their registration obligations without claiming callback execution. + +The R10 frozen matrix accepted 15,642 lawful observations and rejected 1,089 +hostile controls at 99 actual cuts. It includes literal prefixes, numeric and +opaque labels, ambiguity, seed-like literal suffixes, arbitrary bounded reasons +and same-byte rendering. Thirty-five focused tests passed. Six lawful actual +source renderers passed two tests each; actual wrong sibling and missing cause +were rejected at child/calibration, with determinism passing. R11 carried those +matrix results only in their original `Error`-wrapper domain: removing a guard +that was true for every original input preserves that predicate. It separately +passed the actual custom-wrapper/opaque-reason source combination and rejected +wrong-sibling and replacement-cause source mutants at the intended comparison. +It did not claim to rerun the complete matrix for arbitrary wrapper names. + +R11's suspected absent reason dimension was refuted: the generic Suite grammar +already supplies arbitrary reasons. A 2,000-Suite sample contained 1,381 defined +reasons and 1,180 distinct constructed reasons. The redundant weighted-branch +draw is now forwarded too; that cleanup is not evidence of a former absent +whole-campaign axis. All setup failures and zero-test selectors remain excluded +from assertion-rejection counts. + +The timeout law limits every registered test independently. A timed-out test +owes no fixed completed-call count. Actual started work must retain its selected +input and corresponding build/resolve obligations. The fixture now captures its +logical owning test/route at Vitest's public `aroundEach` boundary through Node +AsyncLocalStorage. The global runner name remains raw diagnostic evidence; +advancing to another test cannot reassign an older continuation's work. + +| ID | Executed defect before repair | General repair | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| T1 | Exact-head Vitest 5 CI and local Vitest 4 reject timeout-history determinism; seven differing fields concern slow call attribution/order. | Record semantic relations of actual started work and keep raw runner observations outside the deterministic trace. | +| T2 | Slow isolated constructions also show resolution attribution drift. | Group build/resolve work by logical owner. A determinism test resolves once and may drive twice; other actual executions resolve individually. | +| T3 | A timed-out bound calibration with one actual execution is rejected for failing to complete two builds. | Check selected inputs and work for actual starts; retain input consistency and upper bounds without requiring optional late work to finish. | + +The extended primary grammar reproduced T1–T3 before the fixture repair. A +global resolve-count-equals-driver-count rule was disproved by a lawful trace +with 20 driver calls and 17 resolves. The repaired observation keeps first +logical inputs plus missing/extra work and changed-input relations. A legal +reporting-cut transformation removes only an optional second continuation with +its paired work; raw-versus-cut equivalence must still hold. It cannot repair an +invalid raw report silently. + +Slow late-probe counts retain a deficit/excess relation computed from complete +raw started histories and actual known late errors. Both values must be zero. +Fast-only late counts stay exact. Unknown and secondary errors, test statuses, +registration, loads, exit/signal and report completeness retain their checks. +Raw reports, global runner names and cuts remain available in diagnostic +sidecars. A blanket slow-log filter would not satisfy these obligations. + +The frozen timeout gate passed 48 tests, including 20 hostile controls. Fixed +and random required constructions each measured one hit for ordinary, +reordered, seedless and environment input cuts, one optional-continuation cut +and one presentation-equivalence hit. Three separate actual perturbations were +assertion-rejected: lost logical ownership (two failures), duplicate raw resolve +work (one calibration failure, delta two), and missing raw build work (one +timeout-presentation failure). Types, lint and formatting passed. + +Chronology remains explicit: ordinary/reordered REDs preceded the fixture +repair; seedless/environment witnesses were added afterward and establish +current enforcement only. A registration-phase missing-build experiment reached +no drivers. Its initial survivor classification was corrected to unreached; +it supplies neither a defect nor a kill. This qualification does not alter +production-repair chronology, because no production code changed here. + +The combined executable identities are owner SHA256 +`5b85ea79e738ea399ddd8c85ce5f55a95fa3e6dd213949e71c5f3e1f42621e5e`, +fixture `bad74feb45d92bdd390d48b804dc671c660ab4aa40462a45983c9a75d0c175e9`, +and unchanged production/reporter `a1c132e8`/`1918e8dc`. Three pure-addition +merge conflicts retained both repair groups. Combined TypeScript, lint, +formatting and diff checks pass. Combined focused checks and two independent +final reviews are running at this entry; complete final-head validation remains +pending. Earlier interrupted local runs are not full-suite GREEN results. + +These are finite synthetic and actual Node/Vitest boundary results. They make +no DB, browser, provider, release-time or memory-performance claim. Historical +CHRON-1, the retrospective successful-calibration stdout gate, optional stronger +mutant-only undeclared reach and fresh calibration seeds retain their prior +qualifications. No confirmed defect is deferred by relabeling it a limit. + +The exact combined owner gate passes **49/49** in 223.83 seconds, with 831 +tests registered. From `packages/oracles`, the command is: + +```sh +node ../../node_modules/vitest/vitest.mjs run \ + --config vitest.oracle-framework.config.mjs --maxWorkers=2 \ + -t 'round11 outer wrapper|timeout every receiving cut|reject timeout|round8 callback presentation freedom|reject equal-data origin|reject.*callback' +``` + +This gate exercises both repair groups together, actual outer determinism, +callback and timeout controls, and the canonical origin calibration. It is a +focused GREEN, not a complete framework run. The final workflow runs the whole +framework on both supported Vitest major versions. No campaign budget was +lowered and no dependency or lockfile workaround is included. + +### Original review loss accounting at the combined executable tree + +The original review targeted `329dbd4`. All nine technical findings were valid +there. Their current implementation dispositions are **nine already-fixed**; +the diagnostic follow-up supplies the remaining R6 recorder enforcement. This +table preserves each original cure and durable obligation separately. + +| Original item | Claim and proposed cure | Current evidence and durable obligation | +| ------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | Bound mutant verdict ranking differs from unbound; reuse `runOne`. | Bound/unbound ranking shares `runOne`; recorded violation before later throw/spin and bounded-clock controls remain enforced. | +| 2 | Forced premise is learned from the selected driver log; reconstruct it independently. | Authored selectors/origins determine both forced premises, including equal-data originals and explicit opposite orientations. | +| 3 | Reorder invariance misses the property-fixed-seed mutant; compare full named builds. | Distinct ordinary/astral name families and complete construction inputs reject the fixed-seed substitution while allowing other stable mixers. | +| 4 | Clean selected reach checks missing states but allows undeclared states; validate both directions. | Clean declaration checks cover both directions. Mutant-only undeclared reach remains a separate, unapproved stronger policy. | +| 5 | UTF-16 high-surrogate hashing collides for emoji names; use code points. | Code-point hashing and finite astral full-input controls distinguish the original defect without promising universal hash injectivity. | +| 6 | Corruption and judgment callback throws lose calibration/history names; contextualize them. | Production catches preserve context/cause. Four actual receiving cuts now enforce exact authored identity and finite cause while accepting lawful representation changes. | +| 7 | Explicit `entered: null` is silently inferred; infer only on `undefined`. | Omitted versus malformed controls remain distinct; null/false/zero and inherited/own binding cases retain registration-before-driver enforcement. | +| 8 | An impossible `if (!kind)` branch remains; remove it. | The branch is absent after exact-one-key validation; this is control-flow/build evidence, not an invented behavioral RED. | +| 9 | Transitional warnings repeat, collect-only warns and successful bound calibrations print announcements; deduplicate/silence. | Once-per-property warnings retain exact unbound names and registrations; collection and successful bound runs stay silent. The original announcement gate's retrospective chronology remains recorded. | + +Coordination is also complete: main/F27 was merged normally, `difference` has +bound/unbound parity after designated-law selection, and main's named UNIT +support is integrated. Fresh calibration seeds remain the explicitly separate +follow-up. No published branch was rebased or force-pushed. + +The eight current follow-up IDs remain separate from the original nine: +R8-REPRESENTATION, R9-REP-SEAM, R9-REP-RANGE, R10-REP-REASON, R11-DIAG-1 and +T1–T3. Each has an applied oracle repair and targeted sensitivity evidence above. +The original reviewer merits a strong hire recommendation: nine accurate +findings, independent-judgment attacks, useful cures and sound prioritization. +Later review results remain bounded to their recorded artifacts; full CI and +the final fresh review reports are not implied by this accounting table. + +### Final independent reviews of the combined executable + +Both fresh R12 reviewers found **no new supported defect** on owner `5b85ea79` +and fixture `bad74feb`. Neither changed the live tree. Their interventions were +sequential in isolated copies, and all source/owner/fixture/reporter hashes were +restored. The executable commit is `1fdf731`; this subsequent record-only append +does not change those reviewed bytes. + +The timeout reviewer passed 24/24 final checks, including twenty permanent +controls and the actual fixed/random required receiver units. A real delay +change from 300 ms to 450 ms, with a 100 ms timeout, changed a reordered +calibration from two actual started calls to one. Both shared comparison and +outer determinism passed. An actual discarded extra `selected.build()` was +rejected at child/calibration with build difference one, while determinism +passed. Each campaign measured all four input receivers, continuation cut and +presentation equivalence once. An initial all-skipped selector is retained as +unreached, not a kill. + +The diagnostic reviewer passed 18/18 baseline checks, then 5/5 with actual +source changes combining tabs/mixed spaces, bracketed labels/seed decoration, +custom outer name and category-marker opaque reason. Both matching fixed and +random units measured every callback-failure and presentation receiver once. +Actual wrong sibling, wrong cause and environment-only cause omission each +failed child/calibration while determinism passed. The restored-source gate +passed 2/2. Its public observation hook recorded three complete children, three +presentations and three matching identity/cause failures for each of ordinary, +reordered, seedless and environment receivers. A failed pre-write script path +and redundant clean run are explicitly excluded from source-variant evidence. + +The separate read-only final loss audit found no lost obligation: original nine +already-fixed, eight follow-ups fixed-now, coordination already-fixed. It +retained historical CHRON-1 and every chronology, corrected nonfinding and +policy qualification above. Both R12 reviewers merit strong positive assessment +within their scopes: they independently challenged actual permitted timing and +diagnostic representations, then demonstrated intended comparison rejection +for adjacent wrong implementations. No further confirmed repair remains in +these bounded reviews. Complete final pushed-head CI remains required; this +record does not promote focused results to full-suite success.