From 5304e9ee8b718dd26cbf53b5aaa71c25be41621a Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 10:07:05 -0700 Subject: [PATCH 01/14] feat: bind calibrations to declared histories Law: An explicit history binding runs only that history. The selected clean run and actual reach must pass before exactly the designated law rejects the mutant. Authority: approved maintainer proposal and review, 2026-11-11. Reach: Constructed framework histories exercise direct selection, invalid selectors, isolated constructed drivers, and both forced-probe controls. Fixed seed 1 reached bound-calibration 6 times, registration-error once, and bound-constructed-driver once. The random campaign reached them 3, 4, and 1 times respectively. Existing run budgets remain unchanged. RED: On unchanged source at 49f3bd1, the extended framework self-oracle's 'fixed history: bound calibrations use only' test rejected two driver calls where the binding permits one. GREEN: 22 focused binding regressions and the selected clean-driver-error regression pass. Build, lint, types, format, publint, sherif, 56 core tests, and 4320 ESLint plugin tests pass. Full framework and gap checks are running. Mutant: The same comparison rejects restoring search (driver count 2 > 1), skipping reach, accepting an inapplicable corruption, accepting the wrong law, and cloning a built history that carries identity metadata. Limits: Unbound calibrations retain search with named registration warnings until consumers migrate. Fresh seeds await UNIT replay support. DB migration, a matched DB timing comparison in CI, and the later F27 rebase remain separate. --- .changeset/calibration-history-bindings.md | 7 + README.md | 59 ++ packages/oracles/src/oracle-property.ts | 453 +++++++++-- .../oracle-property-self-oracle-fixture.ts | 101 ++- .../oracle-property-self-oracle-reporter.mjs | 5 + .../tests/oracle-property-self-oracle.test.ts | 726 ++++++++++++++++-- 6 files changed, 1208 insertions(+), 143 deletions(-) create mode 100644 .changeset/calibration-history-bindings.md diff --git a/.changeset/calibration-history-bindings.md b/.changeset/calibration-history-bindings.md new file mode 100644 index 0000000..bb6fb5b --- /dev/null +++ b/.changeset/calibration-history-bindings.md @@ -0,0 +1,7 @@ +--- +'@tanstack/oracles': minor +--- + +Add direct calibration history bindings for constructed, fixed, and enumerated histories. Bound calibrations validate their selected clean run and actual reach, then require the designated law to reject the mutant, without searching other histories. Constructed calibrations use a stable name-derived seed and fresh builds and resolved plans for each driver execution. + +Add `probeHistories` to bind automatic forced-probe controls, with inference from the unique construction entering a state. Unbound calibrations retain search during migration and warn at registration; consumers must migrate before search is removed ahead of the first npm release. diff --git a/README.md b/README.md index 5f6a929..63f3491 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,65 @@ preset's, so merge them: } ``` +## Bind calibrations to histories + +A calibration's `history` selects an existing input directly: + +```ts +calibrations: [ + { + name: 'drops the committed row', + law: 'durable publication', + history: { constructed: 'a committed write' }, + corrupt: dropCommittedRow, + }, +] +``` + +Generated sources accept `{ constructed: name }` and `{ fixed: name }`. +Enumerated sources accept `{ enumerated: index }` (zero-based) or +`{ enumerated: name }`, using `label` or the framework's default data label. +Unknown or duplicate names, invalid indices, and incompatible source kinds fail +registration before a driver runs. + +The selected clean run must pass and actually enter every state the history +declares. The mutation must apply, and the check must reject it under exactly +the named law. A failed clean run, missing reach, inapplicable corruption, +passing mutant, wrong-law rejection, or driver setup error fails the calibration +directly. It never searches another history. + +A selected construction uses one deterministic seed derived from the property +id and calibration name. Inserting or reordering other calibrations leaves this +seed unchanged. Each driver execution builds from that seed afresh and resolves +its plan once. This preserves builder-created metadata, including WeakMap +associations, and avoids reusing consumed plans. Fixed and enumerated values are +copied before resolution. Each calibration remains on the `calibration` route, +so a failed self-check is a framework error in the gap report. Fresh-seed +calibrations will follow the planned `UNIT` replay support; this API adds no new +replay variable. + +For automatic forced-probe controls, opt a state into direct selection: + +```ts +probeHistories: { + 'committed write': { + notEntered: { fixed: 'no writes' }, + // Optional: entered: { constructed: 'a committed write' } + }, +} +``` + +The clean-entered history detects the probe forced not entered. If `entered` +is omitted, exactly one construction must declare the state in `enters`. +The clean-not-entered history detects the probe forced entered and always needs +an explicit selection. Neither control may silently disappear. + +During consumer migration, an unbound authored calibration or entirely unbound +probe state retains candidate search and prints a registration-time warning +naming each affected calibration. Bind both probe controls together with +`probeHistories`. Search will be removed after consumers migrate, before the +first npm release. + ## The `oracle/no-unit-test` rule Each test in an oracle file must reach a part that the file declares: its diff --git a/packages/oracles/src/oracle-property.ts b/packages/oracles/src/oracle-property.ts index 9ef1585..ad0baf9 100644 --- a/packages/oracles/src/oracle-property.ts +++ b/packages/oracles/src/oracle-property.ts @@ -161,10 +161,19 @@ * - **Calibration.** A `calibration` names a wrong answer and the law that * must reject it: a `corrupt` function that alters a recorded trace, as an * observation mutant does, or a mutant `driver`, such as production set up - * wrongly. The calibration test finds a history where the corruption - * applies and the clean trace passes (it skips a candidate whose clean trace - * fails), and requires the check to report that law for the corrupted one - * (rubric, ORC-006). + * wrongly. A `history` binding selects one constructed, fixed, or enumerated + * history at registration. The clean run must pass and prove its declared + * reach; the wrong answer must apply and fail under exactly the named law. + * A failed premise fails directly, without candidate search (maintainer + * decision, 2026-11-11; rubric, ORC-006). Constructed calibrations use a stable + * seed derived from property id and calibration name. Each driver execution + * builds afresh from that seed and resolves once, preserving builder metadata + * and isolating consumed plans. Fixed/enumerated values are copied before + * resolution. During migration only, unbound calibrations warn at registration + * and retain the existing candidate search. `probeHistories` opts a state into + * direct forced controls: its clean-entered history detects forced-not-entered + * (inferred from the unique construction unless selected explicitly), and its + * explicitly selected clean-not-entered history detects forced-entered. * - **Determinism.** A `determinism` test runs some histories twice and * requires the same trace: every fixed and constructed history of a generated * source and three generated histories, or up to eight evenly spaced @@ -490,6 +499,18 @@ export interface ConstructedHistory { */ export type DeclaredState = string | { name: string; probe: string } +/** Select one existing history; a binding never searches for another. */ +export type CalibrationHistory = + { constructed: string } | { fixed: string } | { enumerated: number | string } + +/** Clean premises for the two automatic forced-probe controls. */ +export interface ProbeHistories { + /** Inferred from the unique construction entering this state when omitted. */ + entered?: CalibrationHistory + /** There is no unique default for a history that does not enter a state. */ + notEntered: CalibrationHistory +} + export interface GeneratedHistories { arbitrary: fc.Arbitrary /** @@ -539,6 +560,8 @@ export type Calibration = { name: string /** The law the check must report. */ law: string + /** Run this history directly, with a stable name-derived construction seed. */ + history?: CalibrationHistory } & ( | { /** A corrupted copy of the trace, or undefined when nothing applies. */ @@ -583,6 +606,11 @@ export interface OraclePropertyOptions { expected: TExpected, ) => Verdict calibrations?: ReadonlyArray> + /** + * Opt each probe-bound state into direct forced controls. Entirely unbound + * states retain search with warnings during the consumer migration. + */ + probeHistories?: Readonly> /** * Whether the history enters each probe-bound state, from the plan alone: * a prediction never sees the trace, so it cannot follow production. @@ -854,7 +882,7 @@ function enumeratedTests( return [ ...historyTests, ...(declared ? [reachTest] : []), - ...calibrationTests(options, () => named), + ...calibrationTests(options, () => named, named), determinismTest(options, () => named.filter((_, index) => index % spacing === 0), ), @@ -1586,89 +1614,357 @@ function judge( const lawOf = (verdict: Extract) => verdict.law ?? verdict.checkpoint +/** A seeded build preserves arbitrary context and the builder's identity metadata. */ +function buildConstruction( + id: string, + arbitrary: fc.Arbitrary, + construction: ConstructedHistory, + seed: number, + registering = false, +): THistory { + let history: THistory + try { + const [generated] = fc.sample(arbitrary, { seed, numRuns: 1 }) + history = construction.build(generated!) + } catch (error) { + if (!registering) throw error + throw new Error( + `oracleProperty ${id}: constructed history "${construction.name}" threw while it was built for registration: ${error instanceof Error ? error.message : reportText(error)}`, + { cause: error }, + ) + } + if (!arbitrary.canShrinkWithoutContext(history)) + throw untakeable(id, `constructed history "${construction.name}" built`) + return history +} + +/** Stable across list insertion and reordering; no process-random replay route. */ +function calibrationSeed(id: string, name: string): number { + let seed = 2166136261 + for (const char of `${id}\0${name}`) + seed = Math.imul(seed ^ char.charCodeAt(0), 16777619) + return seed | 0 +} + +interface SelectedHistory { + name: string + enters: ReadonlyArray + build: () => THistory +} + +/** Resolve every selector at registration, through one diagnostic boundary. */ +function selectCalibrationHistory( + options: BoundOptions, + calibration: { name: string }, + selector: unknown, + enumerated?: ReadonlyArray<{ name: string; history: THistory }>, +): SelectedHistory { + const { id } = options + const invalid = (reason: string): never => { + throw new Error( + `${id}: calibration history selector for "${calibration.name}": ${reason}.`, + ) + } + if ( + !selector || + typeof selector !== `object` || + Object.keys(selector).length !== 1 || + !Object.keys(selector).every((key) => + [`constructed`, `fixed`, `enumerated`].includes(key), + ) + ) + return invalid(`expected exactly one of constructed, fixed, or enumerated`) + const source = options.histories as GeneratedHistories + if (`enumerated` in selector) { + if (!enumerated) return invalid(`enumerated requires an enumerated source`) + const key = selector.enumerated + const matches = + typeof key === `number` + ? Number.isInteger(key) && key >= 0 && key < enumerated.length + ? [enumerated[key]!] + : [] + : typeof key === `string` + ? enumerated.filter(({ name }) => name === key) + : [] + if (matches.length !== 1) + return invalid( + `enumerated ${reportText(key)} must identify exactly one history (found ${matches.length})`, + ) + const selected = matches[0]! + return { + name: `enumerated history: ${selected.name}`, + enters: [], + build: () => snapshotOf(selected.history) as THistory, + } + } + const kind = + `constructed` in selector + ? `constructed` + : `fixed` in selector + ? `fixed` + : undefined + if (!kind) return invalid(`expected constructed, fixed, or enumerated`) + if (enumerated) return invalid(`${kind} requires a generated source`) + const key = + kind === `constructed` + ? (selector as { constructed: string }).constructed + : (selector as { fixed: string }).fixed + const matches = ( + kind === `constructed` ? (source.constructed ?? []) : (source.fixed ?? []) + ).filter(({ name }) => name === key) + if (typeof key !== `string` || matches.length !== 1) + return invalid( + `${kind} ${reportText(key)} must identify exactly one history (found ${matches.length})`, + ) + const selected = matches[0]! + const name = `${kind} history: ${selected.name}` + if (`build` in selected) { + const seed = calibrationSeed(id, calibration.name) + return { + name: `${name} (seed ${seed})`, + enters: selected.enters, + build: () => buildConstruction(id, source.arbitrary, selected, seed), + } + } + return { + name, + enters: selected.enters ?? [], + build: () => snapshotOf(selected.history) as THistory, + } +} + /** - * One test per calibration. It tries candidate histories in order until the - * wrong answer applies, and requires the check to report the named law. + * A binding is a premise, never a search hint: require the selected clean run + * and actual reach, then exactly the designated law's rejection. No failed + * premise can be hidden by another history. Legacy search remains during the + * pinned consumers' migration and warns once per unbound calibration at load. */ function calibrationTests( options: BoundOptions, candidates: () => ReadonlyArray<{ name: string; history: THistory }>, + enumerated?: ReadonlyArray<{ name: string; history: THistory }>, ): Array { const { id } = options + for (const state of Object.keys(options.probeHistories ?? {})) + if (!options.probes?.has(state)) + throw new Error( + `${id}: calibration history selector for probe state "${state}": the state is not probe-bound.`, + ) // Each probe-bound state also gets two forced-entry calibrations. const forcedCalibrations = [...(options.probes ?? [])].flatMap( - ([state, probe]) => - [true, false].map((entered) => ({ + ([state, probe]) => { + const binding = options.probeHistories?.[state] + let enteredHistory = binding?.entered + const notEntered: unknown = binding?.notEntered + if (binding && notEntered === undefined) + throw new Error( + `${id}: calibration history selector for probe state "${state}": supply notEntered explicitly.`, + ) + if (binding && !enteredHistory) { + const constructions = + (options.histories as GeneratedHistories).constructed ?? [] + const matching = constructions.filter(({ enters }) => + enters.includes(state), + ) + if (matching.length !== 1) + throw new Error( + `${id}: calibration history selector for probe state "${state}": expected one construction entering it (found ${matching.length}); add one or select entered explicitly.`, + ) + enteredHistory = { constructed: matching[0]!.name } + } + return [true, false].map((entered) => ({ name: `probe ${probe} forced ${entered ? `entered` : `not entered`} (state "${state}")`, law: agreementLaw, forced: { probe, entered }, - })), + state, + // The clean premise has the opposite polarity to the forced value. + history: entered ? binding?.notEntered : enteredHistory, + })) + }, ) return [...(options.calibrations ?? []), ...forcedCalibrations].map( - (calibration) => ({ - name: `calibration: ${calibration.name}`, - route: `calibration`, - replays: [], - run: async () => { - const seen: Array = [] - for (const { name, history } of candidates()) { - const plan = planOf(options, history) - if (`corrupt` in calibration) { - const { observations, driverError, probes, spin } = - await recordGuarded(options.driver, plan) - const corrupted = calibration.corrupt(observations, plan) - if (corrupted === undefined) continue - // A calibration is meaningful only where the true trace passes, so - // a candidate whose clean trace fails, whose driver throws, or - // whose clock spins is skipped; that history fails in its own - // test. + (calibration) => { + const selected = + calibration.history === undefined + ? undefined + : selectCalibrationHistory( + options, + calibration, + calibration.history, + enumerated, + ) + if (!selected) + console.warn( + `${id}: calibration "${calibration.name}" has no history binding; using transitional candidate search. Bind its history before search is removed.`, + ) + return { + name: `calibration: ${calibration.name}`, + route: `calibration`, + replays: [], + run: async () => { + if (selected) { + const context = `${id}: calibration "${calibration.name}" on ${selected.name}` + process.stdout.write(`${context}\n`) + const failed = (reason: string, cause?: unknown): never => { + throw new Error( + `${context}: ${reason}${cause === undefined ? `` : `: ${cause instanceof Error ? cause.message : reportText(cause)}`}`, + { cause }, + ) + } + const freshPlan = (phase: string) => { + try { + return planOf(options, selected.build()) + } catch (error) { + return failed( + `selected ${phase} run failed while building or resolving`, + error, + ) + } + } + const plan = freshPlan(`clean`) + const clean = await recordGuarded(options.driver, plan) + if (clean.driverError || clean.spin) + return failed( + `selected clean run failed`, + clean.driverError?.error ?? clean.spin, + ) + let actual: ReturnType> + try { + actual = judge( + options, + plan, + clean.observations, + undefined, + clean.probes, + ) + } catch (error) { + return failed(`selected clean run failed`, error) + } + if (actual.verdict.kind !== `pass`) + return failed( + `selected clean run failed: reported law "${lawOf(actual.verdict)}"`, + ) + const missing = selected.enters.filter( + (state) => !actual.reached.has(state), + ) + if (missing.length) + return failed( + `selected history did not enter ${missing.join(`, `)}`, + ) if ( - driverError || - spin || - judge(options, plan, observations, undefined, probes).verdict - .kind !== `pass` + `forced` in calibration && + actual.reached.has(calibration.state) === + calibration.forced.entered ) - continue - const wrong = judge( - options, - plan, - corrupted, - undefined, - probes, - ).verdict + return failed( + `selected history did not enter the required opposite clean probe premise`, + ) + let wrong: Verdict + if (`corrupt` in calibration) { + const corrupted = calibration.corrupt( + snapshotOf(clean.observations) as ReadonlyArray, + plan, + ) + if (corrupted === undefined) + return failed(`selected corruption did not apply`) + wrong = judge( + options, + plan, + corrupted, + undefined, + clean.probes, + ).verdict + } else { + // Rebuild, do not clone: builders can attach WeakMap metadata, and + // a resolved plan may already have consumed its resources. + const mutantPlan = freshPlan(`mutant`) + const mutant = await recordGuarded( + `forced` in calibration ? options.driver : calibration.driver, + mutantPlan, + `forced` in calibration ? calibration.forced : undefined, + ) + if (mutant.driverError || mutant.spin) + return failed( + `selected mutant run failed`, + mutant.driverError?.error ?? mutant.spin, + ) + wrong = judge( + options, + mutantPlan, + mutant.observations, + undefined, + mutant.probes, + ).verdict + } if (wrong.kind === `violation` && lawOf(wrong) === calibration.law) return - throw new Error( - `${id}: calibration "${calibration.name}" corrupted the trace of "${name}", but the check ${wrong.kind === `pass` ? `passed it` : `reported law "${lawOf(wrong)}"`}. It must report law "${calibration.law}".`, + return failed( + `but the check ${wrong.kind === `pass` ? `passed it` : `reported law "${lawOf(wrong)}"`}. It must report law "${calibration.law}".`, ) } - // The real driver's trace of the candidate must pass first. - const clean = await recordGuarded(options.driver, plan) - if ( - clean.driverError || - clean.spin || - judge(options, plan, clean.observations, undefined, clean.probes) - .verdict.kind !== `pass` + const seen: Array = [] + for (const { name, history } of candidates()) { + const plan = planOf(options, history) + if (`corrupt` in calibration) { + const { observations, driverError, probes, spin } = + await recordGuarded(options.driver, plan) + const corrupted = calibration.corrupt(observations, plan) + if (corrupted === undefined) continue + // A calibration is meaningful only where the true trace passes, so + // a candidate whose clean trace fails, whose driver throws, or + // whose clock spins is skipped; that history fails in its own + // test. + if ( + driverError || + spin || + judge(options, plan, observations, undefined, probes).verdict + .kind !== `pass` + ) + continue + const wrong = judge( + options, + plan, + corrupted, + undefined, + probes, + ).verdict + if ( + wrong.kind === `violation` && + lawOf(wrong) === calibration.law + ) + return + throw new Error( + `${id}: calibration "${calibration.name}" corrupted the trace of "${name}", but the check ${wrong.kind === `pass` ? `passed it` : `reported law "${lawOf(wrong)}"`}. It must report law "${calibration.law}".`, + ) + } + // The real driver's trace of the candidate must pass first. + const clean = await recordGuarded(options.driver, plan) + if ( + clean.driverError || + clean.spin || + judge(options, plan, clean.observations, undefined, clean.probes) + .verdict.kind !== `pass` + ) + continue + const { verdict } = + `forced` in calibration + ? await runOne( + options, + history, + options.driver, + calibration.forced, + ) + : await runOne(options, history, calibration.driver) + if (verdict.kind === `pass`) continue + if (lawOf(verdict) === calibration.law) return + seen.push(`"${name}" reported law "${lawOf(verdict)}"`) + } + throw new Error( + `${id}: calibration "${calibration.name}" found no history whose clean trace passes and where the check reports law "${calibration.law}"${seen.length ? `; ${seen.slice(0, 3).join(`, `)}` : ``}. Add a passing fixed or constructed history that reaches the state its wrong answer changes.`, ) - continue - const { verdict } = - `forced` in calibration - ? await runOne( - options, - history, - options.driver, - calibration.forced, - ) - : await runOne(options, history, calibration.driver) - if (verdict.kind === `pass`) continue - if (lawOf(verdict) === calibration.law) return - seen.push(`"${name}" reported law "${lawOf(verdict)}"`) - } - throw new Error( - `${id}: calibration "${calibration.name}" found no history whose clean trace passes and where the check reports law "${calibration.law}"${seen.length ? `; ${seen.slice(0, 3).join(`, `)}` : ``}. Add a passing fixed or constructed history that reaches the state its wrong answer changes.`, - ) - }, - }), + }, + } + }, ) } @@ -1840,22 +2136,13 @@ function generatedTests( const examplesFor = (seed: number, registering = false) => { const constructionOf = new Map>() const examples = constructions.map((construction, index) => { - let history: THistory - try { - const [generated] = fc.sample(source.arbitrary, { - seed: seed + index, - numRuns: 1, - }) - history = construction.build(generated!) - } catch (error) { - if (!registering) throw error - throw new Error( - `oracleProperty ${id}: constructed history "${construction.name}" threw while it was built for registration: ${error instanceof Error ? error.message : reportText(error)}`, - { cause: error }, - ) - } - if (!source.arbitrary.canShrinkWithoutContext(history)) - throw untakeable(id, `constructed history "${construction.name}" built`) + const history = buildConstruction( + id, + source.arbitrary, + construction, + seed + index, + registering, + ) constructionOf.set(history, construction) return [history] as [THistory] }) diff --git a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts index 83ed177..2a2a60b 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts +++ b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts @@ -213,8 +213,17 @@ export type SyntheticSpec = { nondeterministic: boolean calibrations: ReadonlyArray<{ name: string - kind: `right` | `wrong-law` | `no-op` | `driver` + kind: `right` | `wrong-law` | `no-op` | `driver` | `inapplicable` | `throws` }> + /** The proposed binding, passed structurally so the unchanged framework can be judged. */ + calibrationHistory?: + | { constructed: string } + | { fixed: string } + | { enumerated: number | string } + /** Preserve generated variation while testing builder identity and consumed plans. */ + isolatedPlans?: boolean + probeBinding?: `inferred` | `explicit` | `no-absence` + duplicateHistoryNames?: boolean /** `observe`: the driver reports reach itself, the framework's default. */ reachVia: `reached` | `observe` /** Each history takes about 300 ms; with `timeout`, tests outlive it. */ @@ -644,11 +653,20 @@ function lateProbe() { * wrong answer a driver calibration supplies. */ function syntheticDriver(spec: SyntheticSpec, mutant = false) { + const consumed = new WeakSet() return async ( formed: Formed, observe: (checkpoint: string, value: unknown) => void, ) => { const history = itemsOf(spec)(formed) + if ( + spec.isolatedPlans && + expect.getState().currentTestName?.includes(`calibration:`) + ) { + if (consumed.has(formed as object)) + throw new Error(`driver received a consumed plan`) + consumed.add(formed as object) + } const log = process.env.SELF_ORACLE_LOG if (log) appendFileSync( @@ -769,12 +787,26 @@ function syntheticCalibrations( return spec.calibrations.map( ({ name, kind }): Calibration => { // A mutant driver records a 3 in place of the first item. - if (kind === `driver`) - return { name, law: `three`, driver: syntheticDriver(spec, true) } + const history = spec.calibrationHistory + if (kind === `driver` || kind === `throws`) + return { + name, + history, + law: `three`, + driver: + kind === `throws` + ? async (plan, observe) => { + await syntheticDriver(spec, true)(plan, observe) + throw new Error(`mutant setup failed`) + } + : syntheticDriver(spec, true), + } return { name, + history, law: kind === `wrong-law` ? `four` : `three`, corrupt: (trace) => { + if (kind === `inapplicable`) return undefined const at = trace.findIndex(({ checkpoint }) => checkpoint.startsWith(`item`), ) @@ -838,7 +870,33 @@ export function syntheticProperty(spec: SyntheticSpec): void { const id = spec.unregistered ? `self-oracle.unregistered` : spec.id const form = formOf(spec) const items = itemsOf(spec) + // A builder may attach metadata by object identity. A plan may own consumed + // resources. Neither cloning a built history nor resolving it twice preserves + // that contract. These sets record those premises, not expected observations. + const built = new WeakSet() + const resolved = new WeakSet() + const resolve = spec.isolatedPlans + ? (history: Formed) => { + if ( + expect.getState().currentTestName?.includes(`calibration:`) && + ((spec.calibrationHistory && + `constructed` in spec.calibrationHistory) || + (spec.probeBinding && + expect + .getState() + .currentTestName?.includes(`forced not entered`))) + ) { + if (!built.has(history as object) || resolved.has(history as object)) + throw new Error( + `calibration lost builder metadata or reused a built history`, + ) + resolved.add(history as object) + } + return form([...items(history)]) + } + : undefined const parts = { + ...(resolve ? { resolve } : {}), model: spec.strictModel ? (history: Formed, trace: ReadonlyArray<{ checkpoint: string }>) => { const recorded = trace.filter(({ checkpoint }) => @@ -877,6 +935,21 @@ export function syntheticProperty(spec: SyntheticSpec): void { }), } : {}), + ...(spec.probeBinding + ? { + probeHistories: { + 'saw-0': + spec.probeBinding === `no-absence` + ? ({} as never) + : { + notEntered: { fixed: `fixed-0` }, + ...(spec.probeBinding === `explicit` + ? { entered: { constructed: `constructed-0` } } + : {}), + }, + }, + } + : {}), driver: spec.slow ? trackedSlow(syntheticDriver(spec)) : syntheticDriver(spec), @@ -915,7 +988,7 @@ export function syntheticProperty(spec: SyntheticSpec): void { // The last history's late probe call comes after its test; the file waits // for it, so the run still reports it. if (spec.probe === `late`) - afterAll(() => new Promise((resolve) => setTimeout(resolve, 50))) + afterAll(() => new Promise((settle) => setTimeout(settle, 50))) const register = () => { if (spec.source.kind === `enumerated`) { const enumerated = spec.source.histories.map(form) @@ -931,10 +1004,15 @@ export function syntheticProperty(spec: SyntheticSpec): void { ? {} : { label: (history: Formed) => - spec.labelStyle?.startsWith(`reserved-`) && - JSON.stringify(items(history)) === first - ? spec.labelStyle.slice(`reserved-`.length) - : `items ${items(history).join(` | `)}${spec.labelStyle === `shell` ? ` ${shellText}` : ``}`, + spec.duplicateHistoryNames && + spec.calibrationHistory && + `enumerated` in spec.calibrationHistory && + typeof spec.calibrationHistory.enumerated === `string` + ? `items ${spec.source.kind === `enumerated` ? spec.source.histories[0]!.join(` | `) : ``}` + : spec.labelStyle?.startsWith(`reserved-`) && + JSON.stringify(items(history)) === first + ? spec.labelStyle.slice(`reserved-`.length) + : `items ${items(history).join(` | `)}${spec.labelStyle === `shell` ? ` ${shellText}` : ``}`, }), ...parts, }) @@ -965,7 +1043,12 @@ export function syntheticProperty(spec: SyntheticSpec): void { build: (generated: Formed) => { if (at === 0 && spec.throwingBuild) throw new Error(`synthetic build failed`) - return form([...prefix, ...items(generated)].slice(0, 3)) + const tail = spec.isolatedPlans + ? items(generated).map((item) => item % 3) + : items(generated) + const history = form([...prefix, ...tail].slice(0, 3)) + built.add(history as object) + return history }, })), fixed: spec.fixed.map((fixed) => ({ diff --git a/packages/oracles/tests/oracle-property-self-oracle-reporter.mjs b/packages/oracles/tests/oracle-property-self-oracle-reporter.mjs index 38d2fe0..560aa63 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-reporter.mjs +++ b/packages/oracles/tests/oracle-property-self-oracle-reporter.mjs @@ -20,9 +20,13 @@ const serialize = (error) => { export default class SelfOracleReporter { /** @type {Map} console output by task id */ console = new Map() + /** @type {string[]} registration-time warnings */ + warnings = [] /** @param {{ taskId?: string, content: unknown }} log */ onUserConsoleLog(log) { + if (String(log.content).includes(`transitional candidate search`)) + this.warnings.push(String(log.content)) if (log.taskId === undefined) return this.console.set( log.taskId, @@ -56,6 +60,7 @@ export default class SelfOracleReporter { process.env.SELF_ORACLE_REPORT ?? `self-oracle-report.json`, JSON.stringify({ files, + warnings: this.warnings, unhandled: (unhandledErrors ?? []).map(serialize), }), ) diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 191ad80..3b7395c 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -61,9 +61,13 @@ * history does (maintainer decision, 2026-11-10, F23: every enumerated history runs, so a * required state needs no constructed history). Run alone, under a `-t` pattern that selects * only it, the reach test judges every history itself and owes the same outcome and counts. - * - `calibration`: a calibration uses the first candidate whose clean trace - * passes, and passes only when its wrong answer (a corrupted trace or a - * mutant driver) fails with the named law. + * - `calibration`: a bound calibration runs only its selected history. Its clean + * run must pass and enter the history's declared states, then its wrong answer + * must fail under exactly the named law. An observation mutant uses one driver + * call; a driver mutant uses two fresh builds and plans from the same seed. + * Invalid selectors fail registration before any driver runs. These are the + * maintainer-approved history-binding contract (2026-11-11). During migration, + * unbound calibrations retain the first-passing-candidate search. * - `determinism`: the determinism test runs every fixed and constructed history * and three generated ones, and fails for a driver whose trace changes, * compared by their data encoding: `-0` is not `0`, and key order and @@ -192,7 +196,14 @@ import type { const allStates = [`saw-0`, `saw-1`, `saw-2`] const stateRoles = [`required`, `measured`, `undeclared`] as const type StateRole = (typeof stateRoles)[number] -const calibrationKinds = [`right`, `wrong-law`, `no-op`, `driver`] as const +const calibrationKinds = [ + `right`, + `wrong-law`, + `no-op`, + `driver`, + `inapplicable`, + `throws`, +] as const type Draft = { source: | { kind: `generated`; runs: number; seed: number } @@ -220,7 +231,11 @@ type Draft = { reachVia: `reached` | `observe` /** `slow`: 300 ms per history under a 100 ms timeout. */ timing: `normal` | `slow` - calibrations: Array<(typeof calibrationKinds)[number]> + calibrations: Array + calibrationHistory: NonNullable | null + isolatedPlans: boolean + probeBinding: `unbound` | `inferred` | `explicit` | `no-absence` + duplicateHistoryNames: boolean /** An id no oracle configuration registers. */ unregistered: boolean /** How the driver calls the probe bound to `saw-0`; `none` binds no probe. */ @@ -275,6 +290,37 @@ const insideOrOutside = (insideWeight: number) => { arbitrary: outside, weight: 1 }, ) const draft: fc.Arbitrary = fc.record({ + calibrationHistory: fc.option( + fc.oneof( + fc.record({ + fixed: fc.constantFrom(`fixed-0`, `fixed-1`, `unknown`, `missing`), + }), + fc.record({ + constructed: fc.constantFrom(`constructed-0`, `unknown`, `missing`), + }), + fc.record({ + enumerated: fc.oneof( + fc.integer({ min: -1, max: 4 }), + fc.constant(1.5), + fc.constant(`items 0 | 1`), + ), + }), + ), + { freq: 20 }, + ), + isolatedPlans: fc.boolean(), + probeBinding: fc.oneof( + { weight: 20, arbitrary: fc.constant(`unbound` as const) }, + { + weight: 1, + arbitrary: fc.constantFrom( + `inferred` as const, + `explicit` as const, + `no-absence` as const, + ), + }, + ), + duplicateHistoryNames: fc.boolean(), source: fc.oneof( fc.record({ kind: fc.constant(`generated` as const), @@ -540,7 +586,7 @@ function resolveSuite({ drafts, seedless }: Suite): Plan { const generated = d.source.kind === `generated` const constructed = generated ? d.constructed.map((construction, at) => ({ - name: `constructed-${at}`, + name: `constructed-${d.duplicateHistoryNames && d.calibrationHistory ? 0 : at}`, ...construction, })) : [] @@ -561,7 +607,7 @@ function resolveSuite({ drafts, seedless }: Suite): Plan { constructed, fixed: generated ? d.fixed.map((fixedHistory, at) => ({ - name: `fixed-${at}`, + name: `fixed-${d.duplicateHistoryNames && d.calibrationHistory ? 0 : at}`, history: fixedHistory.history, ...(fixedHistory.enters ? { enters: fixedHistory.enters } : {}), })) @@ -656,6 +702,14 @@ function resolveSuite({ drafts, seedless }: Suite): Plan { : {}), ...(d.nested ? { nested: true } : {}), ...(d.strictModel ? { strictModel: true } : {}), + ...(d.calibrationHistory + ? { calibrationHistory: d.calibrationHistory } + : {}), + ...(d.isolatedPlans ? { isolatedPlans: true } : {}), + ...(d.probeBinding !== `unbound` + ? { probeBinding: d.probeBinding } + : {}), + ...(d.duplicateHistoryNames ? { duplicateHistoryNames: true } : {}), calibrations: d.calibrations.map((kind, at) => ({ name: `cal-${at}`, kind, @@ -689,6 +743,10 @@ type Kind = | `calibration-passed` | `calibration-wrong-law` | `calibration-none` + | `calibration-clean` + | `calibration-reach` + | `calibration-inapplicable` + | `calibration-mutant-error` | `timeout` | `model-not-total` | `data` @@ -1152,6 +1210,62 @@ function predictSpec( } } } + // The synthetic namespace is deliberately small: names identify the fixed + // or constructed index; enumerated names state their items. This model judges + // selection independently of the framework's resolver. + const binding = spec.calibrationHistory + const selected = + binding && + (`fixed` in binding + ? spec.fixed.find(({ name }) => name === binding.fixed) + : `constructed` in binding + ? spec.constructed.find(({ name }) => name === binding.constructed) + : spec.source.kind === `enumerated` + ? spec.source.histories.find((history, index) => + typeof binding.enumerated === `number` + ? index === binding.enumerated + : `items ${history.join(` | `)}` === binding.enumerated, + ) + : undefined) + const duplicates = + binding && + (`fixed` in binding + ? spec.fixed.filter(({ name }) => name === binding.fixed).length > 1 + : `constructed` in binding + ? spec.constructed.filter(({ name }) => name === binding.constructed) + .length > 1 + : typeof binding.enumerated === `string` && + spec.source.kind === `enumerated` + ? (spec.duplicateHistoryNames && + binding.enumerated === + `items ${spec.source.histories[0]!.join(` | `)}` && + spec.source.histories.length > 1) || + spec.source.histories.filter( + (history) => + `items ${history.join(` | `)}` === binding.enumerated, + ).length > 1 + : false) + if ( + binding && + spec.calibrations.length && + (!selected || + duplicates || + (spec.source.kind === `enumerated` + ? !(`enumerated` in binding) + : `enumerated` in binding)) + ) + return { loads: false, loadError: `calibration history selector` } + if ( + spec.probeBinding && + (!spec.probe || + spec.probeBinding === `no-absence` || + !spec.fixed.some(({ name }) => name === `fixed-0`) || + (spec.probeBinding === `inferred` + ? spec.constructed.filter(({ enters }) => enters.includes(`saw-0`)) + .length !== 1 + : !spec.constructed.some(({ name }) => name === `constructed-0`))) + ) + return { loads: false, loadError: `calibration history selector` } const names = spec.source.kind === `enumerated` ? spec.source.histories.map((history) => @@ -1216,12 +1330,36 @@ function predictSpec( // A calibration needs a candidate whose clean trace passes; it skips the // others (maintainer decision, 2026-11-03). Every history has an item, so // the corruption applies to every candidate. - const calibrate = (name: string, kind: (typeof calibrationKinds)[number]) => { + const calibrate = ( + name: string, + kind: SyntheticSpec[`calibrations`][number][`kind`], + ) => { + if (binding) { + const clean = ran(`calibration: ${name}`)[0] + if (!clean || judged(spec, clean).kind !== `pass`) + return { kind: `calibration-clean` } as Outcome + const enters = + selected && !Array.isArray(selected) && `enters` in selected + ? (selected.enters ?? []) + : [] + if (!reaches(spec, clean, enters)) + return { kind: `calibration-reach` } as Outcome + if (kind === `inapplicable`) + return { kind: `calibration-inapplicable` } as Outcome + if (kind === `throws`) + return { kind: `calibration-mutant-error` } as Outcome + if (kind === `driver` || kind === `right`) + return { kind: `pass` } as Outcome + return { + kind: kind === `no-op` ? `calibration-passed` : `calibration-wrong-law`, + } as Outcome + } const used = ran(`calibration: ${name}`).find( (history) => judged(spec, history).kind === `pass`, ) - if (!used) return { kind: `calibration-none` } as Outcome - if (kind === `right` || kind === `driver`) + if (!used || kind === `inapplicable`) + return { kind: `calibration-none` } as Outcome + if (kind === `right` || kind === `driver` || kind === `throws`) return { kind: `pass` } as Outcome return { kind: kind === `no-op` ? `calibration-passed` : `calibration-wrong-law`, @@ -1308,6 +1446,18 @@ function predictSpec( // reach agreement. for (const [at, name] of forcedCalibrationNames(spec).entries()) { const forced = at === 0 + if (spec.probeBinding) { + const clean = ran(name)[0] + tests.set( + name, + !clean || judged(spec, clean).kind !== `pass` + ? { kind: `calibration-clean` } + : predictsZero(clean) === forced + ? { kind: `calibration-reach` } + : { kind: `pass` }, + ) + continue + } tests.set( name, ran(name).some( @@ -1441,6 +1591,9 @@ const frameworkStates = [ `calibration-applied`, `calibration-rejected`, `calibration-skipped-candidate`, + `bound-calibration`, + `bound-constructed-driver`, + `bound-calibration-registration-error`, `determinism-caught`, `observed-reach`, `timeout`, @@ -1503,6 +1656,8 @@ function frameworkReach( const owed = expected.get(spec.id)! if (!owed.loads) { reached.add(`registration-error`) + if (spec.calibrationHistory) + reached.add(`bound-calibration-registration-error`) if (owed.shapeError) reached.add(`enumerated-shape-error`) if (owed.reservedLabel) reached.add(`reserved-label-error`) if (owed.dataError) reached.add(`history-out-of-domain`) @@ -1520,6 +1675,15 @@ function frameworkReach( continue } const named = spec.fixed.length + spec.constructed.length > 0 + if (spec.calibrationHistory || spec.probeBinding) + reached.add(`bound-calibration`) + if ( + spec.isolatedPlans && + spec.calibrationHistory && + `constructed` in spec.calibrationHistory && + spec.calibrations.some(({ kind }) => kind === `driver`) + ) + reached.add(`bound-constructed-driver`) if (spec.arbitraryMap?.unmapper === true && named) reached.add(`unmapper-history-loads`) if (spec.arbitraryMap?.unmapper === false && !named) @@ -1742,6 +1906,7 @@ type ReportedFile = { type Report = { files: Array stdout: string + warnings: Array unhandled: Array } type Failure = { @@ -1781,6 +1946,7 @@ type SuiteRun = { } > log: Array + warnings: Array /** The run's unhandled errors, by message. */ unhandled: Array } @@ -1850,40 +2016,50 @@ function classify(error: ReportedError | undefined): Failure { : undefined // A data error names the value's path, then its kind. const data = /^(.*): (.*) is not (?:observation|history) data$/.exec(text) - const kind: Kind | `other` = - inner.name === `OracleDataError` - ? `data` - : text.includes(`the framework's encoding limit`) - ? `limit` - : inner.name === `OracleModelError` - ? `model-not-total` - : inner.name === `OracleViolation` - ? `violation` - : /timed out in \d+ms/.test(text) || - /timed out in \d+ms/.test(outer) - ? `timeout` - : text.includes(`did not enter`) && - text.includes(`: fixed history "`) - ? `fixed-reach` - : text.includes(`did not enter`) && - text.includes(`: constructed history "`) - ? `constructed-reach` - : text.includes(`no enumerated history entered`) - ? `enumerated-reach` - : text.includes(`which the property does not declare`) - ? `undeclared-state` - : text.includes(`two runs of`) - ? `determinism` - : text.includes(`but the check passed it`) - ? `calibration-passed` - : text.includes(`but the check reported law`) - ? `calibration-wrong-law` - : text.includes(`found no history`) - ? `calibration-none` - : text.includes(`synthetic driver failed`) || - text.includes(`cannot read a 7`) - ? `setup` - : `other` + const kind: Kind | `other` = text.includes(`selected clean run failed`) + ? `calibration-clean` + : text.includes(`selected history did not enter`) + ? `calibration-reach` + : text.includes(`selected corruption did not apply`) + ? `calibration-inapplicable` + : text.includes(`selected mutant run failed`) + ? `calibration-mutant-error` + : inner.name === `OracleDataError` + ? `data` + : text.includes(`the framework's encoding limit`) + ? `limit` + : inner.name === `OracleModelError` + ? `model-not-total` + : inner.name === `OracleViolation` + ? `violation` + : /timed out in \d+ms/.test(text) || + /timed out in \d+ms/.test(outer) + ? `timeout` + : text.includes(`did not enter`) && + text.includes(`: fixed history "`) + ? `fixed-reach` + : text.includes(`did not enter`) && + text.includes(`: constructed history "`) + ? `constructed-reach` + : text.includes(`no enumerated history entered`) + ? `enumerated-reach` + : text.includes(`which the property does not declare`) + ? `undeclared-state` + : text.includes(`two runs of`) + ? `determinism` + : text.includes(`but the check passed it`) + ? `calibration-passed` + : text.includes(`but the check reported law`) + ? `calibration-wrong-law` + : text.includes(`found no history`) + ? `calibration-none` + : text.includes( + `synthetic driver failed`, + ) || + text.includes(`cannot read a 7`) || + text.includes(`mutant setup failed`) + ? `setup` + : `other` return { kind, // A model error that quotes a machine path, such as an inspected stack. @@ -1972,10 +2148,21 @@ function runVitest( ) // The framework prints reach counts straight to stdout, past Vitest's // console capture, so they arrive in the child's own output. - const { files: reported, unhandled } = JSON.parse( - readFileSync(report, `utf8`), - ) as { files: Array; unhandled?: Array } - return { files: reported, stdout: run.stdout, unhandled: unhandled ?? [] } + const { + files: reported, + unhandled, + warnings, + } = JSON.parse(readFileSync(report, `utf8`)) as { + files: Array + unhandled?: Array + warnings?: Array + } + return { + files: reported, + stdout: run.stdout, + warnings: warnings ?? [], + unhandled: unhandled ?? [], + } } // A test's name within its property, without a nested spec's outer suite. @@ -1993,6 +2180,10 @@ const readLog = (log: string): Array => const entry = JSON.parse(line) as LogEntry return { ...entry, test: unnested(entry.test) } }) + // Vitest may run files concurrently. Each property's call order is + // observable; interleaving independent properties is not a framework + // promise. Stable grouping retains every call and each local order. + .sort((a, b) => a.id.localeCompare(b.id)) : [] // The items of a counterexample that fast-check printed, in any form: the // first bracketed list of integers, where the lossy form's `Number.NaN` is a @@ -2019,7 +2210,7 @@ const statesIn = (report: Report, plan: Plan, files: Array) => ), ) -async function driveSuite( +function driveSuite( plan: Plan, observe: (checkpoint: string, value: unknown) => void, ) { @@ -2042,6 +2233,7 @@ async function driveSuite( const run: SuiteRun = { files: {}, log: readLog(log), + warnings: [...report.warnings].sort(), unhandled: report.unhandled.map((error) => error.message ?? ``), } for (const [index, spec] of plan.specs.entries()) { @@ -2090,6 +2282,45 @@ async function driveSuite( } } observe(`run`, run) + // Adding a calibration ahead of another must not change the latter's seed + // or built history. This is a metamorphic comparison of driver inputs, + // independent of the framework's particular seed hash. + for (const [index, spec] of plan.specs.entries()) { + if ( + !spec.calibrationHistory || + !(`constructed` in spec.calibrationHistory) || + !run.files[spec.id]!.loaded + ) + continue + const file = files[index]! + const original = readFileSync(file, `utf8`) + const reorderedLog = join(directory, `reordered-${index}.jsonl`) + try { + const reordered = { + ...spec, + calibrations: [ + { name: `unrelated calibration`, kind: `right` }, + ...[...spec.calibrations].reverse(), + ], + } + writeFileSync( + file, + `import { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\nsyntheticProperty(${JSON.stringify(reordered)})\n`, + ) + runVitest( + directory, + [file], + { SELF_ORACLE_LOG: reorderedLog }, + `-t 'calibration:'`, + ) + observe(`binding-reordered`, { + id: spec.id, + log: readLog(reorderedLog), + }) + } finally { + writeFileSync(file, original) + } + } if (plan.seedless) { const seedlessLog = join(directory, `seedless-${runCount}.jsonl`) runVitest( @@ -2263,6 +2494,15 @@ function checkSuite( const owed = expected.get(spec.id)! const seen = run.files[spec.id]! if (!owed.loads) { + if ( + (spec.calibrationHistory || spec.probeBinding) && + run.log.some((entry) => entry.id === spec.id) + ) + return violation( + `run`, + `${spec.id} ran a driver despite an invalid calibration selector`, + `calibration`, + ) const law = owed.membership ? `membership` : `registration` if (seen.loaded) return violation( @@ -2292,6 +2532,70 @@ function checkSuite( } if (!seen.loaded) return violation(`run`, `${spec.id} failed to load`, `registration`) + const reordered = trace.find( + ({ checkpoint, value }) => + checkpoint === `binding-reordered` && + (value as { id: string }).id === spec.id, + )?.value as { log: Array } | undefined + if (reordered) { + const tests = new Set( + spec.calibrations.map( + ({ name }) => `${spec.id} > calibration: ${name}`, + ), + ) + const selected = (log: ReadonlyArray) => + log + .filter(({ test }) => tests.has(test)) + .map(({ test, history }) => ({ test, history })) + .sort((a, b) => a.test.localeCompare(b.test)) + if ( + JSON.stringify(selected(run.log)) !== + JSON.stringify(selected(reordered.log)) + ) + return violation( + `run`, + `${spec.id} changed bound histories after calibration insertion/reordering`, + `calibration`, + ) + } + // The compatibility route must be visible before any selected test runs, + // including automatic controls; bound calibrations must never warn search. + const unbound = [ + ...(!spec.calibrationHistory + ? spec.calibrations.map(({ name }) => name) + : []), + ...(!spec.probeBinding + ? forcedCalibrationNames(spec).map((name) => + name.slice(`calibration: `.length), + ) + : []), + ] + for (const name of unbound) + if ( + run.warnings.filter((warning) => + warning.includes( + `${spec.id}: calibration "${name}" has no history binding`, + ), + ).length !== 1 + ) + return violation( + `run`, + `${spec.id} did not warn once at registration for unbound calibration "${name}"`, + `calibration`, + ) + if ( + (spec.calibrationHistory || spec.probeBinding) && + run.warnings.some( + (warning) => + warning.startsWith(`${spec.id}:`) && + !unbound.some((name) => warning.includes(`calibration "${name}"`)), + ) + ) + return violation( + `run`, + `${spec.id} warned about search for a bound calibration`, + `calibration`, + ) const placed = seen.tests.find((test) => test.name.includes(packageDirectory), ) @@ -2313,6 +2617,84 @@ function checkSuite( for (const test of seen.tests) { const outcome = owed.tests.get(test.name)! const failure = test.failure + if ( + spec.probeBinding && + forcedCalibrationNames(spec).includes(test.name) + ) { + const calls = run.log.filter( + (entry) => + entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, + ) + const expectedCalls = + outcome.kind === `calibration-clean` || + outcome.kind === `calibration-reach` + ? 1 + : 2 + if ( + calls.length !== expectedCalls || + calls.some( + ({ history }) => + JSON.stringify(history) !== JSON.stringify(calls[0]!.history), + ) + ) + return violation( + `run`, + `${spec.id} forced control did not run ${expectedCalls} equivalent selected histories`, + `calibration`, + ) + } + if (spec.calibrationHistory && test.name.startsWith(`calibration:`)) { + const calibration = spec.calibrations.find( + ({ name }) => test.name === `calibration: ${name}`, + )! + const calls = run.log.filter( + (entry) => + entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, + ) + const expectedCalls = + (calibration.kind === `driver` || calibration.kind === `throws`) && + outcome.kind !== `calibration-clean` && + outcome.kind !== `calibration-reach` + ? 2 + : 1 + if (calls.length !== expectedCalls) + return violation( + `run`, + `${spec.id} "${test.name}" used ${calls.length} driver calls; binding owes ${expectedCalls}`, + `calibration`, + ) + const binding = spec.calibrationHistory + const first = calls[0]!.history + const selected = + `fixed` in binding + ? spec.fixed.find(({ name }) => name === binding.fixed)?.history + : `enumerated` in binding && spec.source.kind === `enumerated` + ? spec.source.histories.find((history, index) => + typeof binding.enumerated === `number` + ? index === binding.enumerated + : `items ${history.join(` | `)}` === binding.enumerated, + ) + : undefined + const prefix = + `constructed` in binding + ? spec.constructed.find(({ name }) => name === binding.constructed)! + .prefix + : undefined + if ( + (selected && JSON.stringify(first) !== JSON.stringify(selected)) || + (prefix && + JSON.stringify(first.slice(0, prefix.length)) !== + JSON.stringify(prefix)) || + calls.some( + ({ history }) => JSON.stringify(history) !== JSON.stringify(first), + ) + ) + return violation( + `run`, + `${spec.id} "${test.name}" did not run equivalent selected histories`, + `calibration`, + ) + } const observedKind = test.state === `passed` ? `pass` : (failure?.kind ?? `other`) if ( @@ -2605,6 +2987,10 @@ const generatedDraft = (overrides: Partial): Draft => ({ reachVia: `reached`, timing: `normal`, calibrations: [], + calibrationHistory: null, + isolatedPlans: false, + probeBinding: `unbound`, + duplicateHistoryNames: false, unregistered: false, probe: `none`, overReport: false, @@ -2653,11 +3039,248 @@ oracleProperty({ }, histories: { arbitrary: suite, - states: [], - measured: frameworkStates, + states: [ + `bound-calibration`, + `bound-calibration-registration-error`, + `bound-constructed-driver`, + ], + measured: frameworkStates.filter((state) => !state.startsWith(`bound-`)), runs: 3, fixedSeeds: [1], + constructed: [ + { + name: `bound constructed drivers rebuild seeded plans`, + enters: [`bound-constructed-driver`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([6])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + calibrations: [`driver`], + calibrationHistory: { constructed: `constructed-0` }, + }), + ], + }), + }, + { + name: `bound calibrations bypass unrelated histories`, + enters: [`bound-calibration`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + // The decoy fails if scanned. The selected construction keeps a + // generated tail; builder identity and fresh resolution are checked. + fixed: [at([6])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + calibrations: [generated.drafts[0]!.clock ? `driver` : `right`], + calibrationHistory: { constructed: `constructed-0` }, + }), + ], + }), + }, + { + name: `invalid bindings fail before execution`, + enters: [`bound-calibration-registration-error`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([0])], + calibrations: [`right`], + calibrationHistory: { fixed: `unknown` }, + }), + ], + }), + }, + { + name: `bound forced controls retain both clean premises`, + enters: [`bound-calibration`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + probe: `agree`, + probeBinding: generated.drafts[0]!.clock + ? `inferred` + : `explicit`, + }), + ], + }), + }, + ], fixed: [ + fixedHistory( + `selected clean driver errors cannot search for another history`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([6]), at([0])], + calibrations: [`driver`], + calibrationHistory: { fixed: `fixed-0` }, + }), + ), + fixedHistory( + `bound constructed drivers rebuild seeded plans`, + [`bound-constructed-driver`], + generatedDraft({ + fixed: [at([6])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + calibrations: [`driver`], + calibrationHistory: { constructed: `constructed-0` }, + }), + ), + fixedHistory( + `duplicate enumerated labels fail before execution`, + [`bound-calibration-registration-error`], + generatedDraft({ + source: { + kind: `enumerated`, + histories: [ + [0, 1], + [1, 0], + ], + }, + duplicateHistoryNames: true, + calibrations: [`right`], + calibrationHistory: { enumerated: `items 0 | 1` }, + }), + ), + ...([`inferred`, `explicit`] as const).map((probeBinding) => + fixedHistory( + `bound forced controls use ${probeBinding} entered history`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans: true, + probe: `agree`, + probeBinding, + }), + ), + ), + fixedHistory( + `probe inference rejects ambiguous and missing constructions`, + [`registration-error`], + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [0], enters: [`saw-0`] }, + ], + probe: `agree`, + probeBinding: `inferred`, + }), + generatedDraft({ + fixed: [at([1])], + probe: `agree`, + probeBinding: `inferred`, + }), + ), + fixedHistory( + `a bound probe control needs an absent history`, + [`registration-error`], + generatedDraft({ probe: `agree`, probeBinding: `no-absence` }), + ), + fixedHistory( + `fixed and constructed duplicate selectors fail before execution`, + [`bound-calibration-registration-error`], + generatedDraft({ + fixed: [at([0]), at([1])], + duplicateHistoryNames: true, + calibrations: [`right`], + calibrationHistory: { fixed: `fixed-0` }, + }), + generatedDraft({ + constructed: [ + { prefix: [0], enters: [] }, + { prefix: [1], enters: [] }, + ], + duplicateHistoryNames: true, + calibrations: [`right`], + calibrationHistory: { constructed: `constructed-0` }, + }), + ), + fixedHistory( + `bound calibrations use only their selected histories`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([6]), at([0, 1])], + calibrations: [`right`], + calibrationHistory: { fixed: `fixed-1` }, + }), + generatedDraft({ + source: { kind: `enumerated`, histories: [[6], [0, 2]] }, + calibrations: [`driver`], + calibrationHistory: { enumerated: 1 }, + }), + ), + ...( + [`right`, `no-op`, `wrong-law`, `inapplicable`, `throws`] as const + ).map((kind) => + fixedHistory( + `bound ${kind} requires the designated rejection`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([6]), at([0])], + calibrations: [kind], + calibrationHistory: { fixed: `fixed-1` }, + }), + ), + ), + fixedHistory( + `a failed selected clean run cannot search for another`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([3]), at([0])], + calibrations: [`right`], + calibrationHistory: { fixed: `fixed-0` }, + }), + ), + fixedHistory( + `a selected history must prove its declared reach`, + [`bound-calibration`], + generatedDraft({ + fixed: [{ history: [0], enters: [`saw-1`] }], + calibrations: [`right`], + calibrationHistory: { fixed: `fixed-0` }, + }), + ), + ...( + [ + { fixed: `missing` }, + { constructed: `missing` }, + { enumerated: -1 }, + { enumerated: 1.5 }, + { enumerated: 2 }, + { fixed: `fixed-0` }, + ] as const + ).map((binding, index) => + fixedHistory( + `invalid binding ${index} fails at registration`, + [`bound-calibration-registration-error`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [1]] }, + calibrations: [`right`], + calibrationHistory: binding, + }), + ), + ), + fixedHistory( + `enumerated histories can be selected by label`, + [`bound-calibration`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[6], [0, 1]] }, + calibrations: [`right`], + calibrationHistory: { enumerated: `items 0 | 1` }, + }), + ), fixedHistory(`a rerun command escapes a label with |`, [`rerun-probe`], { ...generatedDraft({}), source: { @@ -3337,6 +3960,7 @@ oracleProperty({ reached: frameworkReach, driver: driveSuite, check: checkSuite, + owns: [`src/oracle-property.ts`], // Each history spawns one to five child Vitest runs. timeout: 600_000, }) From 2597e80369937c3a53f858511e547d2b7c10c59d Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 11:00:19 -0700 Subject: [PATCH 02/14] Preserve calibration history identity and strengthen self-oracle checks --- README.md | 7 +- packages/oracles/src/oracle-property.ts | 212 ++++---- .../oracle-property-self-oracle-fixture.ts | 171 +++++-- .../tests/oracle-property-self-oracle.test.ts | 461 +++++++++++++++--- 4 files changed, 636 insertions(+), 215 deletions(-) diff --git a/README.md b/README.md index 63f3491..53a4026 100644 --- a/README.md +++ b/README.md @@ -77,9 +77,10 @@ A selected construction uses one deterministic seed derived from the property id and calibration name. Inserting or reordering other calibrations leaves this seed unchanged. Each driver execution builds from that seed afresh and resolves its plan once. This preserves builder-created metadata, including WeakMap -associations, and avoids reusing consumed plans. Fixed and enumerated values are -copied before resolution. Each calibration remains on the `calibration` route, -so a failed self-check is a framework error in the gap report. Fresh-seed +associations, and avoids reusing consumed plans. Fixed and enumerated selectors +preserve the existing history's identity and semantics. Use `resolve` to create a +fresh mutable execution plan from those histories for each driver run. Each +calibration remains on the `calibration` route, so a failed self-check is a framework error in the gap report. Fresh-seed calibrations will follow the planned `UNIT` replay support; this API adds no new replay variable. diff --git a/packages/oracles/src/oracle-property.ts b/packages/oracles/src/oracle-property.ts index ad0baf9..87cac0e 100644 --- a/packages/oracles/src/oracle-property.ts +++ b/packages/oracles/src/oracle-property.ts @@ -168,8 +168,9 @@ * decision, 2026-11-11; rubric, ORC-006). Constructed calibrations use a stable * seed derived from property id and calibration name. Each driver execution * builds afresh from that seed and resolves once, preserving builder metadata - * and isolating consumed plans. Fixed/enumerated values are copied before - * resolution. During migration only, unbound calibrations warn at registration + * and isolating consumed plans. Fixed/enumerated selectors preserve the + * existing history's identity; use resolve to create a fresh mutable plan per + * run. During migration only, unbound calibrations warn at registration * and retain the existing candidate search. `probeHistories` opts a state into * direct forced controls: its clean-entered history detects forced-not-entered * (inferred from the unique construction unless selected explicitly), and its @@ -1694,7 +1695,7 @@ function selectCalibrationHistory( return { name: `enumerated history: ${selected.name}`, enters: [], - build: () => snapshotOf(selected.history) as THistory, + build: () => selected.history, } } const kind = @@ -1729,10 +1730,106 @@ function selectCalibrationHistory( return { name, enters: selected.enters ?? [], - build: () => snapshotOf(selected.history) as THistory, + build: () => selected.history, } } +interface ForcedCalibration { + name: string + law: string + state: string + forced: ForcedProbe +} + +/** Execute the selected history without falling back to candidate search. */ +async function runBoundCalibration( + options: BoundOptions, + calibration: Calibration | ForcedCalibration, + selected: SelectedHistory, +): Promise { + const { id } = options + const context = `${id}: calibration "${calibration.name}" on ${selected.name}` + process.stdout.write(`${context}\n`) + const failed = (reason: string, cause?: unknown): never => { + throw new Error( + `${context}: ${reason}${cause === undefined ? `` : `: ${cause instanceof Error ? cause.message : reportText(cause)}`}`, + { cause }, + ) + } + const freshPlan = (phase: string) => { + try { + return planOf(options, selected.build()) + } catch (error) { + return failed( + `selected ${phase} run failed while building or resolving`, + error, + ) + } + } + const plan = freshPlan(`clean`) + const clean = await recordGuarded(options.driver, plan) + if (clean.driverError || clean.spin) + return failed( + `selected clean run failed`, + clean.driverError?.error ?? clean.spin, + ) + let actual: ReturnType> + try { + actual = judge(options, plan, clean.observations, undefined, clean.probes) + } catch (error) { + return failed(`selected clean run failed`, error) + } + if (actual.verdict.kind !== `pass`) + return failed( + `selected clean run failed: reported law "${lawOf(actual.verdict)}"`, + ) + const missing = selected.enters.filter((state) => !actual.reached.has(state)) + if (missing.length) + return failed(`selected history did not enter ${missing.join(`, `)}`) + if ( + `forced` in calibration && + actual.reached.has(calibration.state) === calibration.forced.entered + ) + return failed( + `selected history did not enter the required opposite clean probe premise`, + ) + let wrong: Verdict + if (`corrupt` in calibration) { + const corrupted = calibration.corrupt( + snapshotOf(clean.observations) as ReadonlyArray, + plan, + ) + if (corrupted === undefined) + return failed(`selected corruption did not apply`) + wrong = judge(options, plan, corrupted, undefined, clean.probes).verdict + } else { + // Rebuild, do not clone: builders can attach WeakMap metadata, and + // a resolved plan may already have consumed its resources. + const mutantPlan = freshPlan(`mutant`) + const mutant = await recordGuarded( + `forced` in calibration ? options.driver : calibration.driver, + mutantPlan, + `forced` in calibration ? calibration.forced : undefined, + ) + if (mutant.driverError || mutant.spin) + return failed( + `selected mutant run failed`, + mutant.driverError?.error ?? mutant.spin, + ) + wrong = judge( + options, + mutantPlan, + mutant.observations, + undefined, + mutant.probes, + ).verdict + } + if (wrong.kind === `violation` && lawOf(wrong) === calibration.law) return + return failed( + `but the check ${wrong.kind === `pass` ? `passed it` : `reported law "${lawOf(wrong)}"`}. It must report law "${calibration.law}".`, + ) +} + /** * A binding is a premise, never a search hint: require the selected clean run * and actual reach, then exactly the designated law's rejection. No failed @@ -1753,7 +1850,10 @@ function calibrationTests( // Each probe-bound state also gets two forced-entry calibrations. const forcedCalibrations = [...(options.probes ?? [])].flatMap( ([state, probe]) => { - const binding = options.probeHistories?.[state] + const binding = + options.probeHistories && Object.hasOwn(options.probeHistories, state) + ? options.probeHistories[state] + : undefined let enteredHistory = binding?.entered const notEntered: unknown = binding?.notEntered if (binding && notEntered === undefined) @@ -1802,106 +1902,8 @@ function calibrationTests( route: `calibration`, replays: [], run: async () => { - if (selected) { - const context = `${id}: calibration "${calibration.name}" on ${selected.name}` - process.stdout.write(`${context}\n`) - const failed = (reason: string, cause?: unknown): never => { - throw new Error( - `${context}: ${reason}${cause === undefined ? `` : `: ${cause instanceof Error ? cause.message : reportText(cause)}`}`, - { cause }, - ) - } - const freshPlan = (phase: string) => { - try { - return planOf(options, selected.build()) - } catch (error) { - return failed( - `selected ${phase} run failed while building or resolving`, - error, - ) - } - } - const plan = freshPlan(`clean`) - const clean = await recordGuarded(options.driver, plan) - if (clean.driverError || clean.spin) - return failed( - `selected clean run failed`, - clean.driverError?.error ?? clean.spin, - ) - let actual: ReturnType> - try { - actual = judge( - options, - plan, - clean.observations, - undefined, - clean.probes, - ) - } catch (error) { - return failed(`selected clean run failed`, error) - } - if (actual.verdict.kind !== `pass`) - return failed( - `selected clean run failed: reported law "${lawOf(actual.verdict)}"`, - ) - const missing = selected.enters.filter( - (state) => !actual.reached.has(state), - ) - if (missing.length) - return failed( - `selected history did not enter ${missing.join(`, `)}`, - ) - if ( - `forced` in calibration && - actual.reached.has(calibration.state) === - calibration.forced.entered - ) - return failed( - `selected history did not enter the required opposite clean probe premise`, - ) - let wrong: Verdict - if (`corrupt` in calibration) { - const corrupted = calibration.corrupt( - snapshotOf(clean.observations) as ReadonlyArray, - plan, - ) - if (corrupted === undefined) - return failed(`selected corruption did not apply`) - wrong = judge( - options, - plan, - corrupted, - undefined, - clean.probes, - ).verdict - } else { - // Rebuild, do not clone: builders can attach WeakMap metadata, and - // a resolved plan may already have consumed its resources. - const mutantPlan = freshPlan(`mutant`) - const mutant = await recordGuarded( - `forced` in calibration ? options.driver : calibration.driver, - mutantPlan, - `forced` in calibration ? calibration.forced : undefined, - ) - if (mutant.driverError || mutant.spin) - return failed( - `selected mutant run failed`, - mutant.driverError?.error ?? mutant.spin, - ) - wrong = judge( - options, - mutantPlan, - mutant.observations, - undefined, - mutant.probes, - ).verdict - } - if (wrong.kind === `violation` && lawOf(wrong) === calibration.law) - return - return failed( - `but the check ${wrong.kind === `pass` ? `passed it` : `reported law "${lawOf(wrong)}"`}. It must report law "${calibration.law}".`, - ) - } + if (selected) + return runBoundCalibration(options, calibration, selected) const seen: Array = [] for (const { name, history } of candidates()) { const plan = planOf(options, history) diff --git a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts index 2a2a60b..83d24bf 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts +++ b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts @@ -222,7 +222,10 @@ export type SyntheticSpec = { | { enumerated: number | string } /** Preserve generated variation while testing builder identity and consumed plans. */ isolatedPlans?: boolean - probeBinding?: `inferred` | `explicit` | `no-absence` + identityHistories?: boolean + /** Public alpha-renaming of logical saw-0; the production probe id stays fixed. */ + probeName?: string + probeBinding?: `empty` | `inferred` | `explicit` | `no-absence` duplicateHistoryNames?: boolean /** `observe`: the driver reports reach itself, the framework's default. */ reachVia: `reached` | `observe` @@ -337,6 +340,11 @@ class SyntheticFault extends Error { } /** A class instance that holds a history's items and an Error. */ class SyntheticBox { + #valid = true + readItems(): SyntheticHistory { + if (!this.#valid) throw new Error(`invalid synthetic box`) + return this.items + } constructor( readonly items: SyntheticHistory, readonly failure: Error, @@ -440,7 +448,7 @@ const itemsOf = return (history as ReadonlyArray).map(itemOf) } if (spec.form === `boxed`) { - if (history instanceof SyntheticBox) return history.items + if (history instanceof SyntheticBox) return history.readItems() throw new Error(`not a boxed history`) } if (spec.form === `trap`) { @@ -652,15 +660,18 @@ function lateProbe() { * The synthetic driver. `mutant` records a 3 in place of the first item, the * wrong answer a driver calibration supplies. */ -function syntheticDriver(spec: SyntheticSpec, mutant = false) { - const consumed = new WeakSet() +function syntheticDriver( + spec: SyntheticSpec, + mutant = false, + consumed = new WeakSet(), +) { return async ( formed: Formed, observe: (checkpoint: string, value: unknown) => void, ) => { const history = itemsOf(spec)(formed) if ( - spec.isolatedPlans && + (spec.isolatedPlans || spec.identityHistories) && expect.getState().currentTestName?.includes(`calibration:`) ) { if (consumed.has(formed as object)) @@ -783,6 +794,7 @@ function syntheticCheck( function syntheticCalibrations( spec: SyntheticSpec, + consumed: WeakSet, ): Array> { return spec.calibrations.map( ({ name, kind }): Calibration => { @@ -796,10 +808,10 @@ function syntheticCalibrations( driver: kind === `throws` ? async (plan, observe) => { - await syntheticDriver(spec, true)(plan, observe) + await syntheticDriver(spec, true, consumed)(plan, observe) throw new Error(`mutant setup failed`) } - : syntheticDriver(spec, true), + : syntheticDriver(spec, true, consumed), } return { name, @@ -868,33 +880,90 @@ function declaredEnumerated( export function syntheticProperty(spec: SyntheticSpec): void { const id = spec.unregistered ? `self-oracle.unregistered` : spec.id - const form = formOf(spec) + const makeForm = formOf(spec) const items = itemsOf(spec) + const identities = new WeakMap() + const remember = (history: Formed): Formed => { + if (!spec.identityHistories) return history + identities.set(history as object, history) + return history + } + const form = (history: SyntheticHistory): Formed => + remember(makeForm(history)) + // Only the public state name changes: the synthetic model still reasons + // about item zero and the production probe id remains synthetic#saw-0. + const publicState = (state: string) => + spec.source.kind === `generated` && state === `saw-0` + ? (spec.probeName ?? state) + : state // A builder may attach metadata by object identity. A plan may own consumed // resources. Neither cloning a built history nor resolving it twice preserves // that contract. These sets record those premises, not expected observations. const built = new WeakSet() const resolved = new WeakSet() - const resolve = spec.isolatedPlans - ? (history: Formed) => { - if ( - expect.getState().currentTestName?.includes(`calibration:`) && - ((spec.calibrationHistory && - `constructed` in spec.calibrationHistory) || - (spec.probeBinding && - expect - .getState() - .currentTestName?.includes(`forced not entered`))) - ) { - if (!built.has(history as object) || resolved.has(history as object)) - throw new Error( - `calibration lost builder metadata or reused a built history`, + const resolve = + spec.isolatedPlans || spec.identityHistories + ? (history: Formed) => { + // Count invocations separately from driver calls: even a discarded + // resolved plan can have acquired resources. This records input-side + // work only; the independent checker derives the allowed budget. + const log = process.env.SELF_ORACLE_LOG + if (log) + appendFileSync( + `${log}.resolutions`, + `${JSON.stringify({ + id: spec.id, + test: expect.getState().currentTestName ?? ``, + })}\n`, ) - resolved.add(history as object) + if (spec.identityHistories && !identities.has(history as object)) + throw new Error(`selected history lost identity metadata`) + if ( + spec.isolatedPlans && + ((spec.calibrationHistory && + `constructed` in spec.calibrationHistory && + spec.calibrations.some(({ name }) => + expect + .getState() + .currentTestName?.endsWith(` > calibration: ${name}`), + )) || + (spec.probeBinding && + spec.probeBinding !== `empty` && + expect + .getState() + .currentTestName?.includes(`forced not entered`))) + ) { + if ( + !built.has(history as object) || + resolved.has(history as object) + ) + throw new Error( + `calibration lost builder metadata or reused a built history`, + ) + resolved.add(history as object) + } + return makeForm([ + ...(spec.identityHistories + ? items(identities.get(history as object)) + : items(history)), + ]) } - return form([...items(history)]) - } - : undefined + : undefined + // Clean and mutant driver instances share the consumption boundary. A + // resolved plan belongs to one execution even when different drivers use it. + const consumed = new WeakSet() + const runDriver = spec.slow + ? trackedSlow(syntheticDriver(spec, false, consumed)) + : syntheticDriver(spec, false, consumed) + const driver: typeof runDriver = (history, observe) => + runDriver(history, (checkpoint, value) => + observe( + checkpoint, + checkpoint === `reach` && typeof value === `string` + ? publicState(value) + : value, + ), + ) const parts = { ...(resolve ? { resolve } : {}), model: spec.strictModel @@ -925,35 +994,39 @@ export function syntheticProperty(spec: SyntheticSpec): void { : () => undefined, // Without `reached`, the framework counts the driver's `reach` records. ...(spec.reachVia === `reached` - ? { reached: (history: Formed) => reachedStates(spec)(items(history)) } + ? { + reached: (history: Formed) => + reachedStates(spec)(items(history)).map(publicState), + } : {}), ...((spec.probe && spec.predictMode !== `missing`) || spec.predictMode === `extra` ? { predict: (history: Formed) => ({ - 'saw-0': items(history).includes(0), + [publicState(`saw-0`)]: items(history).includes(0), }), } : {}), ...(spec.probeBinding ? { - probeHistories: { - 'saw-0': - spec.probeBinding === `no-absence` - ? ({} as never) - : { - notEntered: { fixed: `fixed-0` }, - ...(spec.probeBinding === `explicit` - ? { entered: { constructed: `constructed-0` } } - : {}), - }, - }, + probeHistories: + spec.probeBinding === `empty` + ? {} + : { + [publicState(`saw-0`)]: + spec.probeBinding === `no-absence` + ? ({} as never) + : { + notEntered: { fixed: `fixed-0` }, + ...(spec.probeBinding === `explicit` + ? { entered: { constructed: `constructed-0` } } + : {}), + }, + }, } : {}), - driver: spec.slow - ? trackedSlow(syntheticDriver(spec)) - : syntheticDriver(spec), - calibrations: syntheticCalibrations(spec), + driver, + calibrations: syntheticCalibrations(spec, consumed), check: ( history: Formed, trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, @@ -1021,9 +1094,12 @@ export function syntheticProperty(spec: SyntheticSpec): void { // With a probe, `saw-0` is a probe-bound state wherever it is declared. const declare = (state: string) => spec.probe && state === `saw-0` - ? { name: state, probe: syntheticProbe } - : state - const arbitrary = arbitraryOf(spec) + ? { name: publicState(state), probe: syntheticProbe } + : publicState(state) + const baseArbitrary = arbitraryOf(spec) + const arbitrary = spec.identityHistories + ? baseArbitrary.map(remember, (value) => value) + : baseArbitrary oracleProperty({ id, law, @@ -1037,7 +1113,7 @@ export function syntheticProperty(spec: SyntheticSpec): void { fixedSeeds: [spec.source.seed], constructed: spec.constructed.map(({ name, prefix, enters }, at) => ({ name, - enters, + enters: enters.map(publicState), // A constructed history extends a generated history, so its histories // depend on the campaign's seed, as real constructed histories do. build: (generated: Formed) => { @@ -1053,6 +1129,7 @@ export function syntheticProperty(spec: SyntheticSpec): void { })), fixed: spec.fixed.map((fixed) => ({ ...fixed, + ...(fixed.enters ? { enters: fixed.enters.map(publicState) } : {}), history: form(fixed.history), })), }, diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 3b7395c..a80f203 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -155,7 +155,12 @@ * oracle. A counterexample of the `errors` form prints its Errors without * the items they hold, so the shrink and replay checks cannot read it back * and skip it. Only generated sources bind a probe, and only - * `saw-0`; an enumerated source declares its states by name. A late probe is judged by whether the run reports it, not by which + * logical `saw-0`, alpha-renamed to saw-0, toString, constructor, or __proto__ + * at the fixture's public state boundary. The model uses logical integer reach + * and translates only printed names; the production probe id is unchanged. + * Resolution may require history identity, and generated boxed histories have + * private slots; each resolve returns a fresh plan. A separate invocation log + * checks exactly one resolution per driver execution, including discarded plans. An enumerated source declares its states by name. A late probe is judged by whether the run reports it, not by which * test Vitest blames. */ import { spawnSync } from 'node:child_process' @@ -234,7 +239,11 @@ type Draft = { calibrations: Array calibrationHistory: NonNullable | null isolatedPlans: boolean - probeBinding: `unbound` | `inferred` | `explicit` | `no-absence` + /** Resolution requires the original history identity and returns a fresh plan. */ + identityHistories: boolean + /** Alpha-renames logical saw-0 at the fixture's public boundary. */ + probeName: `saw-0` | `toString` | `constructor` | `__proto__` + probeBinding: `unbound` | `empty` | `inferred` | `explicit` | `no-absence` duplicateHistoryNames: boolean /** An id no oracle configuration registers. */ unregistered: boolean @@ -309,11 +318,14 @@ const draft: fc.Arbitrary = fc.record({ { freq: 20 }, ), isolatedPlans: fc.boolean(), + identityHistories: fc.boolean(), + probeName: fc.constantFrom(`saw-0`, `toString`, `constructor`, `__proto__`), probeBinding: fc.oneof( { weight: 20, arbitrary: fc.constant(`unbound` as const) }, { weight: 1, arbitrary: fc.constantFrom( + `empty` as const, `inferred` as const, `explicit` as const, `no-absence` as const, @@ -706,6 +718,8 @@ function resolveSuite({ drafts, seedless }: Suite): Plan { ? { calibrationHistory: d.calibrationHistory } : {}), ...(d.isolatedPlans ? { isolatedPlans: true } : {}), + ...(d.identityHistories ? { identityHistories: true } : {}), + probeName: d.probeName, ...(d.probeBinding !== `unbound` ? { probeBinding: d.probeBinding } : {}), @@ -938,7 +952,7 @@ const forcedCalibrationNames = (spec: SyntheticSpec) => spec.probe ? [true, false].map( (entered) => - `calibration: probe ${syntheticProbe} ${entered ? `forced entered` : `forced not entered`} (state "saw-0")`, + `calibration: probe ${syntheticProbe} ${entered ? `forced entered` : `forced not entered`} (state "${spec.probeName ?? `saw-0`}")`, ) : [] // An enumerated history's test name. An unlabelled one is named by the @@ -1257,6 +1271,7 @@ function predictSpec( return { loads: false, loadError: `calibration history selector` } if ( spec.probeBinding && + spec.probeBinding !== `empty` && (!spec.probe || spec.probeBinding === `no-absence` || !spec.fixed.some(({ name }) => name === `fixed-0`) || @@ -1446,13 +1461,25 @@ function predictSpec( // reach agreement. for (const [at, name] of forcedCalibrationNames(spec).entries()) { const forced = at === 0 - if (spec.probeBinding) { + if (spec.probeBinding && spec.probeBinding !== `empty`) { const clean = ran(name)[0] + // The clean opposite-polarity premise also owes every state declared by + // its selected fixed case or construction, not just the probe's state. + const selectedControl = forced + ? spec.fixed.find( + ({ name: selectedName }) => selectedName === `fixed-0`, + ) + : spec.probeBinding === `explicit` + ? spec.constructed.find( + ({ name: selectedName }) => selectedName === `constructed-0`, + ) + : spec.constructed.find(({ enters }) => enters.includes(`saw-0`)) tests.set( name, !clean || judged(spec, clean).kind !== `pass` ? { kind: `calibration-clean` } - : predictsZero(clean) === forced + : !reaches(spec, clean, selectedControl?.enters ?? []) || + predictsZero(clean) === forced ? { kind: `calibration-reach` } : { kind: `pass` }, ) @@ -1490,7 +1517,7 @@ function predictSpec( name, Object.fromEntries( allStates.map((state) => [ - state, + state === `saw-0` ? (spec.probeName ?? state) : state, ran(name).filter((history) => reachedOf(spec, history).includes(state), ).length, @@ -1591,6 +1618,11 @@ const frameworkStates = [ `calibration-applied`, `calibration-rejected`, `calibration-skipped-candidate`, + `bound-fixed-identity`, + `bound-enumerated-identity`, + `bound-combined-controls`, + `unbound-prototype-probe`, + `bound-control-full-reach`, `bound-calibration`, `bound-constructed-driver`, `bound-calibration-registration-error`, @@ -1674,8 +1706,35 @@ function frameworkReach( } continue } + if (spec.identityHistories && spec.calibrationHistory) { + if (`fixed` in spec.calibrationHistory) + reached.add(`bound-fixed-identity`) + if (`enumerated` in spec.calibrationHistory) + reached.add(`bound-enumerated-identity`) + } + if ( + spec.calibrationHistory && + spec.probeBinding && + spec.probeBinding !== `empty` + ) + reached.add(`bound-combined-controls`) + if ( + spec.probe && + spec.probeBinding === `empty` && + spec.probeName !== `saw-0` + ) + reached.add(`unbound-prototype-probe`) + if ( + spec.probeBinding && + spec.probeBinding !== `empty` && + [...owed.tests.values()].some(({ kind }) => kind === `calibration-reach`) + ) + reached.add(`bound-control-full-reach`) const named = spec.fixed.length + spec.constructed.length > 0 - if (spec.calibrationHistory || spec.probeBinding) + if ( + spec.calibrationHistory || + (spec.probeBinding && spec.probeBinding !== `empty`) + ) reached.add(`bound-calibration`) if ( spec.isolatedPlans && @@ -1946,6 +2005,7 @@ type SuiteRun = { } > log: Array + resolutions: Array> warnings: Array /** The run's unhandled errors, by message. */ unhandled: Array @@ -1984,6 +2044,48 @@ const vitest = join( ) let runCount = 0 +function classifyKind( + inner: ReportedError, + text: string, + outer: string, +): Kind | `other` { + if (text.includes(`selected clean run failed`)) return `calibration-clean` + if (text.includes(`selected history did not enter`)) + return `calibration-reach` + if (text.includes(`selected corruption did not apply`)) + return `calibration-inapplicable` + if (text.includes(`selected mutant run failed`)) + return `calibration-mutant-error` + if (inner.name === `OracleDataError`) return `data` + if (text.includes(`the framework's encoding limit`)) return `limit` + if (inner.name === `OracleModelError`) return `model-not-total` + if (inner.name === `OracleViolation`) return `violation` + if (/timed out in \d+ms/.test(text) || /timed out in \d+ms/.test(outer)) + return `timeout` + if (text.includes(`did not enter`) && text.includes(`: fixed history "`)) + return `fixed-reach` + if ( + text.includes(`did not enter`) && + text.includes(`: constructed history "`) + ) + return `constructed-reach` + if (text.includes(`no enumerated history entered`)) return `enumerated-reach` + if (text.includes(`which the property does not declare`)) + return `undeclared-state` + if (text.includes(`two runs of`)) return `determinism` + if (text.includes(`but the check passed it`)) return `calibration-passed` + if (text.includes(`but the check reported law`)) + return `calibration-wrong-law` + if (text.includes(`found no history`)) return `calibration-none` + if ( + text.includes(`synthetic driver failed`) || + text.includes(`cannot read a 7`) || + text.includes(`mutant setup failed`) + ) + return `setup` + return `other` +} + function classify(error: ReportedError | undefined): Failure { if (!error) return { kind: `other` } const outer = error.message ?? `` @@ -2016,50 +2118,7 @@ function classify(error: ReportedError | undefined): Failure { : undefined // A data error names the value's path, then its kind. const data = /^(.*): (.*) is not (?:observation|history) data$/.exec(text) - const kind: Kind | `other` = text.includes(`selected clean run failed`) - ? `calibration-clean` - : text.includes(`selected history did not enter`) - ? `calibration-reach` - : text.includes(`selected corruption did not apply`) - ? `calibration-inapplicable` - : text.includes(`selected mutant run failed`) - ? `calibration-mutant-error` - : inner.name === `OracleDataError` - ? `data` - : text.includes(`the framework's encoding limit`) - ? `limit` - : inner.name === `OracleModelError` - ? `model-not-total` - : inner.name === `OracleViolation` - ? `violation` - : /timed out in \d+ms/.test(text) || - /timed out in \d+ms/.test(outer) - ? `timeout` - : text.includes(`did not enter`) && - text.includes(`: fixed history "`) - ? `fixed-reach` - : text.includes(`did not enter`) && - text.includes(`: constructed history "`) - ? `constructed-reach` - : text.includes(`no enumerated history entered`) - ? `enumerated-reach` - : text.includes(`which the property does not declare`) - ? `undeclared-state` - : text.includes(`two runs of`) - ? `determinism` - : text.includes(`but the check passed it`) - ? `calibration-passed` - : text.includes(`but the check reported law`) - ? `calibration-wrong-law` - : text.includes(`found no history`) - ? `calibration-none` - : text.includes( - `synthetic driver failed`, - ) || - text.includes(`cannot read a 7`) || - text.includes(`mutant setup failed`) - ? `setup` - : `other` + const kind = classifyKind(inner, text, outer) return { kind, // A model error that quotes a machine path, such as an inspected stack. @@ -2185,6 +2244,20 @@ const readLog = (log: string): Array => // promise. Stable grouping retains every call and each local order. .sort((a, b) => a.id.localeCompare(b.id)) : [] +// Resolve callbacks may create resources even when no driver receives the +// result. Their separate input-side log preserves extra and omitted calls. +const readResolutions = (log: string): SuiteRun[`resolutions`] => + existsSync(log) + ? readFileSync(log, `utf8`) + .trim() + .split(`\n`) + .filter(Boolean) + .map((line) => { + const entry = JSON.parse(line) as Pick + return { ...entry, test: unnested(entry.test) } + }) + .sort((a, b) => a.id.localeCompare(b.id)) + : [] // The items of a counterexample that fast-check printed, in any form: the // first bracketed list of integers, where the lossy form's `Number.NaN` is a // 1 and its `-0` a 2. @@ -2233,6 +2306,7 @@ function driveSuite( const run: SuiteRun = { files: {}, log: readLog(log), + resolutions: readResolutions(`${log}.resolutions`), warnings: [...report.warnings].sort(), unhandled: report.unhandled.map((error) => error.message ?? ``), } @@ -2495,7 +2569,8 @@ function checkSuite( const seen = run.files[spec.id]! if (!owed.loads) { if ( - (spec.calibrationHistory || spec.probeBinding) && + (spec.calibrationHistory || + (spec.probeBinding && spec.probeBinding !== `empty`)) && run.log.some((entry) => entry.id === spec.id) ) return violation( @@ -2564,7 +2639,7 @@ function checkSuite( ...(!spec.calibrationHistory ? spec.calibrations.map(({ name }) => name) : []), - ...(!spec.probeBinding + ...(!spec.probeBinding || spec.probeBinding === `empty` ? forcedCalibrationNames(spec).map((name) => name.slice(`calibration: `.length), ) @@ -2584,7 +2659,8 @@ function checkSuite( `calibration`, ) if ( - (spec.calibrationHistory || spec.probeBinding) && + (spec.calibrationHistory || + (spec.probeBinding && spec.probeBinding !== `empty`)) && run.warnings.some( (warning) => warning.startsWith(`${spec.id}:`) && @@ -2619,6 +2695,7 @@ function checkSuite( const failure = test.failure if ( spec.probeBinding && + spec.probeBinding !== `empty` && forcedCalibrationNames(spec).includes(test.name) ) { const calls = run.log.filter( @@ -2630,6 +2707,19 @@ function checkSuite( outcome.kind === `calibration-reach` ? 1 : 2 + if ( + (spec.isolatedPlans || spec.identityHistories) && + run.resolutions.filter( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + ).length !== expectedCalls + ) + return violation( + `run`, + `${spec.id} "${test.name}" did not resolve once per driver execution (owed ${expectedCalls})`, + `calibration`, + ) if ( calls.length !== expectedCalls || calls.some( @@ -2643,10 +2733,10 @@ function checkSuite( `calibration`, ) } - if (spec.calibrationHistory && test.name.startsWith(`calibration:`)) { - const calibration = spec.calibrations.find( - ({ name }) => test.name === `calibration: ${name}`, - )! + const calibration = spec.calibrations.find( + ({ name }) => test.name === `calibration: ${name}`, + ) + if (spec.calibrationHistory && calibration) { const calls = run.log.filter( (entry) => entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, @@ -2657,6 +2747,19 @@ function checkSuite( outcome.kind !== `calibration-reach` ? 2 : 1 + if ( + (spec.isolatedPlans || spec.identityHistories) && + run.resolutions.filter( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + ).length !== expectedCalls + ) + return violation( + `run`, + `${spec.id} "${test.name}" did not resolve once per driver execution (owed ${expectedCalls})`, + `calibration`, + ) if (calls.length !== expectedCalls) return violation( `run`, @@ -2785,7 +2888,12 @@ function checkSuite( } } for (const [campaign, counts] of owed.reach) - if (JSON.stringify(seen.reach[campaign]) !== JSON.stringify(counts)) + // Reach is a dictionary of counts, not an ordered observation. Renaming + // a state may reorder its printed keys without changing any count. + if ( + JSON.stringify(Object.entries(seen.reach[campaign] ?? {}).sort()) !== + JSON.stringify(Object.entries(counts).sort()) + ) return violation( `run`, `${spec.id} "${campaign}" printed reach ${JSON.stringify(seen.reach[campaign] ?? null)}, owed ${JSON.stringify(counts)}`, @@ -2989,6 +3097,8 @@ const generatedDraft = (overrides: Partial): Draft => ({ calibrations: [], calibrationHistory: null, isolatedPlans: false, + identityHistories: false, + probeName: `saw-0`, probeBinding: `unbound`, duplicateHistoryNames: false, unregistered: false, @@ -3040,14 +3150,126 @@ oracleProperty({ histories: { arbitrary: suite, states: [ + `bound-fixed-identity`, + `bound-enumerated-identity`, + `bound-combined-controls`, + `unbound-prototype-probe`, + `bound-control-full-reach`, `bound-calibration`, `bound-calibration-registration-error`, `bound-constructed-driver`, ], - measured: frameworkStates.filter((state) => !state.startsWith(`bound-`)), + measured: frameworkStates.filter( + (state) => + !state.startsWith(`bound-`) && state !== `unbound-prototype-probe`, + ), runs: 3, fixedSeeds: [1], constructed: [ + ...([`fixed`, `enumerated`] as const).map((source) => ({ + name: `selected ${source} preserves identity`, + enters: [ + source === `fixed` + ? `bound-fixed-identity` + : `bound-enumerated-identity`, + ] as const, + build: (generated: Suite) => { + const varied = generated.drafts[0]! + const history = [ + 0, + varied.source.kind === `generated` ? varied.source.seed % 3 : 1, + ] + return { + ...generated, + drafts: [ + generatedDraft({ + ...(source === `fixed` + ? { + fixed: [at(history)], + calibrationHistory: { fixed: `fixed-0` }, + } + : { + source: { kind: `enumerated`, histories: [history] }, + calibrationHistory: { enumerated: 0 }, + }), + form: source === `fixed` && varied.clock ? `boxed` : `plain`, + identityHistories: true, + isolatedPlans: varied.isolatedPlans, + calibrations: [varied.clock ? `driver` : `right`], + }), + ], + } + }, + })), + { + name: `authored bindings coexist with automatic controls`, + enters: [`bound-combined-controls`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + calibrationHistory: { constructed: `constructed-0` }, + calibrations: [generated.drafts[0]!.clock ? `right` : `driver`], + isolatedPlans: generated.drafts[0]!.isolatedPlans, + probe: `agree`, + probeName: generated.drafts[0]!.probeName, + probeBinding: generated.drafts[0]!.clock + ? `inferred` + : `explicit`, + }), + ], + }), + }, + { + name: `empty tables leave arbitrary probe names unbound`, + enters: [`unbound-prototype-probe`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `empty`, + probeName: + generated.drafts[0]!.probeName === `saw-0` + ? `constructor` + : generated.drafts[0]!.probeName, + }), + ], + }), + }, + { + name: `automatic controls require all selected states`, + enters: [`bound-control-full-reach`], + build: (generated) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [ + { + history: [1], + enters: generated.drafts[0]!.clock ? [`saw-2`] : null, + }, + ], + constructed: [ + { + prefix: [0, 0, 0], + enters: generated.drafts[0]!.clock + ? [`saw-0`] + : [`saw-0`, `saw-1`], + }, + ], + probe: `agree`, + probeBinding: generated.drafts[0]!.isolatedPlans + ? `explicit` + : `inferred`, + }), + ], + }), + }, { name: `bound constructed drivers rebuild seeded plans`, enters: [`bound-constructed-driver`], @@ -3116,6 +3338,125 @@ oracleProperty({ }, ], fixed: [ + ...([false, true] as const).map((isolatedPlans) => + fixedHistory( + `review combined controls isolated=${isolatedPlans}`, + [`bound-combined-controls`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + isolatedPlans, + probe: `agree`, + probeBinding: `explicit`, + calibrationHistory: { constructed: `constructed-0` }, + calibrations: [`driver`], + }), + ), + ), + fixedHistory( + `review entered control full reach`, + [`bound-control-full-reach`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0, 0, 0], enters: [`saw-0`, `saw-1`] }], + probe: `agree`, + probeBinding: `inferred`, + }), + ), + fixedHistory( + `review absence control full reach`, + [`bound-control-full-reach`], + generatedDraft({ + fixed: [{ history: [1], enters: [`saw-2`] }], + constructed: [{ prefix: [0, 0, 0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `inferred`, + }), + ), + ...([`saw-0`, `toString`, `constructor`, `__proto__`] as const).map( + (probeName) => + fixedHistory( + `review empty binding table ${probeName}`, + [], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `empty`, + probeName, + }), + ), + ), + ...([`toString`, `constructor`, `__proto__`] as const).map((probeName) => + fixedHistory( + `review own binding table ${probeName}`, + [`bound-calibration`], + generatedDraft({ + fixed: [at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `explicit`, + probeName, + isolatedPlans: true, + }), + ), + ), + fixedHistory( + `review enumerated alpha naming and invalid data precedence`, + [], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [1]] }, + probeName: `toString`, + identityHistories: true, + enumeratedStates: [`required`, `measured`, `measured`], + }), + generatedDraft({ + source: { kind: `enumerated`, histories: [[0]] }, + identityHistories: true, + form: `trap`, + }), + ), + fixedHistory( + `review fixed private slots without resolve`, + [], + generatedDraft({ + fixed: [at([0])], + form: `boxed`, + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`driver`], + }), + ), + fixedHistory( + `review fixed plain identity metadata`, + [`bound-fixed-identity`], + generatedDraft({ + fixed: [at([0])], + identityHistories: true, + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`driver`], + }), + ), + fixedHistory( + `review fixed identity and private slots`, + [`bound-fixed-identity`], + generatedDraft({ + fixed: [at([0])], + form: `boxed`, + identityHistories: true, + calibrationHistory: { fixed: `fixed-0` }, + calibrations: [`driver`], + }), + ), + fixedHistory( + `review enumerated identity metadata`, + [`bound-enumerated-identity`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0]] }, + identityHistories: true, + calibrationHistory: { enumerated: 0 }, + calibrations: [`driver`], + }), + ), fixedHistory( `selected clean driver errors cannot search for another history`, [`bound-calibration`], From 125267154afeb30b02fc337a06de3d9f77d642e4 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 11:01:03 -0700 Subject: [PATCH 03/14] Document calibration binding guarantees --- .changeset/calibration-history-bindings.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/.changeset/calibration-history-bindings.md b/.changeset/calibration-history-bindings.md index bb6fb5b..e05f7b7 100644 --- a/.changeset/calibration-history-bindings.md +++ b/.changeset/calibration-history-bindings.md @@ -2,6 +2,4 @@ '@tanstack/oracles': minor --- -Add direct calibration history bindings for constructed, fixed, and enumerated histories. Bound calibrations validate their selected clean run and actual reach, then require the designated law to reject the mutant, without searching other histories. Constructed calibrations use a stable name-derived seed and fresh builds and resolved plans for each driver execution. - -Add `probeHistories` to bind automatic forced-probe controls, with inference from the unique construction entering a state. Unbound calibrations retain search during migration and warn at registration; consumers must migrate before search is removed ahead of the first npm release. +Add direct `history` bindings for calibrations and `probeHistories` for automatic controls, so bound calibrations validate one selected premise instead of searching. Preserve fixed and enumerated history identity, rebuild constructed histories from stable seeds, and resolve once per driver execution. Retain search for unbound calibrations with registration warnings during migration, before search removal ahead of the first npm release. From 1a1e87f33020f1f5f60af84ff2697b5ff2e3a75e Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 12:23:24 -0700 Subject: [PATCH 04/14] Preserve stdout facts without cross-file scheduling order --- packages/oracles/tests/oracle-property-self-oracle.test.ts | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index de17dcc..d2d4c8d 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -2800,7 +2800,10 @@ function driveSuite( return { ...entry, test: unnested(entry.test) } }) : [], - stdout: report.stdout, + // Independent child files finish in either order. Preserve every line + // and its multiplicity, without treating their interleaving as a law. + // Per-property driver observations retain their original order above. + stdout: report.stdout.split(`\n`).sort().join(`\n`), warnings: [...report.warnings].sort(), unhandled: report.unhandled.map((error) => error.message ?? ``), } From 2dc95d4e838678a03eb345c08f141dbd20a8c4a7 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 12:30:13 -0700 Subject: [PATCH 05/14] Distinguish calibration registration validation phases --- .../tests/oracle-property-self-oracle.test.ts | 60 +++++++++++++++---- 1 file changed, 48 insertions(+), 12 deletions(-) diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index d2d4c8d..4628d70 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -1535,6 +1535,21 @@ function predictSpec( } } } + // Calibration declaration checks precede requirement checks; resolving a + // selected history comes last. This diagnostic order is analyst-chosen, + // like the earlier shape checks, rather than a new product guarantee. + if ( + spec.probeBinding && + spec.probeBinding !== `empty` && + (!spec.probe || + [`no-absence`, `null-binding`, `false-binding`, `zero-binding`].includes( + spec.probeBinding, + ) || + (spec.probeBinding === `inferred` && + spec.constructed.filter(({ enters }) => enters.includes(`saw-0`)) + .length !== 1)) + ) + return { loads: false, loadError: `calibration history selector` } const empty = spec.calibrations.find(({ difference }) => emptyDifference(difference), ) @@ -1592,19 +1607,10 @@ function predictSpec( if ( spec.probeBinding && spec.probeBinding !== `empty` && - (!spec.probe || - [ - `no-absence`, - `null-entered`, - `null-binding`, - `false-binding`, - `zero-binding`, - ].includes(spec.probeBinding) || + (spec.probeBinding === `null-entered` || !spec.fixed.some(({ name }) => name === `fixed-0`) || - (spec.probeBinding === `inferred` - ? spec.constructed.filter(({ enters }) => enters.includes(`saw-0`)) - .length !== 1 - : !spec.constructed.some(({ name }) => name === `constructed-0`))) + (spec.probeBinding === `explicit` && + !spec.constructed.some(({ name }) => name === `constructed-0`))) ) return { loads: false, loadError: `calibration history selector` } const names = @@ -4394,6 +4400,36 @@ oracleProperty({ }), ), ), + ...([`empty text`, `empty pattern`] as const).flatMap((difference) => + ( + [ + `null-binding`, + `false-binding`, + `zero-binding`, + `no-absence`, + `inferred`, + `null-entered`, + `explicit`, + ] as const + ).map((probeBinding) => + fixedHistory( + `registration phases ${probeBinding} with ${difference}`, + [`registration-error`], + generatedDraft({ + fixed: [at([1])], + // Inferred entry lacks a unique construction; explicit entry + // selects a missing one. These fail in different phases. + constructed: + probeBinding === `inferred` || probeBinding === `explicit` + ? [] + : [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding, + calibrations: [cal(`right`, difference)], + }), + ), + ), + ), fixedHistory( `round2 warning repeat and manifest phases`, [`registration-warning-policy`], From a301e1f898a54e7750bb19c24e737d71785896b3 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:32:49 -0700 Subject: [PATCH 06/14] test: close calibration selection and child observation gaps --- .../oracle-property-self-oracle-fixture.ts | 25 +- .../tests/oracle-property-self-oracle.test.ts | 760 ++++++++++++++++-- reviews/calibration-history-self-oracle.md | 147 ++++ 3 files changed, 883 insertions(+), 49 deletions(-) create mode 100644 reviews/calibration-history-self-oracle.md diff --git a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts index 647c88a..a7024bc 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts +++ b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts @@ -280,6 +280,9 @@ export type SyntheticSpec = { | `null-binding` | `false-binding` | `zero-binding` + /** Independent public selectors for the two clean opposite premises. */ + probeEntered?: { fixed: string } | { constructed: string } + probeNotEntered?: { fixed: string } | { constructed: string } duplicateHistoryNames?: boolean /** `observe`: the driver reports reach itself, the framework's default. */ reachVia: `reached` | `observe` @@ -1106,9 +1109,19 @@ export function syntheticProperty(spec: SyntheticSpec): void { )) || (spec.probeBinding && spec.probeBinding !== `empty` && - expect + ((expect .getState() - .currentTestName?.includes(`forced not entered`))) + .currentTestName?.includes(`forced not entered`) && + (spec.probeBinding !== `explicit` || + `constructed` in + (spec.probeEntered ?? { + constructed: `constructed-0`, + }))) || + (expect + .getState() + .currentTestName?.includes(`forced entered`) && + `constructed` in + (spec.probeNotEntered ?? { fixed: `fixed-0` }))))) ) { if ( !built.has(history as object) || @@ -1202,10 +1215,14 @@ export function syntheticProperty(spec: SyntheticSpec): void { : spec.probeBinding === `no-absence` ? ({} as never) : { - notEntered: { fixed: `fixed-0` }, + notEntered: spec.probeNotEntered ?? { + fixed: `fixed-0`, + }, ...(spec.probeBinding === `explicit` ? { - entered: { constructed: `constructed-0` }, + entered: spec.probeEntered ?? { + constructed: `constructed-0`, + }, } : spec.probeBinding === `null-entered` ? { entered: null as never } diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 4628d70..1bb88ed 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -85,8 +85,13 @@ * finite ordinary/astral name families declared here; no universal collision * freedom is claimed. Forced premises are judged from declared input origins * and author-built callback inputs, never the framework's selected driver log. - * Callback errors retain calibration/history context and cause. Registration - * warns once per property/process, and collect-only registration stays silent. + * Callback errors retain calibration/history context and cause. Both forced + * premises independently select fixed or constructed originals; only omitted + * entered selectors infer a unique construction. Equal data never permits + * substituting another named original. Registration warns once per + * property/process without dropping tests, and collect-only stays silent. + * Every real child must finish with a complete report and no unmodeled + * unhandled errors, including replay and registration-only children. * - `determinism`: the determinism test runs every fixed and constructed history * and three generated ones, and fails for a driver whose trace changes, * compared by their data encoding: `-0` is not `0`, and key order and @@ -315,6 +320,8 @@ type Draft = { | `null-binding` | `false-binding` | `zero-binding` + probeEntered: NonNullable + probeNotEntered: NonNullable duplicateHistoryNames: boolean /** An id no oracle configuration registers. */ unregistered: boolean @@ -369,7 +376,15 @@ const insideOrOutside = (insideWeight: number) => { arbitrary: syntheticArbitrary, weight: insideWeight }, { arbitrary: outside, weight: 1 }, ) +const probeSelector = fc.oneof( + fc.record({ fixed: fc.constantFrom(`fixed-0`, `fixed-1`, `missing`) }), + fc.record({ + constructed: fc.constantFrom(`constructed-0`, `constructed-1`, `missing`), + }), +) const draft: fc.Arbitrary = fc.record({ + probeEntered: probeSelector, + probeNotEntered: probeSelector, calibrationHistory: fc.option( fc.oneof( fc.record({ @@ -382,7 +397,7 @@ const draft: fc.Arbitrary = fc.record({ enumerated: fc.oneof( fc.integer({ min: -1, max: 4 }), fc.constant(1.5), - fc.constant(`items 0 | 1`), + fc.constantFrom(`items 0 | 1`, `items 0 | 1 ${shellText}`, `[0,1]`), ), }), ), @@ -855,6 +870,8 @@ function resolveSuite({ drafts, seedless }: Suite): Plan { ...(d.probeBinding !== `unbound` ? { probeBinding: d.probeBinding } : {}), + probeEntered: d.probeEntered, + probeNotEntered: d.probeNotEntered, ...(d.duplicateHistoryNames ? { duplicateHistoryNames: true } : {}), calibrations: (d.calibrationNames === `indexed` ? d.calibrations @@ -1240,6 +1257,32 @@ type BuildEntry = { generated: SyntheticHistory history: SyntheticHistory } +// Named enumeration selectors refer to the label the author supplied, not +// merely the underlying items. The selector string domain below includes +// authored items labels (plain/shell) and one plain-JSON default label. +// Other forms' encoded default labels are outside that string domain. +const authoredEnumerationLabel = ( + spec: SyntheticSpec, + history: SyntheticHistory, +): string | undefined => { + if (spec.labelStyle === `default`) + return spec.form === undefined ? JSON.stringify(history) : undefined + if ( + spec.duplicateHistoryNames && + spec.calibrationHistory && + `enumerated` in spec.calibrationHistory && + typeof spec.calibrationHistory.enumerated === `string` && + spec.source.kind === `enumerated` + ) + return `items ${spec.source.histories[0]!.join(` | `)}` + if ( + spec.labelStyle?.startsWith(`reserved-`) && + spec.source.kind === `enumerated` && + JSON.stringify(history) === JSON.stringify(spec.source.histories[0]) + ) + return spec.labelStyle.slice(`reserved-`.length) + return `items ${history.join(` | `)}${spec.labelStyle === `shell` ? ` ${shellText}` : ``}` +} const authoredInput = ( spec: SyntheticSpec, test: string, @@ -1252,7 +1295,7 @@ const authoredInput = ( return spec.source.histories.find((history, index) => typeof binding.enumerated === `number` ? index === binding.enumerated - : `items ${history.join(` | `)}` === binding.enumerated, + : authoredEnumerationLabel(spec, history) === binding.enumerated, ) if (`constructed` in binding) { const construction = spec.constructed.find( @@ -1274,16 +1317,39 @@ const authoredInput = ( } return undefined } +// Resolve from authored declarations, never from the driver-selected input. +// A unique state declaration is insufficient if another construction shares +// its name: every selected name must identify exactly one original. const selectedControlBinding = (spec: SyntheticSpec, forced: boolean) => forced - ? { fixed: `fixed-0` } - : { - constructed: - spec.probeBinding === `explicit` - ? `constructed-0` - : (spec.constructed.find(({ enters }) => enters.includes(`saw-0`)) - ?.name ?? `missing`), - } + ? (spec.probeNotEntered ?? { fixed: `fixed-0` }) + : spec.probeBinding === `explicit` + ? (spec.probeEntered ?? { constructed: `constructed-0` }) + : { + constructed: + spec.constructed.find(({ enters }) => enters.includes(`saw-0`)) + ?.name ?? `missing`, + } +const selectedControls = (spec: SyntheticSpec, forced: boolean) => { + const binding = selectedControlBinding(spec, forced) + return `fixed` in binding + ? spec.fixed.filter(({ name }) => name === binding.fixed) + : spec.constructed.filter(({ name }) => name === binding.constructed) +} +const authoredOrigin = ( + spec: SyntheticSpec, + binding: NonNullable, +): string | undefined => { + if (`fixed` in binding) return `fixed:${binding.fixed}` + if (`constructed` in binding) return `constructed:${binding.constructed}` + if (spec.source.kind !== `enumerated`) return undefined + const index = spec.source.histories.findIndex((history, at) => + typeof binding.enumerated === `number` + ? at === binding.enumerated + : authoredEnumerationLabel(spec, history) === binding.enumerated, + ) + return index < 0 ? undefined : `enumerated:${index}` +} const isCampaign = (name: string) => name.startsWith(`fixed seed`) || name === `random seed` @@ -1573,7 +1639,8 @@ function predictSpec( ? spec.source.histories.find((history, index) => typeof binding.enumerated === `number` ? index === binding.enumerated - : `items ${history.join(` | `)}` === binding.enumerated, + : authoredEnumerationLabel(spec, history) === + binding.enumerated, ) : undefined) const duplicates = @@ -1585,13 +1652,9 @@ function predictSpec( .length > 1 : typeof binding.enumerated === `string` && spec.source.kind === `enumerated` - ? (spec.duplicateHistoryNames && - binding.enumerated === - `items ${spec.source.histories[0]!.join(` | `)}` && - spec.source.histories.length > 1) || - spec.source.histories.filter( + ? spec.source.histories.filter( (history) => - `items ${history.join(` | `)}` === binding.enumerated, + authoredEnumerationLabel(spec, history) === binding.enumerated, ).length > 1 : false) if ( @@ -1608,9 +1671,8 @@ function predictSpec( spec.probeBinding && spec.probeBinding !== `empty` && (spec.probeBinding === `null-entered` || - !spec.fixed.some(({ name }) => name === `fixed-0`) || - (spec.probeBinding === `explicit` && - !spec.constructed.some(({ name }) => name === `constructed-0`))) + selectedControls(spec, true).length !== 1 || + selectedControls(spec, false).length !== 1) ) return { loads: false, loadError: `calibration history selector` } const names = @@ -1836,15 +1898,7 @@ function predictSpec( ) // The clean opposite-polarity premise also owes every state declared by // its selected fixed case or construction, not just the probe's state. - const selectedControl = forced - ? spec.fixed.find( - ({ name: selectedName }) => selectedName === `fixed-0`, - ) - : spec.probeBinding === `explicit` - ? spec.constructed.find( - ({ name: selectedName }) => selectedName === `constructed-0`, - ) - : spec.constructed.find(({ enters }) => enters.includes(`saw-0`)) + const selectedControl = selectedControls(spec, forced)[0] tests.set( name, !clean || judged(spec, clean).kind !== `pass` @@ -1942,7 +1996,10 @@ function suiteModel( // unhandled error per run the log records. A slow spec's driver runs on // after its test timed out, so its calls may land after the run ended; they // count at most once each. -const lateCalls = (plan: Plan, logged: ReadonlyArray) => { +const lateCalls = ( + plan: Plan, + logged: ReadonlyArray>, +) => { let min = 0 let max = 0 for (const spec of plan.specs) { @@ -1976,6 +2033,14 @@ function owedRoute(name: string, replay = false): string { // about. Each fixed history names the state it witnesses, so every one is reached on // every run; campaigns count them. const frameworkStates = [ + `auxiliary-late-error`, + `bound-enumerated-label-selector`, + `bound-control-fixed-entered`, + `bound-control-constructed-absent`, + `bound-control-inferred-ambiguous`, + `bound-control-explicit-ambiguous`, + `bound-equal-fixed-origin`, + `bound-ranked-mutants`, `bound-independent-premise`, `bound-name-seed`, @@ -2073,11 +2138,51 @@ function frameworkReach( expected: ReturnType, ): Iterable { const reached = new Set() + if ( + trace + .filter(({ checkpoint }) => checkpoint === `child`) + .slice(1) + .some(({ value }) => { + const child = value as { + log: Array> + unhandled: Array + } + return ( + child.log.some( + ({ id, history }) => + history.includes(0) && + plan.specs.some( + (spec) => spec.id === id && spec.probe === `late`, + ), + ) && + child.unhandled.some((message) => + message.includes(`was called after its history ended`), + ) + ) + }) + ) + reached.add(`auxiliary-late-error`) const run = trace.find(({ checkpoint }) => checkpoint === `run`)?.value as SuiteRun | undefined for (const spec of plan.specs) { const owed = expected.get(spec.id)! + if ( + run?.files[spec.id] && + spec.source.kind === `enumerated` && + spec.calibrationHistory && + `enumerated` in spec.calibrationHistory && + typeof spec.calibrationHistory.enumerated === `string` && + [`shell`, `default`].includes(spec.labelStyle ?? ``) + ) + reached.add(`bound-enumerated-label-selector`) if (!owed.loads) { + if ( + run?.files[spec.id]?.loaded === false && + (spec.probeBinding === `explicit` || + spec.probeBinding === `inferred`) && + selectedControls(spec, false).length > 1 + ) + reached.add(`bound-control-${spec.probeBinding}-ambiguous`) if ( [ `null-entered`, @@ -2108,6 +2213,31 @@ function frameworkReach( } continue } + if ( + run?.files[spec.id]?.loaded && + spec.probeBinding === `explicit` && + `fixed` in selectedControlBinding(spec, false) + ) + reached.add(`bound-control-fixed-entered`) + if ( + run?.files[spec.id]?.loaded && + spec.probeBinding && + spec.probeBinding !== `empty` && + `constructed` in selectedControlBinding(spec, true) + ) + reached.add(`bound-control-constructed-absent`) + if ( + run?.files[spec.id]?.loaded && + spec.calibrationHistory && + `fixed` in spec.calibrationHistory && + spec.identityHistories && + spec.fixed.filter( + ({ history }) => + JSON.stringify(history) === + JSON.stringify(authoredInput(spec, ``, spec.calibrationHistory!, [])), + ).length > 1 + ) + reached.add(`bound-equal-fixed-origin`) if ( spec.calibrationHistory && spec.calibrations.some( @@ -2442,6 +2572,8 @@ type Report = { stdout: string warnings: Array unhandled: Array + exit: number | null + signal: string | null } type Failure = { message?: string @@ -2710,6 +2842,8 @@ function runVitest( stdout: run.stdout, warnings: warnings ?? [], unhandled: unhandled ?? [], + exit: run.status, + signal: run.signal, } } @@ -2782,6 +2916,46 @@ function driveSuite( ) mkdirSync(directory, { recursive: true }) try { + const receive: typeof runVitest = (...args) => { + const childLog = + args[2].SELF_ORACLE_LOG ?? join(directory, `child-${runCount}.jsonl`) + const report = runVitest( + args[0], + args[1], + { ...args[2], SELF_ORACLE_LOG: childLog }, + args[3], + args[4], + ) + observe(`child`, { + log: readLog(childLog).map(({ id, history }) => ({ id, history })), + loads: report.files.map(({ file, errors }) => ({ + id: plan.specs[files.indexOf(file)]?.id ?? `unknown`, + errors: errors.map(({ message }) => message ?? ``), + })), + exit: report.exit, + signal: report.signal, + files: report.files.map(({ file }) => + file.replace(`${directory}/`, ``), + ), + expectedFiles: args[1].map((file) => file.replace(`${directory}/`, ``)), + // Synthetic tests owe one primary failure; causes retain its nested + // diagnostic chain. Separate teardown/hook failures must stay visible. + secondaryErrors: report.files.flatMap(({ tests }) => + tests.flatMap(({ name, errors }) => + errors + .slice(1) + .map(({ message }) => ({ test: name, message: message ?? `` })), + ), + ), + unhandled: report.unhandled.map(({ message }) => message ?? ``), + failed: report.files.some( + (file) => + file.errors.length > 0 || + file.tests.some(({ state }) => state === `failed`), + ), + }) + return report + } const log = join(directory, `log.jsonl`) const files = plan.specs.map((spec, index) => { const file = join(directory, `spec-${index}-oracle.test.ts`) @@ -2791,7 +2965,7 @@ function driveSuite( ) return file }) - const report = runVitest(directory, files, { SELF_ORACLE_LOG: log }) + const report = receive(directory, files, { SELF_ORACLE_LOG: log }) const run: SuiteRun = { files: {}, log: readLog(log), @@ -2884,7 +3058,7 @@ function driveSuite( file, `import { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\nsyntheticProperty(${JSON.stringify(reordered)})\n`, ) - runVitest( + receive( directory, [file], { SELF_ORACLE_LOG: reorderedLog }, @@ -2913,7 +3087,7 @@ function driveSuite( file, `import { describe } from 'vitest'\nimport { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\ndescribe('first',()=>{${call}})\ndescribe('second',()=>{${call}})\n`, ) - const repeated = runVitest( + const repeated = receive( directory, [file], {}, @@ -2924,7 +3098,7 @@ function driveSuite( file, `import { describe } from 'vitest'\nimport { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\ndescribe('collect',()=>{process.env.ORACLE_MANIFEST=${JSON.stringify(manifest)};try{${call}}finally{delete process.env.ORACLE_MANIFEST}})\ndescribe('ordinary',()=>{${call}})\n`, ) - const collectThenRun = runVitest( + const collectThenRun = receive( directory, [file], {}, @@ -2938,7 +3112,7 @@ function driveSuite( file, `import { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\n${call}\n`, ) - const collected = runVitest( + const collected = receive( directory, [file], { ORACLE_MANIFEST: collectedManifest }, @@ -2961,6 +3135,17 @@ function driveSuite( collectThenRun: registered(manifest), collected: registered(collectedManifest), }, + tests: { + repeated: repeated.files + .flatMap(({ tests }) => tests.map(({ name }) => name)) + .sort(), + collectThenRun: collectThenRun.files + .flatMap(({ tests }) => tests.map(({ name }) => name)) + .sort(), + collected: collected.files + .flatMap(({ tests }) => tests.map(({ name }) => name)) + .sort(), + }, loaded: { repeated: loaded(repeated), collectThenRun: loaded(collectThenRun), @@ -2977,7 +3162,7 @@ function driveSuite( } if (plan.seedless) { const seedlessLog = join(directory, `seedless-${runCount}.jsonl`) - runVitest( + receive( directory, files, { SELF_ORACLE_LOG: seedlessLog }, @@ -3008,7 +3193,7 @@ function driveSuite( if (!isCampaign(campaign.name) || campaign.failure?.path === undefined) continue const replayLog = join(directory, `replay-${runCount}.jsonl`) - const replay = runVitest(directory, files, { + const replay = receive(directory, files, { ORACLE_SEED: String(campaign.failure.seed), ORACLE_PATH: campaign.failure.path, ORACLE_PROPERTY: spec.id, @@ -3038,7 +3223,7 @@ function driveSuite( ) if (!target) continue const index = plan.specs.indexOf(spec) - const rerun = runVitest( + const rerun = receive( directory, [files[index]!], {}, @@ -3063,7 +3248,7 @@ function driveSuite( const escape = (text: string) => text.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`) const pattern = `^${[...(spec.nested ? [outerSuite] : []), spec.id, `reach`].map(escape).join(`(?: | > )`)}$` - const alone = runVitest( + const alone = receive( directory, [files[index]!], {}, @@ -3136,6 +3321,101 @@ function checkSuite( const run = trace.find(({ checkpoint }) => checkpoint === `run`)?.value as SuiteRun | undefined if (!run) return violation(`run`, `the driver recorded no run`, `trace`) + const children = trace.filter(({ checkpoint }) => checkpoint === `child`) + const owedChildren = + 1 + + trace.reduce( + (count, { checkpoint }) => + count + + (checkpoint === `registration-warnings` + ? 3 + : [ + `binding-reordered`, + `seedless`, + `replay`, + `rerun`, + `reach-alone`, + ].includes(checkpoint) + ? 1 + : 0), + 0, + ) + if (children.length !== owedChildren) + return violation( + `child`, + `recorded ${children.length} child reports, owed ${owedChildren}`, + `trace`, + ) + // Expected synthetic failures exit 1; unexpected process failures do not + // disappear merely because a reporter managed to write a partial result. + const lateMessage = `probe synthetic#saw-0 was called after its history ended. The driver must await the work it starts, so every probe call belongs to one history.` + for (const { value } of children) { + const child = value as { + exit: number | null + signal: string | null + files: Array + expectedFiles: Array + unhandled: Array + failed: boolean + secondaryErrors: Array<{ test: string; message: string }> + log: Array> + loads: Array<{ id: string; errors: Array }> + } + if (child.secondaryErrors.length) + return violation( + `child`, + `unexpected secondary test errors: ${JSON.stringify(child.secondaryErrors)}`, + `trace`, + ) + for (const file of child.loads) { + const owed = expected.get(file.id as SyntheticSpec[`id`]) + if ( + !owed || + (owed.loads + ? file.errors.length !== 0 + : file.errors.length !== 1 || + (owed.loadError !== undefined && + !file.errors[0]!.includes(owed.loadError))) + ) + return violation( + `child`, + `unexpected child load/suite errors: ${JSON.stringify(file)}`, + `registration`, + ) + } + const owedLate = lateCalls(plan, child.log) + const actualLate = child.unhandled.filter( + (message) => message === lateMessage, + ).length + if (actualLate < owedLate.min || actualLate > owedLate.max) + return violation( + `child`, + `child reported ${actualLate} late probe calls, owed ${owedLate.min} to ${owedLate.max}`, + `agreement`, + ) + const unexpected = child.unhandled.filter( + (message) => + message !== lateMessage || + !plan.specs.some(({ probe }) => probe === `late`), + ) + if (unexpected.length) + return violation( + `child`, + `unexpected unhandled errors: ${JSON.stringify(unexpected)}`, + `trace`, + ) + if ( + child.signal !== null || + child.exit !== (child.failed || child.unhandled.length ? 1 : 0) || + JSON.stringify([...child.files].sort()) !== + JSON.stringify([...child.expectedFiles].sort()) + ) + return violation( + `child`, + `incomplete or abnormal child process: ${JSON.stringify(child)}`, + `trace`, + ) + } const probes = trace .filter( ({ checkpoint }) => checkpoint === `replay` || checkpoint === `rerun`, @@ -3165,6 +3445,13 @@ function checkSuite( `${spec.id} loaded, but its id is unregistered, a required state has no constructed history, a constructed history names an undeclared state, a fixed or constructed history is one the arbitrary cannot take back, \`owns\` is not a non-empty literal list of strings, or a history is not data`, law, ) + if (seen.loadErrors.length !== 1 || seen.loadCauses.length !== 1) + return violation( + `run`, + `${spec.id} must report its first load error alone`, + `registration`, + { actual: seen.loadErrors, expected: `one load error` }, + ) const owedText = [owed.loadError, owed.loadFix].filter( (text) => text !== undefined, ) @@ -3243,6 +3530,11 @@ function checkSuite( (value as { id: string }).id === spec.id, )?.value as | { + tests: { + repeated: Array + collectThenRun: Array + collected: Array + } repeated: Array collectThenRun: Array collected: Array @@ -3258,6 +3550,32 @@ function checkSuite( } | undefined if (warningProbe) { + // Each ordinary registration owes the full model-derived test set. + // Preserve suite scope and multiplicity: warning deduplication grants no + // permission to drop the second property's tests or either forced control. + const names = [...owed.tests.keys()] + const phaseNames = (phases: Array) => + phases + .flatMap((phase) => + names.map((name) => `${phase} > ${spec.id} > ${name}`), + ) + .sort() + for (const [phase, expectedNames] of [ + [`repeated`, phaseNames([`first`, `second`])], + [`collectThenRun`, phaseNames([`collect`, `ordinary`])], + [`collected`, names.map((name) => `${spec.id} > ${name}`).sort()], + ] as const) { + if ( + JSON.stringify(warningProbe.tests[phase]) !== + JSON.stringify(expectedNames) + ) + return violation( + `run`, + `${spec.id} ${phase} omitted or duplicated registrations`, + `registration`, + { actual: warningProbe.tests[phase], expected: expectedNames }, + ) + } if ( [ warningProbe.manifests.collectThenRun, @@ -3355,7 +3673,7 @@ function checkSuite( expectedBinding, run.builds, ) - const expectedOrigin = `${`fixed` in expectedBinding ? `fixed` : `constructed`}:${`fixed` in expectedBinding ? expectedBinding.fixed : expectedBinding.constructed}` + const expectedOrigin = authoredOrigin(spec, expectedBinding) if ( calls.some( (entry) => @@ -3474,13 +3792,14 @@ function checkSuite( if ( fullInput && calls.some( - ({ history }) => + ({ history, origin }) => + origin !== authoredOrigin(spec, binding) || JSON.stringify(history) !== JSON.stringify(fullInput), ) ) return violation( `run`, - `${spec.id} selected calibration did not run its full authored input`, + `${spec.id} selected calibration did not run its full authored input and origin`, `calibration`, ) const selected = @@ -3490,7 +3809,8 @@ function checkSuite( ? spec.source.histories.find((history, index) => typeof binding.enumerated === `number` ? index === binding.enumerated - : `items ${history.join(` | `)}` === binding.enumerated, + : authoredEnumerationLabel(spec, history) === + binding.enumerated, ) : undefined const prefix = @@ -3891,6 +4211,8 @@ const generatedDraft = (overrides: Partial): Draft => ({ identityHistories: false, probeName: `saw-0`, probeBinding: `unbound`, + probeEntered: { constructed: `constructed-0` }, + probeNotEntered: { fixed: `fixed-0` }, duplicateHistoryNames: false, unregistered: false, numberDifference: false, @@ -3949,6 +4271,13 @@ oracleProperty({ histories: { arbitrary: suite, states: [ + `auxiliary-late-error`, + `bound-enumerated-label-selector`, + `bound-control-fixed-entered`, + `bound-control-constructed-absent`, + `bound-control-inferred-ambiguous`, + `bound-control-explicit-ambiguous`, + `bound-equal-fixed-origin`, `bound-ranked-mutants`, `bound-independent-premise`, `bound-name-seed`, @@ -3968,6 +4297,7 @@ oracleProperty({ ], measured: frameworkStates.filter( (state) => + state !== `auxiliary-late-error` && !state.startsWith(`bound-`) && state !== `unbound-prototype-probe` && state !== `registration-warning-policy`, @@ -3975,6 +4305,116 @@ oracleProperty({ runs: 3, fixedSeeds: [1], constructed: [ + { + name: `review auxiliary late errors`, + enters: [`auxiliary-late-error`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + probe: `late`, + clock: true, + strictModel: true, + constructed: [{ prefix: [7, 0], enters: [] }], + }), + ], + }), + }, + { + name: `review enumerated selector labels`, + enters: [`bound-enumerated-label-selector`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + source: { kind: `enumerated`, histories: [[0, 1]] }, + labelStyle: generated.drafts[0]!.clock ? `shell` : `default`, + calibrationHistory: { enumerated: `items 0 | 1` }, + calibrations: [`right`], + }), + ], + }), + }, + { + name: `review fixed entered control`, + enters: [`bound-control-fixed-entered`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1]), at([0, generated.drafts[0]!.clock ? 1 : 2])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `explicit`, + probeEntered: { fixed: `fixed-1` }, + identityHistories: true, + }), + ], + }), + }, + { + name: `review constructed absent control`, + enters: [`bound-control-constructed-absent`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { + prefix: [1, generated.drafts[0]!.clock ? 1 : 2, 2], + enters: [], + }, + ], + probe: `agree`, + probeBinding: generated.drafts[0]!.clock + ? `explicit` + : `inferred`, + probeNotEntered: { constructed: `constructed-1` }, + isolatedPlans: true, + }), + ], + }), + }, + ...([`inferred`, `explicit`] as const).map((probeBinding) => ({ + name: `review ${probeBinding} ambiguous control`, + enters: [`bound-control-${probeBinding}-ambiguous` as const], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [generated.drafts[0]!.clock ? 1 : 2], enters: [] }, + ], + probe: `agree`, + probeBinding, + duplicateHistoryNames: true, + calibrationHistory: { fixed: `fixed-0` }, + }), + ], + }), + })), + { + name: `review equal fixed originals`, + enters: [`bound-equal-fixed-origin`], + build: (generated: Suite) => { + const history = [0, generated.drafts[0]!.clock ? 1 : 2] + return { + ...generated, + drafts: [ + generatedDraft({ + fixed: [at([...history]), at([...history])], + calibrationHistory: { fixed: `fixed-1` }, + calibrations: [`driver`], + identityHistories: true, + }), + ], + } + }, + }, { name: `round2 ranked mutant verdicts`, enters: [`bound-ranked-mutants`], @@ -4283,6 +4723,91 @@ oracleProperty({ }, ], fixed: [ + fixedHistory( + `review auxiliary late errors preserve counts`, + [`auxiliary-late-error`], + generatedDraft({ + probe: `late`, + clock: true, + strictModel: true, + constructed: [{ prefix: [7, 0], enters: [] }], + }), + ), + ...([`shell`, `default`] as const).flatMap((labelStyle) => + [true, false].map((valid) => + fixedHistory( + `review ${labelStyle} enumerated selector ${valid ? `exact` : `missing`}`, + [`bound-enumerated-label-selector`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0, 1]] }, + labelStyle, + calibrationHistory: { + enumerated: valid + ? labelStyle === `shell` + ? `items 0 | 1 ${shellText}` + : `[0,1]` + : `items 0 | 1`, + }, + calibrations: [`right`], + identityHistories: true, + }), + ), + ), + ), + ...([`inferred`, `explicit`] as const).map((probeBinding) => + fixedHistory( + `review duplicate selected ${probeBinding} names`, + [`bound-control-${probeBinding}-ambiguous`], + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [1], enters: [] }, + ], + probe: `agree`, + probeBinding, + duplicateHistoryNames: true, + calibrationHistory: { fixed: `fixed-0` }, + }), + ), + ), + fixedHistory( + `review fixed entered selector`, + [`bound-control-fixed-entered`], + generatedDraft({ + fixed: [at([1]), at([0, 1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: `agree`, + probeBinding: `explicit`, + probeEntered: { fixed: `fixed-1` }, + identityHistories: true, + }), + ), + fixedHistory( + `review constructed absent selector`, + [`bound-control-constructed-absent`], + generatedDraft({ + fixed: [at([1])], + constructed: [ + { prefix: [0], enters: [`saw-0`] }, + { prefix: [1, 2, 2], enters: [] }, + ], + probe: `agree`, + probeBinding: `explicit`, + probeNotEntered: { constructed: `constructed-1` }, + isolatedPlans: true, + }), + ), + fixedHistory( + `review equal fixed input origins`, + [`bound-equal-fixed-origin`], + generatedDraft({ + fixed: [at([0]), at([0])], + calibrationHistory: { fixed: `fixed-1` }, + calibrations: [`driver`], + identityHistories: true, + }), + ), ...([`throws`, `clock`] as const).map((kind) => fixedHistory( `round2 ranked ${kind} bound and legacy`, @@ -5565,6 +6090,151 @@ oracleProperty({ reached: frameworkReach, driver: driveSuite, check: checkSuite, + calibrations: [ + { + name: `reject missing child observation`, + history: { constructed: `round2 ranked mutant verdicts` }, + law: `trace`, + corrupt: (trace) => { + const first = trace.findIndex( + ({ checkpoint }) => checkpoint === `child`, + ) + return trace.filter((_, index) => index !== first) + }, + }, + { + name: `reject abnormal child exit`, + history: { constructed: `round2 ranked mutant verdicts` }, + law: `trace`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.find(({ checkpoint }) => checkpoint === `child`)! + .value as { exit: number } + child.exit = 2 + return altered + }, + }, + { + name: `reject secondary auxiliary test errors`, + history: { constructed: `review auxiliary late errors` }, + law: `trace`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.filter( + ({ checkpoint }) => checkpoint === `child`, + )[1]!.value as { + secondaryErrors: Array<{ test: string; message: string }> + } + child.secondaryErrors.push({ + test: `random seed`, + message: `unexpected replay teardown failure`, + }) + return altered + }, + }, + { + name: `reject auxiliary teardown errors`, + history: { constructed: `round2 registration warning phases` }, + law: `registration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.filter( + ({ checkpoint }) => checkpoint === `child`, + )[1]!.value as { + loads: Array<{ errors: Array }> + failed: boolean + exit: number + } + child.loads[0]!.errors.push(`unexpected replay teardown failure`) + child.failed = true + child.exit = 1 + return altered + }, + }, + { + name: `reject auxiliary late error duplicates`, + history: { constructed: `review auxiliary late errors` }, + law: `agreement`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .slice(1) + .map(({ value }) => value as { unhandled: Array }) + .find(({ unhandled }) => unhandled.length > 0)! + child.unhandled.push(child.unhandled[0]!) + return altered + }, + }, + { + name: `reject extra registration diagnostics`, + history: { constructed: `review inferred ambiguous control` }, + law: `registration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const run = altered.find(({ checkpoint }) => checkpoint === `run`)! + .value as SuiteRun + const file = Object.values(run.files)[0]! + file.loadErrors.push(file.loadErrors[0]!) + file.loadCauses.push(file.loadCauses[0]!) + return altered + }, + }, + ...[`authored`, `forced`].map((kind) => ({ + name: `reject lost ${kind} repeated registration`, + history: { constructed: `round2 registration warning phases` }, + law: `registration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const phase = altered.find( + ({ checkpoint }) => checkpoint === `registration-warnings`, + )!.value as { + tests: { repeated: Array } + } + const omitted = phase.tests.repeated.findIndex( + (name) => + name.startsWith(`second >`) && + (kind === `forced` + ? name.includes(`forced entered`) + : name.includes(`calibration: cal-0`)), + ) + phase.tests.repeated.splice(omitted, 1) + return altered + }, + })), + { + name: `reject unexpected child error`, + history: { constructed: `round2 ranked mutant verdicts` }, + law: `trace`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.find(({ checkpoint }) => checkpoint === `child`)! + .value as { + exit: number + unhandled: Array + } + child.exit = 1 + child.unhandled.push(`unexpected registration rejection`) + return altered + }, + }, + { + name: `reject equal-data origin substitution`, + history: { constructed: `review equal fixed originals` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const run = altered.find(({ checkpoint }) => checkpoint === `run`)! + .value as SuiteRun + for (const entry of run.log) + if (entry.test.includes(`calibration: cal-0`)) + entry.origin = `fixed:fixed-0` + return altered + }, + }, + ], owns: [`src/oracle-property.ts`], // Each history spawns one to five child Vitest runs. timeout: 600_000, diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md new file mode 100644 index 0000000..892016e --- /dev/null +++ b/reviews/calibration-history-self-oracle.md @@ -0,0 +1,147 @@ +# Calibration history self-oracle review + +Reviewed production and starting oracle: `2dc95d4e838678a03eb345c08f141dbd20a8c4a7`. +This follow-up changes only the primary self-oracle and its fixture. It changes +no framework production behavior. Four independent reviewers evaluated execution +semantics, judgment independence, grammar, and receiving observations; two then +rechecked the repairs. This record distinguishes their evidence from the final +repair validation below. + +## Laws and experiments + +The README's history-binding contract requires a selected name to identify one +existing original. Both automatic clean premises accept fixed or constructed +selectors; only omitted entered selectors infer a construction. Identity matters +even when two originals have equal data. The registration contract owes the same +test list on every registration: suppressing duplicate warnings does not suppress +tests. Real Vitest must report complete results and every unexpected failure. +The self-oracle separately chooses diagnostic precedence and explicitly promises +to check its first registration error alone; that ordering is not a new public +framework policy. + +The primary model now derives both selected premises from authored declarations, +checks cardinality, and compares input origins, item values and exact build, +resolve and driver budgets. Independent selector axes include non-first names, +fixed entry, constructed absence, and inferred entry. Equal-data fixed originals +remain distinguishable by fixture-side identity associations. Named enumeration +selectors match authored labels, including shell suffixes and the plain JSON +label represented in this bounded string grammar. + +The driver uses actual child Vitest processes for normal runs, replays, reruns, +reach-only runs, seedless runs, reordered calibrations and registration phases. +Each child records status, signal, file completeness, suite/load errors, +secondary test errors, unhandled errors and the input histories that determine +late-call obligations. The check permits expected synthetic failed tests to exit +1 while independently rejecting unexpected errors. Relative file names preserve +identity within a run without recording random temporary directory paths. +Repeated ordinary and collecting registrations retain every scoped test name +with multiplicity and compare against the same model-derived registration list. + +## Finding accounting + +| ID | Verified finding | Disposition and durable enforcement | +| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| I-1 | Model falsely permitted ambiguous selected construction names for inferred/explicit automatic controls. Production correctly rejected them. | Repaired cardinality model; both generated required paths and fixed replays. | +| I-2 | Authored refinement accepted the wrong equal-valued original. | Repaired origin comparison; equal-data distinct originals and corruption calibration. | +| G1 | Grammar omitted fixed clean-entry and constructed clean-absence. | Independent selectors and separately required receiving paths; actual-kind budgets. | +| OBS-1 | Repeated-warning phase discarded registrations. | Scoped complete name lists, shared expectations, authored/forced omission calibrations. | +| OBS-2 | Unrelated child unhandled failures were ignored. | Every receiving child checked; unexpected-error and process/report calibrations. | +| LOADERR-CAND | Expected diagnostic plus extra diagnostics was accepted. | Confirmed by a failing calibration before repair; exact cardinality and permanent duplicate-diagnostic calibration. | +| SIB-1 | Named enumeration model ignored custom/default labels. | Confirmed false failures on correct production; authored-label model and valid/missing neighboring receivers. | +| OBS-2a | Replay-only afterAll failure hid behind an expected failing test/exit1. | Per-file suite/load errors independently judged; auxiliary calibration. | +| OBS-2b | Extra real late-probe error during replay escaped main-only counting. | Input-based late-call bounds checked separately for every child; required actual replay receiver and duplicate-error calibration. | +| OBS-2c | Replay-only afterEach error hid behind the first test error. | Secondary test errors retained and rejected for every child; auxiliary calibration. | +| CHRON-1 | Earlier clean declared-enumerated receiving gate was run after production repair. | Historical process violation remains recorded. Current receiver and hostile omission check exist; retrospective evidence cannot make the earlier gate pre-edit. | + +The raw intake has seven canonical items. Rechecks added four sibling items. +Ten executable gaps are repaired; CHRON-1 remains a historical process finding, +with no claim that today's tests repair the past ordering. No item was deferred, +refuted, waived, or reclassified as a product-law decision. The temporary stdout +announcement note is preserved as chronology, not a final net production defect. +The cross-file warning experiment used different worker processes and did not +reach its premise; its two warnings establish no defect. + +## Sensitivity + +Before repair, the reviewers demonstrated two false model predictions for +ambiguous selectors and four targeted surviving wrong implementations. They +also ran 25 clean execution-semantics controls and rejected incorrect bound +verdict ranking, a narrow throw-only repair (the neighboring clock case failed), +and difference-before-law. These are bounded reviewer results, not a whole-suite +run against each variant. + +After repair, eleven isolated source mutants reject through the primary model +and check at their intended comparison: + +- ignoring an explicit fixed entered selector; +- canonicalizing an equal-data fixed original; +- dropping calibrations on the second registration; +- introducing an unrelated unhandled rejection; +- dropping only a repeated forced control; +- substituting fixed absence for a selected constructed absence; +- adding replay-only afterAll or afterEach failures; +- doubling replay-only late-probe errors; +- accepting an ambiguous selected name; +- ignoring an enumerated label's shell suffix. + +Ten permanent observation-corruption calibrations also challenge registration +multiplicity, selected identity, extra load diagnostics, unknown child errors, +omitted reports, abnormal status, auxiliary suite/test errors and late-call +multiplicity. They all use the same primary law/model/driver/check. + +## Validation and bounds + +At publication, the final full framework/determinism run is still in progress; +no final-head GREEN or merge-readiness claim is made here. The earlier focused +receivers passed 14/14; the independent observation recheck passed 12/12, +including all ten corruption calibrations. All eleven isolated source mutants +are assertion-rejected. Package type checking, changed-file lint/format and the +package build pass. The ordinary core run passed 54 checks; two npm-pack checks +hit the sandbox's read-only default npm cache and are being retried with a +scratch cache. CI and the final local run must validate the published head. +Commands use the installed Node entry point, `--maxWorkers=2`, and no dependency +install workaround or override. A first build command used a nonexistent +package-local Vite entry; rerunning the installed workspace entry succeeded. +These setup failures are not product RED or mutant rejection evidence. + +Required new states are fixed-entry, constructed-absence, inferred ambiguity, +explicit ambiguity, equal-data original selection, enumerated named labels, and +an actual auxiliary late-call receiver. Both normal campaigns must reach each. +The existing registration-warning and bound-state requirements remain in place. +Fixed inputs replay the shared judgment without separate expected answers. + +The independent model imports no production selector or verdict helper. Input +origins and construction callback inputs are fixture-side facts, not answers +learned from production. Selection kind changes build obligations; equal data +with different input identity changes the promised selected original. These +justify the retained distinctions. No concrete missing-source-fact hypothesis +remains in the repaired cuts: the real reporter supplies all the compared facts. +The bounded faults are distinguished by the legal neighbors and mutants above; +no second implementation of production selection was introduced. + +The owner retains its explicit finite synthetic-history, name-family, callback, +regex and scheduler limits. It does not establish all possible JavaScript labels +or errors, browser/provider behavior, downstream DB consumer integration, a DB +release-time improvement, or fresh-seed UNIT support. Real Vitest is the actual +receiving host; no controlled-provider extrapolation is claimed. + +ORC-001 through ORC-006, ORC-008 through ORC-010, ORC-013, ORC-015 and ORC-016 +are covered by the authority, independent model, shared generated/fixed inputs, +receiving observations, neighbor controls and explicit mutants above. ORC-007 +uses the owner's unchanged fixed/random budgets and direct replay checks. +ORC-011's named identity/selection faults are challenged by declaration-origin +relations instead of another copy of production selection. ORC-014 needs no +external-provider handoff for this Node framework claim. This versioned record +and the executable owner provide ORC-012 evidence; CHRON-1 remains an explicit +historical exception to a complete pre-edit receiving-gate claim. + +The four reviewers distinguished current production from oracle integrity and +supplied concrete witnesses or useful positive controls. Strong hire for all +four in their reviewed scopes; none establishes whole-PR correctness alone. + +## Reviewed content identity + +Parent commit: `2dc95d4e838678a03eb345c08f141dbd20a8c4a7`. + +- `packages/oracles/tests/oracle-property-self-oracle.test.ts`: SHA256 `7d1da835c7c07bd0ad6670de47e0893e3e950c79df15e6773d038ca4e6da77d7`. +- `packages/oracles/tests/oracle-property-self-oracle-fixture.ts`: SHA256 `46a7a47ce4e028a45ab0de99119e130aae4bcde238791f3c869970b5739d3962`. From fc4bad8dc2c3da3a80b5002bcc36281b75f12846 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:47:24 -0700 Subject: [PATCH 07/14] test: control host randomness for seedless receiving checks --- .../oracle-property-self-oracle-fixture.ts | 15 ++++++++++++++- .../tests/oracle-property-self-oracle.test.ts | 8 +++++--- reviews/calibration-history-self-oracle.md | 17 ++++++++++++++--- 3 files changed, 33 insertions(+), 7 deletions(-) diff --git a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts index a7024bc..bf07dd6 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts +++ b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts @@ -146,7 +146,7 @@ import { appendFileSync } from 'node:fs' import { runInNewContext } from 'node:vm' import fc from 'fast-check' -import { afterAll, beforeEach, describe, expect, vi } from 'vitest' +import { afterAll, beforeAll, beforeEach, describe, expect, vi } from 'vitest' import { configureOracles, differs, @@ -1054,6 +1054,19 @@ function declaredEnumerated( } export function syntheticProperty(spec: SyntheticSpec): void { + // An ordinary child still selects its seed without ORACLE_SEED. Control + // the host draw so observing its inputs/status does not make the outer + // determinism check compare two different random environments. + if (process.env.SELF_ORACLE_RANDOM_DRAW !== undefined) { + const random = Math.random + const draw = Number(process.env.SELF_ORACLE_RANDOM_DRAW) + beforeAll(() => { + Math.random = () => draw + }) + afterAll(() => { + Math.random = random + }) + } const id = spec.unregistered ? `self-oracle.unregistered` : spec.id const makeForm = formOf(spec) const items = itemsOf(spec) diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 1bb88ed..a1745e3 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -2553,7 +2553,9 @@ function frameworkReach( // Vitest runs that file with a `-t` pattern that selects only its reach // test. // Child runs fix the random campaign's seed (a seed without a path reseeds the -// campaign and selects no replay), so a child run records the same facts every time. +// campaign and selects no replay). The ordinary-route child instead controls +// Math.random's draw while leaving ORACLE_SEED absent. This fixes the host +// input without changing the route, so every recorded fact can repeat. type ReportedError = { name?: string; message?: string; cause?: ReportedError } type ReportedTest = { @@ -3165,13 +3167,13 @@ function driveSuite( receive( directory, files, - { SELF_ORACLE_LOG: seedlessLog }, + { SELF_ORACLE_LOG: seedlessLog, SELF_ORACLE_RANDOM_DRAW: `0.5` }, undefined, false, ) const logged = readLog(seedlessLog) // The routes each random campaign ran under: the same whatever seed it - // drew, so this observation repeats. + // draws under the controlled host RNG, so this observation repeats. observe( `seedless`, Object.fromEntries( diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md index 892016e..7ff55ef 100644 --- a/reviews/calibration-history-self-oracle.md +++ b/reviews/calibration-history-self-oracle.md @@ -54,7 +54,7 @@ with multiplicity and compare against the same model-derived registration list. | CHRON-1 | Earlier clean declared-enumerated receiving gate was run after production repair. | Historical process violation remains recorded. Current receiver and hostile omission check exist; retrospective evidence cannot make the earlier gate pre-edit. | The raw intake has seven canonical items. Rechecks added four sibling items. -Ten executable gaps are repaired; CHRON-1 remains a historical process finding, +Ten review-derived executable gaps are repaired; CHRON-1 remains a historical process finding, with no claim that today's tests repair the past ordering. No item was deferred, refuted, waived, or reclassified as a product-law decision. The temporary stdout announcement note is preserved as chronology, not a final net production defect. @@ -119,6 +119,17 @@ remains in the repaired cuts: the real reporter supplies all the compared facts. The bounded faults are distinguished by the legal neighbors and mutants above; no second implementation of production selection was introduced. +The full local sweep then passed 226 of 227 checks and caught REPAIR-1, an +observation mistake introduced by this repair: the new child recorder included +raw seedless histories and status, so legitimate host randomness broke the +outer determinism comparison. The seedless receiver now controls Math.random's +draw through fixture hooks while leaving ORACLE_SEED absent. It still exercises +ordinary random-route selection and all child error/count assertions. A focused +real-Vitest receiver runs the same model/driver/check for ordinary, late-probe, +and two-property inputs and then the framework's own determinism check: 4/4 +pass. Entropy or seed-distribution quality is not claimed. The complete expanded +suite remains subject to final-head CI; the failed intermediate run is not GREEN. + The owner retains its explicit finite synthetic-history, name-family, callback, regex and scheduler limits. It does not establish all possible JavaScript labels or errors, browser/provider behavior, downstream DB consumer integration, a DB @@ -143,5 +154,5 @@ four in their reviewed scopes; none establishes whole-PR correctness alone. Parent commit: `2dc95d4e838678a03eb345c08f141dbd20a8c4a7`. -- `packages/oracles/tests/oracle-property-self-oracle.test.ts`: SHA256 `7d1da835c7c07bd0ad6670de47e0893e3e950c79df15e6773d038ca4e6da77d7`. -- `packages/oracles/tests/oracle-property-self-oracle-fixture.ts`: SHA256 `46a7a47ce4e028a45ab0de99119e130aae4bcde238791f3c869970b5739d3962`. +- `packages/oracles/tests/oracle-property-self-oracle.test.ts`: SHA256 `c8df3de2569460a80ff1eab23811d21f9ac6fffe6e4d935bdcfdc5ed5f579d0f`. +- `packages/oracles/tests/oracle-property-self-oracle-fixture.ts`: SHA256 `27195bf184b7ae58f9eaea7bab82973aae5876ce292e2b6aa7e6fa02ac20627d`. From d5e615a8f508bae1577147112aa7225222687374 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 16:51:52 -0700 Subject: [PATCH 08/14] test: enforce calibration selection and child report fidelity --- .../oracle-property-self-oracle-fixture.ts | 260 ++-- .../tests/oracle-property-self-oracle.test.ts | 1366 +++++++++++++++-- reviews/calibration-history-self-oracle.md | 73 + 3 files changed, 1446 insertions(+), 253 deletions(-) diff --git a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts index bf07dd6..463045a 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts +++ b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts @@ -125,7 +125,11 @@ * the text `is 3`. A `throws` driver records that wrong observation before * throwing, so ordinary verdict ranking still decides its calibration. A * `clock` driver leaves its item observations unchanged and spins a controlled - * clock; only the mutant execution spins. `corrupt-throws`, `judge-throws`, + * clock; only the mutant execution spins. A `driver-spin` mutant first records + * its wrong item and then spins, and `driver-spin-throws` also throws after the + * spin. Those ordered events distinguish an earlier violation from a later + * clock failure without making the selected clean history fail. + * `corrupt-throws`, `judge-throws`, * and `judge-driver-throws` exercise exceptions from corruption and from the * two mutant judgment routes without supplying any expected result. * @@ -137,6 +141,10 @@ * outside every declaration, either globally or only in authored calibrations. * The phase driver registers the same property under distinct suites to * exercise repeated registration and collect-only reporting in one process. + * Each authored calibration can select its own history, or explicitly remain + * unbound beside bound peers. A fixed declaration can carry unrelated own or + * inherited `build` metadata. That method is never the declaration's source + * kind; if called, its copied result and build log expose the wrong dispatch. * * The synthetic ids come from a fixed pool, which this fixture registers with * `configureOracles`. The framework's own id, `self-oracle.framework`, is @@ -228,6 +236,8 @@ export type SyntheticSpec = { name: string history: SyntheticHistory enters?: ReadonlyArray + /** Unrelated declaration metadata, not a construction. */ + buildMetadata?: `own` | `inherited` }> /** * A `.map` the arbitrary applies, with or without an unmapper, to the whole @@ -252,6 +262,8 @@ export type SyntheticSpec = { | `no-op` | `driver` | `driver-last` + | `driver-spin` + | `driver-spin-throws` | `inapplicable` | `throws` | `clock` @@ -260,6 +272,8 @@ export type SyntheticSpec = { | `judge-driver-throws` /** The difference the calibration requires, as text or a pattern, matching or not; absent, none. */ difference?: CalibrationDifference + /** Absent inherits the property default; null explicitly stays unbound. */ + history?: SyntheticSpec[`calibrationHistory`] | null }> /** The proposed binding, passed structurally so the unchanged framework can be judged. */ calibrationHistory?: @@ -736,7 +750,7 @@ function lateProbe() { */ function syntheticDriver( spec: SyntheticSpec, - mutant?: `first` | `last` | `clock`, + mutant?: `first` | `last` | `clock` | `first-spin` | `first-spin-throws`, consumed = new WeakSet(), origins = new WeakMap(), ) { @@ -784,13 +798,19 @@ function syntheticDriver( for (const state of reachedStates(spec)(history)) observe(`reach`, state) // A real timer first: a clock that an earlier history released must have // given the real timers back, or this history never goes on. - if (spec.clock) await new Promise((resolve) => setTimeout(resolve, 1)) - const clock = - spec.clock || mutant === `clock` ? controlledClock() : undefined + const mutantSpins = + mutant === `clock` || + mutant === `first-spin` || + mutant === `first-spin-throws` + if (spec.clock || mutantSpins) + await new Promise((resolve) => setTimeout(resolve, 1)) + const clock = spec.clock || mutantSpins ? controlledClock() : undefined try { for (const [index, item] of history.entries()) { const mutated = - mutant === `first` + mutant === `first` || + mutant === `first-spin` || + mutant === `first-spin-throws` ? index === 0 : mutant === `last` && index === history.length - 1 observe(`item ${index}`, mutated ? 3 : item) @@ -816,12 +836,14 @@ function syntheticDriver( ) } } - if (mutant === `clock`) { + if (mutantSpins) { const spin = () => { setTimeout(spin, 0) } spin() await vi.advanceTimersByTimeAsync(1000) + if (mutant === `first-spin-throws`) + throw new Error(`mutant setup failed after the spin`) } } finally { clock?.release() @@ -921,91 +943,107 @@ const differences: Record = { 'empty pattern': /(?:)/, } +/** The authored selector passed to production, including explicit migration opt-out. */ +const calibrationHistory = ( + spec: SyntheticSpec, + calibration: SyntheticSpec[`calibrations`][number], +) => + calibration.history === undefined + ? spec.calibrationHistory + : (calibration.history ?? undefined) + function syntheticCalibrations( spec: SyntheticSpec, consumed: WeakSet, origins: WeakMap, ): Array> { - return spec.calibrations.map( - ({ name, kind, difference }): Calibration => { - const history = spec.calibrationHistory - const required = - difference === undefined ? {} : { difference: differences[difference] } - if (kind === `clock`) - return { - name, - history, - law: `bounded clock`, - ...required, - driver: syntheticDriver(spec, `clock`, consumed, origins), - } - if (kind === `judge-driver-throws`) - return { - name, - history, - law: `three`, - ...required, - driver: async (plan, observe) => { - await syntheticDriver( - spec, - `first`, - consumed, - origins, - )(plan, observe) - observe(`synthetic-judgment-fault`, true) - }, - } - // A mutant driver records a 3 in place of the first or the last item. - if (kind === `driver` || kind === `driver-last` || kind === `throws`) - return { - name, - history, - law: `three`, - ...required, - driver: - kind === `throws` - ? async (plan, observe) => { - await syntheticDriver( - spec, - `first`, - consumed, - origins, - )(plan, observe) - throw new Error(`mutant setup failed`) - } - : syntheticDriver( - spec, - kind === `driver-last` ? `last` : `first`, - consumed, - origins, - ), - } + return spec.calibrations.map((calibration): Calibration => { + const { name, kind, difference } = calibration + const history = calibrationHistory(spec, calibration) + const required = + difference === undefined ? {} : { difference: differences[difference] } + if (kind === `clock`) return { name, history, - law: kind === `wrong-law` ? `four` : `three`, + law: `bounded clock`, ...required, - corrupt: (trace) => { - if (kind === `corrupt-throws`) - throw new Error(`synthetic corruption failed`) - if (kind === `judge-throws`) - return [ - ...trace, - { checkpoint: `synthetic-judgment-fault`, value: true }, - ] - if (kind === `inapplicable`) return undefined - const at = trace.findIndex(({ checkpoint }) => - checkpoint.startsWith(`item`), - ) - if (at < 0) return undefined - if (kind === `no-op`) return [...trace] - return trace.map((entry, index) => - index === at ? { ...entry, value: 3 } : entry, - ) + driver: syntheticDriver(spec, `clock`, consumed, origins), + } + if (kind === `driver-spin` || kind === `driver-spin-throws`) + return { + name, + history, + law: `three`, + ...required, + driver: syntheticDriver( + spec, + kind === `driver-spin` ? `first-spin` : `first-spin-throws`, + consumed, + origins, + ), + } + if (kind === `judge-driver-throws`) + return { + name, + history, + law: `three`, + ...required, + driver: async (plan, observe) => { + await syntheticDriver(spec, `first`, consumed, origins)(plan, observe) + observe(`synthetic-judgment-fault`, true) }, } - }, - ) + // A mutant driver records a 3 in place of the first or the last item. + if (kind === `driver` || kind === `driver-last` || kind === `throws`) + return { + name, + history, + law: `three`, + ...required, + driver: + kind === `throws` + ? async (plan, observe) => { + await syntheticDriver( + spec, + `first`, + consumed, + origins, + )(plan, observe) + throw new Error(`mutant setup failed`) + } + : syntheticDriver( + spec, + kind === `driver-last` ? `last` : `first`, + consumed, + origins, + ), + } + return { + name, + history, + law: kind === `wrong-law` ? `four` : `three`, + ...required, + corrupt: (trace) => { + if (kind === `corrupt-throws`) + throw new Error(`synthetic corruption failed`) + if (kind === `judge-throws`) + return [ + ...trace, + { checkpoint: `synthetic-judgment-fault`, value: true }, + ] + if (kind === `inapplicable`) return undefined + const at = trace.findIndex(({ checkpoint }) => + checkpoint.startsWith(`item`), + ) + if (at < 0) return undefined + if (kind === `no-op`) return [...trace] + return trace.map((entry, index) => + index === at ? { ...entry, value: 3 } : entry, + ) + }, + } + }) } // `owns`, the literal list the gap report reads, in the shape the spec names. @@ -1113,13 +1151,18 @@ export function syntheticProperty(spec: SyntheticSpec): void { throw new Error(`selected history lost identity metadata`) if ( spec.isolatedPlans && - ((spec.calibrationHistory && - `constructed` in spec.calibrationHistory && - spec.calibrations.some(({ name }) => + (spec.calibrations.some((calibration) => { + const binding = calibrationHistory(spec, calibration) + return ( + binding !== undefined && + `constructed` in binding && expect .getState() - .currentTestName?.endsWith(` > calibration: ${name}`), - )) || + .currentTestName?.endsWith( + ` > calibration: ${calibration.name}`, + ) + ) + }) || (spec.probeBinding && spec.probeBinding !== `empty` && ((expect @@ -1299,9 +1342,14 @@ export function syntheticProperty(spec: SyntheticSpec): void { : { label: (history: Formed) => spec.duplicateHistoryNames && - spec.calibrationHistory && - `enumerated` in spec.calibrationHistory && - typeof spec.calibrationHistory.enumerated === `string` + spec.calibrations.some((calibration) => { + const selected = calibrationHistory(spec, calibration) + return ( + selected !== undefined && + `enumerated` in selected && + typeof selected.enumerated === `string` + ) + }) ? `items ${spec.source.kind === `enumerated` ? spec.source.histories[0]!.join(` | `) : ``}` : spec.labelStyle?.startsWith(`reserved-`) && JSON.stringify(items(history)) === first @@ -1363,11 +1411,35 @@ export function syntheticProperty(spec: SyntheticSpec): void { return history }, })), - fixed: spec.fixed.map((fixed) => ({ - ...fixed, - ...(fixed.enters ? { enters: fixed.enters.map(publicState) } : {}), - history: markOrigin(form(fixed.history), `fixed:${fixed.name}`), - })), + fixed: spec.fixed.map((fixed) => { + const declaration = { + ...fixed, + ...(fixed.enters ? { enters: fixed.enters.map(publicState) } : {}), + history: markOrigin(form(fixed.history), `fixed:${fixed.name}`), + } + // Fixed declarations are structurally open metadata wrappers. An + // incidental build method must never run or replace their original. + const build = (generated: Formed) => { + const history = makeForm([...fixed.history]) + const log = process.env.SELF_ORACLE_LOG + if (log) + appendFileSync( + `${log}.builds`, + `${JSON.stringify({ + id: spec.id, + test: expect.getState().currentTestName ?? ``, + name: `fixed-metadata:${fixed.name}`, + generated: items(generated), + history: items(history), + })}\n`, + ) + return history + } + if (fixed.buildMetadata === `own`) return { ...declaration, build } + if (fixed.buildMetadata === `inherited`) + Object.setPrototypeOf(declaration, { build }) + return declaration + }), }, ...parts, }) diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index a1745e3..75af543 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -92,6 +92,10 @@ * property/process without dropping tests, and collect-only stays silent. * Every real child must finish with a complete report and no unmodeled * unhandled errors, including replay and registration-only children. + * Each authored calibration selects independently, including bound/unbound + * mixtures. Extra own or inherited metadata on a fixed declaration cannot + * change its selected source kind. A mutant may record a wrong value, then + * spin and optionally throw: that earlier violation still decides. * - `determinism`: the determinism test runs every fixed and constructed history * and three generated ones, and fails for a driver whose trace changes, * compared by their data encoding: `-0` is not `0`, and key order and @@ -248,6 +252,8 @@ const calibrationKinds = [ `no-op`, `driver`, `driver-last`, + `driver-spin`, + `driver-spin-throws`, `clock`, `corrupt-throws`, `judge-throws`, @@ -266,6 +272,8 @@ const calibrationDifferences = [ type CalibrationDraft = { kind: (typeof calibrationKinds)[number] difference: CalibrationDifference | `none` + /** Absent inherits the property default; null explicitly retains search. */ + history?: SyntheticSpec[`calibrationHistory`] | null } type Draft = { source: @@ -274,7 +282,12 @@ type Draft = { /** `exact` declares the constructed histories' states; the others break registration. */ statesMode: `exact` | `extra` | `missing` constructed: Array<{ prefix: SyntheticHistory; enters: Array }> - fixed: Array<{ history: SyntheticHistory; enters: Array | null }> + fixed: Array<{ + history: SyntheticHistory + enters: Array | null + /** Unrelated declaration metadata, never a history construction. */ + buildMetadata?: `own` | `inherited` + }> /** * How a generated source's arbitrary maps its histories: not at all (only * a form maps, with an unmapper), through a `.map` with an unmapper, or @@ -382,6 +395,28 @@ const probeSelector = fc.oneof( constructed: fc.constantFrom(`constructed-0`, `constructed-1`, `missing`), }), ) +// Selection belongs to each calibration. The property default remains useful +// for migrations; an explicit null chooses search beside selected neighbors. +const calibrationSelector = fc.oneof( + fc.record({ + fixed: fc.constantFrom(`fixed-0`, `fixed-1`, `unknown`, `missing`), + }), + fc.record({ + constructed: fc.constantFrom( + `constructed-0`, + `constructed-1`, + `unknown`, + `missing`, + ), + }), + fc.record({ + enumerated: fc.oneof( + fc.integer({ min: -1, max: 4 }), + fc.constant(1.5), + fc.constantFrom(`items 0 | 1`, `items 0 | 1 ${shellText}`, `[0,1]`), + ), + }), +) const draft: fc.Arbitrary = fc.record({ probeEntered: probeSelector, probeNotEntered: probeSelector, @@ -450,10 +485,17 @@ const draft: fc.Arbitrary = fc.record({ { maxLength: 2 }, ), fixed: fc.array( - fc.record({ - history: insideOrOutside(4), - enters: fc.option(fc.subarray(allStates), { freq: 2 }), - }), + fc.record( + { + history: insideOrOutside(4), + enters: fc.option(fc.subarray(allStates), { freq: 2 }), + buildMetadata: fc.option( + fc.constantFrom(`own` as const, `inherited` as const), + { nil: undefined, freq: 8 }, + ), + }, + { requiredKeys: [`history`, `enters`] }, + ), { maxLength: 2 }, ), // Rare: a `.map` with or without an unmapper, which decides whether the @@ -517,21 +559,31 @@ const draft: fc.Arbitrary = fc.record({ calibrations: fc.array( fc.oneof( fc.constantFrom(...calibrationKinds), - fc.record({ - kind: fc.constantFrom(...calibrationKinds), - difference: fc.oneof( - // Rare: an empty difference fails the whole property's load. - { weight: 9, arbitrary: fc.constant(`none` as const) }, - { weight: 9, arbitrary: fc.constantFrom(...calibrationDifferences) }, - { - weight: 1, - arbitrary: fc.constantFrom( - `empty text` as const, - `empty pattern` as const, - ), - }, - ), - }), + fc.record( + { + kind: fc.constantFrom(...calibrationKinds), + history: fc.option(fc.option(calibrationSelector, { nil: null }), { + nil: undefined, + freq: 4, + }), + difference: fc.oneof( + // Rare: an empty difference fails the whole property's load. + { weight: 9, arbitrary: fc.constant(`none` as const) }, + { + weight: 9, + arbitrary: fc.constantFrom(...calibrationDifferences), + }, + { + weight: 1, + arbitrary: fc.constantFrom( + `empty text` as const, + `empty pattern` as const, + ), + }, + ), + }, + { requiredKeys: [`kind`, `difference`] }, + ), ), { maxLength: 3 }, ), @@ -729,9 +781,16 @@ function resolveSuite({ drafts, seedless }: Suite): Plan { seedless, specs: drafts.map((d, index) => { const generated = d.source.kind === `generated` + const hasBinding = d.calibrations.some((calibration) => + typeof calibration === `string` + ? d.calibrationHistory !== null + : calibration.history === undefined + ? d.calibrationHistory !== null + : calibration.history !== null, + ) const constructed = generated ? d.constructed.map((construction, at) => ({ - name: `constructed-${d.duplicateHistoryNames && d.calibrationHistory ? 0 : at}`, + name: `constructed-${d.duplicateHistoryNames && hasBinding ? 0 : at}`, ...construction, })) : [] @@ -752,9 +811,12 @@ function resolveSuite({ drafts, seedless }: Suite): Plan { constructed, fixed: generated ? d.fixed.map((fixedHistory, at) => ({ - name: `fixed-${d.duplicateHistoryNames && d.calibrationHistory ? 0 : at}`, + name: `fixed-${d.duplicateHistoryNames && hasBinding ? 0 : at}`, history: fixedHistory.history, ...(fixedHistory.enters ? { enters: fixedHistory.enters } : {}), + ...(fixedHistory.buildMetadata + ? { buildMetadata: fixedHistory.buildMetadata } + : {}), })) : [], ...(generated && d.arbitraryMap !== `none` @@ -879,9 +941,9 @@ function resolveSuite({ drafts, seedless }: Suite): Plan { (item) => item !== undefined, ) ).map((entry, at) => { - const { kind, difference } = + const { kind, difference, history } = typeof entry === `string` - ? { kind: entry, difference: `none` as const } + ? { kind: entry, difference: `none` as const, history: undefined } : entry return { name: @@ -892,6 +954,7 @@ function resolveSuite({ drafts, seedless }: Suite): Plan { : `cal-${at}`, kind, ...(difference === `none` ? {} : { difference }), + ...(history === undefined ? {} : { history }), } }), } @@ -937,6 +1000,8 @@ type Outcome = { law?: `three` | `four` | `reach agreement` | `bounded clock` /** A violation whose driver threw afterwards carries that error. */ cause?: `driver` + /** The exact finite driver error a violation carries. */ + causeMessage?: string /** The checkpoint a framework-made violation names. */ checkpoint?: string /** For a predicted entry that did not happen: whether the probe was called. */ @@ -1002,7 +1067,21 @@ function verdictOf(history: SyntheticHistory, spec: SyntheticSpec): Outcome { (history[stopAt] === 6 || spec.throwOnSpin === true || history.slice(stopAt + 1).includes(6)) - const carried = driverThrows ? { cause: `driver` as const } : {} + const driverThrowAt = history.findIndex( + (item) => item === 6 || (clock && spec.throwOnSpin === true && item === 7), + ) + const carried = driverThrows + ? { + cause: `driver` as const, + causeMessage: + history[driverThrowAt] === 7 + ? `synthetic driver failed after the spin at item ${driverThrowAt}` + : `synthetic driver failed at item ${driverThrowAt}`, + } + : {} + // A model-not-total error wraps the judgment error, while retaining the + // driver error separately. Its inner cause does not use violation shape. + const modelCarried = driverThrows ? { cause: `driver` as const } : {} const judgedItems = stopAt >= 0 ? history.slice(0, stopAt + 1) : history const modelThrows = spec.strictModel ? stopAt >= 0 && stopAt < history.length - 1 @@ -1021,7 +1100,7 @@ function verdictOf(history: SyntheticHistory, spec: SyntheticSpec): Outcome { stopAt >= 0 && (stopAt < history.length - 1 || trailing || driverThrows) ) - return { kind: `model-not-total`, ...carried } + return { kind: `model-not-total`, ...modelCarried } return { kind: `setup` } } // A check that compares data past the encoding limit fails as that limit, @@ -1033,6 +1112,7 @@ function verdictOf(history: SyntheticHistory, spec: SyntheticSpec): Outcome { return { kind: `violation`, law: item === 3 ? `three` : `four`, + checkpoint: `item ${at}`, ...(history.slice(at + 1).some(stops) ? carried : {}), } if (item === 6) return { kind: `setup` } @@ -1247,9 +1327,50 @@ const emptyDifference = (difference: CalibrationDifference | undefined) => const driverCalibration = ( kind: SyntheticSpec[`calibrations`][number][`kind`], ) => - [`driver`, `driver-last`, `throws`, `clock`, `judge-driver-throws`].includes( - kind, + [ + `driver`, + `driver-last`, + `driver-spin`, + `driver-spin-throws`, + `throws`, + `clock`, + `judge-driver-throws`, + ].includes(kind) +// Binding is authored input, not an observation of the framework's selection. +// Explicit unbound entries remain distinct from inheritance of the default. +const bindingOf = ( + spec: SyntheticSpec, + calibration: SyntheticSpec[`calibrations`][number], +) => + calibration.history === undefined + ? spec.calibrationHistory + : (calibration.history ?? undefined) +const bindingsOf = (spec: SyntheticSpec) => + spec.calibrations + .map((calibration) => bindingOf(spec, calibration)) + .filter((binding) => binding !== undefined) +const unboundCalibrationNames = (spec: SyntheticSpec) => [ + ...spec.calibrations + .filter((calibration) => !bindingOf(spec, calibration)) + .map(({ name }) => name), + ...(!spec.probeBinding || spec.probeBinding === `empty` + ? forcedCalibrationNames(spec).map((name) => + name.slice(`calibration: `.length), + ) + : []), +] +const boundTest = (spec: SyntheticSpec, name: string) => { + const calibration = spec.calibrations.find( + (entry) => name === `calibration: ${entry.name}`, ) + return calibration + ? bindingOf(spec, calibration) !== undefined + : Boolean( + spec.probeBinding && + spec.probeBinding !== `empty` && + forcedCalibrationNames(spec).includes(name), + ) +} type BuildEntry = { id: string test: string @@ -1269,9 +1390,10 @@ const authoredEnumerationLabel = ( return spec.form === undefined ? JSON.stringify(history) : undefined if ( spec.duplicateHistoryNames && - spec.calibrationHistory && - `enumerated` in spec.calibrationHistory && - typeof spec.calibrationHistory.enumerated === `string` && + bindingsOf(spec).some( + (binding) => + `enumerated` in binding && typeof binding.enumerated === `string`, + ) && spec.source.kind === `enumerated` ) return `items ${spec.source.histories[0]!.join(` | `)}` @@ -1389,15 +1511,13 @@ const enumeratedReach = ( judgedHistory: (history: SyntheticHistory) => boolean, ): { outcome: Outcome; counts: Record } => { const counts = Object.fromEntries( - [...declaredOf(spec)!] - .sort() - .map((state) => [ - state, - histories.filter( - (history) => - judgedHistory(history) && reachedOf(spec, history).includes(state), - ).length, - ]), + [...declaredOf(spec)!].map((state) => [ + state, + histories.filter( + (history) => + judgedHistory(history) && reachedOf(spec, history).includes(state), + ).length, + ]), ) const missed = spec.enumeratedStates!.states.some( (state) => counts[state] === 0, @@ -1628,10 +1748,10 @@ function predictSpec( // The synthetic namespace is deliberately small: names identify the fixed // or constructed index; enumerated names state their items. This model judges // selection independently of the framework's resolver. - const binding = spec.calibrationHistory - const selected = - binding && - (`fixed` in binding + const selectedDeclaration = ( + binding: NonNullable, + ) => + `fixed` in binding ? spec.fixed.find(({ name }) => name === binding.fixed) : `constructed` in binding ? spec.constructed.find(({ name }) => name === binding.constructed) @@ -1642,10 +1762,11 @@ function predictSpec( : authoredEnumerationLabel(spec, history) === binding.enumerated, ) - : undefined) - const duplicates = - binding && - (`fixed` in binding + : undefined + const duplicateSelection = ( + binding: NonNullable, + ) => + `fixed` in binding ? spec.fixed.filter(({ name }) => name === binding.fixed).length > 1 : `constructed` in binding ? spec.constructed.filter(({ name }) => name === binding.constructed) @@ -1656,15 +1777,16 @@ function predictSpec( (history) => authoredEnumerationLabel(spec, history) === binding.enumerated, ).length > 1 - : false) + : false if ( - binding && - spec.calibrations.length && - (!selected || - duplicates || - (spec.source.kind === `enumerated` - ? !(`enumerated` in binding) - : `enumerated` in binding)) + bindingsOf(spec).some( + (binding) => + !selectedDeclaration(binding) || + duplicateSelection(binding) || + (spec.source.kind === `enumerated` + ? !(`enumerated` in binding) + : `enumerated` in binding), + ) ) return { loads: false, loadError: `calibration history selector` } if ( @@ -1705,9 +1827,7 @@ function predictSpec( for (const name of all) tests.set( name, - binding && name.startsWith(`calibration:`) - ? { kind: `calibration-clean` } - : data, + boundTest(spec, name) ? { kind: `calibration-clean` } : data, ) const { outcome, counts } = enumeratedReach( spec, @@ -1726,9 +1846,7 @@ function predictSpec( for (const name of all) tests.set( name, - binding && name.startsWith(`calibration:`) - ? { kind: `calibration-clean` } - : data, + boundTest(spec, name) ? { kind: `calibration-clean` } : data, ) return { loads: true, @@ -1750,10 +1868,11 @@ function predictSpec( } // The source changes how the premise is chosen, never how a mutant verdict ranks. const calibrate = ( - name: string, - kind: SyntheticSpec[`calibrations`][number][`kind`], - difference?: CalibrationDifference, + calibration: SyntheticSpec[`calibrations`][number], ): Outcome => { + const { name, kind, difference } = calibration + const binding = bindingOf(spec, calibration) + const selected = binding && selectedDeclaration(binding) const chosen = binding ? authoredInput(spec, `calibration: ${name}`, binding, builds) : undefined @@ -1828,7 +1947,12 @@ function predictSpec( // 2026-11-03). A campaign runs its constructed histories first, in order. const campaign = (test: string): Outcome => { for (const [at, history] of ran(test).entries()) { - const { cause: _shrinkable, ...verdict } = judged(spec, history) + const { + cause: _shrinkableCause, + causeMessage: _shrinkableCauseMessage, + checkpoint: _shrinkableCheckpoint, + ...verdict + } = judged(spec, history) const construction = spec.constructed[at] // Without a budget, a constructed history's failure shrinks to the // end. No smaller history is a constructed one, so a missed state @@ -1882,8 +2006,8 @@ function predictSpec( ) } } - for (const { name, kind, difference } of spec.calibrations) - tests.set(`calibration: ${name}`, calibrate(name, kind, difference)) + for (const calibration of spec.calibrations) + tests.set(`calibration: ${calibration.name}`, calibrate(calibration)) // A forced-probe calibration needs a candidate whose clean trace passes and // whose prediction differs from the forced probe; the check must then report // reach agreement. @@ -2032,7 +2156,30 @@ function owedRoute(name: string, replay = false): string { // The self-oracle's own reach: states of the framework that its laws are // about. Each fixed history names the state it witnesses, so every one is reached on // every run; campaigns count them. +const round3FrameworkStates = [ + `bound-fixed-own-metadata-authored`, + `bound-fixed-own-metadata-forced-entered`, + `bound-fixed-own-metadata-forced-not-entered`, + `bound-fixed-inherited-metadata-authored`, + `bound-fixed-inherited-metadata-forced-entered`, + `bound-fixed-inherited-metadata-forced-not-entered`, + `bound-authored-distinct-selectors`, + `bound-authored-search-neighbor`, + `bound-mutant-violation-before-spin`, + `legacy-mutant-violation-before-spin`, + `bound-enumerated-label-rejected`, + `enumerated-reach-key-order-neighbor`, + `bound-registration-warning-silence`, + `receiving-run`, + `receiving-seedless`, + `receiving-replay`, + `receiving-rerun`, + `receiving-reach-alone`, + `receiving-binding-reordered`, + `receiving-registration-warnings`, +] as const const frameworkStates = [ + ...round3FrameworkStates, `auxiliary-late-error`, `bound-enumerated-label-selector`, `bound-control-fixed-entered`, @@ -2138,6 +2285,20 @@ function frameworkReach( expected: ReturnType, ): Iterable { const reached = new Set() + // Count the real host receiving cut, not a plan flag or a derived summary. + // Registration-only children owe a loaded test inventory; every other cut + // additionally owes a test that actually ran in that child. + for (const { checkpoint, value } of trace) { + if (checkpoint !== `child`) continue + const child = value as ChildRun + if ( + child.loads.some(({ errors }) => errors.length === 0) && + child.tests.length > 0 && + (child.context.kind === `registration-warnings` || + child.tests.some(({ state }) => state !== `skipped`)) + ) + reached.add(`receiving-${child.context.kind}`) + } if ( trace .filter(({ checkpoint }) => checkpoint === `child`) @@ -2166,15 +2327,45 @@ function frameworkReach( SuiteRun | undefined for (const spec of plan.specs) { const owed = expected.get(spec.id)! + const reachOnly = trace.find( + ({ checkpoint, value }) => + checkpoint === `reach-alone` && (value as ReachAlone).id === spec.id, + )?.value as ReachAlone | undefined + if ( + spec.enumeratedStates && + declaredOf(spec)!.join(`,`) !== [...declaredOf(spec)!].sort().join(`,`) && + reachOnly?.state === `passed` + ) + reached.add(`enumerated-reach-key-order-neighbor`) + const file = run?.files[spec.id] + const ranCalibration = (name: string) => + file?.loaded && + file.tests.some( + (test) => test.name === name && test.state !== `skipped`, + ) && + run?.log.some( + (entry) => + entry.id === spec.id && entry.test === `${spec.id} > ${name}`, + ) if ( - run?.files[spec.id] && + file && spec.source.kind === `enumerated` && - spec.calibrationHistory && - `enumerated` in spec.calibrationHistory && - typeof spec.calibrationHistory.enumerated === `string` && + bindingsOf(spec).some( + (binding) => + `enumerated` in binding && typeof binding.enumerated === `string`, + ) && [`shell`, `default`].includes(spec.labelStyle ?? ``) - ) - reached.add(`bound-enumerated-label-selector`) + ) { + if ( + owed.loads && + spec.calibrations.some(({ name }) => + ranCalibration(`calibration: ${name}`), + ) + ) + reached.add(`bound-enumerated-label-selector`) + else if (!owed.loads && file.loaded === false) + reached.add(`bound-enumerated-label-rejected`) + } if (!owed.loads) { if ( run?.files[spec.id]?.loaded === false && @@ -2193,7 +2384,7 @@ function frameworkReach( ) reached.add(`bound-null-selector`) reached.add(`registration-error`) - if (spec.calibrationHistory) + if (bindingsOf(spec).length) reached.add(`bound-calibration-registration-error`) if (owed.shapeError) reached.add(`enumerated-shape-error`) if (owed.reservedLabel) reached.add(`reserved-label-error`) @@ -2213,6 +2404,58 @@ function frameworkReach( } continue } + for (const declaration of spec.fixed) { + if (!declaration.buildMetadata) continue + const prefix = `bound-fixed-${declaration.buildMetadata}-metadata` + if ( + spec.calibrations.some((calibration) => { + const binding = bindingOf(spec, calibration) + return ( + binding && + `fixed` in binding && + binding.fixed === declaration.name && + ranCalibration(`calibration: ${calibration.name}`) + ) + }) + ) + reached.add(`${prefix}-authored`) + if (spec.probeBinding === `explicit`) + for (const [at, name] of forcedCalibrationNames(spec).entries()) { + const binding = selectedControlBinding(spec, at === 0) + if ( + `fixed` in binding && + binding.fixed === declaration.name && + ranCalibration(name) + ) + reached.add( + `${prefix}-${at === 0 ? `forced-entered` : `forced-not-entered`}`, + ) + } + } + const authored = spec.calibrations.filter(({ name }) => + ranCalibration(`calibration: ${name}`), + ) + if ( + new Set( + authored + .map((calibration) => bindingOf(spec, calibration)) + .filter(Boolean) + .map((binding) => JSON.stringify(binding)), + ).size > 1 + ) + reached.add(`bound-authored-distinct-selectors`) + if ( + authored.some((calibration) => bindingOf(spec, calibration)) && + authored.some((calibration) => !bindingOf(spec, calibration)) + ) + reached.add(`bound-authored-search-neighbor`) + for (const calibration of authored) + if ([`driver-spin`, `driver-spin-throws`].includes(calibration.kind)) + reached.add( + bindingOf(spec, calibration) + ? `bound-mutant-violation-before-spin` + : `legacy-mutant-violation-before-spin`, + ) if ( run?.files[spec.id]?.loaded && spec.probeBinding === `explicit` && @@ -2228,18 +2471,20 @@ function frameworkReach( reached.add(`bound-control-constructed-absent`) if ( run?.files[spec.id]?.loaded && - spec.calibrationHistory && - `fixed` in spec.calibrationHistory && spec.identityHistories && - spec.fixed.filter( - ({ history }) => - JSON.stringify(history) === - JSON.stringify(authoredInput(spec, ``, spec.calibrationHistory!, [])), - ).length > 1 + bindingsOf(spec).some( + (binding) => + `fixed` in binding && + spec.fixed.filter( + ({ history }) => + JSON.stringify(history) === + JSON.stringify(authoredInput(spec, ``, binding, [])), + ).length > 1, + ) ) reached.add(`bound-equal-fixed-origin`) if ( - spec.calibrationHistory && + bindingsOf(spec).length > 0 && spec.calibrations.some( ({ kind }) => kind === `throws` || kind === `clock`, ) @@ -2253,10 +2498,10 @@ function frameworkReach( reached.add(`bound-independent-premise`) if (spec.calibrationNames !== `indexed` && spec.calibrations.length > 1) reached.add(`bound-name-seed`) - if (spec.calibrationHistory && spec.extraReach) + if (bindingsOf(spec).length && spec.extraReach) reached.add(`bound-extra-state`) if ( - spec.calibrationHistory && + bindingsOf(spec).length > 0 && spec.calibrations.some(({ kind }) => [`corrupt-throws`, `judge-throws`, `judge-driver-throws`].includes( kind, @@ -2264,20 +2509,35 @@ function frameworkReach( ) ) reached.add(`bound-error-context`) - if (spec.warningProbe) reached.add(`registration-warning-policy`) + if (spec.warningProbe) { + reached.add(`registration-warning-policy`) + if ( + unboundCalibrationNames(spec).length === 0 && + trace.some( + ({ checkpoint, value }) => + checkpoint === `registration-warnings` && + (value as { id: string; loaded: Record }).id === + spec.id && + Object.values( + (value as { loaded: Record }).loaded, + ).every(Boolean), + ) + ) + reached.add(`bound-registration-warning-silence`) + } if ( - spec.calibrationHistory && + bindingsOf(spec).length > 0 && spec.calibrations.some(({ difference }) => difference !== undefined) ) reached.add(`bound-difference`) - if (spec.identityHistories && spec.calibrationHistory) { - if (`fixed` in spec.calibrationHistory) + if (spec.identityHistories) { + if (bindingsOf(spec).some((binding) => `fixed` in binding)) reached.add(`bound-fixed-identity`) - if (`enumerated` in spec.calibrationHistory) + if (bindingsOf(spec).some((binding) => `enumerated` in binding)) reached.add(`bound-enumerated-identity`) } if ( - spec.calibrationHistory && + bindingsOf(spec).length > 0 && spec.probeBinding && spec.probeBinding !== `empty` ) @@ -2296,15 +2556,18 @@ function frameworkReach( reached.add(`bound-control-full-reach`) const named = spec.fixed.length + spec.constructed.length > 0 if ( - spec.calibrationHistory || + bindingsOf(spec).length > 0 || (spec.probeBinding && spec.probeBinding !== `empty`) ) reached.add(`bound-calibration`) if ( spec.isolatedPlans && - spec.calibrationHistory && - `constructed` in spec.calibrationHistory && - spec.calibrations.some(({ kind }) => kind === `driver`) + spec.calibrations.some((calibration) => { + const binding = bindingOf(spec, calibration) + return ( + binding && `constructed` in binding && calibration.kind === `driver` + ) + }) ) reached.add(`bound-constructed-driver`) if (spec.arbitraryMap?.unmapper === true && named) @@ -2588,6 +2851,8 @@ type Failure = { diagnosis?: `never-reached` | `false-condition` cause?: `driver` counterexample?: string + /** The concrete reported candidate, when its finite data form shows its items. */ + counterexampleHistory?: SyntheticHistory seed?: number path?: string rerun?: string @@ -2623,6 +2888,35 @@ type SuiteRun = { /** The run's unhandled errors, by message. */ unhandled: Array } +// Every real child has a named receiving cut. The context is supplied by the +// driver, independently of the test states Vitest reports, so absent tests or +// an unexpected first error cannot authorize their own outcome. +type ChildContext = + | { kind: `run` | `seedless` } + | { kind: `binding-reordered`; id: string } + | { kind: `registration-warnings`; id: string; phases: Array } + | { kind: `replay`; id: string; target: string } + | { kind: `rerun`; id: string; target: string } + | { kind: `reach-alone`; id: string } +type ChildRun = { + context: ChildContext + exit: number | null + signal: string | null + files: Array + expectedFiles: Array + loads: Array<{ id: string; errors: Array }> + tests: Array<{ + id: string + name: string + state: string + errors: Array + }> + log: Array + builds: Array + secondaryErrors: Array<{ test: string; message: string }> + unhandled: Array + failed: boolean +} type ProbeRun = { id: string /** The probed test: a campaign for replay, a fixed or enumerated history for rerun. */ @@ -2908,6 +3202,48 @@ const statesIn = (report: Report, plan: Plan, files: Array) => ), ) +const readBuilds = (log: string): Array => + existsSync(`${log}.builds`) + ? readFileSync(`${log}.builds`, `utf8`) + .trim() + .split(`\n`) + .filter(Boolean) + .map((line) => { + const entry = JSON.parse(line) as BuildEntry + return { ...entry, test: unnested(entry.test) } + }) + // Independent files may finish in either order. Keep every build and + // each property's local order, as the invocation log does. + .sort((a, b) => a.id.localeCompare(b.id)) + : [] + +// Only semantic diagnostic facts enter the outer observation. Host stacks, +// temporary paths, elapsed times and a campaign wrapper's random seed are not +// promised failure differences. Error multiplicity remains observable. +const diagnosticFacts = ({ + kind, + law, + checkpoint, + diagnosis, + cause, + causeMessage, + counterexample, + dataPath, + dataKind, +}: Failure): Failure => ({ + kind, + ...(law === undefined ? {} : { law }), + ...(checkpoint === undefined ? {} : { checkpoint }), + ...(diagnosis === undefined ? {} : { diagnosis }), + ...(cause === undefined ? {} : { cause }), + ...(causeMessage === undefined ? {} : { causeMessage }), + ...(counterexample !== undefined && counterexampleItems(counterexample) + ? { counterexampleHistory: counterexampleItems(counterexample)! } + : {}), + ...(dataPath === undefined ? {} : { dataPath }), + ...(dataKind === undefined ? {} : { dataKind }), +}) + function driveSuite( plan: Plan, observe: (checkpoint: string, value: unknown) => void, @@ -2918,7 +3254,10 @@ function driveSuite( ) mkdirSync(directory, { recursive: true }) try { - const receive: typeof runVitest = (...args) => { + const receive = ( + context: ChildContext, + ...args: Parameters + ): Report => { const childLog = args[2].SELF_ORACLE_LOG ?? join(directory, `child-${runCount}.jsonl`) const report = runVitest( @@ -2929,7 +3268,20 @@ function driveSuite( args[4], ) observe(`child`, { - log: readLog(childLog).map(({ id, history }) => ({ id, history })), + context, + log: readLog(childLog), + builds: readBuilds(childLog), + tests: report.files + .flatMap(({ file, tests }) => { + const id = plan.specs[files.indexOf(file)]?.id ?? `unknown` + return tests.map(({ name, state, errors }) => ({ + id, + name: short(id, name), + state, + errors: errors.map((error) => diagnosticFacts(classify(error))), + })) + }) + .sort((a, b) => a.id.localeCompare(b.id)), loads: report.files.map(({ file, errors }) => ({ id: plan.specs[files.indexOf(file)]?.id ?? `unknown`, errors: errors.map(({ message }) => message ?? ``), @@ -2955,7 +3307,7 @@ function driveSuite( file.errors.length > 0 || file.tests.some(({ state }) => state === `failed`), ), - }) + } satisfies ChildRun) return report } const log = join(directory, `log.jsonl`) @@ -2967,21 +3319,14 @@ function driveSuite( ) return file }) - const report = receive(directory, files, { SELF_ORACLE_LOG: log }) + const report = receive({ kind: `run` }, directory, files, { + SELF_ORACLE_LOG: log, + }) const run: SuiteRun = { files: {}, log: readLog(log), resolutions: readResolutions(`${log}.resolutions`), - builds: existsSync(`${log}.builds`) - ? readFileSync(`${log}.builds`, `utf8`) - .trim() - .split(`\n`) - .filter(Boolean) - .map((line) => { - const entry = JSON.parse(line) as BuildEntry - return { ...entry, test: unnested(entry.test) } - }) - : [], + builds: readBuilds(log), // Independent child files finish in either order. Preserve every line // and its multiplicity, without treating their interleaving as a law. // Per-property driver observations retain their original order above. @@ -3040,8 +3385,10 @@ function driveSuite( // independent of the framework's particular seed hash. for (const [index, spec] of plan.specs.entries()) { if ( - !spec.calibrationHistory || - !(`constructed` in spec.calibrationHistory) || + !spec.calibrations.some((calibration) => { + const binding = bindingOf(spec, calibration) + return binding && `constructed` in binding + }) || !run.files[spec.id]!.loaded ) continue @@ -3061,6 +3408,7 @@ function driveSuite( `import { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\nsyntheticProperty(${JSON.stringify(reordered)})\n`, ) receive( + { kind: `binding-reordered`, id: spec.id }, directory, [file], { SELF_ORACLE_LOG: reorderedLog }, @@ -3090,6 +3438,11 @@ function driveSuite( `import { describe } from 'vitest'\nimport { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\ndescribe('first',()=>{${call}})\ndescribe('second',()=>{${call}})\n`, ) const repeated = receive( + { + kind: `registration-warnings`, + id: spec.id, + phases: [`first`, `second`], + }, directory, [file], {}, @@ -3101,6 +3454,11 @@ function driveSuite( `import { describe } from 'vitest'\nimport { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\ndescribe('collect',()=>{process.env.ORACLE_MANIFEST=${JSON.stringify(manifest)};try{${call}}finally{delete process.env.ORACLE_MANIFEST}})\ndescribe('ordinary',()=>{${call}})\n`, ) const collectThenRun = receive( + { + kind: `registration-warnings`, + id: spec.id, + phases: [`collect`, `ordinary`], + }, directory, [file], {}, @@ -3115,6 +3473,7 @@ function driveSuite( `import { syntheticProperty } from ${JSON.stringify(join(here, `oracle-property-self-oracle-fixture.ts`))}\n${call}\n`, ) const collected = receive( + { kind: `registration-warnings`, id: spec.id, phases: [] }, directory, [file], { ORACLE_MANIFEST: collectedManifest }, @@ -3165,6 +3524,7 @@ function driveSuite( if (plan.seedless) { const seedlessLog = join(directory, `seedless-${runCount}.jsonl`) receive( + { kind: `seedless` }, directory, files, { SELF_ORACLE_LOG: seedlessLog, SELF_ORACLE_RANDOM_DRAW: `0.5` }, @@ -3195,12 +3555,17 @@ function driveSuite( if (!isCampaign(campaign.name) || campaign.failure?.path === undefined) continue const replayLog = join(directory, `replay-${runCount}.jsonl`) - const replay = receive(directory, files, { - ORACLE_SEED: String(campaign.failure.seed), - ORACLE_PATH: campaign.failure.path, - ORACLE_PROPERTY: spec.id, - SELF_ORACLE_LOG: replayLog, - }) + const replay = receive( + { kind: `replay`, id: spec.id, target: campaign.name }, + directory, + files, + { + ORACLE_SEED: String(campaign.failure.seed), + ORACLE_PATH: campaign.failure.path, + ORACLE_PROPERTY: spec.id, + SELF_ORACLE_LOG: replayLog, + }, + ) const index = plan.specs.indexOf(spec) const random = replay.files .find((file) => file.file === files[index]) @@ -3226,6 +3591,7 @@ function driveSuite( if (!target) continue const index = plan.specs.indexOf(spec) const rerun = receive( + { kind: `rerun`, id: spec.id, target: target.name }, directory, [files[index]!], {}, @@ -3251,6 +3617,7 @@ function driveSuite( text.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`) const pattern = `^${[...(spec.nested ? [outerSuite] : []), spec.id, `reach`].map(escape).join(`(?: | > )`)}$` const alone = receive( + { kind: `reach-alone`, id: spec.id }, directory, [files[index]!], {}, @@ -3315,6 +3682,210 @@ const lawOfKind = (kind: string, name: string) => ? `reach` : lawOfTest(name) +const matchesOutcome = ( + outcome: Outcome, + failure: Failure | undefined, + kind: string, + campaign: boolean, +) => + kind === outcome.kind && + (outcome.law === undefined || failure?.law === outcome.law) && + (outcome.checkpoint === undefined || + failure?.checkpoint === outcome.checkpoint) && + failure?.diagnosis === outcome.diagnosis && + failure?.dataPath === outcome.dataPath && + failure?.dataKind === outcome.dataKind && + (campaign || failure?.cause === outcome.cause) && + (campaign || + (outcome.kind === `violation` + ? failure?.causeMessage === outcome.causeMessage + : outcome.causeMessage === undefined || + failure?.causeMessage === outcome.causeMessage)) + +// A concrete campaign candidate owes its own diagnostics, not those of the +// first failing history before shrinking. Missing a construction's state is +// separate from the candidate's row-value law and can have a passing trace. +const campaignCandidateOutcome = ( + spec: SyntheticSpec, + concrete: SyntheticHistory, + campaign: Outcome, +): Outcome => { + const precise = dataOutcome(spec) ?? judged(spec, concrete) + return campaign.kind === `constructed-reach` && precise.kind === `pass` + ? campaign + : precise +} + +// The same independent synthetic model judges every receiving child. The +// context determines selection, never the reported state. Full registration +// remains owed when a filter or path replay makes most tests stand aside. +function childExpectation( + plan: Plan, + child: ChildRun, + expected: ReturnType, +) { + const context = child.context + const selectedSpecs = + `id` in context + ? plan.specs.filter( + ({ id }) => id === context.id || context.kind === `replay`, + ) + : plan.specs + let failed = false + for (const original of selectedSpecs) { + const spec: SyntheticSpec = + context.kind === `binding-reordered` + ? { + ...original, + calibrations: [ + { name: `unrelated calibration`, kind: `right` }, + ...[...original.calibrations].reverse(), + ], + } + : original + const prediction = predictSpec(spec, child.log, child.builds) + if (!prediction.loads) failed = true + const names = prediction.loads ? [...prediction.tests.keys()] : [] + const owedNames = + context.kind === `registration-warnings` && context.phases.length + ? context.phases.flatMap((phase) => + names.map((name) => `${phase} > ${name}`), + ) + : names + const seen = child.tests.filter(({ id }) => id === spec.id) + const actualNames = seen.map(({ name }) => name).sort() + if (JSON.stringify(actualNames) !== JSON.stringify([...owedNames].sort())) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} omitted or duplicated registrations`, + `registration`, + { actual: actualNames, expected: [...owedNames].sort() }, + ), + } + if (!prediction.loads) continue + for (const test of seen) { + const selected = + context.kind === `registration-warnings` + ? false + : context.kind === `binding-reordered` + ? test.name.startsWith(`calibration:`) + : context.kind === `replay` + ? spec.id === context.id && test.name === `random seed` + : context.kind === `rerun` + ? test.name === context.target + : context.kind === `reach-alone` + ? test.name === `reach` + : true + if (!selected) { + if (test.state !== `skipped` || test.errors.length !== 0) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} unselected test ${JSON.stringify(test.name)} did not stand aside`, + `registration`, + ), + } + continue + } + const originalPrediction = expected.get(spec.id)! + if (!originalPrediction.loads) + return { + failure: violation( + `child`, + `${spec.id} ran despite its rejected declaration`, + `registration`, + ), + } + const ordinary = + context.kind === `replay` + ? originalPrediction.tests.get(context.target)! + : prediction.tests.get(test.name)! + const outcome = + context.kind === `reach-alone` + ? reachAloneOutcome(spec, ordinary) + : ordinary + if (outcome.kind !== `pass`) failed = true + const failure = test.errors[0] + const kind = test.state === `passed` ? `pass` : (failure?.kind ?? `other`) + const owesFailure = outcome.kind !== `pass` + if ( + test.state !== (owesFailure ? `failed` : `passed`) || + test.errors.length !== (owesFailure ? 1 : 0) || + !matchesOutcome(outcome, failure, kind, isCampaign(test.name)) + ) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} ${JSON.stringify(test.name)} ${test.state} as ${JSON.stringify(test.errors)}, owed ${JSON.stringify(outcome)}`, + lawOfKind(outcome.kind === `pass` ? kind : outcome.kind, test.name), + { actual: test.errors, expected: outcome }, + ), + } + if (isCampaign(test.name) && failure) { + const calls = child.log.filter( + ({ id, test: name }) => + id === spec.id && name === `${spec.id} > ${test.name}`, + ) + // A path replay runs its one requested concrete history directly. + // The input-side log also supplies its items when an Error history's + // inspection cannot be read back as an array of integers. + if (context.kind === `replay` && calls.length !== 1) + return { + failure: violation( + `child`, + `${spec.id} replay ran ${calls.length} histories, owed exactly one`, + `replay`, + ), + } + const concrete = + context.kind === `replay` + ? calls[0]!.history + : failure.counterexampleHistory + if ( + context.kind === `replay` && + failure.counterexampleHistory && + JSON.stringify(failure.counterexampleHistory) !== + JSON.stringify(concrete) + ) + return { + failure: violation( + `child`, + `${spec.id} replay reported a different concrete candidate than it ran`, + `replay`, + ), + } + if (concrete) { + if ( + !calls.some( + ({ history }) => + JSON.stringify(history) === JSON.stringify(concrete), + ) + ) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} reported a candidate the driver never ran`, + `shrink`, + ), + } + const diagnostic = campaignCandidateOutcome(spec, concrete, outcome) + if (!matchesOutcome(diagnostic, failure, kind, false)) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} campaign lost its concrete candidate's diagnostic facts: ${JSON.stringify(failure)}, owed ${JSON.stringify(diagnostic)} for ${JSON.stringify(concrete)}`, + context.kind === `replay` ? `replay` : `shrink`, + { actual: failure, expected: diagnostic }, + ), + } + } + } + } + } + return { failed } +} + function checkSuite( plan: Plan, trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, @@ -3352,17 +3923,7 @@ function checkSuite( // disappear merely because a reporter managed to write a partial result. const lateMessage = `probe synthetic#saw-0 was called after its history ended. The driver must await the work it starts, so every probe call belongs to one history.` for (const { value } of children) { - const child = value as { - exit: number | null - signal: string | null - files: Array - expectedFiles: Array - unhandled: Array - failed: boolean - secondaryErrors: Array<{ test: string; message: string }> - log: Array> - loads: Array<{ id: string; errors: Array }> - } + const child = value as ChildRun if (child.secondaryErrors.length) return violation( `child`, @@ -3385,6 +3946,8 @@ function checkSuite( `registration`, ) } + const modeled = childExpectation(plan, child, expected) + if (modeled.failure) return modeled.failure const owedLate = lateCalls(plan, child.log) const actualLate = child.unhandled.filter( (message) => message === lateMessage, @@ -3408,7 +3971,7 @@ function checkSuite( ) if ( child.signal !== null || - child.exit !== (child.failed || child.unhandled.length ? 1 : 0) || + child.exit !== (modeled.failed || child.unhandled.length ? 1 : 0) || JSON.stringify([...child.files].sort()) !== JSON.stringify([...child.expectedFiles].sort()) ) @@ -3431,7 +3994,7 @@ function checkSuite( const seen = run.files[spec.id]! if (!owed.loads) { if ( - (spec.calibrationHistory || + (bindingsOf(spec).length || (spec.probeBinding && spec.probeBinding !== `empty`)) && run.log.some((entry) => entry.id === spec.id) ) @@ -3504,9 +4067,11 @@ function checkSuite( } if ( spec.calibrationNames !== `indexed` && - spec.calibrationHistory && - `constructed` in spec.calibrationHistory && - spec.calibrations.length > 1 + spec.calibrations.length > 1 && + spec.calibrations.every((calibration) => { + const binding = bindingOf(spec, calibration) + return binding && `constructed` in binding + }) ) { const inputs = spec.calibrations.map( ({ name }) => @@ -3595,12 +4160,13 @@ function checkSuite( warning.startsWith(`${spec.id}:`) && warning.includes(`history binding`), ) + const expectedWarnings = unboundCalibrationNames(spec).length ? 1 : 0 if ( !warningProbe.loaded.repeated || !warningProbe.loaded.collectThenRun || !warningProbe.loaded.collected || - forId(warningProbe.repeated).length !== 1 || - forId(warningProbe.collectThenRun).length !== 1 || + forId(warningProbe.repeated).length !== expectedWarnings || + forId(warningProbe.collectThenRun).length !== expectedWarnings || forId(warningProbe.collected).length !== 0 ) return violation( @@ -3611,16 +4177,7 @@ function checkSuite( } // The compatibility route must be visible before any selected test runs, // including automatic controls; bound calibrations must never warn search. - const unbound = [ - ...(!spec.calibrationHistory - ? spec.calibrations.map(({ name }) => name) - : []), - ...(!spec.probeBinding || spec.probeBinding === `empty` - ? forcedCalibrationNames(spec).map((name) => - name.slice(`calibration: `.length), - ) - : []), - ] + const unbound = unboundCalibrationNames(spec) const warnings = run.warnings.filter( (warning) => warning.startsWith(`${spec.id}:`) && @@ -3738,7 +4295,8 @@ function checkSuite( const calibration = spec.calibrations.find( ({ name }) => test.name === `calibration: ${name}`, ) - if (spec.calibrationHistory && calibration) { + const calibrationBinding = calibration && bindingOf(spec, calibration) + if (calibrationBinding) { const calls = run.log.filter( (entry) => entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, @@ -3750,8 +4308,8 @@ function checkSuite( ? 2 : 1 const expectedConstruction = - `constructed` in spec.calibrationHistory - ? spec.calibrationHistory.constructed + `constructed` in calibrationBinding + ? calibrationBinding.constructed : undefined const expectedBuilds = expectedConstruction === undefined ? 0 : expectedCalls @@ -3788,7 +4346,7 @@ function checkSuite( `${spec.id} "${test.name}" used ${calls.length} driver calls; binding owes ${expectedCalls}`, `calibration`, ) - const binding = spec.calibrationHistory + const binding = calibrationBinding const first = calls[0]!.history const fullInput = authoredInput(spec, test.name, binding, run.builds) if ( @@ -3836,8 +4394,7 @@ function checkSuite( ) } if ( - spec.calibrationHistory && - calibration && + calibrationBinding && test.state === `failed` && ([`corrupt-throws`, `judge-throws`, `judge-driver-throws`].includes( calibration.kind, @@ -3845,9 +4402,9 @@ function checkSuite( (spec.odd && dataOutcome(spec))) ) { const selectedKind = - `constructed` in spec.calibrationHistory + `constructed` in calibrationBinding ? `constructed` - : `fixed` in spec.calibrationHistory + : `fixed` in calibrationBinding ? `fixed` : `enumerated` const invalidData = dataOutcome(spec) @@ -3985,7 +4542,7 @@ function checkSuite( } } if ( - (spec.calibrationHistory || + (bindingsOf(spec).length || (spec.probeBinding && spec.probeBinding !== `empty`)) && run.stdout .split(`\n`) @@ -4095,7 +4652,12 @@ function checkSuite( { actual: alone ?? null, expected: outcome }, ) const counts = outcome.kind === `pass` && owed.reach.get(`enumerated`) - if (counts && JSON.stringify(alone.reach) !== JSON.stringify(counts)) + if ( + counts && + JSON.stringify( + Object.entries((alone.reach ?? {}) as Record).sort(), + ) !== JSON.stringify(Object.entries(counts).sort()) + ) return violation( `reach-alone`, `${spec.id}: its reach test run alone printed ${JSON.stringify(alone.reach ?? null)}, owed ${JSON.stringify(counts)}`, @@ -4155,7 +4717,66 @@ function checkSuite( `${probe.id}: reproduced ${JSON.stringify(probe.failure)}, first failure ${JSON.stringify(probe.original)}`, checkpoint, ) - if (checkpoint !== `replay`) continue + if (checkpoint === `rerun`) { + const prediction = expected.get(probe.id as SyntheticSpec[`id`])! + if (!prediction.loads) + return violation( + checkpoint, + `${probe.id}: rerun of a rejected declaration`, + `registration`, + ) + const outcome = prediction.tests.get(probe.target)! + if (!matchesOutcome(outcome, probe.failure, probe.failure.kind, false)) + return violation( + checkpoint, + `${probe.id}: rerun lost its modeled diagnostic facts: ${JSON.stringify(probe.failure)}, owed ${JSON.stringify(outcome)}`, + `rerun`, + { actual: diagnosticFacts(probe.failure), expected: outcome }, + ) + continue + } + const spec = plan.specs.find(({ id }) => id === probe.id)! + const prediction = expected.get(spec.id)! + if (!prediction.loads) + return violation( + checkpoint, + `${probe.id}: replay of a rejected declaration`, + `registration`, + ) + const receiving = children + .map(({ value }) => value as ChildRun) + .find( + ({ context }) => + context.kind === `replay` && + context.id === probe.id && + context.target === probe.target, + ) + const concreteInputs = receiving?.log + .filter( + ({ id, test }) => id === spec.id && test === `${spec.id} > random seed`, + ) + .map(({ history }) => history) + if ( + concreteInputs?.length !== 1 || + JSON.stringify(probe.ran) !== JSON.stringify(concreteInputs) + ) + return violation( + checkpoint, + `${probe.id}: replay summary omitted or changed its one received concrete history`, + `replay`, + ) + const diagnostic = campaignCandidateOutcome( + spec, + concreteInputs[0]!, + prediction.tests.get(probe.target)!, + ) + if (!matchesOutcome(diagnostic, probe.failure, probe.failure.kind, false)) + return violation( + checkpoint, + `${probe.id}: replay summary lost its concrete candidate's diagnostic facts: ${JSON.stringify(probe.failure)}, owed ${JSON.stringify(diagnostic)}`, + `replay`, + { actual: diagnosticFacts(probe.failure), expected: diagnostic }, + ) const replayRoute = owedRoute(`random seed`, true) if (probe.routes?.some((route) => route !== replayRoute)) return violation( @@ -4258,10 +4879,109 @@ const seedlessHistory = ( ...drafts: Array ) => ({ name, enters, history: { drafts, seedless: true } }) const at = (history: SyntheticHistory) => ({ history, enters: null }) +const receivingCutStates = [ + `receiving-run`, + `receiving-seedless`, + `receiving-replay`, + `receiving-rerun`, + `receiving-reach-alone`, + `receiving-binding-reordered`, + `receiving-registration-warnings`, +] as const +const receivingCutDrafts = (tail: 1 | 2 = 1): Array => [ + generatedDraft({ + fixed: [at([3, 6])], + constructed: [ + { prefix: [0, tail], enters: [`saw-0`] }, + { prefix: [3, 6], enters: [] }, + ], + calibrationHistory: { constructed: `constructed-0` }, + calibrations: [`right`, `driver`], + isolatedPlans: true, + warningProbe: true, + }), + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [tail]] }, + enumeratedStates: [`measured`, `measured`, `measured`], + }), +] const cal = ( kind: CalibrationDraft[`kind`], difference: CalibrationDraft[`difference`] = `none`, -): CalibrationDraft => ({ kind, difference }) + history?: CalibrationDraft[`history`], +): CalibrationDraft => ({ + kind, + difference, + ...(history === undefined ? {} : { history }), +}) + +// One wrapper dimension crosses all three public selection receivers. Its +// metadata must never execute, regardless of the selected clean polarity. +const metadataDraft = ( + buildMetadata: `own` | `inherited` | undefined, + tail = 1, + receiver: + `all` | `authored` | `forced-entered` | `forced-not-entered` = `all`, +) => + generatedDraft({ + fixed: [ + { + history: [0, tail], + enters: null, + ...(buildMetadata && receiver !== `forced-entered` + ? { buildMetadata } + : {}), + }, + { + history: [tail], + enters: null, + ...(buildMetadata && + (receiver === `all` || receiver === `forced-entered`) + ? { buildMetadata } + : {}), + }, + ], + calibrationHistory: { fixed: `fixed-0` }, + calibrations: + receiver === `all` || receiver === `authored` ? [`driver`] : [], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + probe: receiver === `authored` ? `none` : `agree`, + probeBinding: receiver === `authored` ? `unbound` : `explicit`, + probeEntered: { fixed: `fixed-0` }, + probeNotEntered: { fixed: `fixed-1` }, + identityHistories: true, + }) +const mixedBindingsDraft = (second: `fixed` | `constructed`) => + generatedDraft({ + fixed: [at([0]), at([1])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + // The last calibration explicitly opts out of this property default. + calibrationHistory: { fixed: `fixed-1` }, + calibrations: [ + cal(`driver`, `none`, { fixed: `fixed-0` }), + cal( + `driver`, + `none`, + second === `fixed` + ? { fixed: `fixed-1` } + : { constructed: `constructed-0` }, + ), + cal(`driver`, `none`, null), + ], + isolatedPlans: true, + }) +const spinDraft = ( + kind: `driver-spin` | `driver-spin-throws`, + bound: boolean, + tail = 1, +) => + generatedDraft({ + fixed: [at([0, tail])], + calibrations: [ + cal(kind, `text match`, bound ? { fixed: `fixed-0` } : null), + ], + identityHistories: true, + }) oracleProperty({ id: `self-oracle.framework`, @@ -4273,6 +4993,7 @@ oracleProperty({ histories: { arbitrary: suite, states: [ + ...round3FrameworkStates, `auxiliary-late-error`, `bound-enumerated-label-selector`, `bound-control-fixed-entered`, @@ -4300,6 +5021,7 @@ oracleProperty({ measured: frameworkStates.filter( (state) => state !== `auxiliary-late-error` && + !round3FrameworkStates.some((required) => required === state) && !state.startsWith(`bound-`) && state !== `unbound-prototype-probe` && state !== `registration-warning-policy`, @@ -4307,6 +5029,93 @@ oracleProperty({ runs: 3, fixedSeeds: [1], constructed: [ + { + name: `round3 every actual receiving cut`, + enters: [...receivingCutStates], + build: (generated: Suite) => ({ + ...generated, + seedless: true, + drafts: receivingCutDrafts(generated.drafts[0]!.clock ? 1 : 2), + }), + }, + { + name: `round3 replayed clock failure retains driver cause`, + enters: [`receiving-replay`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + clock: true, + throwOnSpin: true, + constructed: [{ prefix: [7], enters: [] }], + }), + ], + }), + }, + { + name: `round3 named reach counts permit key reorder`, + enters: [`enumerated-reach-key-order-neighbor`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [0, 1], [2]] }, + enumeratedStates: generated.drafts[0]!.clock + ? [`measured`, `measured`, `required`] + : [`measured`, `required`, `measured`], + }), + ], + }), + }, + ...([`own`, `inherited`] as const).flatMap((buildMetadata) => + ([`authored`, `forced-entered`, `forced-not-entered`] as const).map( + (receiver) => ({ + name: `round3 ${buildMetadata} fixed metadata ${receiver}`, + enters: [`bound-fixed-${buildMetadata}-metadata-${receiver}`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + metadataDraft( + buildMetadata, + generated.drafts[0]!.clock ? 1 : 2, + receiver, + ), + ], + }), + }), + ), + ), + { + name: `round3 independently selected authored calibrations`, + enters: [ + `bound-authored-distinct-selectors`, + `bound-authored-search-neighbor`, + ], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + mixedBindingsDraft( + generated.drafts[0]!.clock ? `fixed` : `constructed`, + ), + ], + }), + }, + { + name: `round3 recorded violation before mutant clock spin`, + enters: [ + `bound-mutant-violation-before-spin`, + `legacy-mutant-violation-before-spin`, + ], + build: (generated: Suite) => { + const kind = generated.drafts[0]!.clock + ? `driver-spin` + : `driver-spin-throws` + return { + ...generated, + drafts: [spinDraft(kind, true), spinDraft(kind, false)], + } + }, + }, { name: `review auxiliary late errors`, enters: [`auxiliary-late-error`], @@ -4325,6 +5134,25 @@ oracleProperty({ { name: `review enumerated selector labels`, enters: [`bound-enumerated-label-selector`], + build: (generated: Suite) => ({ + ...generated, + drafts: [ + generatedDraft({ + source: { kind: `enumerated`, histories: [[0, 1]] }, + labelStyle: generated.drafts[0]!.clock ? `shell` : `default`, + calibrationHistory: { + enumerated: generated.drafts[0]!.clock + ? `items 0 | 1 ${shellText}` + : `[0,1]`, + }, + calibrations: [`right`], + }), + ], + }), + }, + { + name: `round3 rejected named enumeration selector`, + enters: [`bound-enumerated-label-rejected`], build: (generated: Suite) => ({ ...generated, drafts: [ @@ -4520,7 +5348,10 @@ oracleProperty({ }, { name: `round2 registration warning phases`, - enters: [`registration-warning-policy`], + enters: [ + `registration-warning-policy`, + `bound-registration-warning-silence`, + ], build: (generated: Suite) => ({ ...generated, drafts: [ @@ -4531,6 +5362,12 @@ oracleProperty({ warningProbe: true, calibrations: [`right`, `driver`], }), + generatedDraft({ + fixed: [at([0, generated.drafts[0]!.clock ? 1 : 2])], + calibrationHistory: { fixed: `fixed-0` }, + warningProbe: true, + calibrations: [`right`, `driver`], + }), ], }), }, @@ -4725,6 +5562,76 @@ oracleProperty({ }, ], fixed: [ + seedlessHistory( + `round3 every actual receiving cut`, + [...receivingCutStates], + ...receivingCutDrafts(), + ), + fixedHistory( + `round3 replayed clock failure retains driver cause`, + [`receiving-replay`], + generatedDraft({ + clock: true, + throwOnSpin: true, + constructed: [{ prefix: [7], enters: [] }], + }), + ), + fixedHistory( + `round3 named reach count key order neighbor`, + [`enumerated-reach-key-order-neighbor`], + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [0, 1], [2]] }, + enumeratedStates: [`measured`, `measured`, `required`], + }), + ), + fixedHistory( + `round3 fully bound warning phases stay silent`, + [`bound-registration-warning-silence`], + generatedDraft({ + fixed: [at([0])], + calibrationHistory: { fixed: `fixed-0` }, + warningProbe: true, + calibrations: [`right`, `driver`], + }), + ), + fixedHistory( + `round3 plain fixed declaration metadata neighbor`, + [], + metadataDraft(undefined), + ), + ...([`own`, `inherited`] as const).flatMap((buildMetadata) => + ([`authored`, `forced-entered`, `forced-not-entered`] as const).map( + (receiver) => + fixedHistory( + `round3 ${buildMetadata} fixed metadata ${receiver} receiver`, + [`bound-fixed-${buildMetadata}-metadata-${receiver}`], + metadataDraft(buildMetadata, 1, receiver), + ), + ), + ), + ...([`fixed`, `constructed`] as const).map((second) => + fixedHistory( + `round3 distinct ${second} and unbound authored selectors`, + [ + `bound-authored-distinct-selectors`, + `bound-authored-search-neighbor`, + ], + mixedBindingsDraft(second), + ), + ), + ...([`driver-spin`, `driver-spin-throws`] as const).flatMap((kind) => + [true, false].map((bound) => + fixedHistory( + `round3 ${kind} after recorded violation ${bound ? `bound` : `legacy`} receiver`, + [ + bound + ? `bound-mutant-violation-before-spin` + : `legacy-mutant-violation-before-spin`, + ], + spinDraft(kind, bound), + ), + ), + ), fixedHistory( `review auxiliary late errors preserve counts`, [`auxiliary-late-error`], @@ -4739,7 +5646,11 @@ oracleProperty({ [true, false].map((valid) => fixedHistory( `review ${labelStyle} enumerated selector ${valid ? `exact` : `missing`}`, - [`bound-enumerated-label-selector`], + [ + valid + ? `bound-enumerated-label-selector` + : `bound-enumerated-label-rejected`, + ], generatedDraft({ source: { kind: `enumerated`, histories: [[0, 1]] }, labelStyle, @@ -6093,6 +7004,143 @@ oracleProperty({ driver: driveSuite, check: checkSuite, calibrations: [ + { + name: `reject replay summary cause loss`, + history: { fixed: `round3 replayed clock failure retains driver cause` }, + law: `replay`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const probe = altered.find(({ checkpoint }) => checkpoint === `replay`)! + .value as ProbeRun + if (probe.failure?.cause !== `driver`) return undefined + delete probe.failure.cause + delete probe.failure.causeMessage + return altered + }, + }, + { + name: `reject rerun wrong cause ordinal`, + history: { + fixed: `a driver error after a violation is the violation's cause`, + }, + law: `verdict`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `rerun`)! + const test = child.tests.find(({ state }) => state === `failed`)! + test.errors[0]!.causeMessage = `synthetic driver failed at item 2` + return altered + }, + }, + { + name: `reject replay concrete cause loss`, + history: { fixed: `round3 replayed clock failure retains driver cause` }, + law: `replay`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `replay`)! + const test = child.tests.find(({ state }) => state === `failed`)! + if (test.errors[0]?.cause !== `driver`) return undefined + delete test.errors[0].cause + delete test.errors[0].causeMessage + return altered + }, + }, + ...[`omit`, `duplicate`].map((mutation) => ({ + name: `reject ${mutation} auxiliary registration`, + history: { fixed: `a replayed campaign failure reproduces its path` }, + law: `registration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `replay`)! + const omittedAt = child.tests.findIndex( + ({ state }) => state === `skipped`, + ) + if (mutation === `omit`) child.tests.splice(omittedAt, 1) + else child.tests.push(structuredClone(child.tests[omittedAt]!)) + return altered + }, + })), + { + name: `reject seedless first primary error`, + history: { + fixed: `an ordinary run's random campaign runs under random seed`, + }, + law: `determinism`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `seedless`)! + const test = child.tests.find(({ name }) => name === `determinism`)! + test.state = `failed` + test.errors = [{ kind: `other` }] + child.failed = true + child.exit = 1 + return altered + }, + }, + { + name: `reject rerun recorder cause loss`, + history: { + fixed: `a driver error after a violation is the violation's cause`, + }, + law: `verdict`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `rerun`)! + const test = child.tests.find(({ state }) => state === `failed`)! + delete test.errors[0]!.cause + delete test.errors[0]!.causeMessage + return altered + }, + }, + { + name: `reject rerun probe cause loss`, + history: { + fixed: `a driver error after a violation is the violation's cause`, + }, + law: `rerun`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const probe = altered.find(({ checkpoint }) => checkpoint === `rerun`)! + .value as ProbeRun + delete probe.failure!.cause + delete probe.failure!.causeMessage + return altered + }, + }, + { + name: `reject changed reach-only count`, + history: { + fixed: `an enumerated source counts the histories that entered each declared state`, + }, + law: `reach`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const alone = altered.find( + ({ checkpoint }) => checkpoint === `reach-alone`, + )!.value as ReachAlone + const counts = alone.reach as Record + counts[`saw-0`] = (counts[`saw-0`] ?? 0) + 1 + return altered + }, + }, { name: `reject missing child observation`, history: { constructed: `round2 ranked mutant verdicts` }, diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md index 7ff55ef..bc0120b 100644 --- a/reviews/calibration-history-self-oracle.md +++ b/reviews/calibration-history-self-oracle.md @@ -156,3 +156,76 @@ Parent commit: `2dc95d4e838678a03eb345c08f141dbd20a8c4a7`. - `packages/oracles/tests/oracle-property-self-oracle.test.ts`: SHA256 `c8df3de2569460a80ff1eab23811d21f9ac6fffe6e4d935bdcfdc5ed5f579d0f`. - `packages/oracles/tests/oracle-property-self-oracle-fixture.ts`: SHA256 `27195bf184b7ae58f9eaea7bab82973aae5876ce292e2b6aa7e6fa02ac20627d`. + +## Fresh review and oracle-first repair from fc4bad8 + +Starting reviewed head: `fc4bad8dc2c3da3a80b5002bcc36281b75f12846`. +The previous full-head CI passed both Vitest versions. A fresh independent +round nevertheless demonstrated one production defect and six oracle defects. +The repair begins with all seven findings together, before changing production. + +The public binding law preserves an explicitly selected original and gives each +authored calibration its own selector. Incidental declaration metadata does not +change a fixed history into a construction. The verdict law gives an earlier +recorded violation precedence over a later controlled-clock spin, including a +supplied mutant driver. The receiving law requires full registration, expected +test outcomes, faithful diagnostic causes, and named reach counts independent +of their print order. README and the production header state these promises. +No product-law change is proposed. + +The primary oracle repair adds declaration-wrapper metadata, independent +per-calibration binding, and ordered mutant-driver events. Required real Vitest +receivers include authored selection and both automatic clean-premise paths, +mixed authored bindings, and bound/legacy wrong-observation-before-spin. Full +child records must carry every registered name and independently judged primary +outcome through normal, seedless, replay, rerun and other auxiliary cuts. A +rerun owes diagnostic facts from its concrete reproduced history; shrinking may +legitimately remove an unrelated throw. Count dictionaries permit key reordering +while retaining exact names and values. + +Pre-edit production identity is SHA256 +`4d4c9ac25a83ebc49e1604519176418f1618c5f70bf9b9231e555bd2b5e8ee39` +for `packages/oracles/src/oracle-property.ts`. Required receiving RED and +neighbor results must be recorded before its source-kind dispatch is repaired. +Hostile variants then challenge each new distinction. This section is the +pre-edit inventory, not a claim of completed validation. The historical CHRON-1 +entry above remains unchanged. + +### Pre-edit gate at fc4bad8 + +All oracle changes preceded the production repair. The final combined command +used installed Node Vitest from `packages/oracles`, framework config, +`--maxWorkers=2`, and selector +`fixed history: round3|calibration: reject (omit|duplicate|seedless first|rerun|replay|changed reach)`. +On unchanged production it reports **six intended RED and twenty GREEN** out of +253 registered checks (227 skipped), in 45.08 seconds. RED separately compares +own/inherited metadata on authored, forced-entered and forced-not-entered +receiving paths. Every failure is a calibration comparison observing lost +original identity; no setup failure supplies this gate. The plain neighbor, +mixed selectors, bound/legacy spin and throw, dictionary order, warning phases, +seven actual receiving cuts, concrete replay causes and nine corruption controls +are GREEN. Package TypeScript checking passes. + +The primary grammar separately declares each metadata receiving path, bound and +legacy wrong-observation-before-spin, mixed authored selectors/search, successful +and rejected named enumeration selectors, legal count dictionary order, silent +fully-bound warning phases, and all seven actual child receiving contexts. Full +fixed/random reach counts remain to be measured after production repair. + +Isolated source/fixture/reporter variants already reject first-selector sharing, +supplied-driver spin ranking, seedless primary errors, missing replay and +reach-only registrations, duplicate reach-only registrations, reordered-child +primary errors, lost rerun/replay causes, wrong cause ordinals and unexpected +causes. Reversing only count-key printing is accepted. These are bounded +assertion results, not full-suite mutation claims. The warning-phase extension +also caught an oracle mistake: fully-bound properties emit no migration warning, +but the old phase model demanded one. Its false RED preceded the independent +per-entry warning model repair and both bound/unbound GREEN neighbors. + +Concrete diagnostics are judged from the history actually reproduced. A +campaign's initial checkpoint or cause may change while shrinking. Replay's +input-side log supplies its one concrete history; an ordinary parseable reported +counterexample must have appeared in that test's invocation log. Exact finite +driver causes and item checkpoints are compared at that concrete cut. Nested +judgment causes and arbitrary AggregateError metadata remain outside this +synthetic diagnostic grammar; no universal error-serialization claim is made. From b09a40378742b2023433acf4915d3602045022a9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 16:55:17 -0700 Subject: [PATCH 09/14] fix: classify calibration histories by selector kind --- .changeset/calibration-history-bindings.md | 2 +- packages/oracles/src/oracle-property.ts | 27 ++++++++++++---------- reviews/calibration-history-self-oracle.md | 10 ++++++++ 3 files changed, 26 insertions(+), 13 deletions(-) diff --git a/.changeset/calibration-history-bindings.md b/.changeset/calibration-history-bindings.md index 5ea3bae..7f1490d 100644 --- a/.changeset/calibration-history-bindings.md +++ b/.changeset/calibration-history-bindings.md @@ -2,4 +2,4 @@ '@tanstack/oracles': minor --- -Add direct `history` bindings for calibrations and `probeHistories` for automatic controls, so bound calibrations validate one selected premise instead of searching. Preserve fixed and enumerated history identity, rebuild constructed histories from stable seeds, and resolve once per driver execution. Apply ordinary verdict ranking and optional difference requirements to bound mutants, validate clean declared reach, and retain selected-history error context. Retain search during migration with one warning per property outside collect-only runs, before search removal ahead of the first npm release. +Add direct `history` bindings for calibrations and `probeHistories` for automatic controls, so bound calibrations validate one selected premise instead of searching. Preserve fixed and enumerated history identity, including fixed declarations with unrelated `build` metadata, rebuild constructed histories from stable seeds, and resolve once per driver execution. Apply ordinary verdict ranking and optional difference requirements to bound mutants, validate clean declared reach, and retain selected-history error context. Retain search during migration with one warning per property outside collect-only runs, before search removal ahead of the first npm release. diff --git a/packages/oracles/src/oracle-property.ts b/packages/oracles/src/oracle-property.ts index b5aac3e..0beba8e 100644 --- a/packages/oracles/src/oracle-property.ts +++ b/packages/oracles/src/oracle-property.ts @@ -1741,25 +1741,28 @@ function selectCalibrationHistory( kind === `constructed` ? (selector as { constructed: string }).constructed : (selector as { fixed: string }).fixed - const matches = ( - kind === `constructed` ? (source.constructed ?? []) : (source.fixed ?? []) - ).filter(({ name }) => name === key) - if (typeof key !== `string` || matches.length !== 1) - return invalid( - `${kind} ${reportText(key)} must identify exactly one history (found ${matches.length})`, - ) - const selected = matches[0]! - const name = `${kind} history: ${selected.name}` - if (`build` in selected) { + const select = ( + declarations: ReadonlyArray, + ) => { + const matches = declarations.filter(({ name }) => name === key) + if (typeof key !== `string` || matches.length !== 1) + return invalid( + `${kind} ${reportText(key)} must identify exactly one history (found ${matches.length})`, + ) + return matches[0]! + } + if (kind === `constructed`) { + const selected = select(source.constructed ?? []) const seed = calibrationSeed(id, calibration.name) return { - name: `${name} (seed ${seed})`, + name: `constructed history: ${selected.name} (seed ${seed})`, enters: selected.enters, build: () => buildConstruction(id, source.arbitrary, selected, seed), } } + const selected = select(source.fixed ?? []) return { - name, + name: `fixed history: ${selected.name}`, enters: selected.enters ?? [], build: () => selected.history, } diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md index bc0120b..0b525a0 100644 --- a/reviews/calibration-history-self-oracle.md +++ b/reviews/calibration-history-self-oracle.md @@ -229,3 +229,13 @@ counterexample must have appeared in that test's invocation log. Exact finite driver causes and item checkpoints are compared at that concrete cut. Nested judgment causes and arbitrary AggregateError metadata remain outside this synthetic diagnostic grammar; no universal error-serialization claim is made. + +### Source-kind repair after the gate + +Oracle-only commit `d5e615a` precedes the production repair. Selection now +uses the selector kind to look up fixed or constructed declarations, preserving +fixed originals even when the declaration has unrelated `build` metadata. The +same gate passes **26/26** after repair (227 skipped), in 49.59 seconds. Package +TypeScript, scoped ESLint and the Vite package/declaration build pass. Full +fixed/random validation and a new independent review round follow this commit; +this focused result alone does not establish PR readiness. From 68ed5412a4acbb5520f0d3cc052b0c8ad308b192 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 17:18:59 -0700 Subject: [PATCH 10/14] test: enforce warnings and campaign budgets on every child --- .../tests/oracle-property-self-oracle.test.ts | 419 ++++++++++++++++-- reviews/calibration-history-self-oracle.md | 53 +++ 2 files changed, 438 insertions(+), 34 deletions(-) diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 75af543..81d87bc 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -90,6 +90,8 @@ * entered selectors infer a unique construction. Equal data never permits * substituting another named original. Registration warns once per * property/process without dropping tests, and collect-only stays silent. + * Every receiving child preserves that warning and each unbound name once; + * neither property order nor name order changes that promise. * Every real child must finish with a complete report and no unmodeled * unhandled errors, including replay and registration-only children. * Each authored calibration selects independently, including bound/unbound @@ -100,6 +102,11 @@ * and three generated ones, and fails for a driver whose trace changes, * compared by their data encoding: `-0` is not `0`, and key order and * Map and Set order count, which a check that iterates them can see. + * - `campaign budget`: a passing generated campaign runs its requested random + * sample count plus one example per construction, independently in every + * fixed and random lane. Child processes discard the parent's ORACLE_ + * settings, so their multiplier is one. Path replay runs exactly one + * history; failures, shrinking, and timeouts owe no fixed call count. * - `agreement`: for a probe-bound state, a history whose `predict` says it * enters the state while the history does not enter it, or does not enter * it while the history does, violates law `reach agreement` at checkpoint @@ -371,7 +378,12 @@ type Draft = { } // `seedless`: one more child run, without `ORACLE_SEED`, records the routes // of each random campaign as an ordinary run draws its seed. -type Suite = { drafts: Array; seedless: boolean } +type Suite = { + drafts: Array + seedless: boolean + /** A controlled host draw, independent of the framework's seed selection. */ + seedlessDraw?: number +} // Outside the synthetic language: an item 5, or four items. const outside = fc.oneof( @@ -756,13 +768,21 @@ const draft: fc.Arbitrary = fc.record({ { weight: 1, arbitrary: fc.constant(true) }, ), }) -const suite: fc.Arbitrary = fc.record({ - drafts: fc.array(draft, { minLength: 1, maxLength: 2 }), - seedless: fc.oneof( - { weight: 5, arbitrary: fc.constant(false) }, - { weight: 1, arbitrary: fc.constant(true) }, - ), -}) +const cleanHostDraw = 0.5000000016298145 +const suite: fc.Arbitrary = fc.record( + { + drafts: fc.array(draft, { minLength: 1, maxLength: 2 }), + seedless: fc.oneof( + { weight: 5, arbitrary: fc.constant(false) }, + { weight: 1, arbitrary: fc.constant(true) }, + ), + seedlessDraw: fc.option(fc.constantFrom(0.5, cleanHostDraw), { + nil: undefined, + freq: 8, + }), + }, + { requiredKeys: [`drafts`, `seedless`] }, +) // --------------------------------------------------------------------------- // Resolution: ids, names, and declared states @@ -774,20 +794,29 @@ const suite: fc.Arbitrary = fc.record({ // constructed history that names an undeclared state). An enumerated draft has no // constructed or fixed histories. -type Plan = { specs: Array; seedless: boolean } +type Plan = { + specs: Array + seedless: boolean + seedlessDraw?: number +} -function resolveSuite({ drafts, seedless }: Suite): Plan { +function resolveSuite({ drafts, seedless, seedlessDraw }: Suite): Plan { return { seedless, + ...(seedlessDraw === undefined ? {} : { seedlessDraw }), specs: drafts.map((d, index) => { const generated = d.source.kind === `generated` - const hasBinding = d.calibrations.some((calibration) => - typeof calibration === `string` - ? d.calibrationHistory !== null - : calibration.history === undefined + const hasBinding = + (d.probe !== `none` && + d.probeBinding !== `unbound` && + d.probeBinding !== `empty`) || + d.calibrations.some((calibration) => + typeof calibration === `string` ? d.calibrationHistory !== null - : calibration.history !== null, - ) + : calibration.history === undefined + ? d.calibrationHistory !== null + : calibration.history !== null, + ) const constructed = generated ? d.constructed.map((construction, at) => ({ name: `constructed-${d.duplicateHistoryNames && hasBinding ? 0 : at}`, @@ -1359,6 +1388,46 @@ const unboundCalibrationNames = (spec: SyntheticSpec) => [ ) : []), ] +const registeredSpec = (spec: SyntheticSpec, context: ChildContext) => + context.kind === `binding-reordered` + ? { + ...spec, + calibrations: [ + { name: `unrelated calibration`, kind: `right` as const }, + ...[...spec.calibrations].reverse(), + ], + } + : spec +const propertyWarnings = (warnings: ReadonlyArray, id: string) => + warnings.filter((warning) => warning.startsWith(`${id}:`)) +const warningNameCount = (warning: string, name: string) => { + const escaped = name.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`) + return [ + ...warning.matchAll( + new RegExp(`(?, + collectOnly: boolean, +) => { + const unbound = collectOnly ? [] : unboundCalibrationNames(spec) + const own = propertyWarnings(warnings, spec.id) + if (own.length !== (unbound.length ? 1 : 0)) return true + if (own.length === 0) return false + const names = [ + ...spec.calibrations.map(({ name }) => name), + ...forcedCalibrationNames(spec).map((name) => + name.slice(`calibration: `.length), + ), + ] + return names.some( + (name) => + warningNameCount(own[0]!, name) !== (unbound.includes(name) ? 1 : 0), + ) +} const boundTest = (spec: SyntheticSpec, name: string) => { const calibration = spec.calibrations.find( (entry) => name === `calibration: ${entry.name}`, @@ -1614,6 +1683,9 @@ function predictSpec( spec: SyntheticSpec, logged: ReadonlyArray, builds: ReadonlyArray = [], + // The full ordinary run supplies only history/name association. Filtered + // children's own logs still determine every invocation and outcome. + nameLog: ReadonlyArray = logged, ): SpecExpectation { const ran = (test: string) => logged @@ -1800,7 +1872,7 @@ function predictSpec( const names = spec.source.kind === `enumerated` ? spec.source.histories.map((history) => - enumeratedName(spec, history, logged), + enumeratedName(spec, history, nameLog), ) : [ `fixed seed ${spec.source.seed}`, @@ -1976,7 +2048,7 @@ function predictSpec( for (const history of spec.source.histories) { const verdict = verdictOf(history, spec) tests.set( - enumeratedName(spec, history, logged), + enumeratedName(spec, history, nameLog), declared && judgedKind(verdict) && reachedOf(spec, history).some((state) => !declared.includes(state)) @@ -2156,7 +2228,31 @@ function owedRoute(name: string, replay = false): string { // The self-oracle's own reach: states of the framework that its laws are // about. Each fixed history names the state it witnesses, so every one is reached on // every run; campaigns count them. +const warningReceivingKinds = [ + `run`, + `seedless`, + `replay`, + `rerun`, + `reach-alone`, + `binding-reordered`, + `registration-warnings`, +] as const +const passingBudgetLanes = [ + { context: `run`, lane: `fixed` }, + { context: `run`, lane: `random` }, + { context: `seedless`, lane: `fixed` }, + { context: `seedless`, lane: `random` }, +] as const +const round4FrameworkStates = [ + ...warningReceivingKinds.map((kind) => `warning-receiving-${kind}` as const), + ...passingBudgetLanes.map( + ({ context, lane }) => `passing-budget-${context}-${lane}` as const, + ), + `passing-budget-constructed`, + `campaign-budget-failure-neighbor`, +] as const const round3FrameworkStates = [ + ...round4FrameworkStates, `bound-fixed-own-metadata-authored`, `bound-fixed-own-metadata-forced-entered`, `bound-fixed-own-metadata-forced-not-entered`, @@ -2298,6 +2394,47 @@ function frameworkReach( child.tests.some(({ state }) => state !== `skipped`)) ) reached.add(`receiving-${child.context.kind}`) + for (const original of plan.specs) { + if ( + `id` in child.context && + child.context.id !== original.id && + child.context.kind !== `replay` + ) + continue + const spec = registeredSpec(original, child.context) + if (!predictSpec(spec, child.log, child.builds).loads) continue + if ( + unboundCalibrationNames(spec).length > 0 && + !( + child.context.kind === `registration-warnings` && + child.context.phases.length === 0 + ) && + !warningMismatch(spec, child.warnings, false) + ) + reached.add(`warning-receiving-${child.context.kind}`) + if (spec.source.kind !== `generated`) continue + for (const test of child.tests.filter( + ({ id, name }) => id === spec.id && isCampaign(name), + )) { + const calls = child.log.filter( + ({ id, test: name }) => + id === spec.id && name === `${spec.id} > ${test.name}`, + ) + if (test.state === `failed` && calls.length > 0) + reached.add(`campaign-budget-failure-neighbor`) + if ( + (child.context.kind === `run` || child.context.kind === `seedless`) && + test.state === `passed` && + calls.length === spec.source.runs + spec.constructed.length + ) { + reached.add( + `passing-budget-${child.context.kind}-${test.name === `random seed` ? `random` : `fixed`}`, + ) + if (spec.constructed.length > 0) + reached.add(`passing-budget-constructed`) + } + } + } } if ( trace @@ -2913,6 +3050,8 @@ type ChildRun = { }> log: Array builds: Array + /** Registration warnings retained for every process, including filtered runs. */ + warnings: Array secondaryErrors: Array<{ test: string; message: string }> unhandled: Array failed: boolean @@ -3271,6 +3410,8 @@ function driveSuite( context, log: readLog(childLog), builds: readBuilds(childLog), + // Independent property files may register in either order. + warnings: [...report.warnings].sort(), tests: report.files .flatMap(({ file, tests }) => { const id = plan.specs[files.indexOf(file)]?.id ?? `unknown` @@ -3527,7 +3668,10 @@ function driveSuite( { kind: `seedless` }, directory, files, - { SELF_ORACLE_LOG: seedlessLog, SELF_ORACLE_RANDOM_DRAW: `0.5` }, + { + SELF_ORACLE_LOG: seedlessLog, + SELF_ORACLE_RANDOM_DRAW: String(plan.seedlessDraw ?? 0.5), + }, undefined, false, ) @@ -3723,6 +3867,7 @@ function childExpectation( plan: Plan, child: ChildRun, expected: ReturnType, + nameLog: ReadonlyArray, ) { const context = child.context const selectedSpecs = @@ -3733,17 +3878,8 @@ function childExpectation( : plan.specs let failed = false for (const original of selectedSpecs) { - const spec: SyntheticSpec = - context.kind === `binding-reordered` - ? { - ...original, - calibrations: [ - { name: `unrelated calibration`, kind: `right` }, - ...[...original.calibrations].reverse(), - ], - } - : original - const prediction = predictSpec(spec, child.log, child.builds) + const spec = registeredSpec(original, context) + const prediction = predictSpec(spec, child.log, child.builds, nameLog) if (!prediction.loads) failed = true const names = prediction.loads ? [...prediction.tests.keys()] : [] const owedNames = @@ -3764,6 +3900,16 @@ function childExpectation( ), } if (!prediction.loads) continue + const collectOnly = + context.kind === `registration-warnings` && context.phases.length === 0 + if (warningMismatch(spec, child.warnings, collectOnly)) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} lost its once-per-process unbound calibration warning or names: ${JSON.stringify(propertyWarnings(child.warnings, spec.id))}`, + `calibration`, + ), + } for (const test of seen) { const selected = context.kind === `registration-warnings` @@ -3822,6 +3968,29 @@ function childExpectation( { actual: test.errors, expected: outcome }, ), } + if ( + spec.source.kind === `generated` && + isCampaign(test.name) && + outcome.kind === `pass` + ) { + const actualCalls = child.log.filter( + ({ id, test: name }) => + id === spec.id && name === `${spec.id} > ${test.name}`, + ).length + const owedCalls = + context.kind === `replay` + ? 1 + : spec.source.runs + spec.constructed.length + if (actualCalls !== owedCalls) + return { + failure: violation( + `child`, + `${spec.id} ${context.kind} ${JSON.stringify(test.name)} passed after ${actualCalls} driver calls, owed ${owedCalls} from its declared campaign budget`, + `campaign budget`, + { actual: actualCalls, expected: owedCalls }, + ), + } + } if (isCampaign(test.name) && failure) { const calls = child.log.filter( ({ id, test: name }) => @@ -3946,7 +4115,7 @@ function checkSuite( `registration`, ) } - const modeled = childExpectation(plan, child, expected) + const modeled = childExpectation(plan, child, expected, run.log) if (modeled.failure) return modeled.failure const owedLate = lateCalls(plan, child.log) const actualLate = child.unhandled.filter( @@ -4896,15 +5065,39 @@ const receivingCutDrafts = (tail: 1 | 2 = 1): Array => [ { prefix: [3, 6], enters: [] }, ], calibrationHistory: { constructed: `constructed-0` }, - calibrations: [`right`, `driver`], + calibrations: [ + `right`, + { kind: `driver`, difference: `none`, history: null }, + ], isolatedPlans: true, warningProbe: true, }), generatedDraft({ source: { kind: `enumerated`, histories: [[0], [tail]] }, enumeratedStates: [`measured`, `measured`, `measured`], + calibrations: [`right`, `driver`], }), ] +const campaignBudgetSuite = ( + runs: 2 | 3 = 2, + constructions: 0 | 1 | 2 = 1, + tail: 1 | 2 = 1, +): Suite => ({ + seedless: true, + seedlessDraw: cleanHostDraw, + drafts: [ + generatedDraft({ + source: { kind: `generated`, runs, seed: 7 }, + constructed: [ + { prefix: [0, tail, 1], enters: [`saw-0`] }, + { prefix: [tail, 0, tail], enters: [`saw-0`] }, + ].slice(0, constructions), + }), + generatedDraft({ + constructed: [{ prefix: [3, 6], enters: [] }], + }), + ], +}) const cal = ( kind: CalibrationDraft[`kind`], difference: CalibrationDraft[`difference`] = `none`, @@ -5031,13 +5224,34 @@ oracleProperty({ constructed: [ { name: `round3 every actual receiving cut`, - enters: [...receivingCutStates], + enters: [ + ...receivingCutStates, + ...warningReceivingKinds.map( + (kind) => `warning-receiving-${kind}` as const, + ), + ], build: (generated: Suite) => ({ ...generated, seedless: true, drafts: receivingCutDrafts(generated.drafts[0]!.clock ? 1 : 2), }), }, + { + name: `round4 successful campaign budgets and failing neighbor`, + enters: [ + ...passingBudgetLanes.map( + ({ context, lane }) => `passing-budget-${context}-${lane}` as const, + ), + `passing-budget-constructed`, + `campaign-budget-failure-neighbor`, + ], + build: (generated: Suite) => + campaignBudgetSuite( + 2, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? 1 : 2, + ), + }, { name: `round3 replayed clock failure retains driver cause`, enters: [`receiving-replay`], @@ -5564,9 +5778,41 @@ oracleProperty({ fixed: [ seedlessHistory( `round3 every actual receiving cut`, - [...receivingCutStates], + [ + ...receivingCutStates, + ...warningReceivingKinds.map( + (kind) => `warning-receiving-${kind}` as const, + ), + ], ...receivingCutDrafts(), ), + { + name: `round4 successful campaign budgets and failing neighbor`, + enters: [ + ...passingBudgetLanes.map( + ({ context, lane }) => `passing-budget-${context}-${lane}` as const, + ), + `passing-budget-constructed`, + `campaign-budget-failure-neighbor`, + ], + history: campaignBudgetSuite(), + }, + { + name: `round4 successful campaign budget without constructions`, + enters: passingBudgetLanes.map( + ({ context, lane }) => `passing-budget-${context}-${lane}` as const, + ), + history: campaignBudgetSuite(2, 0), + }, + { + name: `round4 larger successful fixed and seedless budgets`, + enters: [ + `passing-budget-run-fixed`, + `passing-budget-seedless-fixed`, + `passing-budget-seedless-random`, + ], + history: campaignBudgetSuite(3, 2), + }, fixedHistory( `round3 replayed clock failure retains driver cause`, [`receiving-replay`], @@ -7004,6 +7250,111 @@ oracleProperty({ driver: driveSuite, check: checkSuite, calibrations: [ + ...warningReceivingKinds.flatMap((context) => + ([`omit`, `duplicate`] as const).map((mutation) => ({ + name: `reject ${mutation} ${context} registration warning`, + history: { fixed: `round3 every actual receiving cut` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find( + ({ context: cut, warnings }) => + cut.kind === context && warnings.length > 0, + )! + if (mutation === `omit`) child.warnings.splice(0, 1) + else child.warnings.push(child.warnings[0]!) + return altered + }, + })), + ), + ...([`omit`, `duplicate`] as const).map((mutation) => ({ + name: `reject ${mutation} unbound warning name`, + history: { fixed: `round3 every actual receiving cut` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context }) => context.kind === `seedless`)! + child.warnings = child.warnings.map((warning) => + warning.replace( + `"cal-1"`, + mutation === `omit` ? `"missing calibration"` : `"cal-1", "cal-1"`, + ), + ) + return altered + }, + })), + { + name: `reject warning from all-bound registration`, + history: { fixed: `round3 fully bound warning phases stay silent` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const child = altered.find(({ checkpoint }) => checkpoint === `child`)! + .value as ChildRun + child.warnings.push( + `${syntheticIds[0]}: calibrations "cal-0" have no history binding`, + ) + return altered + }, + }, + { + name: `reject warning from collect-only registration`, + history: { fixed: `round3 every actual receiving cut` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const children = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + const collected = children.find( + ({ context }) => + context.kind === `registration-warnings` && + context.phases.length === 0, + )! + collected.warnings.push( + children.find(({ context }) => context.kind === `run`)!.warnings[0]!, + ) + return altered + }, + }, + ...passingBudgetLanes.flatMap(({ context, lane }) => + ([`few`, `many`] as const).map((mutation) => ({ + name: `reject too ${mutation} successful ${context} ${lane} campaign calls`, + history: { + fixed: `round4 successful campaign budgets and failing neighbor`, + }, + law: `campaign budget`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered + .filter(({ checkpoint }) => checkpoint === `child`) + .map(({ value }) => value as ChildRun) + .find(({ context: cut }) => cut.kind === context)! + const callAt = child.log.findIndex( + ({ id, test }) => + id === syntheticIds[0] && + (lane === `random` + ? test === `${id} > random seed` + : test.startsWith(`${id} > fixed seed `)), + ) + if (mutation === `few`) child.log.splice(callAt, 1) + else child.log.push(structuredClone(child.log[callAt]!)) + return altered + }, + })), + ), { name: `reject replay summary cause loss`, history: { fixed: `round3 replayed clock failure retains driver cause` }, diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md index 0b525a0..fd7fc06 100644 --- a/reviews/calibration-history-self-oracle.md +++ b/reviews/calibration-history-self-oracle.md @@ -239,3 +239,56 @@ same gate passes **26/26** after repair (227 skipped), in 49.59 seconds. Package TypeScript, scoped ESLint and the Vite package/declaration build pass. Full fixed/random validation and a new independent review round follow this commit; this focused result alone does not establish PR readiness. + +## Fresh review and receiving repairs from b09a403 + +Four fresh independent reviewers found two P2 primary-oracle gaps and no new +production defect. Auxiliary children discarded registration warnings, and +passing campaign counts were learned from the executions being checked. +README's once-per-process warning promise and the header/config's equal +campaign-budget promise already supply the laws. Both are enforcement repairs. + +The owner now records every child's warning facts and independently judges +names and multiplicity for its effective declaration and collecting phase. +Required warning-positive receivers cover run, seedless, replay, rerun, +reach-alone, reordered bindings and registration phases separately. Passing +campaigns owe the authored run budget plus constructed examples; failed, +shrinking and timed-out campaigns retain their variable execution counts. +Seedless host draws remain controlled inputs, with ORACLE_SEED absent. Required +states separately cover successful fixed/random lanes in seeded and seedless +children, construction examples, and a failing neighbor. The count model never +uses observed reach totals as its owed budget. + +Same-head CI also caught two oracle regressions from the preceding expansion. +A filtered rerun cannot learn skipped default data labels from its own invocation +log. Auxiliary inventories now use the full ordinary input/name association, +consistent with the owner's existing default-label distinctness limit. Each +child's execution and build logs remain separate for outcomes and counts. +The per-entry binding refactor also stopped constructing duplicate declaration +names for bound probe controls with no authored calibrations. Effective probe +bindings now restore that grammar, preserving the required ambiguity states. +Neither issue required a source change. + +On unchanged b09a403 production, six combined warning/budget controls pass, +all five CI regression receivers pass, and 26 permanent corruption calibrations +pass. These share the primary model/driver/check. Real-child missing-warning and +fixed-budget-one source mutants are assertion-rejected; legal warning-name +reversal passes. The default-label repair still rejects omitted skipped +registrations on both Error and bigint receivers. The duplicate-name grammar +still rejects first-match selection on both inferred and explicit controls. +Package TypeScript passes. Raw RED/GREEN, malformed reports, source patches, +controls and setup attempts remain in the task audit. + +The preceding CI Vitest5 framework run reported 7 failures/246 passes: two +campaigns, two fixed cases and one diagnostic calibration share the ambiguity +premise; two default-label reruns share the inventory defect. These are two +oracle defects, not seven source bugs. The old local full sweep was superseded +and interrupted after these same-head RED results; it is not GREEN. A complete +new full sweep and another fresh review round must validate this owner. + +Warning judgment permits name/property ordering. The executed allowed control +uses a fixed reversal; it does not establish arbitrary variation between repeat +executions, whose raw warning spelling also reaches outer determinism. This is +a remaining representation question for the next independent review, not an +asserted full closure of every legal warning representation. Source and fixture +remain byte-identical to b09a403. Historical CHRON-1 remains unchanged. From f3f39a02d804cc3c74a3b1a9ba9333aad693a34d Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 17:41:52 -0700 Subject: [PATCH 11/14] test: preserve warning identity and legal name ordering --- .../tests/oracle-property-self-oracle.test.ts | 319 +++++++++++++++--- reviews/calibration-history-self-oracle.md | 64 ++++ 2 files changed, 340 insertions(+), 43 deletions(-) diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 81d87bc..775de36 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -383,6 +383,8 @@ type Suite = { seedless: boolean /** A controlled host draw, independent of the framework's seed selection. */ seedlessDraw?: number + /** Exercise equivalent warning-name presentations across repeated drives. */ + permuteWarnings?: boolean } // Outside the synthetic language: an item 5, or four items. @@ -780,6 +782,7 @@ const suite: fc.Arbitrary = fc.record( nil: undefined, freq: 8, }), + permuteWarnings: fc.option(fc.boolean(), { nil: undefined, freq: 12 }), }, { requiredKeys: [`drafts`, `seedless`] }, ) @@ -798,12 +801,19 @@ type Plan = { specs: Array seedless: boolean seedlessDraw?: number + permuteWarnings?: boolean } -function resolveSuite({ drafts, seedless, seedlessDraw }: Suite): Plan { +function resolveSuite({ + drafts, + seedless, + seedlessDraw, + permuteWarnings, +}: Suite): Plan { return { seedless, ...(seedlessDraw === undefined ? {} : { seedlessDraw }), + ...(permuteWarnings ? { permuteWarnings: true } : {}), specs: drafts.map((d, index) => { const generated = d.source.kind === `generated` const hasBinding = @@ -1400,13 +1410,33 @@ const registeredSpec = (spec: SyntheticSpec, context: ChildContext) => : spec const propertyWarnings = (warnings: ReadonlyArray, id: string) => warnings.filter((warning) => warning.startsWith(`${id}:`)) -const warningNameCount = (warning: string, name: string) => { - const escaped = name.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`) - return [ - ...warning.matchAll( - new RegExp(`(?, + ids: ReadonlyArray, +) => + warnings.filter((warning) => !ids.some((id) => warning.startsWith(`${id}:`))) +// Decode the actual rendered list, without consulting expected names. The +// finite name grammar includes inner quotes in forced-control state labels, +// but no name contains the list delimiter `", "`. Malformed lists stay raw. +const warningParts = (warning: string) => { + const match = + /^(.*?: calibrations )"(.*)"( have no history binding;.*)$/s.exec(warning) + return match + ? { prefix: match[1]!, names: match[2]!.split(`", "`), suffix: match[3]! } + : undefined +} +const renderWarning = ( + parts: NonNullable>, + names: ReadonlyArray, +) => + `${parts.prefix}${names.map((name) => `"${name}"`).join(`, `)}${parts.suffix}` +const semanticWarning = (warning: string) => { + const parts = warningParts(warning) + return parts ? renderWarning(parts, [...parts.names].sort()) : warning +} +const reversedWarning = (warning: string) => { + const parts = warningParts(warning) + return parts ? renderWarning(parts, [...parts.names].reverse()) : warning } const warningMismatch = ( spec: SyntheticSpec, @@ -1417,15 +1447,10 @@ const warningMismatch = ( const own = propertyWarnings(warnings, spec.id) if (own.length !== (unbound.length ? 1 : 0)) return true if (own.length === 0) return false - const names = [ - ...spec.calibrations.map(({ name }) => name), - ...forcedCalibrationNames(spec).map((name) => - name.slice(`calibration: `.length), - ), - ] - return names.some( - (name) => - warningNameCount(own[0]!, name) !== (unbound.includes(name) ? 1 : 0), + const actual = warningParts(own[0]!)?.names + return ( + actual === undefined || + JSON.stringify([...actual].sort()) !== JSON.stringify([...unbound].sort()) ) } const boundTest = (spec: SyntheticSpec, name: string) => { @@ -2244,6 +2269,7 @@ const passingBudgetLanes = [ { context: `seedless`, lane: `random` }, ] as const const round4FrameworkStates = [ + `warning-order-equivalence`, ...warningReceivingKinds.map((kind) => `warning-receiving-${kind}` as const), ...passingBudgetLanes.map( ({ context, lane }) => `passing-budget-${context}-${lane}` as const, @@ -2381,6 +2407,15 @@ function frameworkReach( expected: ReturnType, ): Iterable { const reached = new Set() + if ( + trace.some( + ({ checkpoint, value }) => + checkpoint === `warning-permutation` && + (value as { lists: number; stable: boolean }).lists > 0 && + (value as { stable: boolean }).stable, + ) + ) + reached.add(`warning-order-equivalence`) // Count the real host receiving cut, not a plan flag or a derived summary. // Registration-only children owe a loaded test inventory; every other cut // additionally owes a test that actually ran in that child. @@ -2973,6 +3008,7 @@ type Report = { files: Array stdout: string warnings: Array + warningPermutation?: { lists: number; stable: boolean } unhandled: Array exit: number | null signal: string | null @@ -3089,6 +3125,7 @@ const vitest = join( `vitest.mjs`, ) let runCount = 0 +let warningPresentationCount = 0 function classifyKind( inner: ReportedError, @@ -3215,6 +3252,9 @@ function runVitest( shellTail?: string, // Without a seed, the random campaigns draw theirs, as in an ordinary run. seeded = true, + // An authored representation control alternates across repeated drives. + // Its raw report stays unchanged; projection sees either legal ordering. + warningPresentation?: boolean, ): Report { const report = join(directory, `report-${runCount++}.json`) // The child reads the default `ORACLE` variables, so none of the parent's @@ -3272,10 +3312,39 @@ function runVitest( unhandled?: Array warnings?: Array } + const presentedWarnings = (warnings ?? []).map((warning) => + warningPresentation === true ? reversedWarning(warning) : warning, + ) + const projectedWarnings = presentedWarnings.map(semanticWarning) + const warningPermutation = + warningPresentation === undefined + ? undefined + : { + lists: presentedWarnings.filter( + (warning) => reversedWarning(warning) !== warning, + ).length, + stable: + JSON.stringify([...projectedWarnings].sort()) === + JSON.stringify( + presentedWarnings + .map(reversedWarning) + .map(semanticWarning) + .sort(), + ), + } + if (warningPresentation !== undefined) + writeFileSync( + `${report}.presented-warnings.json`, + JSON.stringify({ + reversed: warningPresentation, + warnings: presentedWarnings, + }), + ) return { files: reported, stdout: run.stdout, - warnings: warnings ?? [], + warnings: projectedWarnings, + ...(warningPermutation ? { warningPermutation } : {}), unhandled: unhandled ?? [], exit: run.status, signal: run.signal, @@ -3387,6 +3456,9 @@ function driveSuite( plan: Plan, observe: (checkpoint: string, value: unknown) => void, ) { + const warningPresentation = plan.permuteWarnings + ? warningPresentationCount++ % 2 === 0 + : undefined const directory = join( packageDirectory, `.self-oracle-${process.pid}-${runCount++}`, @@ -3405,7 +3477,13 @@ function driveSuite( { ...args[2], SELF_ORACLE_LOG: childLog }, args[3], args[4], + warningPresentation, ) + if (report.warningPermutation) + observe(`warning-permutation`, { + context, + ...report.warningPermutation, + }) observe(`child`, { context, log: readLog(childLog), @@ -3876,6 +3954,19 @@ function childExpectation( ({ id }) => id === context.id || context.kind === `replay`, ) : plan.specs + if ( + unattributedWarnings( + child.warnings, + selectedSpecs.map(({ id }) => id), + ).length + ) + return { + failure: violation( + `child`, + `registration reported a warning for no declared property: ${JSON.stringify(child.warnings)}`, + `calibration`, + ), + } let failed = false for (const original of selectedSpecs) { const spec = registeredSpec(original, context) @@ -4063,6 +4154,27 @@ function checkSuite( const run = trace.find(({ checkpoint }) => checkpoint === `run`)?.value as SuiteRun | undefined if (!run) return violation(`run`, `the driver recorded no run`, `trace`) + if ( + unattributedWarnings( + run.warnings, + plan.specs.map(({ id }) => id), + ).length + ) + return violation( + `run`, + `ordinary warnings name an undeclared property: ${JSON.stringify(run.warnings)}`, + `calibration`, + ) + for (const { checkpoint, value } of trace) + if ( + checkpoint === `warning-permutation` && + !(value as { stable: boolean }).stable + ) + return violation( + `child`, + `permuting actual warning names changed their semantic observation`, + `calibration`, + ) const children = trace.filter(({ checkpoint }) => checkpoint === `child`) const owedChildren = 1 + @@ -4323,20 +4435,20 @@ function checkSuite( `${spec.id} did not complete its collect-only registration in each phase`, `calibration`, ) - const forId = (warnings: Array) => - warnings.filter( - (warning) => - warning.startsWith(`${spec.id}:`) && - warning.includes(`history binding`), - ) - const expectedWarnings = unboundCalibrationNames(spec).length ? 1 : 0 if ( !warningProbe.loaded.repeated || !warningProbe.loaded.collectThenRun || !warningProbe.loaded.collected || - forId(warningProbe.repeated).length !== expectedWarnings || - forId(warningProbe.collectThenRun).length !== expectedWarnings || - forId(warningProbe.collected).length !== 0 + [ + warningProbe.repeated, + warningProbe.collectThenRun, + warningProbe.collected, + ].some( + (warnings) => unattributedWarnings(warnings, [spec.id]).length > 0, + ) || + warningMismatch(spec, warningProbe.repeated, false) || + warningMismatch(spec, warningProbe.collectThenRun, false) || + warningMismatch(spec, warningProbe.collected, true) ) return violation( `run`, @@ -4346,16 +4458,8 @@ function checkSuite( } // The compatibility route must be visible before any selected test runs, // including automatic controls; bound calibrations must never warn search. - const unbound = unboundCalibrationNames(spec) - const warnings = run.warnings.filter( - (warning) => - warning.startsWith(`${spec.id}:`) && - warning.includes(`history binding`), - ) - if ( - warnings.length !== (unbound.length ? 1 : 0) || - unbound.some((name) => !warnings[0]?.includes(name)) - ) + const warnings = propertyWarnings(run.warnings, spec.id) + if (warningMismatch(spec, run.warnings, false)) return violation( `run`, `${spec.id} did not report unbound calibrations once per property: ${JSON.stringify(warnings)}`, @@ -5068,6 +5172,7 @@ const receivingCutDrafts = (tail: 1 | 2 = 1): Array => [ calibrations: [ `right`, { kind: `driver`, difference: `none`, history: null }, + { kind: `right`, difference: `none`, history: null }, ], isolatedPlans: true, warningProbe: true, @@ -5098,6 +5203,16 @@ const campaignBudgetSuite = ( }), ], }) +const warningRepresentationSuite = (): Suite => ({ + drafts: [ + generatedDraft({ + source: { kind: `enumerated`, histories: [[0], [1]] }, + calibrations: [`right`, `driver`], + }), + ], + seedless: false, + permuteWarnings: true, +}) const cal = ( kind: CalibrationDraft[`kind`], difference: CalibrationDraft[`difference`] = `none`, @@ -5226,6 +5341,7 @@ oracleProperty({ name: `round3 every actual receiving cut`, enters: [ ...receivingCutStates, + `warning-order-equivalence`, ...warningReceivingKinds.map( (kind) => `warning-receiving-${kind}` as const, ), @@ -5233,6 +5349,7 @@ oracleProperty({ build: (generated: Suite) => ({ ...generated, seedless: true, + permuteWarnings: true, drafts: receivingCutDrafts(generated.drafts[0]!.clock ? 1 : 2), }), }, @@ -5776,16 +5893,26 @@ oracleProperty({ }, ], fixed: [ - seedlessHistory( - `round3 every actual receiving cut`, - [ + { + name: `round3 every actual receiving cut`, + enters: [ ...receivingCutStates, + `warning-order-equivalence`, ...warningReceivingKinds.map( (kind) => `warning-receiving-${kind}` as const, ), ], - ...receivingCutDrafts(), - ), + history: { + drafts: receivingCutDrafts(), + seedless: true, + permuteWarnings: true, + }, + }, + { + name: `round5 warning-order equivalence`, + enters: [`warning-order-equivalence`], + history: warningRepresentationSuite(), + }, { name: `round4 successful campaign budgets and failing neighbor`, enters: [ @@ -7250,6 +7377,112 @@ oracleProperty({ driver: driveSuite, check: checkSuite, calibrations: [ + ...([`child`, `run`, `phase`] as const).map((receiver) => ({ + name: `reject unrelated ${receiver} warning property`, + history: { + fixed: + receiver === `phase` + ? `round3 every actual receiving cut` + : `round5 warning-order equivalence`, + }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const unknown = `self-oracle.unrelated: calibrations "cal-0" have no history binding; using transitional candidate search.` + if (receiver === `phase`) { + const phase = altered.find( + ({ checkpoint }) => checkpoint === `registration-warnings`, + )!.value as { repeated: Array } + phase.repeated.push(unknown) + } else { + const observation = altered.find( + ({ checkpoint }) => checkpoint === receiver, + )!.value as { warnings: Array } + observation.warnings.push(unknown) + } + return altered + }, + })), + ...([`suffix`, `replacement`, `malformed`] as const).map((mutation) => ({ + name: `reject ${mutation} actual warning identity`, + history: { fixed: `round5 warning-order equivalence` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find(({ checkpoint }) => checkpoint === `child`)! + .value as ChildRun + child.warnings = child.warnings.map((warning) => { + if (mutation === `malformed`) + return warning.replace(`: calibrations `, `: malformed names `) + const parts = warningParts(warning)! + return renderWarning( + parts, + parts.names.map((name, index) => + mutation === `suffix` + ? `${name}-extra` + : index === 0 + ? `replacement calibration` + : name, + ), + ) + }) + return altered + }, + })), + { + name: `reject suffixed ordinary warning summary identity`, + history: { fixed: `round5 warning-order equivalence` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const run = altered.find(({ checkpoint }) => checkpoint === `run`)! + .value as SuiteRun + run.warnings = run.warnings.map((warning) => { + const parts = warningParts(warning)! + return renderWarning( + parts, + parts.names.map((name) => `${name}-extra`), + ) + }) + return altered + }, + }, + { + name: `reject suffixed registration phase summary identity`, + history: { fixed: `round3 every actual receiving cut` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const phase = altered.find( + ({ checkpoint }) => checkpoint === `registration-warnings`, + )!.value as { repeated: Array } + phase.repeated = phase.repeated.map((warning) => { + const parts = warningParts(warning)! + return renderWarning( + parts, + parts.names.map((name) => `${name}-extra`), + ) + }) + return altered + }, + }, + { + name: `reject lost warning permutation equivalence`, + history: { fixed: `round5 warning-order equivalence` }, + law: `calibration`, + corrupt: (trace) => { + const altered = structuredClone([...trace]) + const permutation = altered.find( + ({ checkpoint }) => checkpoint === `warning-permutation`, + )!.value as { stable: boolean } + permutation.stable = false + return altered + }, + }, ...warningReceivingKinds.flatMap((context) => ([`omit`, `duplicate`] as const).map((mutation) => ({ name: `reject ${mutation} ${context} registration warning`, diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md index fd7fc06..86d7bdb 100644 --- a/reviews/calibration-history-self-oracle.md +++ b/reviews/calibration-history-self-oracle.md @@ -292,3 +292,67 @@ executions, whose raw warning spelling also reaches outer determinism. This is a remaining representation question for the next independent review, not an asserted full closure of every legal warning representation. Source and fixture remain byte-identical to b09a403. Historical CHRON-1 remains unchanged. + +## Fresh warning-observation review from 68ed541 + +Four new scoped reviews found two P2 self-oracle defects. Warning-name order is +not promised, but its raw spelling reached outer determinism. Alternating only +valid name order passed the shared judgment and failed outer determinism. The +warning-name matcher also accepted `cal-0-extra` as `cal-0`: both the ordinary +case and outer determinism stayed green under that wrong-name renderer. These +are distinct observation/formulation and enforcement defects under the README's +once-per-property/process, exact-unbound-names promise. + +The owner now decodes the actual finite quoted-name list, retaining duplicates +and unknown names, and sorts its entries before any child, ordinary-run or +registration-phase observation. The authored expected-name multiset remains +independent of the warning renderer. Malformed lists stay observable and fail; +unknown-property warnings fail at each full-list checkpoint. The finite grammar +allows inner state-name quotes but excludes the list delimiter inside a name; +no claim about arbitrary future warning prose or arbitrary JavaScript strings +is made. + +An authored representation dimension alternates valid name presentation across +real executions. The existing seven-context receiver carries multiple unbound +names and measures nontrivial permutation reach. Its ordinary case and actual +outer determinism reuse the same model, driver and check. Raw reports remain +unchanged; sidecars preserve the presented ordering. Permanent corruptions add +wrong suffixed/replaced names, malformed lists and unrelated-property warnings, +while retaining missing/duplicate warnings and names, all-bound silence and +collect-only silence. No framework production or fixture edit is part of this +repair. + +The frozen pre-repair experiment passed forward2/2 and consistent reverse2/2, +but alternating representation failed outer determinism. The wrong suffix +renderer survived2/2 before repair. With the repaired observation it fails the +intended `child` / `calibration` comparison; the actual ordinary/determinism +representation control passes2/2. Four live neighboring receivers pass. Package +TypeScript and scoped lint pass. Final warning-calibration completion is +recorded in the next entry, without claiming a whole-suite GREEN. + +Reviewed owner SHA256: +`18a5c3e091cf7b218f2d08fad99df51aa548ec2daca4ab7df27973d074b776ac`. +Production and fixture hashes remain those of b09a403. Independent grammar, +seed/premise/identity, and execution-semantics reviewers found no new supported +defect in their bounded scopes. The independence review accepted an alternative +stable hash, preserving implementation freedom. Their negative results are not +whole-framework correctness claims. Historical CHRON-1 remains unchanged. + +The semantics review also preserved an optional policy question: a deliberately +wrong driver may enter an undeclared state while its designated visible-result +violation is accepted as a successful calibration. The current contract scopes +declared-reach validation to the clean run and describes shared verdict rules +for recorded violation/clock/error priority. No stronger mutant-reach obligation +has been established or silently added. A maintainer clarification was requested +separately; the executed trace and its authority qualification remain in the +audit. + +The completed frozen warning sweep passes **27/27** in 184.04 seconds: nine +identity/property/permutation controls and eighteen retained warning controls +across seven receiving contexts. Together with the four positive receivers, +two actual ordinary/outer-determinism checks and the rejected suffix renderer, +this completes the two warning findings within the stated finite grammar. An +independent reviewer is challenging this frozen repair before main integration. +The superseded full68ed541 run completed both campaigns, including required +warning/budget reach, then was explicitly interrupted before the incoming +source/fixture integration. It is incomplete, not a full-suite GREEN. From c18a30ff53a6c9a62ef868fb9f7a607aaf6ab914 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 18:54:15 -0700 Subject: [PATCH 12/14] test: enforce bound callback diagnostics across receiving runs --- .../tests/oracle-property-self-oracle.test.ts | 433 ++++++++++++++++-- reviews/calibration-history-self-oracle.md | 85 ++++ 2 files changed, 469 insertions(+), 49 deletions(-) diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index d29160e..7cc45cf 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -104,7 +104,12 @@ * finite ordinary/astral name families declared here; no universal collision * freedom is claimed. Forced premises are judged from declared input origins * and author-built callback inputs, never the framework's selected driver log. - * Callback errors retain calibration/history context and cause. Both forced + * Executed bound callback errors retain the calibration name, the selected + * authored history's kind and name, and the original cause in every active + * receiving child. Public original-history name/origin associations identify + * opaque default enumeration labels; callback text cannot authorize selection. + * Seed annotations, punctuation, stacks and name hashing are not laws. + * Both forced * premises independently select fixed or constructed originals; only omitted * entered selectors infer a unique construction. Equal data never permits * substituting another named original. Registration warns once per @@ -906,7 +911,7 @@ const replayEnvironment: fc.Arbitrary = fc.record({ ), }) const cleanHostDraw = 0.5000000016298145 -const suite: fc.Arbitrary = fc.record( +const genericSuite: fc.Arbitrary = fc.record( { drafts: fc.array(draft, { minLength: 1, maxLength: 2 }), seedless: fc.oneof( @@ -2632,7 +2637,17 @@ const passingUnitStates = passingBudgetLanes.flatMap(({ context, lane }) => (unit) => `passing-unit-${context}-${lane}-${unit}` as const, ), ) +const callbackReceivingKinds = [ + `run`, + `binding-reordered`, + `seedless`, + `environment`, +] as const +const callbackReceivingStates = callbackReceivingKinds.map( + (kind) => `callback-failure-${kind}` as const, +) const round4FrameworkStates = [ + ...callbackReceivingStates, `host-rng-scope`, `host-rng-forwarding`, `receiving-host-scope`, @@ -2866,6 +2881,52 @@ function frameworkReach( !warningMismatch(spec, child.warnings, false) ) reached.add(`warning-receiving-${child.context.kind}`) + if ( + callbackReceivingKinds.some((kind) => kind === child.context.kind) && + child.loads.some( + (load) => load.id === spec.id && load.errors.length === 0, + ) + ) { + const prediction = predictSpec(spec, child.log, child.builds) + const nameLog = + ( + trace.find((entry) => entry.checkpoint === `run`)?.value as + SuiteRun | undefined + )?.log ?? [] + if ( + prediction.loads && + child.tests.some((test) => { + if ( + test.id !== spec.id || + test.state !== `failed` || + test.errors.length !== 1 || + test.errors[0]?.kind !== `setup` + ) + return false + const outcome = prediction.tests.get(test.name) + const diagnostic = + outcome && callbackExpectation(spec, test.name, outcome, nameLog) + const calibration = spec.calibrations.find( + (entry) => test.name === `calibration: ${entry.name}`, + ) + if ( + !diagnostic || + !calibration || + !callbackDiagnosticMatches(diagnostic, test.errors[0]) + ) + return false + const calls = child.log.filter( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + ) + return ( + calls.length === (driverCalibration(calibration.kind) ? 2 : 1) + ) + }) + ) + reached.add(`callback-failure-${child.context.kind}`) + } if (spec.source.kind !== `generated`) continue const units = child.tests.filter( ({ id, name }) => id === spec.id && isCampaign(name), @@ -3650,6 +3711,8 @@ type Report = { type Failure = { message?: string causeMessage?: string + /** Semantic bound-callback identity; seed annotations and stacks are excluded. */ + callbackContext?: { calibration: string; kind: string; name: string } kind: Kind | `other` dataPath?: string dataKind?: string @@ -3865,6 +3928,26 @@ function classifyKind( return `other` } +// Read the reported identity without asking authored bindings what it should be. +// The construction's seed annotation is diagnostic representation, not identity. +const reportedCallbackContext = ( + message: string, +): Failure[`callbackContext`] => { + const match = + /calibration "([^"]+)" on (constructed|fixed|enumerated) history:\s*(.*?)(?=:\s*selected |$)/s.exec( + message, + ) + if (!match) return undefined + return { + calibration: match[1]!, + kind: match[2]!, + name: + match[2] === `constructed` + ? match[3]!.trim().replace(/ \(seed -?\d+\)$/, ``) + : match[3]!.trim(), + } +} + function classify(error: ReportedError | undefined): Failure { if (!error) return { kind: `other` } const outer = error.message ?? `` @@ -3909,6 +3992,9 @@ function classify(error: ReportedError | undefined): Failure { return { kind, message: text, + ...(reportedCallbackContext(text) + ? { callbackContext: reportedCallbackContext(text) } + : {}), ...(inner.cause?.message ? { causeMessage: inner.cause.message } : {}), // A model error that quotes a machine path, such as an inspected stack. ...(inner.name === `OracleModelError` && text.includes(packageDirectory) @@ -4095,6 +4181,19 @@ const readLog = (log: string): Array => // promise. Stable grouping retains every call and each local order. .sort((a, b) => a.id.localeCompare(b.id)) : [] +// A wrong-answer control changes every ordinary-run representation of its +// invocation. Summary and full child observations are separate data objects; +// either valid projection must retain the same deliberately wrong origin. +const ordinaryInvocationLogs = ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, +): Array> => + trace.flatMap(({ checkpoint, value }) => + checkpoint === `run` + ? [(value as SuiteRun).log] + : checkpoint === `child` && (value as ChildRun).context.kind === `run` + ? [(value as ChildRun).log] + : [], + ) // Resolve callbacks may create resources even when no driver receives the // result. Their separate input-side log preserves extra and omitted calls. const readResolutions = (log: string): SuiteRun[`resolutions`] => @@ -4159,6 +4258,7 @@ const diagnosticFacts = ({ diagnosis, cause, causeMessage, + callbackContext, counterexample, dataPath, dataKind, @@ -4169,6 +4269,7 @@ const diagnosticFacts = ({ ...(diagnosis === undefined ? {} : { diagnosis }), ...(cause === undefined ? {} : { cause }), ...(causeMessage === undefined ? {} : { causeMessage }), + ...(callbackContext === undefined ? {} : { callbackContext }), ...(counterexample !== undefined && counterexampleItems(counterexample) ? { counterexampleHistory: counterexampleItems(counterexample)! } : {}), @@ -4852,6 +4953,103 @@ const matchesOutcome = ( : outcome.causeMessage === undefined || failure?.causeMessage === outcome.causeMessage)) +// Bound callback errors owe the calibration and selected history names (original +// authorized review item 6), plus the callback's original cause. Authored bindings +// decide identity; reported callback text never decides which input was selected. +// Opaque default enumeration labels use the original direct history test's public +// name associated with its authored index/origin. Missing or ambiguous associations +// fail this relation. This does not prescribe the framework's data-label codec. +const callbackExpectation = ( + spec: SyntheticSpec, + test: string, + outcome: Outcome, + nameLog: ReadonlyArray, +) => { + const calibration = spec.calibrations.find( + ({ name }) => test === `calibration: ${name}`, + ) + const binding = calibration && bindingOf(spec, calibration) + const invalidData = dataOutcome(spec) + if ( + !calibration || + !binding || + outcome.kind === `pass` || + !( + (outcome.kind === `setup` && + [`corrupt-throws`, `judge-throws`, `judge-driver-throws`].includes( + calibration.kind, + )) || + (spec.odd && invalidData) + ) + ) + return undefined + const kind = + `constructed` in binding + ? `constructed` + : `fixed` in binding + ? `fixed` + : `enumerated` + let selectedName: string | undefined + if (`constructed` in binding) selectedName = binding.constructed + else if (`fixed` in binding) selectedName = binding.fixed + else if (spec.source.kind === `enumerated`) { + const index = spec.source.histories.findIndex((history, at) => + typeof binding.enumerated === `number` + ? at === binding.enumerated + : authoredEnumerationLabel(spec, history) === binding.enumerated, + ) + const history = spec.source.histories[index] + if (history) { + selectedName = authoredEnumerationLabel(spec, history) + if (selectedName === undefined) { + const originalNames = [ + ...new Set( + nameLog + .filter( + (entry) => + entry.id === spec.id && + entry.origin === `enumerated:${index}` && + !frameworkTest(entry.test.slice(spec.id.length + 3)), + ) + .map((entry) => entry.test.slice(spec.id.length + 3)), + ), + ] + if (originalNames.length === 1) selectedName = originalNames[0] + } + } + } + return { + context: { calibration: calibration.name, kind, name: selectedName }, + causeMessage: + calibration.kind === `corrupt-throws` + ? `synthetic corruption failed` + : [`judge-throws`, `judge-driver-throws`].includes(calibration.kind) + ? `synthetic judgment failed` + : undefined, + causeIncludes: + invalidData?.kind === `data` + ? [invalidData.dataPath!, invalidData.dataKind!] + : invalidData?.kind === `limit` + ? [`the framework's encoding limit`] + : [], + } +} +const callbackDiagnosticMatches = ( + diagnostic: ReturnType, + failure: Failure | undefined, +) => + !diagnostic || + (diagnostic.context.name !== undefined && + failure?.callbackContext?.calibration === diagnostic.context.calibration && + failure.callbackContext.kind === diagnostic.context.kind && + failure.callbackContext.name === diagnostic.context.name && + (diagnostic.causeMessage !== undefined + ? failure.causeMessage === diagnostic.causeMessage + : diagnostic.causeIncludes.length > 0 && + diagnostic.causeIncludes.every((part) => + failure.causeMessage?.includes(part), + ))) + // A concrete campaign candidate owes its own diagnostics, not those of the // first failing history before shrinking. Missing a construction's state is // separate from the candidate's row-value law and can have a passing trace. @@ -5124,7 +5322,11 @@ function childExpectation( if ( test.state !== (owesFailure ? `failed` : `passed`) || test.errors.length !== (owesFailure ? 1 : 0) || - !matchesOutcome(outcome, failure, kind, isCampaign(test.name)) + !matchesOutcome(outcome, failure, kind, isCampaign(test.name)) || + !callbackDiagnosticMatches( + callbackExpectation(spec, test.name, outcome, nameLog), + failure, + ) ) return { failure: violation( @@ -5826,47 +6028,16 @@ function checkSuite( ) } if ( - calibrationBinding && - test.state === `failed` && - ([`corrupt-throws`, `judge-throws`, `judge-driver-throws`].includes( - calibration.kind, - ) || - (spec.odd && dataOutcome(spec))) - ) { - const selectedKind = - `constructed` in calibrationBinding - ? `constructed` - : `fixed` in calibrationBinding - ? `fixed` - : `enumerated` - const invalidData = dataOutcome(spec) - const expectedCause = - calibration.kind === `corrupt-throws` - ? `synthetic corruption failed` - : [`judge-throws`, `judge-driver-throws`].includes(calibration.kind) - ? `synthetic judgment failed` - : undefined - const causeMatches = - expectedCause !== undefined - ? failure?.causeMessage === expectedCause - : invalidData?.kind === `data` - ? failure?.causeMessage?.includes(invalidData.dataPath!) && - failure.causeMessage.includes(invalidData.dataKind!) - : invalidData?.kind === `limit` && - failure?.causeMessage?.includes( - `the framework's encoding limit`, - ) - if ( - !failure?.message?.includes(`calibration "${calibration.name}"`) || - !failure.message.includes(`${selectedKind} history:`) || - !causeMatches + !callbackDiagnosticMatches( + callbackExpectation(spec, test.name, outcome, ordinaryLog), + failure, + ) + ) + return violation( + `run`, + `${spec.id} callback failure lost its authored calibration/history identity or cause`, + `calibration`, ) - return violation( - `run`, - `${spec.id} callback failure lost its calibration/history context or cause`, - `calibration`, - ) - } const observedKind = test.state === `passed` ? `pass` : (failure?.kind ?? `other`) if ( @@ -6433,6 +6604,25 @@ function checkEnvironment( return failed( `${spec.id} failed to load with ${JSON.stringify(seen.loadErrors)} under ${JSON.stringify(environmentOf(plan))}, owed ${JSON.stringify(owedError)}`, ) + if (seen.loaded && owed.loads && !replays) { + const ordinary = trace.find(({ checkpoint }) => checkpoint === `run`) + ?.value as SuiteRun | undefined + for (const test of seen.tests) { + const outcome = owed.tests.get(test.name) + if ( + outcome && + !callbackDiagnosticMatches( + callbackExpectation(spec, test.name, outcome, ordinary?.log ?? []), + test.failure, + ) + ) + return violation( + `environment`, + `${spec.id} callback failure lost its authored calibration/history identity or cause`, + `calibration`, + ) + } + } if (!seen.loaded || !replays) continue // A path replay: only the selected unit runs, if any. const running = seen.tests.filter((test) => test.state !== `skipped`) @@ -6591,6 +6781,85 @@ const replayOf = ( ...environment, }) const at = (history: SyntheticHistory) => ({ history, enters: null }) +// This legal grammar branch crosses callback kind with selected source kind. +// Distinct same-kind names make truthful selected identity observable. A clean +// constructed-bound neighbor admits the reordered receiver for fixed selections. +// Enumerated declarations have no constructed binding, so that receiver is absent. +const callbackKinds = [ + `corrupt-throws`, + `judge-throws`, + `judge-driver-throws`, +] as const +const callbackSelections = [`constructed`, `fixed`, `enumerated`] as const +const callbackDiagnosticSuite = ( + kind: (typeof callbackKinds)[number], + selection: (typeof callbackSelections)[number], + tail: 1 | 2 = 1, + prefix: Prefix = `ORACLE`, +): Suite => ({ + seedless: true, + prefix, + environment: replayOf({ path: `absent`, property: `absent`, unit: `absent` }), + drafts: [ + selection === `enumerated` + ? generatedDraft({ + source: { + kind: `enumerated`, + histories: [ + [0, tail], + [2, 2], + ], + }, + enumeratedStates: [`measured`, `measured`, `measured`], + calibrations: [ + { kind, difference: `none`, history: { enumerated: 0 } }, + ], + identityHistories: true, + warningProbe: true, + }) + : generatedDraft({ + fixed: [at([0, tail]), at([2, 2])], + constructed: [ + { prefix: [0, tail], enters: [`saw-0`] }, + { prefix: [2, 2], enters: [`saw-2`] }, + ], + calibrations: [ + { + kind, + difference: `none`, + history: + selection === `fixed` + ? { fixed: `fixed-0` } + : { constructed: `constructed-0` }, + }, + { + kind: `right`, + difference: `none`, + history: { constructed: `constructed-1` }, + }, + ], + isolatedPlans: true, + identityHistories: true, + warningProbe: true, + }), + ], +}) +const suite: fc.Arbitrary = fc.oneof( + { weight: 8, arbitrary: genericSuite }, + { + weight: 1, + arbitrary: fc + .record({ + kind: fc.constantFrom(...callbackKinds), + selection: fc.constantFrom(...callbackSelections), + tail: fc.constantFrom(1 as const, 2 as const), + prefix: fc.constantFrom(`ORACLE` as const, `SYNTH_ORACLE` as const), + }) + .map(({ kind, selection, tail, prefix }) => + callbackDiagnosticSuite(kind, selection, tail, prefix), + ), + }, +) const receivingCutStates = [ `receiving-run`, `receiving-seedless`, @@ -6877,6 +7146,7 @@ oracleProperty({ arbitrary: suite, states: [ ...mainUnitRequiredStates, + ...callbackReceivingStates, ...round3FrameworkStates, `auxiliary-late-error`, `bound-enumerated-label-selector`, @@ -6908,6 +7178,7 @@ oracleProperty({ (state) => state !== `auxiliary-late-error` && !mainUnitRequiredStates.some((required) => required === state) && + !callbackReceivingStates.some((required) => required === state) && !round3FrameworkStates.some((required) => required === state) && !state.startsWith(`bound-`) && state !== `unbound-prototype-probe` && @@ -6917,6 +7188,22 @@ oracleProperty({ fixedSeeds: [1], constructed: [ ...mainUnitRequiredConstructed, + ...callbackKinds.map((kind, ordinal) => ({ + name: `round7 callback diagnostics ${kind}`, + enters: + ordinal === 2 + ? callbackReceivingStates.filter( + (state) => state !== `callback-failure-binding-reordered`, + ) + : [...callbackReceivingStates], + build: (generated: Suite) => + callbackDiagnosticSuite( + kind, + callbackSelections[ordinal]!, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + })), ...([0, 2] as const).map((entering) => ({ name: `inferred control with ${entering} distinct entering constructions`, enters: [ @@ -7523,6 +7810,16 @@ oracleProperty({ }, ], fixed: [ + ...callbackKinds.map((kind, ordinal) => ({ + name: `round7 callback diagnostics ${kind}`, + enters: + ordinal === 2 + ? callbackReceivingStates.filter( + (state) => state !== `callback-failure-binding-reordered`, + ) + : [...callbackReceivingStates], + history: callbackDiagnosticSuite(kind, callbackSelections[ordinal]!), + })), { name: `custom prefix preserves every receiving cut`, enters: [ @@ -9211,6 +9508,33 @@ oracleProperty({ driver: driveSuite, check: checkSuite, calibrations: [ + ...callbackReceivingKinds.flatMap((receiver) => + ([`context`, `selected name`, `cause`] as const).map((fault) => ({ + name: `reject callback ${receiver} lost ${fault}`, + history: { fixed: `round7 callback diagnostics corrupt-throws` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find( + (entry) => + entry.checkpoint === `child` && + (entry.value as ChildRun).context.kind === receiver, + )?.value as ChildRun | undefined + const failure = child?.tests.find( + (test) => test.name === `calibration: cal-0`, + )?.errors[0] + if (!failure?.callbackContext) return undefined + if (fault === `context`) + failure.callbackContext.calibration = `unrelated calibration` + if (fault === `selected name`) + failure.callbackContext.name = `constructed-1` + if (fault === `cause`) delete failure.causeMessage + return altered + }, + })), + ), ...( [`draw`, `host use`, `forwarding`, `malformed`, `child seed`] as const ).map((fault) => ({ @@ -9778,13 +10102,24 @@ oracleProperty({ name: `reject equal-data origin substitution`, history: { constructed: `review equal fixed originals` }, law: `calibration`, - corrupt: (trace) => { + corrupt: (trace, plan) => { const altered = structuredClone([...trace]) - const run = altered.find(({ checkpoint }) => checkpoint === `run`)! - .value as SuiteRun - for (const entry of run.log) - if (entry.test.includes(`calibration: cal-0`)) - entry.origin = `fixed:fixed-0` + const spec = plan.specs[0]! + const calibration = spec.calibrations[0]! + const binding = bindingOf(spec, calibration)! + const selectedOrigin = authoredOrigin(spec, binding) + const wrongOrigin = `fixed:${spec.fixed[0]!.name}` + const test = `${spec.id} > calibration: ${calibration.name}` + const entries = ordinaryInvocationLogs(altered) + .flat() + .filter((entry) => entry.id === spec.id && entry.test === test) + if ( + selectedOrigin === wrongOrigin || + entries.length === 0 || + entries.some((entry) => entry.origin !== selectedOrigin) + ) + return undefined + for (const entry of entries) entry.origin = wrongOrigin return altered }, }, diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md index 52787a5..5caf0c4 100644 --- a/reviews/calibration-history-self-oracle.md +++ b/reviews/calibration-history-self-oracle.md @@ -472,3 +472,88 @@ determinism still passes, isolating enforcement from nondeterminism. Package types, scoped lint and diff checks pass. Production and fixture retain the hashes above. Full frozen-head validation and three fresh owner reviewers start next; no whole-PR readiness claim is made by these focused results. + +## Fresh merged-head review: callback diagnostics and calibration instrument + +Three fresh reviewers examined immutable `8b4543aa5fd0e887ddf77ed2765fe43d76eb116c`. +Independence and grammar/reach reviews found no new supported defect in their +bounded scopes. They executed lawful name-mixer/forwarding variants, original +identity/forced-premise attacks, exact UNIT stream/budget attacks and actual +receiving reach. The observation review found two oracle defects; its context +and cause subclaims are retained separately below. Current production preserves +the disputed diagnostics. No production or fixture repair is indicated. + +The law comes from original user-authorized review item 6: failures retain the +calibration and selected history names. The accepted owner also retains the +original cause. The README supplies neighboring selection semantics; it does +not prescribe this callback diagnostic template. Exact punctuation, hash/seed +arithmetic, stack paths and an unbound-callback context promise are not added. + +| ID | Verified gap | Repair and evidence | +| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| R7-OBS-CONTEXT | Reordered, seedless and environment callback errors could lose their selected calibration/history context while the shared judgment passed. | Semantic identity survives every active child/summary projection and is compared with authored selection. Actual reordered source mutant now fails child/calibration. | +| R7-OBS-CAUSE | Those auxiliary errors could lose or replace their original cause. | The same independent diagnostic judgment requires the finite callback cause; separate cause-loss source mutant now fails child/calibration. | +| R7-OBS-SELECTED-NAME | Ordinary and reordered errors could falsely name another declared history of the same kind. | Exact authored selected name is checked; a same-kind source substitution now fails. | +| R7-CI-ORIGIN | Permanent origin corruption changed a stale summary log while the comparison consumed the ordinary child log. | The control now changes the exact semantic invocation in both representations; identity comparison is unchanged. Same calibration passes and a real wrong-origin source still fails run/calibration. | +| R7-SIB-CALLBACK-REPRESENTATION | Initial repair falsely rejected harmless separator whitespace. | An actual legal renderer failed before normalization and passes after it; wrong selected names still fail. | + +All findings were grouped before owner repair. The existing Suite arbitrary now +has a weighted legal callback branch crossing three callback kinds, three +selection kinds, two tails and both prefixes. Fixed inputs and required +constructions use the same model, real driver and comparison. Callbacks actually +execute in ordinary, reordered, seedless and seed-only environment paths; each +has required measured callback-failure reach. Registration phases, path replay, +rerun and reach-alone keep their full registration/skipped checks and do not +claim callback execution. Across nine clean kind/selection neighbors, actual +counts are ordinary 9, reordered 6, seedless 9 and environment 9. Enumerated +sources have no reordered-construction receiver under this driver. + +Three bad-source variants survive the frozen owner and fail the repaired +comparison; clean source passes both. Thirty-six independent wrong diagnostics +are rejected across four child cuts plus ordinary/environment summaries. +Opaque default enumeration names are associated with the public direct-history +name of the authored index/origin, independently of callback text. Five opaque +forms pass; fifteen wrong-name, missing-association and ambiguous-association +controls fail. The codec and name hash are not copied into the expectation. + +The callback scratch candidate passed eighteen focused checks, followed by five +on the final whitespace-normalized bytes, and all thirty-six controls were +rerun. One earlier scratch run overlapped source-mutant experiments; its four +failures are contaminated evidence, superseded by the restored-source clean +gates. No setup/runtime failure is counted as an assertion rejection. + +Exact-head Vitest 5 CI on 8b4543a passed 729 of 730 checks and exposed the stale +origin corruption. The same calibration failed on installed Vitest 4, ruling +out a version-only flake. Captured traces show the summary changed to fixed-0 +while the canonical child retained correctly selected fixed-1; changing the +canonical or both logs triggers the intended identity assertion. The nearby +permanent-control audit found no other log/build/resolve corruption with this +stale target. Ten focused origin and adjacent controls pass after the repair; +a genuine wrong-fixed-original source mutant still fails the primary law. + +The superseded local 8b4543a run completed both campaigns' forty-five named +units, retaining their full reach counts, then was explicitly interrupted with +exit 130 after the confirmed CI/focused failure. It is not a full-suite GREEN. +Core/package, plugin, Vitest 5 core and both gap-oracle jobs on that head passed; +the consumer-install CI path supplies the witness blocked by local registry403. + +Combined live owner SHA256 is +`b6efd19d8dd086ed73d65c36e3b471685cc96789d055dcfcb82d94c972d4b3ad`. +The actual combined focused gate passes **19/19** in 84.90 seconds, including +callback fixed histories/outer determinism, twelve corruptions and the origin +control. Full package TypeScript, scoped lint, formatting and diff checks pass. +Production/fixture/reporter retain the a1c/e2d/1918 identities above. Complete +final-head framework validation and fresh focused re-review start next; this +entry makes no full-framework GREEN or whole-PR readiness claim. + +Loss accounting: two raw observation findings retain three independently +verified obligations, plus one CI instrument finding and one repair-discovered +representation sibling. All have applied repairs and focused enforcement; none +is deferred or resolved by a policy change. Other reviewer scopes retain their +negative evidence and limits. Strong hire for each reviewer in its demonstrated +scope, particularly the observation review's actual source survivors and the +CI diagnosis that distinguished an inert instrument from missing law enforcement. +Historical CHRON-1 remains irreversible; optional mutant-only undeclared-state +policy remains a separate unapproved proposal. The finite synthetic grammar, +Node/Vitest receiving boundary and absence of downstream DB/browser/performance +claims remain unchanged. From 1fdf7311251f2f0f5367b8f0239309960cf61dc0 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 20:13:48 -0700 Subject: [PATCH 13/14] test: preserve semantic diagnostics and timeout ownership --- .../oracle-property-self-oracle-fixture.ts | 66 +- .../tests/oracle-property-self-oracle.test.ts | 1504 ++++++++++++++++- reviews/calibration-history-self-oracle.md | 128 ++ 3 files changed, 1619 insertions(+), 79 deletions(-) diff --git a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts index de05db2..78d4b18 100644 --- a/packages/oracles/tests/oracle-property-self-oracle-fixture.ts +++ b/packages/oracles/tests/oracle-property-self-oracle-fixture.ts @@ -17,7 +17,9 @@ * A constructed history builds its history as its prefix followed by a generated history, * cut to three items. The driver appends every history it runs, with the running test's name, to * the file named by SELF_ORACLE_LOG, so the self-oracle knows which - * histories each campaign ran. A `nondeterministic` spec also records, last, a + * histories each campaign ran. The logical owning name comes from the runner's + * public aroundEach boundary, retained across late continuations; a slow raw log + * separately retains the global runner name for timeout diagnostics. A `nondeterministic` spec also records, last, a * counter that changes on every run. A spec reports reach through the * property's `reached` part or, as the framework's default, through the * driver's `observe("reach", state)`. A `slow` spec's driver takes about @@ -152,10 +154,19 @@ * registered beside them, so the self-oracle's registration law can reject an * id outside the registry. */ +import { AsyncLocalStorage } from 'node:async_hooks' import { appendFileSync } from 'node:fs' import { runInNewContext } from 'node:vm' import fc from 'fast-check' -import { afterAll, beforeAll, beforeEach, describe, expect, vi } from 'vitest' +import { + afterAll, + aroundEach, + beforeAll, + beforeEach, + describe, + expect, + vi, +} from 'vitest' import { configureOracles, differs, @@ -467,10 +478,29 @@ const itemOf = (failure: unknown): number => { return code ?? 0 } let noise = 0 +// The public aroundEach boundary supplies an immutable owning test context. +// Continuations retain it after their timeout, although Vitest's global +// currentTestName may then name the next test. This is input provenance only: +// the fixture does not change cancellation, deadlines, or execution order. +const owningTest = new AsyncLocalStorage<{ name: string; route?: string }>() +aroundEach((runTest, { task }) => { + const names = [task.name] + for ( + let parent = task.suite; + parent && parent !== task.file; + parent = parent.suite + ) + names.unshift(parent.name) + return owningTest.run({ name: names.join(` > `) }, runTest) +}) +const owningTestName = () => + owningTest.getStore()?.name ?? expect.getState().currentTestName // The route registered for the running test, as the gap report reads it. let route: string | undefined globalThis.__oracleRoute = (registered) => { - route = registered + const owner = owningTest.getStore() + if (owner) owner.route = registered + else route = registered } beforeEach(() => { route = undefined @@ -577,7 +607,7 @@ const reachedStates = (spec: SyntheticSpec) => (history: SyntheticHistory) => [ ...(spec.extraReach === true || (spec.extraReach === `calibration` && spec.calibrations.some(({ name }) => - expect.getState().currentTestName?.endsWith(` > calibration: ${name}`), + owningTestName()?.endsWith(` > calibration: ${name}`), )) ? [`synthetic-extra`] : []), @@ -782,7 +812,7 @@ function syntheticDriver( const history = itemsOf(spec)(formed) if ( (spec.isolatedPlans || spec.identityHistories) && - expect.getState().currentTestName?.includes(`calibration:`) + owningTestName()?.includes(`calibration:`) ) { if (consumed.has(formed as object)) throw new Error(`driver received a consumed plan`) @@ -792,7 +822,7 @@ function syntheticDriver( if (log) appendFileSync( log, - `${JSON.stringify({ id: spec.id, test: expect.getState().currentTestName ?? ``, history, origin: origins.get(formed as object), route: route ?? null })}\n`, + `${JSON.stringify({ id: spec.id, test: owningTestName() ?? ``, ...(spec.slow ? { runnerTest: expect.getState().currentTestName ?? `` } : {}), history, origin: origins.get(formed as object), route: owningTest.getStore()?.route ?? route ?? null })}\n`, ) if (spec.odd) observe(`odd`, { reason: oddValue(spec.odd) }) if (spec.slow) await new Promise((resolve) => setTimeout(resolve, 300)) @@ -800,7 +830,7 @@ function syntheticDriver( spec.slowShrink && history.includes(4) && / > (?:fixed seed -?\d+|random seed)(?:: .*)?$/.test( - expect.getState().currentTestName ?? ``, + owningTestName() ?? ``, ) ) await new Promise((resolve) => setTimeout(resolve, 2000)) @@ -1168,7 +1198,7 @@ export function syntheticProperty(spec: SyntheticSpec): void { `${log}.resolutions`, `${JSON.stringify({ id: spec.id, - test: expect.getState().currentTestName ?? ``, + test: owningTestName() ?? ``, })}\n`, ) if (spec.identityHistories && !identities.has(history as object)) @@ -1180,26 +1210,20 @@ export function syntheticProperty(spec: SyntheticSpec): void { return ( binding !== undefined && `constructed` in binding && - expect - .getState() - .currentTestName?.endsWith( - ` > calibration: ${calibration.name}`, - ) + owningTestName()?.endsWith( + ` > calibration: ${calibration.name}`, + ) ) }) || (spec.probeBinding && spec.probeBinding !== `empty` && - ((expect - .getState() - .currentTestName?.includes(`forced not entered`) && + ((owningTestName()?.includes(`forced not entered`) && (spec.probeBinding !== `explicit` || `constructed` in (spec.probeEntered ?? { constructed: `constructed-0`, }))) || - (expect - .getState() - .currentTestName?.includes(`forced entered`) && + (owningTestName()?.includes(`forced entered`) && `constructed` in (spec.probeNotEntered ?? { fixed: `fixed-0` }))))) ) { @@ -1428,7 +1452,7 @@ export function syntheticProperty(spec: SyntheticSpec): void { `${log}.builds`, `${JSON.stringify({ id: spec.id, - test: expect.getState().currentTestName ?? ``, + test: owningTestName() ?? ``, name, generated: items(generated), history: items(history), @@ -1454,7 +1478,7 @@ export function syntheticProperty(spec: SyntheticSpec): void { `${log}.builds`, `${JSON.stringify({ id: spec.id, - test: expect.getState().currentTestName ?? ``, + test: owningTestName() ?? ``, name: `fixed-metadata:${fixed.name}`, generated: items(generated), history: items(history), diff --git a/packages/oracles/tests/oracle-property-self-oracle.test.ts b/packages/oracles/tests/oracle-property-self-oracle.test.ts index 7cc45cf..d266401 100644 --- a/packages/oracles/tests/oracle-property-self-oracle.test.ts +++ b/packages/oracles/tests/oracle-property-self-oracle.test.ts @@ -108,7 +108,17 @@ * authored history's kind and name, and the original cause in every active * receiving child. Public original-history name/origin associations identify * opaque default enumeration labels; callback text cannot authorize selection. - * Seed annotations, punctuation, stacks and name hashing are not laws. + * Seed annotations, punctuation, stacks and name hashing are not laws. The + * receiving grammar varies quoted/bracketed calibration names, optional + * parenthesized/bracketed seed suffixes, and independently positive whitespace at structural + * separators, including incoming raw context variants before presentation. The + * generated separator sample includes one/two/three spaces, tabs and mixed gaps. + * Opaque callback reason prose varies independently, including empty text and + * words used by other diagnostics; actual exception cause fixes its kind. + * The outer diagnostic wrapper name varies independently and does not change + * callback kind; original cause name/object identity is not judged. + * A whole context frame must decode to one available authored identity; + * this finite renderer language is not arbitrary human-language parsing. * Both forced * premises independently select fixed or constructed originals; only omitted * entered selectors infer a unique construction. Equal data never permits @@ -245,7 +255,15 @@ * property. The model learns an unlabelled enumerated history's test name * from the log, so it checks that names are distinct, not their text. A slow * spec's late probe calls and routes are not counted exactly: its driver runs - * on after its test times out. That shrinking keeps a model error apart + * on after its test times out. Every child projects away slow invocation + * runner attribution and multiplicity, while retaining logical owning tests, + * each owner's first input, raw log/report diagnostics and per-started-call + * build/resolve/input relations. Determinism resolves one plan for both drives. + * Slow+late processes observe the deficit/excess against their raw started-call + * bounds; fast-only processes retain exact late multiplicity. The check rejects + * either nonzero relation and every unrelated error. Status, registration, + * process completeness and first build/resolve inputs remain compared independently. + * That shrinking keeps a model error apart * from a setup error has no witness: no shrink in this language turns one * into the other. The gap report's observation hook, * `forwardCut`, and `differs` have no law here; `withinTurns` has its own @@ -278,7 +296,9 @@ * four patterns, including the empty ones, judged against the difference a wrong answer * reports, `item k is 3`, where k is the first item or a `driver-last` candidate's last item. A * pattern with the global or sticky flag is not drawn. A difference that is not text is drawn - * only as a number. A calibration's failure message is judged by its kind, not by its text. + * only as a number. Calibration failure kinds are compared semantically. Callback identity + * and original cause are checked separately; exact diagnostic punctuation and seed decoration + * are not promised beyond the bounded presentation grammar exercised below. */ import { spawnSync } from 'node:child_process' import { @@ -289,6 +309,7 @@ import { writeFileSync, } from 'node:fs' import { createRequire } from 'node:module' +import { tmpdir } from 'node:os' import { dirname, join } from 'node:path' import { fileURLToPath } from 'node:url' import fc from 'fast-check' @@ -455,13 +476,36 @@ type Draft = { } // `seedless`: one more child run, without `ORACLE_SEED`, records the routes // of each random campaign as an ordinary run draws its seed. +type CallbackGap = 1 | 2 | 3 | `\t` | ` \t ` +type CallbackPresentation = { + calibration: `quoted` | `bracketed` + seed: `parenthesized` | `bracketed` | `omitted` + spacing: + | `plain` + | `padded` + | [ + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + ] +} type Suite = { + /** Equivalent bounded context renderings; alternate varies between two per drive. */ + callbackPresentation?: CallbackPresentation | `alternate` + callbackIncomingPresentation?: CallbackPresentation + callbackReason?: string + callbackWrapperName?: string drafts: Array seedless: boolean /** A controlled host draw, independent of the framework's seed selection. */ seedlessDraw?: number /** Exercise equivalent warning-name presentations across repeated drives. */ permuteWarnings?: boolean + /** Alternate legal post-timeout call attribution and order across drives. */ + timeoutPresentation?: boolean /** A controlled outer host draw, checked independently from receiving children. */ hostRngDraw?: 0.25 | 0.75 /** Equivalent outer-host wrappers; alternate one/two forwards across drives. */ @@ -513,6 +557,8 @@ const calibrationSelector = fc.oneof( constructed: fc.constantFrom( `constructed-0`, `constructed-1`, + `c`, + `c.1`, `unknown`, `missing`, ), @@ -911,9 +957,59 @@ const replayEnvironment: fc.Arbitrary = fc.record({ ), }) const cleanHostDraw = 0.5000000016298145 +const callbackPresentationArbitrary = fc.record({ + calibration: fc.constantFrom(`quoted` as const, `bracketed` as const), + seed: fc.constantFrom( + `parenthesized` as const, + `bracketed` as const, + `omitted` as const, + ), + spacing: fc.oneof( + fc.constantFrom(`plain` as const, `padded` as const), + fc.tuple( + ...Array.from({ length: 6 }, () => + fc.constantFrom( + 1 as const, + 2 as const, + 3 as const, + `\t` as const, + ` \t ` as const, + ), + ), + ) as fc.Arbitrary< + [ + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + ] + >, + ), +}) const genericSuite: fc.Arbitrary = fc.record( { drafts: fc.array(draft, { minLength: 1, maxLength: 2 }), + callbackWrapperName: fc.option(fc.string({ maxLength: 64 }), { + nil: undefined, + freq: 12, + }), + callbackReason: fc.option(fc.string({ maxLength: 192 }), { + nil: undefined, + freq: 12, + }), + callbackIncomingPresentation: fc.option(callbackPresentationArbitrary, { + nil: undefined, + freq: 12, + }), + callbackPresentation: fc.option( + fc.oneof( + fc.constant(`alternate` as const), + callbackPresentationArbitrary, + ), + { nil: undefined, freq: 12 }, + ), seedless: fc.oneof( { weight: 5, arbitrary: fc.constant(false) }, { weight: 1, arbitrary: fc.constant(true) }, @@ -942,6 +1038,7 @@ const genericSuite: fc.Arbitrary = fc.record( { weight: 1, arbitrary: replayEnvironment }, ), permuteWarnings: fc.option(fc.boolean(), { nil: undefined, freq: 12 }), + timeoutPresentation: fc.option(fc.boolean(), { nil: undefined, freq: 12 }), }, { requiredKeys: [`drafts`, `seedless`] }, ) @@ -957,10 +1054,16 @@ const genericSuite: fc.Arbitrary = fc.record( // constructed or fixed histories. type Plan = { + callbackPresentation?: CallbackPresentation | `alternate` + callbackIncomingPresentation?: CallbackPresentation + callbackReason?: string + callbackWrapperName?: string specs: Array seedless: boolean seedlessDraw?: number permuteWarnings?: boolean + /** Alternate legal post-timeout call attribution and order across drives. */ + timeoutPresentation?: boolean hostRngDraw?: 0.25 | 0.75 hostRngForwarding?: 1 | 2 | `alternate` prefix: Prefix @@ -981,18 +1084,28 @@ const constructedName = (names: Draft[`constructedNames`], at: number) => function resolveSuite({ drafts, + callbackPresentation, + callbackIncomingPresentation, + callbackReason, + callbackWrapperName, seedless, seedlessDraw, permuteWarnings, + timeoutPresentation, hostRngDraw, hostRngForwarding, prefix = `ORACLE`, environment = null, }: Suite): Plan { return { + ...(callbackPresentation ? { callbackPresentation } : {}), + ...(callbackIncomingPresentation ? { callbackIncomingPresentation } : {}), + ...(callbackReason === undefined ? {} : { callbackReason }), + ...(callbackWrapperName === undefined ? {} : { callbackWrapperName }), seedless, ...(seedlessDraw === undefined ? {} : { seedlessDraw }), ...(permuteWarnings ? { permuteWarnings: true } : {}), + ...(timeoutPresentation ? { timeoutPresentation: true } : {}), ...(hostRngDraw === undefined ? {} : { hostRngDraw }), ...(hostRngForwarding === undefined ? {} : { hostRngForwarding }), prefix, @@ -1250,6 +1363,8 @@ type LogEntry = { /** The route registered for the test when the history ran. */ route: string | null origin?: string + /** Raw diagnostic attribution; logical test ownership is `test`. */ + runnerTest?: string } const allowed = (item: number) => [0, 1, 2, 3, 4, 6, 7].includes(item) @@ -2646,7 +2761,11 @@ const callbackReceivingKinds = [ const callbackReceivingStates = callbackReceivingKinds.map( (kind) => `callback-failure-${kind}` as const, ) +const callbackRepresentationStates = callbackReceivingKinds.map( + (kind) => `callback-representation-${kind}` as const, +) const round4FrameworkStates = [ + ...callbackRepresentationStates, ...callbackReceivingStates, `host-rng-scope`, `host-rng-forwarding`, @@ -2684,7 +2803,19 @@ const round3FrameworkStates = [ `receiving-registration-warnings`, `receiving-environment`, ] as const +const timeoutReceivingKinds = [ + `run`, + `binding-reordered`, + `seedless`, + `environment`, +] as const +const timeoutReceivingStates = timeoutReceivingKinds.map( + (kind) => `timeout-input-${kind}` as const, +) const frameworkStates = [ + ...timeoutReceivingStates, + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, ...round3FrameworkStates, `auxiliary-late-error`, `bound-enumerated-label-selector`, @@ -2856,6 +2987,19 @@ function frameworkReach( for (const { checkpoint, value } of trace) { if (checkpoint !== `child`) continue const child = value as ChildRun + const timeoutKind = timeoutReceivingKinds.find( + (kind) => kind === child.context.kind, + ) + if ( + timeoutKind && + timeoutInputsMatch(plan, child) && + child.timeoutInputs?.some((owner) => + child.log.some( + (entry) => entry.id === owner.id && entry.test === owner.test, + ), + ) + ) + reached.add(`timeout-input-${timeoutKind}`) if ( child.loads.some(({ errors }) => errors.length === 0) && child.tests.length > 0 && @@ -2924,8 +3068,11 @@ function frameworkReach( calls.length === (driverCalibration(calibration.kind) ? 2 : 1) ) }) - ) + ) { reached.add(`callback-failure-${child.context.kind}`) + if (child.callbackPresented === true) + reached.add(`callback-representation-${child.context.kind}`) + } } if (spec.source.kind !== `generated`) continue const units = child.tests.filter( @@ -3662,6 +3809,23 @@ function frameworkReach( } if (trace.some(({ checkpoint }) => checkpoint === `rerun`)) reached.add(`rerun-probe`) + if ( + trace.some( + ({ checkpoint, value }) => + checkpoint === `timeout-presentation` && + (value as { exercised: boolean }).exercised, + ) + ) + reached.add(`timeout-presentation-equivalence`) + if ( + trace.some( + ({ checkpoint, value }) => + checkpoint === `timeout-presentation` && + (value as { cutExercised: boolean; stable: boolean }).cutExercised && + (value as { stable: boolean }).stable, + ) + ) + reached.add(`timeout-continuation-cut`) return reached } @@ -3761,6 +3925,8 @@ type SuiteRun = { warnings: Array /** The run's unhandled errors, by message. */ unhandled: Array + lateAgreement?: LateAgreement + timeoutInputs?: Array } // Every real child has a named receiving cut. The context is supplied by the // driver, independently of the test states Vitest reports, so absent tests or @@ -3799,7 +3965,10 @@ type ChildRun = { warnings: Array secondaryErrors: Array<{ test: string; message: string }> unhandled: Array + lateAgreement?: LateAgreement + timeoutInputs?: Array failed: boolean + callbackPresented?: boolean } type ProbeRun = { id: string @@ -3877,13 +4046,16 @@ const vitest = join( ) let runCount = 0 let warningPresentationCount = 0 +let timeoutPresentationCount = 0 let hostForwardingCount = 0 +let callbackPresentationCount = 0 function classifyKind( inner: ReportedError, text: string, outer: string, ): Kind | `other` { + if (callbackExceptionCause(inner)) return `setup` if (text.includes(`selected clean run failed`)) return `calibration-clean` if (text.includes(`selected history did not enter`)) return `calibration-reach` @@ -3928,27 +4100,247 @@ function classifyKind( return `other` } -// Read the reported identity without asking authored bindings what it should be. -// The construction's seed annotation is diagnostic representation, not identity. -const reportedCallbackContext = ( - message: string, -): Failure[`callbackContext`] => { - const match = - /calibration "([^"]+)" on (constructed|fixed|enumerated) history:\s*(.*?)(?=:\s*selected |$)/s.exec( - message, - ) - if (!match) return undefined +// This input transformation changes opaque reason prose after the actual primary +// callback frame, retaining the rendered identity and original finite cause. It +// does not consult the expected selected binding or restrict the reason's words. +const callbackExceptionCause = (error: ReportedError) => + [`synthetic corruption failed`, `synthetic judgment failed`].includes( + error.cause?.message ?? ``, + ) +// Outer wrapper names are presentation input. Preserve the actual original +// cause and all identity/message bytes while changing only the wrapper's name. +const incomingCallbackWrapper = ( + error: ReportedError | undefined, + name: string | undefined, +) => + error && + name !== undefined && + [`synthetic corruption failed`, `synthetic judgment failed`].includes( + error.cause?.message ?? ``, + ) + ? { ...error, name } + : error +const incomingCallbackReason = ( + error: ReportedError | undefined, + reason: string | undefined, + candidates: ReadonlyArray, +) => { + if (!error?.message || reason === undefined || !callbackExceptionCause(error)) + return error + const observed = observedCallbackFrame(error.message, candidates) + if (!observed) return error + const separator = /^:\s+/.exec(error.message.slice(observed.end)) + const suffix = error.message.lastIndexOf(`: ${error.cause!.message}`) + if (!separator || suffix < observed.end) return error + return { + ...error, + message: + error.message.slice(0, observed.end + separator[0].length) + + reason + + error.message.slice(suffix), + } +} + +const incomingCallbackError = ( + error: ReportedError | undefined, + presentation: CallbackPresentation | undefined, + candidates: ReadonlyArray, +) => { + if (!error?.message || !presentation) return error + const observed = observedCallbackFrame(error.message, candidates) + if (!observed) return error + const spaces = + presentation.spacing === `plain` + ? [1, 1, 1, 1, 1, 1] + : presentation.spacing === `padded` + ? [3, 3, 3, 3, 3, 3] + : presentation.spacing + const gaps = spaces.map((count) => + typeof count === `number` ? ` `.repeat(count) : count, + ) + const { calibration, kind, name } = observed.context + const label = + presentation.calibration === `quoted` + ? `"${calibration}"` + : `[${calibration}]` + const seed = + observed.seed === undefined || presentation.seed === `omitted` + ? `` + : gaps[5] + + (presentation.seed === `bracketed` + ? `[seed ${observed.seed}]` + : `(seed ${observed.seed})`) + const frame = `calibration${gaps[0]}${label}${gaps[1]}on${gaps[2]}${kind}${gaps[3]}history:${gaps[4]}${name}${seed}` return { - calibration: match[1]!, - kind: match[2]!, - name: - match[2] === `constructed` - ? match[3]!.trim().replace(/ \(seed -?\d+\)$/, ``) - : match[3]!.trim(), + ...error, + message: + error.message.slice(0, observed.start) + + frame + + error.message.slice(observed.end), + } +} + +// Presentation consumes the actual finite raw frame independently of its selected +// binding. Raw source reports and original causes remain unchanged. Incoming raw +// variants separately challenge this receiver before it renders a requested form. +const renderCallbackContext = ( + context: { calibration: string; kind: string; name: string }, + presentation: CallbackPresentation, + seed?: string, +) => { + const calibration = + presentation.calibration === `quoted` + ? `"${context.calibration}"` + : `[${context.calibration}]` + const spaces = + presentation.spacing === `plain` + ? [1, 1, 1, 1, 1, 1] + : presentation.spacing === `padded` + ? [3, 3, 3, 3, 3, 3] + : presentation.spacing + const gaps = spaces.map((count) => + typeof count === `number` ? ` `.repeat(count) : count, + ) + const annotation = + context.kind !== `constructed` || + seed === undefined || + presentation.seed === `omitted` + ? `` + : presentation.seed === `parenthesized` + ? `${gaps[5]}(seed ${seed})` + : `${gaps[5]}[seed ${seed}]` + return `calibration${gaps[0]}${calibration}${gaps[1]}on${gaps[2]}${context.kind}${gaps[3]}history:${gaps[4]}${context.name}${annotation}` +} +const presentCallbackError = ( + error: ReportedError | undefined, + presentation: CallbackPresentation | undefined, + candidates: ReadonlyArray, +) => { + if (!error?.message || !presentation) return { error, applied: false } + const observed = observedCallbackFrame(error.message, candidates) + if (!observed) return { error, applied: false } + // The consumed identity is the actual complete observed frame, never the + // expected selected binding. Successful rendering is an executed operation + // even when the source already uses this rendering and bytes stay equal. + const message = + error.message.slice(0, observed.start) + + renderCallbackContext(observed.context, presentation, observed.seed) + + error.message.slice(observed.end) + return { error: { ...error, message }, applied: true } +} + +// Decode a whole context frame against the available authored name universe, +// not the selected binding. Exact literal names remain exact inside their frame. +// The bounded rendering grammar varies wrappers, optional seed decoration, and +// positive structural whitespace. Distinct matching identities, unknown text, or multiple frames remain +// unrecognized; the shared comparison then rejects them. This is not an arbitrary +// English-message parser or a product promise about the frozen source's template. +type CallbackCandidate = NonNullable +const callbackPresentations: Array = [ + `quoted`, + `bracketed`, +].flatMap((calibration) => + [`parenthesized`, `bracketed`, `omitted`].flatMap((seed) => + [`plain`, `padded`].map( + (spacing) => ({ calibration, seed, spacing }) as CallbackPresentation, + ), + ), +) +const callbackCandidateUniverse = ( + plan: Plan, + nameLog: ReadonlyArray, +): Array => { + const candidates: Array = [] + for (const spec of plan.specs) { + const calibrations = [ + ...spec.calibrations.map(({ name }) => name), + ...forcedCalibrationNames(spec).map((name) => + name.slice(`calibration: `.length), + ), + ] + const histories = [ + ...spec.fixed.map(({ name }) => ({ kind: `fixed`, name })), + ...spec.constructed.map(({ name }) => ({ kind: `constructed`, name })), + ...(spec.source.kind !== `enumerated` + ? [] + : spec.source.histories.flatMap((history, index) => { + const literal = authoredEnumerationLabel(spec, history) + const names = + literal === undefined + ? [ + ...new Set( + nameLog + .filter( + (entry) => + entry.id === spec.id && + entry.origin === `enumerated:${index}` && + !frameworkTest( + entry.test.slice(spec.id.length + 3), + ), + ) + .map((entry) => entry.test.slice(spec.id.length + 3)), + ), + ] + : [literal] + return names.map((name) => ({ kind: `enumerated`, name })) + })), + ] + for (const calibration of calibrations) + for (const history of histories) + candidates.push({ calibration, ...history }) } + return [ + ...new Map( + candidates.map((candidate) => [JSON.stringify(candidate), candidate]), + ).values(), + ] +} +const escapeCallbackPattern = (text: string) => + text.replace(/[.*+?^${}()|[\]\\]/g, `\\$&`) +const observedCallbackFrame = ( + message: string, + candidates: ReadonlyArray, +) => { + // The reason/cause region is opaque. It cannot provide missing identity names. + const prefix = /^.*?: /s.exec(message) + if (!prefix) return undefined + const start = prefix[0].length + const frame = message.slice(start) + // Only structural separators vary. Literal spaces/punctuation inside authored + // names are escaped intact, including a name's numeric or seed-like suffix. + const gap = `\\s+` + const matches = candidates.flatMap((candidate) => { + const calibration = escapeCallbackPattern(candidate.calibration) + const label = `(?:"${calibration}"|\\[${calibration}\\])` + const suffix = + candidate.kind === `constructed` + ? `(?:${gap}(?:\\(seed (-?\\d+)\\)|\\[seed (-?\\d+)\\]))?` + : `` + const pattern = `^calibration${gap}${label}${gap}on${gap}${escapeCallbackPattern(candidate.kind)}${gap}history:${gap}${escapeCallbackPattern(candidate.name)}${suffix}(?=:\\s)` + const match = new RegExp(pattern).exec(frame) + return match + ? [ + { + context: candidate, + seed: match[1] ?? match[2], + start, + end: start + match[0].length, + }, + ] + : [] + }) + return matches.length === 1 ? matches[0] : undefined } +const reportedCallbackContext = ( + message: string, + candidates: ReadonlyArray, +): Failure[`callbackContext`] => + observedCallbackFrame(message, candidates)?.context -function classify(error: ReportedError | undefined): Failure { +function classify( + error: ReportedError | undefined, + candidates: ReadonlyArray = [], +): Failure { if (!error) return { kind: `other` } const outer = error.message ?? `` // Only fast-check's wrapper carries the framework's failure as its cause; @@ -3989,12 +4381,13 @@ function classify(error: ReportedError | undefined): Failure { // A data error names the value's path, then its kind. const data = /^(.*): (.*) is not (?:observation|history) data$/.exec(text) const kind = classifyKind(inner, text, outer) + const callbackContext = reportedCallbackContext(text, candidates) return { kind, - message: text, - ...(reportedCallbackContext(text) - ? { callbackContext: reportedCallbackContext(text) } - : {}), + // Context rendering is not a deterministic difference; keep its decoded + // actual identity while raw reports/sidecars retain the original text. + message: callbackContext ? JSON.stringify(callbackContext) : text, + ...(callbackContext ? { callbackContext } : {}), ...(inner.cause?.message ? { causeMessage: inner.cause.message } : {}), // A model error that quotes a machine path, such as an inspected stack. ...(inner.name === `OracleModelError` && text.includes(packageDirectory) @@ -4159,13 +4552,222 @@ const unnested = (name: string) => name.startsWith(`${outerSuite} > `) ? name.slice(outerSuite.length + 3) : name const short = (id: string, name: string) => unnested(name).replace(`${id} > `, ``) -// A slow spec can log under whichever test runs after its timeout. The raw -// child record retains these calls; this derived stream summary omits their -// host-dependent test attribution. +// A slow spec can start calls after its test times out. Attribution, order +// and count are then host-dependent. The deterministic observation keeps its +// late-error interval relation; the full raw log/report remain diagnostics. const settledLog = (plan: Plan, log: Array) => log.filter( ({ id }) => !plan.specs.some((spec) => spec.slow && spec.id === id), ) +// Model-derived sufficient statistics of a raw process report. This relation +// combines actual started-call input evidence with actual late-error output. +// It imports no production policy: fast calls owe one error each, while slow +// calls may still be running when the reporter closes. Only the checker +// decides whether the observed deficit/excess violates the law. +type LateAgreement = { belowMinimum: number; aboveMaximum: number } +const lateMessage = `probe synthetic#saw-0 was called after its history ended. The driver must await the work it starts, so every probe call belongs to one history.` +const usesLateRelation = (plan: Plan) => + plan.specs.some((spec) => spec.slow && spec.probe === `late`) +// Timed-out tests owe no completed call count. For each logical owner, retain +// its first input and the relations of every actually started execution. A +// determinant resolves one shared plan; other executions resolve individually. +// A selected constructed calibration builds once for each actual execution. +// Thus an optional late second call changes no observation, while extra or +// missing resolution/build work, or changed selected input, remains visible. +type TimeoutInputAgreement = { + id: string + test: string + resolutionDifference: number + buildDifference: number + changedInput: boolean + changedBuild: boolean + excessDeterminismCalls: number + excessCalibrationCalls: number +} +const invocationObservation = ( + plan: Plan, + log: Array, + unhandled: Array, + builds: Array = [], + resolutions: SuiteRun[`resolutions`] = [], +): Pick< + ChildRun, + | `log` + | `unhandled` + | `lateAgreement` + | `builds` + | `resolutions` + | `timeoutInputs` +> => { + const slow = (id: string) => + plan.specs.some((spec) => spec.id === id && spec.slow) + const firstInputs = ( + entries: Array, + ) => { + const seen = new Set() + return entries + .filter((entry) => { + if (!slow(entry.id) || entry.test === ``) return true + const key = JSON.stringify([entry.id, entry.test, entry.name]) + if (seen.has(key)) return false + seen.add(key) + return true + }) + .sort( + (a, b) => + a.id.localeCompare(b.id) || + (slow(a.id) ? a.test.localeCompare(b.test) : 0), + ) + } + const timeoutInputs: Array = [] + for (const spec of plan.specs.filter((candidate) => candidate.slow)) { + const tests = new Set( + [...log, ...builds, ...resolutions] + .filter(({ id, test }) => id === spec.id && test !== ``) + .map(({ test }) => test), + ) + for (const test of [...tests].sort()) { + const name = short(spec.id, test) + const calls = log.filter( + (entry) => entry.id === spec.id && entry.test === test, + ) + const constructed = builds.filter( + (entry) => entry.id === spec.id && entry.test === test, + ) + const resolved = resolutions.filter( + (entry) => entry.id === spec.id && entry.test === test, + ) + const forced = forcedCalibrationNames(spec).indexOf(name) + const calibration = spec.calibrations.find( + (entry) => `calibration: ${entry.name}` === name, + ) + const binding = + forced >= 0 && spec.probeBinding && spec.probeBinding !== `empty` + ? selectedControlBinding(spec, forced === 0) + : calibration + ? bindingOf(spec, calibration) + : undefined + const deterministic = name.startsWith(`determinism`) + const same = (a: unknown, b: unknown) => + JSON.stringify(a) === JSON.stringify(b) + timeoutInputs.push({ + id: spec.id, + test, + resolutionDifference: + spec.isolatedPlans || spec.identityHistories + ? resolved.length - + (deterministic ? Number(calls.length > 0) : calls.length) + : 0, + buildDifference: binding + ? constructed.length - (`constructed` in binding ? calls.length : 0) + : 0, + changedInput: Boolean( + (binding || deterministic) && + calls.some( + (entry) => + !same( + [entry.history, entry.origin], + [calls[0]!.history, calls[0]!.origin], + ), + ), + ), + changedBuild: Boolean( + binding && + constructed.some( + (entry) => + !same( + [entry.name, entry.generated, entry.history], + [ + constructed[0]!.name, + constructed[0]!.generated, + constructed[0]!.history, + ], + ), + ), + ), + excessDeterminismCalls: deterministic + ? Math.max(0, calls.length - 2) + : 0, + excessCalibrationCalls: binding + ? Math.max( + 0, + calls.length - + (forced >= 0 || + (calibration && driverCalibration(calibration.kind)) + ? 2 + : 1), + ) + : 0, + }) + } + } + const owed = lateCalls(plan, log) + const actual = unhandled.filter((message) => message === lateMessage).length + return { + log: firstInputs(log).map((entry) => { + const projected = { ...entry } + delete projected.runnerTest + return projected + }), + builds: firstInputs(builds), + resolutions: firstInputs(resolutions), + ...(plan.specs.some((spec) => spec.slow) ? { timeoutInputs } : {}), + // Only this known late-probe multiplicity has a timing-dependent bound. + // Unrelated errors remain exact, including similar but nonidentical text. + unhandled: usesLateRelation(plan) + ? unhandled.filter((message) => message !== lateMessage) + : unhandled, + ...(usesLateRelation(plan) + ? { + lateAgreement: { + belowMinimum: Math.max(0, owed.min - actual), + aboveMaximum: Math.max(0, actual - owed.max), + }, + } + : {}), + } +} +const timeoutInputsMatch = ( + plan: Plan, + seen: Pick, +) => { + if (!plan.specs.some((spec) => spec.slow)) return true + const owners = [ + ...new Set( + [...seen.log, ...seen.builds, ...seen.resolutions] + .filter( + ({ id, test }) => + test !== `` && + plan.specs.some((spec) => spec.id === id && spec.slow), + ) + .map(({ id, test }) => JSON.stringify([id, test])), + ), + ].sort() + return ( + seen.timeoutInputs !== undefined && + JSON.stringify( + seen.timeoutInputs + .map(({ id, test }) => JSON.stringify([id, test])) + .sort(), + ) === JSON.stringify(owners) && + seen.timeoutInputs.every( + (entry) => + entry.resolutionDifference === 0 && + entry.buildDifference === 0 && + !entry.changedInput && + !entry.changedBuild && + entry.excessDeterminismCalls === 0 && + entry.excessCalibrationCalls === 0, + ) + ) +} +const lateAgreementMatches = ( + plan: Plan, + seen: { lateAgreement?: LateAgreement }, +) => + !usesLateRelation(plan) || + (seen.lateAgreement?.belowMinimum === 0 && + seen.lateAgreement.aboveMaximum === 0) const readLog = (log: string): Array => existsSync(log) ? readFileSync(log, `utf8`) @@ -4350,6 +4952,90 @@ function driveSuite( plan: Plan, observe: (checkpoint: string, value: unknown) => void, ) { + const timeoutPresentation = plan.timeoutPresentation + ? timeoutPresentationCount++ % 2 === 0 + : false + const presentation = + plan.callbackPresentation === `alternate` + ? callbackPresentationCount++ % 2 === 0 + ? { + calibration: `bracketed` as const, + seed: `parenthesized` as const, + spacing: `padded` as const, + } + : { + calibration: `quoted` as const, + seed: `bracketed` as const, + spacing: `plain` as const, + } + : plan.callbackPresentation + const presentedDiagnostics: Array<{ before: string; after: string }> = [] + let reasonInputCount = 0 + let wrapperInputCount = 0 + const classifyForSuite = ( + error: ReportedError | undefined, + nameLog?: ReadonlyArray, + ): Failure => { + const candidates = callbackCandidateUniverse( + plan, + nameLog ?? readLog(join(directory, `log.jsonl`)), + ) + const reasoned = incomingCallbackReason( + error, + plan.callbackReason, + candidates, + ) + if ( + reasoned?.message && + error?.message && + reasoned.message !== error.message + ) { + writeFileSync( + join(directory, `callback-reason-${reasonInputCount++}.json`), + JSON.stringify( + { before: error.message, after: reasoned.message }, + null, + 2, + ), + ) + } + const wrapped = incomingCallbackWrapper(reasoned, plan.callbackWrapperName) + if (wrapped?.name !== reasoned?.name) + writeFileSync( + join(directory, `callback-wrapper-${wrapperInputCount++}.json`), + JSON.stringify( + { + before: reasoned?.name, + after: wrapped?.name, + message: wrapped?.message, + cause: wrapped?.cause, + }, + null, + 2, + ), + ) + const incoming = incomingCallbackError( + wrapped, + plan.callbackIncomingPresentation, + candidates, + ) + const { error: presented, applied } = presentCallbackError( + incoming, + presentation, + candidates, + ) + if (applied && incoming?.message && presented?.message) { + presentedDiagnostics.push({ + before: incoming.message, + after: presented.message, + }) + writeFileSync( + join(directory, `callback-presentations.json`), + JSON.stringify(presentedDiagnostics, null, 2), + ) + } + return classify(presented, candidates) + } const warningPresentation = plan.permuteWarnings ? warningPresentationCount++ % 2 === 0 : undefined @@ -4378,11 +5064,138 @@ function driveSuite( context, ...report.warningPermutation, }) + const rawLog = readLog(childLog) + // These calls have already outlived their owning test. Reordering them + // and changing the current runner name leaves every timeout obligation + // unchanged; the receiving report and late-error capacity stay intact. + const slowCalls = rawLog.filter(({ id }) => + plan.specs.some((spec) => spec.id === id && spec.slow), + ) + const presentedLog = timeoutPresentation + ? [ + ...settledLog(plan, rawLog), + ...[...new Set(slowCalls.map((entry) => entry.test))] + .reverse() + .flatMap((test) => + slowCalls + .filter((entry) => entry.test === test) + .map((entry) => ({ + ...entry, + runnerTest: `${entry.id} > post-timeout continuation`, + })), + ), + ] + : rawLog + const rawUnhandled = report.unhandled.map(({ message }) => message ?? ``) + const rawBuilds = readBuilds(childLog) + const rawResolutions = readResolutions(`${childLog}.resolutions`) + const cutLog = [...presentedLog] + const cutBuilds = [...rawBuilds] + const cutResolutions = [...rawResolutions] + let cutExercised = false + for (const spec of plan.specs.filter((candidate) => candidate.slow)) { + for (const test of new Set( + slowCalls + .filter(({ id }) => id === spec.id) + .map((entry) => entry.test), + )) { + const name = short(spec.id, test) + const calibration = spec.calibrations.find( + (entry) => `calibration: ${entry.name}` === name, + ) + const forced = Boolean( + spec.probeBinding && + spec.probeBinding !== `empty` && + forcedCalibrationNames(spec).includes(name), + ) + const bound = forced || (calibration && bindingOf(spec, calibration)) + const calls = cutLog.filter( + (entry) => entry.id === spec.id && entry.test === test, + ) + if (calls.length < 2 || (!name.startsWith(`determinism`) && !bound)) + continue + // Only the optional continuation after the first slow call is cut. + // Its paired input work is omitted at the same legal reporting cut. + cutLog.splice(cutLog.indexOf(calls.at(-1)!), 1) + if (!name.startsWith(`determinism`)) { + const resolved = cutResolutions.lastIndexOf( + [...cutResolutions] + .reverse() + .find((entry) => entry.id === spec.id && entry.test === test)!, + ) + if (resolved >= 0) cutResolutions.splice(resolved, 1) + const built = cutBuilds.lastIndexOf( + [...cutBuilds] + .reverse() + .find((entry) => entry.id === spec.id && entry.test === test)!, + ) + if (built >= 0) cutBuilds.splice(built, 1) + } + cutExercised = true + } + } + const cutBounds = lateCalls(plan, cutLog) + const actualLate = rawUnhandled.filter( + (message) => message === lateMessage, + ).length + let keptLate = 0 + const cutLate = Math.max( + cutBounds.min, + Math.min(cutBounds.max, actualLate), + ) + const cutUnhandled = rawUnhandled.filter( + (message) => message !== lateMessage || ++keptLate <= cutLate, + ) + const rawObservation = invocationObservation( + plan, + rawLog, + rawUnhandled, + rawBuilds, + rawResolutions, + ) + const cutObservation = invocationObservation( + plan, + cutLog, + cutUnhandled, + cutBuilds, + cutResolutions, + ) + const observation = timeoutPresentation ? cutObservation : rawObservation + // A diagnostic is deliberately outside the deterministic trace; no raw + // evidence is destroyed by its semantic projection. + if (slowCalls.length || usesLateRelation(plan)) { + const diagnostics = join( + tmpdir(), + `oracle-property-timeout-${process.pid}`, + ) + mkdirSync(diagnostics, { recursive: true }) + writeFileSync( + join(diagnostics, `${runCount}-${context.kind}.json`), + JSON.stringify({ + context, + log: rawLog, + presentedLog, + builds: rawBuilds, + resolutions: rawResolutions, + cutLog, + cutBuilds, + cutResolutions, + cutUnhandled, + report, + }), + ) + } + if (plan.timeoutPresentation && context.kind === `run`) + observe(`timeout-presentation`, { + exercised: slowCalls.length > 0, + cutExercised, + stable: + JSON.stringify(rawObservation) === JSON.stringify(cutObservation), + }) + const presentationsBeforeChild = presentedDiagnostics.length observe(`child`, { context, - log: readLog(childLog), - builds: readBuilds(childLog), - resolutions: readResolutions(`${childLog}.resolutions`), + ...observation, // Independent property files may register in either order. warnings: [...report.warnings].sort(), tests: report.files @@ -4393,7 +5206,7 @@ function driveSuite( name: short(id, test.name), state: test.state, errors: test.errors.map((error) => - diagnosticFacts(classify(error)), + diagnosticFacts(classifyForSuite(error, readLog(childLog))), ), seedLogs: seedLogsOf(test, id), })) @@ -4404,6 +5217,12 @@ function driveSuite( errors: errors.map(({ message }) => message ?? ``), causes: errors.map(({ cause }) => cause?.message ?? ``), })), + ...(presentation + ? { + callbackPresented: + presentedDiagnostics.length > presentationsBeforeChild, + } + : {}), exit: report.exit, signal: report.signal, files: report.files.map(({ file }) => @@ -4419,7 +5238,6 @@ function driveSuite( .map(({ message }) => ({ test: name, message: message ?? `` })), ), ), - unhandled: report.unhandled.map(({ message }) => message ?? ``), failed: report.files.some( (file) => file.errors.length > 0 || @@ -4443,13 +5261,16 @@ function driveSuite( }) const run: SuiteRun = { files: {}, - log: settledLog(plan, readLog(log)), - resolutions: readResolutions(`${log}.resolutions`), - builds: readBuilds(log), + ...invocationObservation( + plan, + readLog(log), + report.unhandled.map((error) => error.message ?? ``), + readBuilds(log), + readResolutions(`${log}.resolutions`), + ), // Independent files may finish in either order; every line remains. stdout: report.stdout.split(`\n`).sort().join(`\n`), warnings: [...report.warnings].sort(), - unhandled: report.unhandled.map((error) => error.message ?? ``), } for (const [index, spec] of plan.specs.entries()) { const reported = report.files.find((file) => file.file === files[index]) @@ -4472,7 +5293,7 @@ function driveSuite( name: short(spec.id, test.name), state: test.state, ...(test.state === `failed` - ? { failure: classify(test.errors[0]) } + ? { failure: classifyForSuite(test.errors[0]) } : {}), seedLogs: seedLogsOf(test, spec.id), })), @@ -4514,7 +5335,13 @@ function driveSuite( }) observe(`binding-reordered`, { id: spec.id, - log: readLog(reorderedLog), + log: invocationObservation( + plan, + readLog(reorderedLog), + [], + readBuilds(reorderedLog), + readResolutions(`${reorderedLog}.resolutions`), + ).log, }) } finally { writeFileSync(file, original) @@ -4703,7 +5530,7 @@ function driveSuite( probed, states: replay ? statesIn(replay, plan, files) : [], ...(random?.state === `failed` - ? { failure: classify(random.errors[0]) } + ? { failure: classifyForSuite(random.errors[0]) } : {}), ran: replayed.map((entry) => entry.history), routes: replayed.map((entry) => entry.route), @@ -4747,7 +5574,7 @@ function driveSuite( original: target.failure!, probed: target.name, states: statesIn(rerun, plan, files), - ...(ran ? { failure: classify(ran.errors[0]) } : {}), + ...(ran ? { failure: classifyForSuite(ran.errors[0]) } : {}), } satisfies ProbeRun) } } @@ -4776,7 +5603,7 @@ function driveSuite( name: short(spec.id, test.name), state: test.state, ...(test.state === `failed` - ? { failure: classify(test.errors[0]) } + ? { failure: classifyForSuite(test.errors[0]) } : {}), })), }, @@ -4885,7 +5712,7 @@ test('host seed diagnostic',()=>{ id: spec.id, state: test?.state ?? `missing`, ...(test?.state === `failed` - ? { failure: classify(test.errors[0]) } + ? { failure: classifyForSuite(test.errors[0]) } : {}), ...(printed ? { reach: JSON.parse(printed) as unknown } : {}), others: statesIn(alone, plan, files).filter( @@ -5454,6 +6281,30 @@ function checkSuite( `permuting actual warning names changed their semantic observation`, `calibration`, ) + if ( + plan.timeoutPresentation && + plan.specs.some((spec) => spec.slow) && + !trace.some( + ({ checkpoint, value }) => + checkpoint === `timeout-presentation` && + (value as { exercised: boolean }).exercised, + ) + ) + return violation( + `child`, + `timeout presentation was not exercised`, + `timeout`, + ) + for (const { checkpoint, value } of trace) + if ( + checkpoint === `timeout-presentation` && + !(value as { stable: boolean }).stable + ) + return violation( + `child`, + `legal timeout presentation changed its semantic observation`, + `timeout`, + ) const children = trace.filter(({ checkpoint }) => checkpoint === `child`) const owedChildren = 1 + @@ -5482,7 +6333,6 @@ function checkSuite( ) // Expected synthetic failures exit 1; unexpected process failures do not // disappear merely because a reporter managed to write a partial result. - const lateMessage = `probe synthetic#saw-0 was called after its history ended. The driver must await the work it starts, so every probe call belongs to one history.` const ordinaryLog = children .map(({ value }) => value as ChildRun) @@ -5553,11 +6403,26 @@ function checkSuite( } const modeled = childExpectation(plan, child, expected, ordinaryLog) if (modeled.failure) return modeled.failure + if (!timeoutInputsMatch(plan, child)) + return violation( + `child`, + `timed-out input work disagrees: ${JSON.stringify(child.timeoutInputs ?? null)}`, + `calibration`, + ) + if (!lateAgreementMatches(plan, child)) + return violation( + `child`, + `late probe interval disagrees: ${JSON.stringify(child.lateAgreement ?? null)}`, + `agreement`, + ) const owedLate = lateCalls(plan, child.log) const actualLate = child.unhandled.filter( (message) => message === lateMessage, ).length - if (actualLate < owedLate.min || actualLate > owedLate.max) + if ( + !usesLateRelation(plan) && + (actualLate < owedLate.min || actualLate > owedLate.max) + ) return violation( `child`, `child reported ${actualLate} late probe calls, owed ${owedLate.min} to ${owedLate.max}`, @@ -5576,7 +6441,12 @@ function checkSuite( ) if ( child.signal !== null || - child.exit !== (modeled.failed || child.unhandled.length ? 1 : 0) || + child.exit !== + (modeled.failed || + child.unhandled.length || + (usesLateRelation(plan) && owedLate.min > 0) + ? 1 + : 0) || JSON.stringify([...child.files].sort()) !== JSON.stringify([...child.expectedFiles].sort()) ) @@ -5850,7 +6720,13 @@ function checkSuite( if ( spec.probeBinding && spec.probeBinding !== `empty` && - forcedCalibrationNames(spec).includes(test.name) + forcedCalibrationNames(spec).includes(test.name) && + (!spec.slow || + ordinaryLog.some( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + )) ) { const calls = ordinaryLog.filter( (entry) => @@ -5879,9 +6755,10 @@ function checkSuite( `${spec.id} forced control ran a different declared premise`, `calibration`, ) - const expectedCalls = - outcome.kind === `calibration-clean` || - outcome.kind === `calibration-reach` + const expectedCalls = spec.slow + ? 1 + : outcome.kind === `calibration-clean` || + outcome.kind === `calibration-reach` ? 1 : 2 const expectedBuilds = @@ -5930,15 +6807,24 @@ function checkSuite( ({ name }) => test.name === `calibration: ${name}`, ) const calibrationBinding = calibration && bindingOf(spec, calibration) - if (calibrationBinding) { + if ( + calibrationBinding && + (!spec.slow || + ordinaryLog.some( + (entry) => + entry.id === spec.id && + entry.test === `${spec.id} > ${test.name}`, + )) + ) { const calls = ordinaryLog.filter( (entry) => entry.id === spec.id && entry.test === `${spec.id} > ${test.name}`, ) - const expectedCalls = - driverCalibration(calibration.kind) && - outcome.kind !== `calibration-clean` && - outcome.kind !== `calibration-reach` + const expectedCalls = spec.slow + ? 1 + : driverCalibration(calibration.kind) && + outcome.kind !== `calibration-clean` && + outcome.kind !== `calibration-reach` ? 2 : 1 const expectedConstruction = @@ -6385,7 +7271,19 @@ function checkSuite( message.includes(`was called after its history ended`), ).length const owedLate = lateCalls(plan, ordinaryLog) - if (late < owedLate.min || late > owedLate.max) + if (!timeoutInputsMatch(plan, run)) + return violation( + `run`, + `timed-out input work disagrees: ${JSON.stringify(run.timeoutInputs ?? null)}`, + `calibration`, + ) + if (!lateAgreementMatches(plan, run)) + return violation( + `run`, + `late probe interval disagrees: ${JSON.stringify(run.lateAgreement ?? null)}`, + `agreement`, + ) + if (!usesLateRelation(plan) && (late < owedLate.min || late > owedLate.max)) return violation( `run`, `the run reported ${late} late probe calls, owed ${owedLate.min === owedLate.max ? owedLate.min : `${owedLate.min} to ${owedLate.max}`}`, @@ -6844,6 +7742,47 @@ const callbackDiagnosticSuite = ( }), ], }) +// The same callback grammar supplies alternate public representations. Single +// originals distinguish name presence from choosing among siblings; prefixed +// names distinguish complete identity from an incidental substring. Default +// enumeration labels keep numeric data beside construction-only seed metadata. +const callbackRepresentationSuite = ( + kind: (typeof callbackKinds)[number], + selection: (typeof callbackSelections)[number], + siblings: boolean, + presentation: Suite[`callbackPresentation`], + tail: 1 | 2 = 1, + prefix: Prefix = `ORACLE`, +): Suite => { + const value = callbackDiagnosticSuite(kind, selection, tail, prefix) + value.callbackPresentation = presentation + const d = value.drafts[0]! + if (!siblings) { + d.calibrations = d.calibrations.slice(0, 1) + d.fixed = d.fixed.slice(0, 1) + d.constructed = selection === `constructed` ? d.constructed.slice(0, 1) : [] + if (d.source.kind === `enumerated`) + d.source.histories = d.source.histories.slice(0, 1) + } + if (selection === `enumerated`) d.labelStyle = `default` + else { + d.constructedNames = `prefixed` + d.calibrations = d.calibrations.map((entry) => + typeof entry === `string` || + !entry.history || + !(`constructed` in entry.history) + ? entry + : { + ...entry, + history: { + constructed: + entry.history.constructed === `constructed-0` ? `c` : `c.1`, + }, + }, + ) + } + return value +} const suite: fc.Arbitrary = fc.oneof( { weight: 8, arbitrary: genericSuite }, { @@ -6854,9 +7793,41 @@ const suite: fc.Arbitrary = fc.oneof( selection: fc.constantFrom(...callbackSelections), tail: fc.constantFrom(1 as const, 2 as const), prefix: fc.constantFrom(`ORACLE` as const, `SYNTH_ORACLE` as const), + siblings: fc.boolean(), + presentation: fc.oneof( + fc.constantFrom(...callbackPresentations), + callbackPresentationArbitrary, + ), + incoming: fc.option(callbackPresentationArbitrary, { nil: undefined }), + reason: fc.option(fc.string({ maxLength: 192 }), { nil: undefined }), + wrapperName: fc.option(fc.string({ maxLength: 64 }), { + nil: undefined, + }), }) - .map(({ kind, selection, tail, prefix }) => - callbackDiagnosticSuite(kind, selection, tail, prefix), + .map( + ({ + kind, + selection, + tail, + prefix, + siblings, + presentation, + incoming, + reason, + wrapperName, + }) => ({ + ...callbackRepresentationSuite( + kind, + selection, + siblings, + presentation, + tail, + prefix, + ), + callbackIncomingPresentation: incoming, + callbackReason: reason, + callbackWrapperName: wrapperName, + }), ), }, ) @@ -6912,6 +7883,48 @@ const campaignBudgetSuite = ( }), ], }) +const timeoutRepresentationSuite = (tail: 1 | 2 = 1): Suite => ({ + seedless: false, + timeoutPresentation: true, + drafts: [ + generatedDraft({ timing: `slow`, probe: `late`, fixed: [at([0, tail])] }), + generatedDraft({ probe: `late`, fixed: [at([0, tail])] }), + ], +}) +const timeoutInputSuite = (bound: boolean, tail: 1 | 2 = 1): Suite => ({ + seedless: false, + timeoutPresentation: true, + drafts: [ + generatedDraft({ + timing: `slow`, + probe: `late`, + probeBinding: `explicit`, + isolatedPlans: true, + fixed: [at([1, tail])], + constructed: [{ prefix: [0], enters: [`saw-0`] }], + ...(bound + ? { + calibrations: [`driver`], + calibrationHistory: { constructed: `constructed-0` }, + } + : {}), + }), + ], +}) +const timeoutReceivingSuite = (tail: 1 | 2 = 1): Suite => { + const base = timeoutInputSuite(true, tail) + return { + ...base, + seedless: true, + prefix: tail === 1 ? `ORACLE` : `SYNTH_ORACLE`, + environment: replayOf({ + path: `absent`, + property: `absent`, + unit: `absent`, + }), + drafts: base.drafts.map((entry) => ({ ...entry, warningProbe: true })), + } +} const warningRepresentationSuite = (): Suite => ({ drafts: [ generatedDraft({ @@ -7145,8 +8158,12 @@ oracleProperty({ histories: { arbitrary: suite, states: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ...timeoutReceivingStates, ...mainUnitRequiredStates, ...callbackReceivingStates, + ...callbackRepresentationStates, ...round3FrameworkStates, `auxiliary-late-error`, `bound-enumerated-label-selector`, @@ -7176,9 +8193,13 @@ oracleProperty({ ], measured: frameworkStates.filter( (state) => + state !== `timeout-presentation-equivalence` && + state !== `timeout-continuation-cut` && + !timeoutReceivingStates.some((required) => required === state) && state !== `auxiliary-late-error` && !mainUnitRequiredStates.some((required) => required === state) && !callbackReceivingStates.some((required) => required === state) && + !callbackRepresentationStates.some((required) => required === state) && !round3FrameworkStates.some((required) => required === state) && !state.startsWith(`bound-`) && state !== `unbound-prototype-probe` && @@ -7187,7 +8208,121 @@ oracleProperty({ runs: 3, fixedSeeds: [1], constructed: [ + { + name: `timeout every receiving cut`, + enters: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ...timeoutReceivingStates, + ], + build: (generated: Suite) => + timeoutReceivingSuite(generated.drafts[0]!.clock ? 1 : 2), + }, + ...([false, true] as const).map((bound) => ({ + name: `timeout ${bound ? `bound` : `isolated`} construction`, + enters: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ] as const, + build: (generated: Suite) => + timeoutInputSuite(bound, generated.drafts[0]!.clock ? 1 : 2), + })), + { + name: `timeout continuation presentation`, + enters: [`timeout-presentation-equivalence`], + build: (generated: Suite) => + timeoutRepresentationSuite(generated.drafts[0]!.clock ? 1 : 2), + }, ...mainUnitRequiredConstructed, + { + name: `round8 callback presentation freedom`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + build: (generated: Suite) => + callbackRepresentationSuite( + generated.drafts[0]!.clock + ? `corrupt-throws` + : `judge-driver-throws`, + `constructed`, + generated.drafts[0]!.isolatedPlans, + `alternate`, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + }, + { + name: `round11 outer callback wrapper`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + build: (generated: Suite) => ({ + ...callbackRepresentationSuite( + generated.drafts[0]!.clock + ? `corrupt-throws` + : `judge-driver-throws`, + `constructed`, + generated.drafts[0]!.isolatedPlans, + `alternate`, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + callbackIncomingPresentation: + generated.callbackIncomingPresentation ?? { + calibration: `bracketed`, + seed: `parenthesized`, + spacing: [2, `\t`, 3, ` \t `, 2, 1], + }, + callbackReason: generated.callbackReason ?? `but the check passed it`, + callbackWrapperName: + generated.callbackWrapperName ?? `CalibrationFailure`, + }), + }, + { + name: `round10 opaque callback reason`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + build: (generated: Suite) => ({ + ...callbackRepresentationSuite( + generated.drafts[0]!.clock + ? `corrupt-throws` + : `judge-driver-throws`, + `constructed`, + generated.drafts[0]!.isolatedPlans, + `alternate`, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + callbackIncomingPresentation: + generated.callbackIncomingPresentation ?? { + calibration: `bracketed`, + seed: `parenthesized`, + spacing: [2, `\t`, 3, ` \t `, 2, 1], + }, + callbackReason: + generated.callbackReason ?? + (generated.drafts[0]!.clock + ? `but the check passed it` + : `calibration [unrelated] on fixed history: c.1: selected another context`), + }), + }, + { + name: `round9 incoming callback context`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + build: (generated: Suite) => ({ + ...callbackRepresentationSuite( + generated.drafts[0]!.clock + ? `corrupt-throws` + : `judge-driver-throws`, + `constructed`, + generated.drafts[0]!.isolatedPlans, + `alternate`, + generated.drafts[0]!.clock ? 1 : 2, + generated.drafts[0]!.isolatedPlans ? `ORACLE` : `SYNTH_ORACLE`, + ), + callbackIncomingPresentation: + generated.callbackIncomingPresentation ?? { + calibration: `bracketed`, + seed: `parenthesized`, + spacing: [3, 3, 3, 3, 3, 1], + }, + }), + }, ...callbackKinds.map((kind, ordinal) => ({ name: `round7 callback diagnostics ${kind}`, enters: @@ -7810,6 +8945,145 @@ oracleProperty({ }, ], fixed: [ + { + name: `timeout every receiving cut`, + enters: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ...timeoutReceivingStates, + ], + history: timeoutReceivingSuite(), + }, + ...([false, true] as const).map((bound) => ({ + name: `timeout ${bound ? `bound` : `isolated`} construction`, + enters: [ + `timeout-presentation-equivalence`, + `timeout-continuation-cut`, + ] as const, + history: timeoutInputSuite(bound), + })), + { + name: `timeout continuation presentation`, + enters: [`timeout-presentation-equivalence`], + history: timeoutRepresentationSuite(), + }, + ...([`wrapper`, `reason`, `combined`] as const).map((axis) => ({ + name: `round11 outer wrapper ${axis}`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + ...(axis === `reason` + ? {} + : { callbackWrapperName: `CalibrationFailure` }), + ...(axis === `wrapper` + ? {} + : { callbackReason: `but the check passed it` }), + }, + })), + ...( + [`but the check passed it`, `but the check reported law`] as const + ).map((reason) => ({ + name: `round10 opaque reason ${reason}`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + callbackReason: reason, + }, + })), + ...([false, true] as const).map((identityWords) => ({ + name: identityWords + ? `round10 opaque reason contains identity words` + : `round10 opaque callback reason`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + callbackIncomingPresentation: { + calibration: `bracketed` as const, + seed: `parenthesized` as const, + spacing: [2, `\t`, 3, ` \t `, 2, 1] as [ + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + CallbackGap, + ], + }, + callbackReason: identityWords + ? `calibration [unrelated] on fixed history: c.1: selected another context` + : `corruption callback failed`, + }, + })), + { + name: `round9 incoming positive whitespace`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + callbackIncomingPresentation: { + calibration: `bracketed`, + seed: `parenthesized`, + spacing: [2, `\t`, 3, ` \t `, 2, 1], + }, + }, + }, + ...([false, true] as const).map((mixed) => ({ + name: mixed + ? `round9 incoming mixed separators` + : `round9 incoming bracketed context`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: { + ...callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + callbackIncomingPresentation: { + calibration: `bracketed` as const, + seed: `parenthesized` as const, + spacing: mixed + ? ([3, 3, 3, 3, 3, 1] as [ + 1 | 3, + 1 | 3, + 1 | 3, + 1 | 3, + 1 | 3, + 1 | 3, + ]) + : (`plain` as const), + }, + }, + })), + { + name: `round8 callback presentation freedom`, + enters: [...callbackReceivingStates, ...callbackRepresentationStates], + history: callbackRepresentationSuite( + `corrupt-throws`, + `constructed`, + true, + `alternate`, + ), + }, ...callbackKinds.map((kind, ordinal) => ({ name: `round7 callback diagnostics ${kind}`, enters: @@ -9508,6 +10782,120 @@ oracleProperty({ driver: driveSuite, check: checkSuite, calibrations: [ + ...timeoutReceivingKinds.map((receiver) => ({ + name: `reject timeout ${receiver} input work`, + history: { fixed: `timeout every receiving cut` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find( + ({ checkpoint, value }) => + checkpoint === `child` && + (value as ChildRun).context.kind === receiver, + )!.value as ChildRun + child.timeoutInputs![0]!.resolutionDifference = 1 + return altered + }, + })), + ...( + [ + `resolution`, + `construction`, + `changed input`, + `changed build`, + `determinism calls`, + `calibration calls`, + `missing relation`, + `selected origin`, + ] as const + ).map((fault) => ({ + name: `reject timeout input ${fault}`, + history: { fixed: `timeout bound construction` }, + law: `calibration`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find( + ({ checkpoint, value }) => + checkpoint === `child` && + (value as ChildRun).context.kind === `run`, + )!.value as ChildRun + const owner = child.timeoutInputs!.find(({ test }) => + test.endsWith(` > calibration: cal-0`), + )! + if (fault === `resolution`) owner.resolutionDifference = 1 + if (fault === `construction`) owner.buildDifference = 1 + if (fault === `changed input`) owner.changedInput = true + if (fault === `changed build`) owner.changedBuild = true + if (fault === `determinism calls`) owner.excessDeterminismCalls = 1 + if (fault === `calibration calls`) owner.excessCalibrationCalls = 1 + if (fault === `missing relation`) + child.timeoutInputs!.splice(child.timeoutInputs!.indexOf(owner), 1) + if (fault === `selected origin`) + for (const logged of ordinaryInvocationLogs(altered)) { + const entry = logged.find(({ test }) => + test.endsWith(` > calibration: cal-0`), + )! + entry.origin = `constructed:unrelated` + } + return altered + }, + })), + ...( + [ + `missing late`, + `excess late`, + `status`, + `timeout`, + `registration`, + `exit`, + `unhandled`, + `presentation`, + ] as const + ).map((fault) => ({ + name: `reject timeout continuation ${fault}`, + history: { fixed: `timeout continuation presentation` }, + law: + fault === `missing late` || fault === `excess late` + ? `agreement` + : fault === `registration` + ? `registration` + : fault === `exit` || fault === `unhandled` + ? `trace` + : `timeout`, + corrupt: ( + trace: ReadonlyArray<{ checkpoint: string; value: unknown }>, + ) => { + const altered = structuredClone([...trace]) + const child = altered.find( + ({ checkpoint, value }) => + checkpoint === `child` && + (value as ChildRun).context.kind === `run`, + )!.value as ChildRun + const timedOut = child.tests.find( + ({ id, state }) => id === syntheticIds[0] && state === `failed`, + )! + if (fault === `missing late`) child.lateAgreement!.belowMinimum = 1 + if (fault === `excess late`) child.lateAgreement!.aboveMaximum = 1 + if (fault === `status`) timedOut.state = `passed` + if (fault === `timeout`) timedOut.errors[0] = { kind: `other` } + if (fault === `registration`) + child.tests.splice(child.tests.indexOf(timedOut), 1) + if (fault === `exit`) child.exit = 0 + if (fault === `unhandled`) + child.unhandled.push(`unrelated timeout teardown error`) + if (fault === `presentation`) + ( + altered.find( + ({ checkpoint }) => checkpoint === `timeout-presentation`, + )!.value as { stable: boolean } + ).stable = false + return altered + }, + })), ...callbackReceivingKinds.flatMap((receiver) => ([`context`, `selected name`, `cause`] as const).map((fault) => ({ name: `reject callback ${receiver} lost ${fault}`, diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md index 5caf0c4..978404d 100644 --- a/reviews/calibration-history-self-oracle.md +++ b/reviews/calibration-history-self-oracle.md @@ -557,3 +557,131 @@ Historical CHRON-1 remains irreversible; optional mutant-only undeclared-state policy remains a separate unapproved proposal. The finite synthetic grammar, Node/Vitest receiving boundary and absence of downstream DB/browser/performance claims remain unchanged. + +## Diagnostic representation and timeout observation follow-up + +This follow-up starts from `c18a30f`. Production retains SHA256 +`a1c132e8e096228e34aad07ba39c20df88facfe998e8c8e4e155ff75b213e01b`. +These repairs change the self-oracle and its synthetic fixture. They change no +framework behavior or accepted diagnostic promise. Latest main, including #36, +was integrated by normal merge `3a58b96`; its gap CLI files are disjoint. + +The diagnostic law requires truthful authored calibration and selected-history +identity plus the finite original callback cause. It permits representation +changes to punctuation, structural whitespace, optional seed annotations, the +outer wrapper's name and opaque reason prose. Literal characters within names +remain significant. An unknown or ambiguous complete identity cannot authorize +its own expected selection. Original cause object identity, class and metadata +remain outside the established law. + +| ID | Executed defect before repair | General repair | +| ----------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | +| R8-REPRESENTATION | Lawful bracketed calibration labels and alternative seed annotations falsely fail the callback comparison. | Decode complete actual frames against available literal identities; derive expected selection independently from authored bindings. | +| R9-REP-SEAM | Already-bracketed source output passes semantic checks but reports zero presentation reach. | Count an actual rendering operation even when its output bytes equal its input. | +| R9-REP-RANGE | Mixed spacing, two spaces and tabs preserve identity/cause but fail decoding. | Generate and accept positive structural whitespace while retaining literal name characters. | +| R10-REP-REASON | Replacing the reason's first word loses identity; category-marker prose can change the verdict. | Locate the complete frame separator without inspecting reason words; the actual finite callback cause determines exception kind. | +| R11-DIAG-1 | Custom outer wrapper name plus opaque category-marker reason fails, although both marginals pass. | Remove the arbitrary outer-name guard; generate wrapper names independently of reason and frame presentation. | + +Every group extended the primary Suite grammar/real receiving driver before +changing its observation judgment. No production edit was needed. R10's +unchanged-observation gate rejected four legal fixed histories while their four +outer determinism checks passed. R11's unchanged guard rejected the combined +history while both marginals and all three determinism checks passed: one +intended false RED and five passes. The repaired R11 gate passes six of six. +Required constructions vary callback and selection kinds, siblings, prefixes, +incoming presentation and bounded arbitrary reason/wrapper text. All four +active receivers require actual callback failure and presentation reach: +ordinary, reordered, seedless and environment. Registration-only receivers +retain their registration obligations without claiming callback execution. + +The R10 frozen matrix accepted 15,642 lawful observations and rejected 1,089 +hostile controls at 99 actual cuts. It includes literal prefixes, numeric and +opaque labels, ambiguity, seed-like literal suffixes, arbitrary bounded reasons +and same-byte rendering. Thirty-five focused tests passed. Six lawful actual +source renderers passed two tests each; actual wrong sibling and missing cause +were rejected at child/calibration, with determinism passing. R11 carried those +matrix results only in their original `Error`-wrapper domain: removing a guard +that was true for every original input preserves that predicate. It separately +passed the actual custom-wrapper/opaque-reason source combination and rejected +wrong-sibling and replacement-cause source mutants at the intended comparison. +It did not claim to rerun the complete matrix for arbitrary wrapper names. + +R11's suspected absent reason dimension was refuted: the generic Suite grammar +already supplies arbitrary reasons. A 2,000-Suite sample contained 1,381 defined +reasons and 1,180 distinct constructed reasons. The redundant weighted-branch +draw is now forwarded too; that cleanup is not evidence of a former absent +whole-campaign axis. All setup failures and zero-test selectors remain excluded +from assertion-rejection counts. + +The timeout law limits every registered test independently. A timed-out test +owes no fixed completed-call count. Actual started work must retain its selected +input and corresponding build/resolve obligations. The fixture now captures its +logical owning test/route at Vitest's public `aroundEach` boundary through Node +AsyncLocalStorage. The global runner name remains raw diagnostic evidence; +advancing to another test cannot reassign an older continuation's work. + +| ID | Executed defect before repair | General repair | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| T1 | Exact-head Vitest 5 CI and local Vitest 4 reject timeout-history determinism; seven differing fields concern slow call attribution/order. | Record semantic relations of actual started work and keep raw runner observations outside the deterministic trace. | +| T2 | Slow isolated constructions also show resolution attribution drift. | Group build/resolve work by logical owner. A determinism test resolves once and may drive twice; other actual executions resolve individually. | +| T3 | A timed-out bound calibration with one actual execution is rejected for failing to complete two builds. | Check selected inputs and work for actual starts; retain input consistency and upper bounds without requiring optional late work to finish. | + +The extended primary grammar reproduced T1–T3 before the fixture repair. A +global resolve-count-equals-driver-count rule was disproved by a lawful trace +with 20 driver calls and 17 resolves. The repaired observation keeps first +logical inputs plus missing/extra work and changed-input relations. A legal +reporting-cut transformation removes only an optional second continuation with +its paired work; raw-versus-cut equivalence must still hold. It cannot repair an +invalid raw report silently. + +Slow late-probe counts retain a deficit/excess relation computed from complete +raw started histories and actual known late errors. Both values must be zero. +Fast-only late counts stay exact. Unknown and secondary errors, test statuses, +registration, loads, exit/signal and report completeness retain their checks. +Raw reports, global runner names and cuts remain available in diagnostic +sidecars. A blanket slow-log filter would not satisfy these obligations. + +The frozen timeout gate passed 48 tests, including 20 hostile controls. Fixed +and random required constructions each measured one hit for ordinary, +reordered, seedless and environment input cuts, one optional-continuation cut +and one presentation-equivalence hit. Three separate actual perturbations were +assertion-rejected: lost logical ownership (two failures), duplicate raw resolve +work (one calibration failure, delta two), and missing raw build work (one +timeout-presentation failure). Types, lint and formatting passed. + +Chronology remains explicit: ordinary/reordered REDs preceded the fixture +repair; seedless/environment witnesses were added afterward and establish +current enforcement only. A registration-phase missing-build experiment reached +no drivers. Its initial survivor classification was corrected to unreached; +it supplies neither a defect nor a kill. This qualification does not alter +production-repair chronology, because no production code changed here. + +The combined executable identities are owner SHA256 +`5b85ea79e738ea399ddd8c85ce5f55a95fa3e6dd213949e71c5f3e1f42621e5e`, +fixture `bad74feb45d92bdd390d48b804dc671c660ab4aa40462a45983c9a75d0c175e9`, +and unchanged production/reporter `a1c132e8`/`1918e8dc`. Three pure-addition +merge conflicts retained both repair groups. Combined TypeScript, lint, +formatting and diff checks pass. Combined focused checks and two independent +final reviews are running at this entry; complete final-head validation remains +pending. Earlier interrupted local runs are not full-suite GREEN results. + +These are finite synthetic and actual Node/Vitest boundary results. They make +no DB, browser, provider, release-time or memory-performance claim. Historical +CHRON-1, the retrospective successful-calibration stdout gate, optional stronger +mutant-only undeclared reach and fresh calibration seeds retain their prior +qualifications. No confirmed defect is deferred by relabeling it a limit. + +The exact combined owner gate passes **49/49** in 223.83 seconds, with 831 +tests registered. From `packages/oracles`, the command is: + +```sh +node ../../node_modules/vitest/vitest.mjs run \ + --config vitest.oracle-framework.config.mjs --maxWorkers=2 \ + -t 'round11 outer wrapper|timeout every receiving cut|reject timeout|round8 callback presentation freedom|reject equal-data origin|reject.*callback' +``` + +This gate exercises both repair groups together, actual outer determinism, +callback and timeout controls, and the canonical origin calibration. It is a +focused GREEN, not a complete framework run. The final workflow runs the whole +framework on both supported Vitest major versions. No campaign budget was +lowered and no dependency or lockfile workaround is included. From 81391d3944b37b38d0f0d0ae6eae6ec8c1c1531d Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 20:18:18 -0700 Subject: [PATCH 14/14] docs: record clean diagnostic and timeout reviews --- reviews/calibration-history-self-oracle.md | 71 ++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/reviews/calibration-history-self-oracle.md b/reviews/calibration-history-self-oracle.md index 978404d..7673122 100644 --- a/reviews/calibration-history-self-oracle.md +++ b/reviews/calibration-history-self-oracle.md @@ -685,3 +685,74 @@ callback and timeout controls, and the canonical origin calibration. It is a focused GREEN, not a complete framework run. The final workflow runs the whole framework on both supported Vitest major versions. No campaign budget was lowered and no dependency or lockfile workaround is included. + +### Original review loss accounting at the combined executable tree + +The original review targeted `329dbd4`. All nine technical findings were valid +there. Their current implementation dispositions are **nine already-fixed**; +the diagnostic follow-up supplies the remaining R6 recorder enforcement. This +table preserves each original cure and durable obligation separately. + +| Original item | Claim and proposed cure | Current evidence and durable obligation | +| ------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | Bound mutant verdict ranking differs from unbound; reuse `runOne`. | Bound/unbound ranking shares `runOne`; recorded violation before later throw/spin and bounded-clock controls remain enforced. | +| 2 | Forced premise is learned from the selected driver log; reconstruct it independently. | Authored selectors/origins determine both forced premises, including equal-data originals and explicit opposite orientations. | +| 3 | Reorder invariance misses the property-fixed-seed mutant; compare full named builds. | Distinct ordinary/astral name families and complete construction inputs reject the fixed-seed substitution while allowing other stable mixers. | +| 4 | Clean selected reach checks missing states but allows undeclared states; validate both directions. | Clean declaration checks cover both directions. Mutant-only undeclared reach remains a separate, unapproved stronger policy. | +| 5 | UTF-16 high-surrogate hashing collides for emoji names; use code points. | Code-point hashing and finite astral full-input controls distinguish the original defect without promising universal hash injectivity. | +| 6 | Corruption and judgment callback throws lose calibration/history names; contextualize them. | Production catches preserve context/cause. Four actual receiving cuts now enforce exact authored identity and finite cause while accepting lawful representation changes. | +| 7 | Explicit `entered: null` is silently inferred; infer only on `undefined`. | Omitted versus malformed controls remain distinct; null/false/zero and inherited/own binding cases retain registration-before-driver enforcement. | +| 8 | An impossible `if (!kind)` branch remains; remove it. | The branch is absent after exact-one-key validation; this is control-flow/build evidence, not an invented behavioral RED. | +| 9 | Transitional warnings repeat, collect-only warns and successful bound calibrations print announcements; deduplicate/silence. | Once-per-property warnings retain exact unbound names and registrations; collection and successful bound runs stay silent. The original announcement gate's retrospective chronology remains recorded. | + +Coordination is also complete: main/F27 was merged normally, `difference` has +bound/unbound parity after designated-law selection, and main's named UNIT +support is integrated. Fresh calibration seeds remain the explicitly separate +follow-up. No published branch was rebased or force-pushed. + +The eight current follow-up IDs remain separate from the original nine: +R8-REPRESENTATION, R9-REP-SEAM, R9-REP-RANGE, R10-REP-REASON, R11-DIAG-1 and +T1–T3. Each has an applied oracle repair and targeted sensitivity evidence above. +The original reviewer merits a strong hire recommendation: nine accurate +findings, independent-judgment attacks, useful cures and sound prioritization. +Later review results remain bounded to their recorded artifacts; full CI and +the final fresh review reports are not implied by this accounting table. + +### Final independent reviews of the combined executable + +Both fresh R12 reviewers found **no new supported defect** on owner `5b85ea79` +and fixture `bad74feb`. Neither changed the live tree. Their interventions were +sequential in isolated copies, and all source/owner/fixture/reporter hashes were +restored. The executable commit is `1fdf731`; this subsequent record-only append +does not change those reviewed bytes. + +The timeout reviewer passed 24/24 final checks, including twenty permanent +controls and the actual fixed/random required receiver units. A real delay +change from 300 ms to 450 ms, with a 100 ms timeout, changed a reordered +calibration from two actual started calls to one. Both shared comparison and +outer determinism passed. An actual discarded extra `selected.build()` was +rejected at child/calibration with build difference one, while determinism +passed. Each campaign measured all four input receivers, continuation cut and +presentation equivalence once. An initial all-skipped selector is retained as +unreached, not a kill. + +The diagnostic reviewer passed 18/18 baseline checks, then 5/5 with actual +source changes combining tabs/mixed spaces, bracketed labels/seed decoration, +custom outer name and category-marker opaque reason. Both matching fixed and +random units measured every callback-failure and presentation receiver once. +Actual wrong sibling, wrong cause and environment-only cause omission each +failed child/calibration while determinism passed. The restored-source gate +passed 2/2. Its public observation hook recorded three complete children, three +presentations and three matching identity/cause failures for each of ordinary, +reordered, seedless and environment receivers. A failed pre-write script path +and redundant clean run are explicitly excluded from source-variant evidence. + +The separate read-only final loss audit found no lost obligation: original nine +already-fixed, eight follow-ups fixed-now, coordination already-fixed. It +retained historical CHRON-1 and every chronology, corrected nonfinding and +policy qualification above. Both R12 reviewers merit strong positive assessment +within their scopes: they independently challenged actual permitted timing and +diagnostic representations, then demonstrated intended comparison rejection +for adjacent wrong implementations. No further confirmed repair remains in +these bounded reviews. Complete final pushed-head CI remains required; this +record does not promote focused results to full-suite success.