From 64448e11e103f72d3ddb3ff4b389c7b5d42ddfb3 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 13:31:47 -0700 Subject: [PATCH 01/76] docs(oracles): ship setup, design, and review skills with the oracle checklist The package now carries what an adopter's coding agent needs, inside node_modules: - skills/oracles-setup: install both pinned packages, configure lint, write the AGENTS.md block, make the skills findable through repository or global pointer skills, and run a first oracle and gap check. - skills/oracles-design: law sheet first, model choice, a witness for every distinction, histories and reach, driver and check, RED, GREEN, hostile mutant, and closing every gap with no exits. - skills/oracles-review: rebuild one observation, the shortcut table, the weaknesses tools cannot see, and the finding format. - docs/oracle-checklist.md: ORC-001 to ORC-016 (version 1), ported from TanStack DB's rubric with the same IDs and a table of every change. - docs/glossary.md, docs/guide.md, and five instrument cards. - READMEs for both packages; `files` now includes skills and docs. The skills teach `oracleProperty({ owns })` and no `law.owner`, so this waits for those framework changes before it merges. Co-authored-by: Isaac --- README.md | 12 +- packages/eslint-plugin-oracles/README.md | 57 +++ packages/oracles/README.md | 152 ++++++ packages/oracles/docs/glossary.md | 90 ++++ packages/oracles/docs/guide.md | 463 ++++++++++++++++++ .../docs/instruments/adversarial-review.md | 76 +++ packages/oracles/docs/instruments/index.md | 37 ++ .../oracles/docs/instruments/law-discovery.md | 76 +++ .../docs/instruments/law-restructuring.md | 50 ++ .../docs/instruments/mutant-gap-hunt.md | 105 ++++ .../oracles/docs/instruments/tension-scan.md | 37 ++ packages/oracles/docs/oracle-checklist.md | 321 ++++++++++++ packages/oracles/package.json | 2 + .../oracles/skills/oracles-design/SKILL.md | 251 ++++++++++ .../oracles/skills/oracles-review/SKILL.md | 163 ++++++ .../oracles/skills/oracles-setup/SKILL.md | 195 ++++++++ 16 files changed, 2084 insertions(+), 3 deletions(-) create mode 100644 packages/eslint-plugin-oracles/README.md create mode 100644 packages/oracles/README.md create mode 100644 packages/oracles/docs/glossary.md create mode 100644 packages/oracles/docs/guide.md create mode 100644 packages/oracles/docs/instruments/adversarial-review.md create mode 100644 packages/oracles/docs/instruments/index.md create mode 100644 packages/oracles/docs/instruments/law-discovery.md create mode 100644 packages/oracles/docs/instruments/law-restructuring.md create mode 100644 packages/oracles/docs/instruments/mutant-gap-hunt.md create mode 100644 packages/oracles/docs/instruments/tension-scan.md create mode 100644 packages/oracles/docs/oracle-checklist.md create mode 100644 packages/oracles/skills/oracles-design/SKILL.md create mode 100644 packages/oracles/skills/oracles-review/SKILL.md create mode 100644 packages/oracles/skills/oracles-setup/SKILL.md diff --git a/README.md b/README.md index 50a83b0..ebc1f44 100644 --- a/README.md +++ b/README.md @@ -9,8 +9,12 @@ every commit with [pkg.pr.new](https://pkg.pr.new). ## Packages -- `packages/oracles` (`@tanstack/oracles`): the oracle framework, and the - `oracle-gaps` CLI. The CLI lists the production behaviour that a +To adopt it, start with [`packages/oracles/README.md`](packages/oracles/README.md): +the package ships setup, design, and review skills for coding agents, the +oracle checklist, and a glossary. + +- `packages/oracles` (`@tanstack/oracles`): the oracle framework, its skills + and docs, and the `oracle-gaps` CLI. The CLI lists the production behavior that a project's owning oracles leave unexercised or unjudged. A project runs it as `oracle-gaps --package --source `, with `--env-prefix` set to the prefix that the project passes to @@ -33,7 +37,9 @@ required. The rule reports: A framework implementation module is exempt. -A file that still migrates can be listed in `settings.oracles.legacyCampaigns`: +A TanStack DB file that still migrates can be listed in +`settings.oracles.legacyCampaigns`. The setting exists only for that +migration and will be removed; other repositories must not use it: ```js settings: { diff --git a/packages/eslint-plugin-oracles/README.md b/packages/eslint-plugin-oracles/README.md new file mode 100644 index 0000000..cc83f3d --- /dev/null +++ b/packages/eslint-plugin-oracles/README.md @@ -0,0 +1,57 @@ +# @tanstack/eslint-plugin-oracles + +Lint rules for oracle files written with `@tanstack/oracles`. They report the +shortcuts that make an oracle pass without enforcing its law, wherever a +syntactic check can see them. They need no type information, so an agent's +post-edit hook can run them on one file quickly. + +Install it at the same pkg.pr.new commit as `@tanstack/oracles`. The +`oracles-setup` skill in that package configures it. + +## Configs + +| Config | Files | Rules | +| --------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| `configs.recommended` | Files whose names contain `oracle` (`.ts`, `.tsx`, `.js`, `.jsx`, `.mjs`) | `no-disabled`, `no-unit-test`, `oracle-property-only`, `pure-model`, `replayable-seed` | +| `configs.production` | `packages/*/src/**` | `probe-form` | + +The `production` config matches only a workspace layout. For a single package +with `src/`, add the rule with your own glob: + +```js +import oracles from '@tanstack/eslint-plugin-oracles' + +export default [ + oracles.configs.recommended, + { + files: ['**/*oracle*.{ts,tsx,js,jsx,mjs}'], + linterOptions: { noInlineConfig: true }, + }, + { + files: ['src/**/*.{ts,tsx,js,jsx,mjs}'], + plugins: { oracle: oracles }, + rules: { 'oracle/probe-form': 'error' }, + }, +] +``` + +`noInlineConfig` stops a directive comment from switching a rule off in an +oracle file. Fix a report; do not disable the rule. + +## Rules + +| Rule | Reports | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `oracle/no-disabled` | An oracle test that stops running: `.only`, a skipped body, a switch-off condition, or one run with no fixed history. | +| `oracle/no-unit-test` | A test in an oracle file that reaches no declared check or model: an ordinary unit test. | +| `oracle/oracle-property-only` | A raw campaign (`fc.assert`, `fc.check`, or a `.prop` registration) or a legacy campaign API in an oracle file. Only `oracleProperty` declares a generated-history law. | +| `oracle/pure-model` | A model, check, or reach classifier that reads a clock, ambient randomness, or a timer. | +| `oracle/probe-form` | A state probe in production code that is not in the one form the production build strips. | +| `oracle/replayable-seed` | A property seed that the shared replay variables cannot select. | + +The decided bans on snapshot matchers (B1), `fc.pre` (B3), production imports +in a model (B4), and identical comparison operands (B7) are not rules yet. +Reviews check them until they ship. + +`settings.oracles.legacyCampaigns` exists only for TanStack DB's migration, +and it will be removed. Do not use it in another repository. diff --git a/packages/oracles/README.md b/packages/oracles/README.md new file mode 100644 index 0000000..ae2a197 --- /dev/null +++ b/packages/oracles/README.md @@ -0,0 +1,152 @@ +# @tanstack/oracles + +Oracle testing for TypeScript on Vitest and fast-check: + +1. State a law. +2. Generate the histories where the law matters. +3. Check production against an independent model. + +The gap check then lists the owned production behavior that no law judges. + +Not on npm yet. Install a commit from pkg.pr.new (see [Start](#start)). + +## Start + +Give your coding agent this prompt in the repository: + +> Install `https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@` as a +> dev dependency, then read +> `node_modules/@tanstack/oracles/skills/oracles-setup/SKILL.md` in full and +> follow it. + +The setup skill installs this package and `@tanstack/eslint-plugin-oracles` +at the same commit, adds the lint preset, and writes a block in `AGENTS.md`. +It offers to make the skills findable, in the repository or globally for your +coding agents, and walks through a first oracle and gap check. + +## What ships in this package + +| Path | What it is | +| -------------------------- | ----------------------------------------------------------------------------------------------- | +| `skills/oracles-setup/` | Setup, upgrade, and making the skills findable. | +| `skills/oracles-design/` | How to design, write, or change an oracle: law sheet first, model, histories, RED, GREEN, gaps. | +| `skills/oracles-review/` | How to review oracle work. | +| `docs/oracle-checklist.md` | The complete requirements, by ID (ORC-001 to ORC-016). | +| `docs/guide.md` | Techniques for each design and review step. | +| `docs/glossary.md` | The oracle terms. | +| `docs/instruments/` | Short reasoning cards that the skills route to. | +| `oracle-gaps/` | The gap check (`oracle-gaps` binary). | + +## An oracle + +One `oracleProperty` call holds the oracle's parts. + +```ts +import fc from 'fast-check' +import { differs, oracleProperty, pass } from '@tanstack/oracles' +import { clamp } from '../src/clamp' + +type History = { n: number; max: number } + +const histories = fc.record({ + n: fc.integer({ min: -100, max: 100 }), + max: fc.integer({ min: 0, max: 50 }), +}) + +oracleProperty({ + id: 'clamp-bound', + owns: ['src/clamp.ts'], + law: { + statement: + 'clamp(n, max) returns max when n is above max, and n otherwise.', + authority: 'README, "clamp"', + limits: 'Integers in [-100, 100] for n and [0, 50] for max.', + }, + histories: { + arbitrary: histories, + runs: 100, + fixedSeeds: [1], + states: ['above max', 'at max'], + measured: ['below max'], + constructed: [ + { + name: 'n above max', + enters: ['above max'], + build: ({ max }) => ({ n: max + 1, max }), + }, + { + name: 'n at max', + enters: ['at max'], + build: ({ max }) => ({ n: max, max }), + }, + ], + }, + // The law restated without production. + model: (plan: History) => ({ result: Math.min(plan.n, plan.max) }), + // The real entry point; the driver records and decides nothing. + driver: (plan, observe) => observe('result', clamp(plan.n, plan.max)), + check: (_plan, observations, expected) => + differs( + 'result', + 'clamp bound', + observations.find((o) => o.checkpoint === 'result')?.value, + expected.result, + ) ?? pass, + reached: (plan) => [ + plan.n > plan.max + ? 'above max' + : plan.n === plan.max + ? 'at max' + : 'below max', + ], + calibrations: [ + { + name: 'returns n unclamped', + law: 'clamp bound', + corrupt: (trace, plan) => + plan.n > plan.max + ? trace.map((o) => + o.checkpoint === 'result' ? { ...o, value: plan.n } : o, + ) + : undefined, + }, + ], +}) +``` + +A generated property runs one test per fixed seed and one random test. It +also runs every constructed history, a determinism test, and each +calibration. Each run reports how many histories entered each state. + +## API + +| Export | Purpose | +| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `oracleProperty(options)` | Registers an oracle. Options: `id`, `owns`, `law` (`statement`, `authority`, `limits`), `histories` (an enumerated list, or `{ arbitrary, runs, fixedSeeds, states, measured, constructed, fixed }`), `resolve`, `model`, `driver`, `check`, `reached`, `predict`, `calibrations`, `label`, `timeout`. | +| `pass`, `violation(...)`, `differs(checkpoint, law, actual, expected)` | The check's verdicts. `differs` returns a violation with a diff, or `undefined`, so comparisons chain with `??`. | +| `forwardCut(...)` | Checks an observation against an ordered list of allowed states that may be skipped but never revisited. | +| `useOracleClock(...)` | Runs the rest of a driver on virtual timers and `Date`, with a bound on fired timers (law `bounded clock`). | +| `scheduleFrom(...)`, `withinTurns(...)` | Controlled scheduling helpers for drivers. | +| `configureOracles({ envPrefix, registry, retiredVariables })` | Optional. Changes the environment-variable prefix or limits the property ids. Call it before any oracle module is imported. | +| `OracleViolation`, `OracleModelError`, `OracleDataError` | The framework's failure types. | + +Observations and enumerated histories must be data: primitives, plain +arrays and objects, Maps, Sets, and Errors. A driver converts anything else +before it records it. + +`@tanstack/oracles/declaration` is an entry point for browser or native +modules. It loads none of the main entry's Node built-ins. + +## The gap check + +```sh +pnpm exec oracle-gaps --package --source +``` + +The gap check mutates the owned source and runs the oracles against each +mutant. Its report lists every gap with one +instruction. The other options and the report format are still changing. +The skills describe how to work with the report. + +A gap closes only when a law rejects the mutant or the code is removed. There +are no equivalent mutants. diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md new file mode 100644 index 0000000..b0c7c53 --- /dev/null +++ b/packages/oracles/docs/glossary.md @@ -0,0 +1,90 @@ +# Oracle glossary + +This glossary owns the terms for oracle work with `@tanstack/oracles`. The +project's own glossary or architecture documents own its domain terms. Use one +term for one concept. Do not rotate synonyms for style. + +## Laws and evidence + +| Term | Meaning | +| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | +| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | +| limits | What a property does not establish. Unresolved parts of a law go here. | +| law sheet | The first output of oracle design: each law with its authority, status, rival, distinguishing witness, and checkpoint. A human reviews it before any code. | +| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | +| rival | A plausible different law that predicts a different observation for some history. | +| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | +| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | +| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | +| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | +| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | + +## The parts of an `oracleProperty` + +| Term | Meaning | +| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| history | Starting conditions plus a sequence of actions or events, including timing. | +| history grammar | The values, actions, constraints, and schedules that can form legal histories: the fast-check arbitrary and its resolver. | +| plan | The concrete actions that `resolve(history)` gives. The driver runs it, and the model and check read it. It never holds expected observations. | +| model | The part that gives the expected result of a plan without production. It is pure. It may read the inputs that the driver recorded in the trace, never production's outputs. | +| driver | The part that runs a plan through the real production entry point and records observations. It returns nothing, so it cannot decide a verdict. | +| observation | One recorded public fact, recorded with `observe(checkpoint, value)`. It must be data. | +| trace | The ordered observations of one history. | +| checkpoint | The exact point where the contract owes an observation. | +| check | The part that compares the trace with the model and reports a violation. Only the check can fail a law. | +| violation | A check's report that a history broke a law at a checkpoint. | +| calibration | A named wrong answer (a corrupted trace or a mutant driver) and the law that must reject it. | +| campaign | One run of a generated `oracleProperty`: a fixed-seed test or the random test. | +| replay | Rerunning a recorded failure from the command that the failure prints. | +| shrinking | Reducing a failing history while it still fails the same way. | +| bounded exhaustiveness | Running every case inside stated finite bounds, with an enumerated history source. | + +## Reach + +| Term | Meaning | +| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| reach | The declared states that a history entered. | +| declared state | A state in `states`. A constructed history must enter it. | +| measured state | A state in `measured`. The run reports how often histories enter it, but nothing requires it. | +| constructed history | A history built from a generated one so that it enters named declared states. Only constructed histories make reach required. | +| fixed history | A named history, such as a regression. It is not reach. | +| state probe | A production call, `probing && globalThis.__oracleProbe?.(name, condition)`, that reports where production decides to enter a state. Production builds strip it. | +| probe-only code | The probe statements and their guard. The gap check does not mutate them. | + +## Gaps + +| Term | Meaning | +| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| owns | The production files or functions an oracle answers for, listed in the `owns` option of `oracleProperty`. The gap check judges owned code only. | +| gap check | The loop that runs the oracle files against production and against mutants of the owned code. It reports every gap with one instruction. | +| mutant | A small deliberate change to owned production code. | +| kill | A mutant that a law rejects through a registered `oracleProperty` route. A plain test in an oracle file is an unregistered kill, and it does not count. | +| survivor | A mutant that no law rejects. Every survivor is a gap. | +| gap | Owned behavior that the oracles do not judge. It closes only when a law kills it or the code is removed. | +| exit | Any reasoning that puts a gap out of reach: calling a mutant equivalent or unobservable, or accepting a gap. None is allowed. | +| framework bug | A gap that no kill or removal can close. The agent stops work on that gap and reports its key and evidence. | +| exclusion control | ORC-004's control: a nearby invalid history that the grammar rejects. It is required, and distinct from a quiet exclusion, which is a finding. | +| dead code | Code whose removal no legal history can observe. Remove it. | +| invalid mutant | A mutant that no oracle could possibly detect. The framework must not generate one, so meeting one is a framework bug to report. The author never decides that a mutant is invalid, and the label does not close a gap. | +| unowned | Production code that no oracle owns. The gap check reports it as a warning, not as judged. | +| RED | The extended oracle failing on unchanged production. It shows that the oracle can see the defect. | +| GREEN | The same oracle passing after the change. | +| hostile mutant | A deliberate wrong implementation, often the original defect, that the oracle must reject before a repair is called closed. | + +There is no "equivalent mutant" and no "unobservable effect". A survivor +shows a missing history, observation, or law, or dead code. An effect that +no check sees is an effect that the driver does not record. + +## Model alignment rule + +For every model state, action, and observation, apply one of these rules: + +1. If it represents a production concept, use the production term. +2. If it combines production concepts, name the abstraction and list what it + combines. +3. If it splits one production concept, qualify each part and state the split. +4. If it exists only in the model, say so. Do not give it a production name. + +Aligned names do not mean copied logic. A model should not use production +queues, caches, or helpers merely to share their names. diff --git a/packages/oracles/docs/guide.md b/packages/oracles/docs/guide.md new file mode 100644 index 0000000..410084b --- /dev/null +++ b/packages/oracles/docs/guide.md @@ -0,0 +1,463 @@ +# Oracle guide: techniques + +These techniques support `oracles-design` and `oracles-review`. They add no +requirements; the [checklist](oracle-checklist.md) holds them all, and this +guide cites it by ID. Terms follow the [glossary](glossary.md). + +## The model + +### Make the model easy to distrust + +Prefer a pure function or a small state transition. Keep production queues, +caches, classifiers, and helpers out of the expected result (ORC-002). A +reader should be able to challenge the model without learning the +implementation first. + +Name a partial model honestly: if it predicts values but not callback counts, +say so in `limits` (ORC-001). + +Keep the model's records separate from production's records. If production +mutates a shared object, the expected answer can change with it, and both sides +agree for the wrong reason. Copy the fields the model needs. + +Independent code also needs independent reasoning. A model that calls +production's comparator cannot see a comparator defect. Reuse ordinary test +mechanics freely, but name any semantic helper that both sides trust. A small +model is easier to inspect. It is not correct because it is small. + +### Decompose the model by law + +A model can become as hard to trust as production. Do not keep adding state +until it is a second implementation. + +When the contract has independent laws, treat the oracle as a small graph: + +- A node owns one rule and the least state needed to predict it. +- An edge records a real dependency between two rules. +- The driver can feed the same action to several nodes. +- The check composes their observations at the checkpoint. + +A failure then names the law that diverged, and a reviewer can inspect each +rule alone. Plain functions and Maps are usually enough. + +Do not split state that a legal next action can observe only as a whole. If two +states look equal to the proposed nodes, try one legal next action. If it can +produce different promised results, add the missing edge, or keep that state in +one model. Decomposition does not mean one model per assertion. Group facts +that form one state machine, and keep independent policies separate. + +### Choose the judgment that fits + +A full reference works when the answer is cheap to compute independently. +Other promises need other judgments: + +| Method | Useful judgment | Limit | +| ----------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------- | +| Simple model | Recompute the allowed result, or step a small abstract state. | The model can omit a distinction or encode the wrong rule. | +| Differential comparison | Run the same work through another implementation or formulation. | Both paths can share a bug. Their semantics must actually agree. | +| Metamorphic relation | Transform a history, and check a relation between the two results. | The transformation's premises must hold. | +| Partial invariant | Check one promise, such as "no request completes twice". | A pass says nothing about other properties. | + +Choose the smallest judgment that answers the question. Combining +complementary checks can help. Making every oracle compute every observation +usually does not. + +### Keep only state that can matter + +Removing fields until a model looks simple is not a design method. Test each +merge of states instead (ORC-008). + +Example: a handle supports a row. Replacing handle `a` with `b` moves that +support. Releasing a retired handle does nothing. + +| History | Supported rows | Live support count | +| --------------------------------------------- | -------------- | ------------------ | +| Acquire `a` for `x` | `{x}` | 1 | +| Acquire `a`, then replace it with `b` for `x` | `{x}` | 1 | + +The two states look equal. Now release `a`. The first history loses support +for `x`. The second keeps it through `b`. A model with only rows and a count +cannot answer both. + +So: find two states that the model treats as equal, and try a legal next +action. If their promised observations differ, or an action is legal in only +one, the model erased something that matters. The witness justifies this one +distinction. It does not prove the model handles every race. + +### Know where the expected answer comes from + +Two questions sit behind "the model says so". Who established the behavior? +Which state determines the observation? + +- A finite model can lose permitted values. For example, a model that holds + scores only up to three makes `score > 3` an empty predicate. Fix the model, + not production. +- A model can read the wrong state. For example, it can use locally edited + values where the contract uses source values. + +Before you change production to satisfy a red oracle, check the contract, the +model's representation, and the comparison. An unresolved product choice goes +to the user as an `open` law. Judge a reviewer's accusation and their proposed +fix separately: a correct concern does not make its repair correct. + +Projection needs the same care. Dropping internal metadata from a comparison +can be valid. Dropping a field the contract promises is not, whatever it is +called. Write down one difference the comparison may ignore and one it must +keep. + +## Histories + +### Describe the input domain as a grammar + +Before choosing arbitraries, describe the language of legal histories. Separate +three kinds of fact: + +- **Dynamics:** the actions that move the system. +- **Constraints:** invariants, interfaces, and forbidden combinations. +- **Boundary conditions:** the starting state, scale, provider behavior, and + value domains where the rules apply. + +Mark which rules come from the contract and which you inferred. Map +containment, overlap, and dependency. Do not force overlapping concerns into a +tree to make the generator neat. + +ORC-004's four controls apply to any claim of generated coverage: + +1. **Reconstruction:** can the grammar rebuild every known valid witness? +2. **Ablation:** does removing each axis, rule, or overlap lose a promised case + or admit a forbidden one? +3. **Range:** does an independent marginal case need only new values, or does + it expose a missing rule? +4. **Exclusion:** does the grammar reject a nearby invalid history? + +Do not take a flat Cartesian product of unrelated axes because the tool makes +it easy. Two false greens are common. A grammar that cannot rebuild a known bug +omits a path. A grammar that generates valid and invalid histories without +telling them apart hides policy in filters. + +### Build legal histories + +Generate only actions that are legal in the current state. Choose the action +kind first, then a key that it applies to. Put legality in the grammar or in +`resolve`, so every shrunk history stays legal; `fc.pre` is banned (B3). + +Design action dependencies so that smaller histories stay meaningful under +shrinking. + +The structural cases that matter become constructed histories, for example: + +- no rows +- a width of zero +- a tie at a boundary +- repeated changes to one key + +Each constructed history names the declared state it enters (ORC-015). Then +let random generation vary the values and lengths around them. A larger run +count is not a substitute for a constructed history. + +### Generate histories that reach the problem + +Once the model can tell the relevant states apart, the histories must put +production into them. Random payloads alone do not create reuse, failed +startup, or work after recovery. + +Look for relationships and transitions: + +- the same key versus a fresh key +- one owner versus a surviving sibling +- rejection before versus after acquisition +- release before versus after completion +- recovery followed by another operation + +Keep healthy peers alive where isolation is part of the promise. A history +that ends at "recovery succeeded" may never show broken state on the next use. + +Fresh disjoint keys can simplify a driver but exclude the reused-key +interaction that the law needs. Repeated `true` reports do not cover +`true → false → true` while earlier work is pending. These are different +histories, not different random values. + +Widening a generator can lose reach in two ways: + +- A rewrite can make a previously possible history impossible, even while it + adds a dimension. +- A larger domain can make a useful history rarer at the same run budget. + +Check possibility with a constructed history, and frequency with the reach +counts. + +### Control the event, not its returned Promise + +An async operation can have several boundaries: + +```text +request invoked → response delivered → writes applied → result visible + ↘ returned Promise settles +``` + +The contract says which boundaries come before which. The driver must control +and observe the boundaries that the law depends on. + +If an operation applies its work eagerly, delaying its returned Promise does +not delay that work. An unconditional `await` can let a queued microtask +repair a mismatch before the check sees it. A driver that reads only settled +state cannot expose an earlier wrong observation. + +Put an explicit hold and release at the event you need. For a law about +visibility before application: + +1. Production requests data. +2. The driver receives the response, and holds delivery of its writes. +3. The driver records what is visible while delivery is held. +4. The driver releases delivery, and production applies the writes. +5. The driver records what is visible at the promised checkpoint. + +The hold must be real. A driver that delivers the writes and only delays +returning from its handler controls a different event. A scheduler helps only +within the boundaries it controls. A seed cannot schedule external I/O that the +driver does not intercept. + +### Check that timing helpers establish their premises + +A driver often needs helpers such as "wait until the request is blocked" or +"let the message reach the worker". Each helper establishes a premise. If it +does less than its name says, the oracle can pass without ever reaching the +history. + +Example: a helper released a lock in the same turn that it sent a close +message. The message was still crossing a task boundary, so the earlier +request finished first. Close never ran during the wait, and the mutants it +targeted passed. + +For each timing helper, do one of these: + +- Assert the premise at its checkpoint, for example from a lock inventory or a + delivery counter. +- Add a calibration that only a history meeting the premise can reject + (ORC-006). + +A helper that waits a fixed time proves no delivery. It still needs one of +these checks. + +### One law can need two schedules + +Enumerating more positions is not always enough. Example: a request settles +after four completion obligations, `r0` to `r3`. Success must wait for every +obligation, and a rejection must not wait for unrelated later work. For a +selected obligation `r1`, two schedules ask different questions: + +| Question | Schedule | Observation | +| ---------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------ | +| Can settlement ignore `r1`? | Hold `r1`; complete `r0`, `r2`, and `r3`. | The request is still pending. | +| Does a rejection of `r1` wait for peers? | Complete `r0`; hold `r2` and `r3`; reject `r1`. | The request rejects at once, with the partial result of `r0` only. | + +Completing every peer first isolates the first question but removes the +pending sibling that the second question needs. At the last obligation there is +no later peer, so that case cannot distinguish the delay. + +### Keep the driver on its side of the boundary + +A test provider can fix the consumer by accident. Example: a paging provider +removed already-delivered rows before it applied the requested limit. That +supplied progress the consumer never requested, and hid a repeated-page bug. + +State what the request asks, what the provider promises, and which progress +production must arrange. The same applies to reentry. If a callback triggers a +second operation only under a condition, assert that the callback and the +trigger happened. "If reached, check it" is not evidence that reentry was +tested. + +When one law must hold in two environments, give each environment its own +driver, and share the law, model, histories, and check (ORC-014). + +## The driver and the check + +### Observe what the contract promises + +Choose observations from the promise. Membership, values, order, +multiplicity, callback boundaries, error identity, settlement timing, durable +storage, and resource ownership are different facts. Record the ones the law +needs, not all of them. + +- If an unprojected sort field changes, compare the new order, not only whether + the result changed. +- If a callback count matters, check the count before looping over the + callbacks. A loop over no callbacks checks nothing. +- If a failed write must not reach storage later, read the stored data after a + later successful operation. + +Each comparison should separate the specific wrong behavior from the allowed +one. + +### Let observations represent violations + +Make generated inputs legal. Do not make recorded outputs unable to be +illegal. + +Suppose production completes one request twice. A map keyed by request that +keeps the last completion hides the violation. Record both events, then compare +the promised count. Do not fold the raw list back into a map before the check +sees it. + +Likewise: + +- Do not reset a change tracker before checking the parts that must stay + unchanged. +- Do not discard values that "should" never appear. +- Do not normalize away an order that the contract promises. + +Normalize only what the comparison permits. + +### Check incremental results and publication separately + +An incremental system emits changes, while a model often computes a whole +result. Compare compatible objects: accumulate the changes, then compare that +state with a full recomputation at the promised checkpoint. Check individual +batches too when the contract makes their boundaries visible. + +A correct final result cannot show that every earlier publication was +coherent. Choose checkpoints that can expose the promised failure, such as a +torn publication of old and new halves. Do not add delays until the result is +stable. + +When scheduling allows several results, check for **one allowed explanation of +the whole observed history**. Do not approve each field against a different +allowed execution: pieces that are each allowed may never occur together. A +bounded oracle checks sampled histories. It is not a proof of every behavior. + +### Make the comparison prove its usefulness + +Suppose an update check asserts only that the result changed. An empty result +passes it. Compare against the exact expected value instead. + +Then calibrate the check against known wrong answers (ORC-006): + +```ts +// The allowed answer passes; a wrong order and an arbitrary change fail. +check(plan, [{ checkpoint: 'ids', value: [1, 2] }], { ids: [1, 2] }) // pass +check(plan, [{ checkpoint: 'ids', value: [2, 1] }], { ids: [1, 2] }) // violation +check(plan, [{ checkpoint: 'ids', value: [] }], { ids: [1, 2] }) // violation +``` + +In an `oracleProperty`, write these as `calibrations`, so the framework runs +them. Calibration proves the check is sensitive. Reach proves the path ran. +These are different claims: a perfect comparison on a path that never runs +protects nothing. + +### Compare formulations when a model might share the bug + +A plain model can copy a wrong semantic assumption. A second formulation gives +another way to disagree. This applies when a plausible shared fault and a +meaningfully different formulation have been named (ORC-011). + +- First state how the two formulations match on empty inputs, duplicates, + order, and projection. A transformation that changes the answer cannot + show that the two formulations agree. +- A metamorphic relation transforms a history and checks a known relation + between the results. Examples: incremental equals recompute, partitions that + recombine to the whole, a round trip. +- Agreement between two paths does not prove that they are independent. Paths + that share an operator or a copied code path share its faults. A matching + count can hide wrong rows. Use each formulation for the faults it can + separate. + +## Evidence + +### Test the test + +"The oracle passed" combines several claims: + +1. Was the property selected? +2. Did it run? +3. Did it reach the intended path? +4. Did its comparison execute? +5. Would that comparison reject the fault under discussion? + +Treat them as separate evidence. Reach counts, calibrations, and the gap +check answer them for an `oracleProperty`. + +For a mutant, record what happened, not only "killed": + +- A comparison rejected the intended wrong answer. +- A timeout exposed a missing progress obligation. +- Setup failed before the comparison. +- Execution failed outside the comparison after setup, such as a crash. +- The changed code never ran. +- The mutant survived. This is a gap (see the + [mutant gap hunt](instruments/mutant-gap-hunt.md), step 3). + +These outcomes call for different conclusions. If deleting one clause leaves +the same obligation enforced elsewhere, survival does not show that the +obligation is unnecessary. If an oracle kills a compound change, it may not +distinguish each part. + +### Preserve the violation, not merely a red exit + +Shrinking keeps the first failure kind (ORC-010). Keep the violated law, the +checkpoint, and the original history beside the reduced one. A replay that +ran but did not reproduce the mismatch is not a reproduction. + +Capture evidence before cleanup can change it. Copy what the law needs before +a rollback runs. A serializer can lose object identity or reorder arrays. Keep +the original form when normalization would erase the disputed distinction. + +A challenge that shaped the design is a regression case afterwards, not a +held-out test of that design. + +### Vary machinery and premises separately + +When comparing implementation candidates, hold the law, the legal histories, +and the checkpoints fixed. If removing an index or cache changes no public +result, it is dead code, unless a law states the work it saves, such as a +count of reads. State that law and measure it, or remove the code. + +If a premise changes instead, derive which guarantees still follow. Examples: +delivery may duplicate, ordering may tie, or the client may see only part of +the source. A failure under a changed premise may call for more provider data +or a law decision, not another algorithm. + +### Simplify machinery without changing the question + +Before removing or combining oracles, name what will remain: the valid +history, the expected outcome, the production path, the observation, and the +oracle that receives them. "Retry the parent" turning into "delete the parent" +changes the behavior, even when the replacement is shorter and green. + +Other reductions are legitimate. An expectation can contradict the contract. +Two branches can enforce the same law. Removing a redundant branch need not +remove its law. + +Share stable mechanics, such as cleanup, where they are truly interchangeable. +Keep domain truth small and reviewable. Several independent models should not +grow into one miniature implementation to reduce duplication. + +### Reusable boundary laws + +Consider these laws when the contract has the matching boundary. They are +prompts, not requirements. If one does not apply, say why in `limits`; do not +add an empty case. + +- **Real-provider conformance:** freeze representative values from each + supported provider version. Show that the driver's stand-in accepts them + before it stands in for that provider (ORC-014). +- **Minimal ambiguity:** include the smallest valid input for every branch of a + classifier. Rich values that carry several redundant signals do not cover a + one-field collision. +- **Name invariance:** renaming a user-controlled name must not change a result + unless the contract gives the name meaning. +- **Representation symmetry:** two forms of one input that the contract treats + as the same must give the same + public result or the same documented error. +- **Await-boundary transitions:** hold each relevant `await`. Change ownership, + leadership, generation, abort, cleanup, or restart state, then release it. + Compare the result with the contract for that transition. +- **Local and remote agreement:** data sent across a transport must mean the + same on both sides. Live local references stay local. +- **Partial-construction cleanup:** fail each construction step after it + acquires a resource. Keep the primary error, and show that every acquired + resource is released exactly once. +- **Value and work:** when bounded work is promised, check the exact result and + a deterministic work count. A correct result alone does not establish the + work law. + +An unchecked prompt is not evidence that the law it names holds. diff --git a/packages/oracles/docs/instruments/adversarial-review.md b/packages/oracles/docs/instruments/adversarial-review.md new file mode 100644 index 0000000..992e37d --- /dev/null +++ b/packages/oracles/docs/instruments/adversarial-review.md @@ -0,0 +1,76 @@ +# Adversarial review + +**Use when:** a model, oracle, or proposed law needs a concrete challenge. + +**Inputs:** one artifact or candidate, its sources, claimed domain, and success +criterion. Prefer a fresh reviewer that has not seen the author's desired +verdict. Without that separation, report a self-critique rather than independent +validation. + +## Operation + +Start with a plausible way this account could be wrong. Select the relevant +levels; they are different questions, not a mandatory sequence: + +| Level | Attack | +| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Enforcement | Find a legal history or wrong output that the model, generator, driver, or recorder cannot distinguish. Can the intended production path remain unexercised while the test passes? | +| Formulation | Find an omitted precondition, excessive guarantee, wrong checkpoint, or ambiguous observation. Try a correct alternative implementation as well as a wrong one. | +| Suitability | Question whether the promised behavior is desirable, safe, or conventional. Name the external contract, standard, user consequence, or engineering reason that makes the objection applicable. | +| Composition | Find obligations that conflict, duplicate each other, or require a shared ordering or authority rule. Use the tension or restructuring card if that is the material uncertainty. | +| Concepts | Question the chosen identity, state, ownership, or boundary distinctions. Describe the same case without relying on its current names and check what is lost or becomes visible. | + +For a suspected information gap, use [law discovery](law-discovery.md#operation), +step 5. + +### Locate the repair + +The attack level names what to challenge. Once evidence shows a problem, use +this table to locate the repair: + +| Finding or question | Possible repair location | +| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | +| Production disagrees with an accepted law. | Production code. | +| The reference predicts behavior the accepted law does not allow. | Reference model. | +| An accepted law over-promises, under-promises, or conflicts with another law. | Authorized contract decision, then model and test updates. | +| The law needs information the production path lacks at the promised checkpoint. | Provider, protocol, architecture, or an authorized contract decision. | +| The check rejects an allowed intermediate observation or accepts a wrong one. | Evaluation: checkpoint, history grammar, driver, recorder, or comparison. | +| Correct behavior still fails the user's need. | Requirement and its suitability for the intended use. | + +These locations can overlap. Name the evidence for each before choosing a +repair; an assertion change cannot resolve a code defect or supply missing +information. + +An accepted contract remains an obligation even if the current path lacks the +information needed to meet it. A repair might require additional provider +data. Moving the promised checkpoint or narrowing the guarantee needs an +authorized contract decision; changing an assertion supplies no data. + +For each material attack, name the broken claim and construct a small legal +history or concrete failure scene. State the competing expected observations, +the evidence that could decide between them, and a possible repair condition. +Check that the attack is within scope and that its standard actually applies. +An outside-domain case may motivate a scope change; label that proposal. + +When execution is authorized and practical, run the discriminating experiment. +A hostile control must reach the intended comparison to demonstrate rejection. +Use the oracle guide's calibration and mutant-outcome rules. A source-level +argument can remain useful when execution is unavailable; label it accurately. +For a systematic search across plausible wrong implementations, use the +[mutant gap hunt](mutant-gap-hunt.md) to design semantic edits and classify +survivors. + +**Return:** supported defects, design objections, and untested hypotheses as +separate kinds of finding, with sources, witnesses, outcomes, and repair +conditions. For bugs, identify the test gap that permitted the class of failure. + +**Control:** explain what would defeat the criticism. Check whether the attack +would also reject a valid implementation or depend on a condition the contract +excludes. Do not presume a defect merely because the stance is adversarial. + +**Distortion:** generic pessimism, irrelevant external standards, author capture, +or treating a preferred design as evidence of a current correctness bug. + +**Stop when:** the material attacks have evidence or an explicit unresolved +experiment. No supported finding is a valid result. Challenging a law does not +silently replace it. diff --git a/packages/oracles/docs/instruments/index.md b/packages/oracles/docs/instruments/index.md new file mode 100644 index 0000000..5e6b2c5 --- /dev/null +++ b/packages/oracles/docs/instruments/index.md @@ -0,0 +1,37 @@ +# Design and review instruments + +Use an instrument when a specific uncertainty makes ordinary implementation or +review insufficient. Read only the relevant card. Choose and use it within the +user's authorized task; these cards do not require a separate approval for each +reasoning step or a fixed workflow. + +| Uncertainty | Instrument | Result | +| --------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| What behavior should this subsystem promise? | [Law discovery](law-discovery.md) | A sourced candidate law, competing predictions, and a distinguishing witness. | +| Could the implementation, model, or law itself be wrong? | [Adversarial review](adversarial-review.md) | Concrete attacks at named levels, their evidence, and repair conditions. | +| Would a plausible wrong implementation still pass the oracle? | [Mutant gap hunt](mutant-gap-hunt.md) | A semantic mutant and what its survival shows: a missing history, observation, or law, dead code, a law question, or a framework bug. | +| Do sound obligations conflict under the same conditions? | [Tension scan](tension-scan.md) | A traced collision, missing distinction, or no supported tension. | +| Should these laws or concepts be merged, split, replaced, or removed? | [Law restructuring](law-restructuring.md) | An alternative organization and a check of what it preserves or loses. | + +These are reasoning aids, not additional oracle conformance requirements. The +[checklist](../oracle-checklist.md) owns testing requirements. Use the +[glossary](../glossary.md) and the relevant architecture and contract sources. +The `oracles-design` and `oracles-review` skills route oracle work here; the cards can also help +with architecture and other design questions. + +Keep these apart: source statements, observed behavior, generated examples, +hypotheses, and design judgments. An existing contract constrains correctness +claims, and a design proposal may challenge it. Only the authority +appropriate to the task can change a contract. An instrument result does not +change product expectations by itself. + +Record the question, the operation, the source pointers, the concrete +witness, and the open part in the task's normal notes. No finding, no +supported tension, and a failed simplification are useful results. A +reasoning witness is not an executed test. Passing sampled histories is not +proof. + +Adapted from Field Lab's assumption and boundary probes, hostile failure +assay, tension statement, structural recombinator, and position-preservation +assay. The adversarial review card also permits criticism of a candidate's +standard and concepts. No Field Lab install is needed. diff --git a/packages/oracles/docs/instruments/law-discovery.md b/packages/oracles/docs/instruments/law-discovery.md new file mode 100644 index 0000000..d15ed8f --- /dev/null +++ b/packages/oracles/docs/instruments/law-discovery.md @@ -0,0 +1,76 @@ +# Law discovery + +**Use when:** a bug, example, or implementation exists but the promised behavior +is unclear; several supplied reports may share a law; or an existing law might +conflate distinct obligations. + +**Inputs:** the question, contract sources, known examples, the existing oracle +and its limits, and the public observation that matters. A missing authority is an +input gap, not permission to infer a promise from production. + +With several reports or examples, ask whether one caller-visible obligation +predicts their different symptoms. Name its common preconditions. Check an +apparent outlier and a nearby case that the obligation should permit. If those +cases need different promises, split the proposal. A descriptive cluster is a +lead, not evidence. + +## Operation + +1. State the obligation in domain terms before describing machinery: for which + legal inputs or histories, under which preconditions, what must be observable + at which checkpoint? Include what can still be pending or unspecified. +2. Trace its authority. Separate accepted contract, observed current behavior, + mathematical consequence, and proposed design. Explain why the obligation + matters to a caller. A mathematical rule still needs an argument that it is + the right rule for this API. +3. Form a plausible rival interpretation. Derive both predictions for the + smallest legal case that separates them, without consulting production for + the expected answer. If authority does not select a prediction, retain the + design question instead of turning one guess into an assertion. +4. Compare an ordinary case with a nearby boundary case. Vary a relevant + condition such as cardinality, ordering, identity, authority, or observation + time. Keep other conditions fixed where possible. Check the boundary is + within the claimed domain. +5. Identify the checkpoint where the contract requires the proposed result; a + permitted pending state does not establish a gap. Check whether the result + is determined by the information available to the responsible production + path there. Seek two legal underlying histories that give the path the same + initial state and inputs through that checkpoint but require incompatible + public observations. If they exist, name the missing information or provider + obligation. A candidate law may be scoped explicitly; do not silently narrow + an accepted promise, which requires authority to revise. +6. Describe the smallest independent model that enacts the obligation and limits + from step 1 and makes the rival and boundary predictions from steps 3–4. + Retain the source facts needed to examine any information gap from step 5, + even when the production path has not received them. + For any proposed merge of model states, ask whether a legal next action can + distinguish them. Map model abstractions to the project vocabulary. + +The information check asks what production can receive. The model-state test +asks what the reference must remember. Example: a path must publish a full top +ten after a deletion, but it receives only the old top ten and the deletion. +Two datasets can then need different results while the path sees the same +inputs. If the path can request the replacement rows, and the provider +supplies them before publication, the path has the information. Judge it at +its own checkpoint. + +**Return:** the candidate law and status, its source, rival predictions, a +concrete distinguishing witness, and the next experiment or unresolved decision. +For an information gap, include the indistinguishable histories and the +unresolved source, protocol, or scope decision. +For a simple law, a short paragraph and a worked example suffice. + +**Control:** try a legal alternative implementation. Does the proposed law +reject it only because its scheduling or representation differs? Also try a +plausible wrong result: could the proposed observation distinguish it? If an +information gap is claimed, try a nearby path that supplies the missing input, +such as a provider backfill. Check whether that input distinguishes the +histories before the checkpoint; if it does, the original impossibility +argument no longer applies to that path. + +**Distortion:** freezing an implementation accident into a law; treating a vivid +edge case as representative; or overstating what is owed at an intermediate cut. + +**Stop when:** the law is precise enough to enact independently, or a named +missing authority or fact prevents choosing between predictions. Continue other +authorized work while keeping that choice unresolved. diff --git a/packages/oracles/docs/instruments/law-restructuring.md b/packages/oracles/docs/instruments/law-restructuring.md new file mode 100644 index 0000000..e93b4d0 --- /dev/null +++ b/packages/oracles/docs/instruments/law-restructuring.md @@ -0,0 +1,50 @@ +# Law restructuring + +**Use when:** laws overlap, need a missing distinction, or might be better +expressed by merging, splitting, replacing, or removing them. + +**Inputs:** the original laws with their authority, motivations, examples, and +limits. Include the oracles that enforce them. Do not simplify labels without first +recovering the obligations they represent. + +## Operation + +1. Decompose the laws into traceable obligations, conditions, distinctions, + and promised observations. Check that these parts can reconstruct each + original claim; invented categories can lose the essential constraint. + If public wording and an oracle disagree, retain both claims and + the unresolved authority. A passing test alone cannot retire public wording. +2. Propose a different organization and state the operation: merge under a + common principle, split by authority or checkpoint, replace a concept, or + retire an obligation with an explicit reason. Fewer words or model fields + alone do not establish improvement. +3. Map every original obligation to where it is preserved, narrowed, strengthened, + rejected, or still unresolved in the candidate. State what insight originally + made the obligation necessary. A claim may be defeated, but needs a reason + and a design decision rather than disappearing from the mapping. +4. Derive old and proposed predictions for a witness exercising each distinction + that the reorganization could erase. Compare both allowed and forbidden + outcomes. Check whether the change accidentally forbids a valid implementation + or admits one of the formerly rejected wrong designs. +5. When an independent preservation check is useful and authorized, give a fresh + agent the original obligation, its sources, and one candidate. Hide sibling + candidates and the author's preferred outcome. Ask what insight survives, + what is defeated, and what smallest repair is needed. Without that separation, + describe the result as a trace audit, not independent validation. + +**Return:** the proposed organization, the old-to-new obligation mapping, +witnesses and their evidence status, losses, and unresolved design decisions. +Keeping the existing laws separate is a valid result. + +**Control:** reconstruct each original obligation from the candidate and check +its discriminating witness. A semantic change may be justified, but it is a +contract change, not a refactor. Merging prose does not require merging oracles +whose models or production boundaries differ. + +**Distortion:** neat categories that erase obligations, speculative connections +presented as established, or preserving familiar wording instead of useful +insight. + +**Stop when:** the alternative's preservation and losses are explicit, or the +source obligations cannot be reconstructed. Do not adopt a replacement merely +because this operation produced it. diff --git a/packages/oracles/docs/instruments/mutant-gap-hunt.md b/packages/oracles/docs/instruments/mutant-gap-hunt.md new file mode 100644 index 0000000..0ee5740 --- /dev/null +++ b/packages/oracles/docs/instruments/mutant-gap-hunt.md @@ -0,0 +1,105 @@ +# Mutant gap hunt + +**Use when:** a recent refactor, a suspicious green oracle, a survivor whose +missing history is not obvious, or an open review question leaves doubt that +the oracle would reject a plausible wrong behavior. One carefully chosen +mutant can answer a useful question; a campaign is optional. + +**Inputs:** + +- the changed code and the commits that shaped it +- the claimed law and its authority +- the oracle's prose and its `limits` +- the gap check's report, and any open review questions +- the public observation that could distinguish a mistake + +Keep test bodies and assertions out of the design brief. + +## Operation + +1. **Design plausible mistakes before you read the tests.** + - Start with semantic, type-plausible edits near the changed boundaries: a + lost operand, a wrong prior value, a premature clear, a duplicated event, + a changed order, or a skipped callback. + - Turn each open review question into a candidate. + - If code is called dead, try removing it. + - Spread candidates across the affected files and boundaries; don't + cluster them where edits are easy. + - Hand-write candidates at boundaries a designer may miss, such as wrappers, + lazy getters, and caches. + - If you inspect the assertions before drafting, label the design as + self-generated. +2. **Make each edit replayable.** + - Record each mutant in the gap check's authored form: + `{ id, source, find, replace, intent, author }`. `find` must occur + exactly once in `source` at the pinned revision, and the result must + parse. + - Change one decision per mutant. A compound kill does not say which part + the comparison rejected. + - Run each candidate against an unchanged, pinned baseline, and restore + the code exactly. + - A crash outside the comparison, a timeout, a setup failure, or an + unreached edit is not a kill. Record which one it was, not a score. +3. **Interrogate every survivor.** A survivor is a gap. Find which one of + these it shows: + - **A missing history:** no legal history reaches the mutated decision, + or none reaches it under the condition where it matters. Name the history + and add the generator dimension or constructed history for it. + - **A missing observation:** the mutant changes something the driver does + not record. Record it at the checkpoint the contract owes it. + - **A missing or loose law:** the change is observed, but no law rejects + it. State the law and find its authority (use + [law discovery](law-discovery.md)). + - **Dead code:** no legal history can observe the decision. Remove the code. + - **A law question:** the observations differ, but the contract seems to + allow both. Do not record that as acceptable. Give the user the two + observations and the history that shows them, and ask whether the law + should decide the choice. + - **A framework bug:** none of the above fits, such as a mutant that no + oracle could detect. Stop work on that gap, and report its key and + evidence. Calling a mutant invalid does not close the gap. + + There is no "equivalent" class, and no class where the agent accepts a + survivor. Probe the nearest unusual legal history before you call a path + unreachable. + + To confirm a missing history or observation, run one focused probe on both + revisions. Expect a pass on the baseline and a failure on the mutant at the + intended assertion. If the contract predicts the mutant's result instead, + the baseline is RED: investigate a product bug. Then remove the temporary + probe, and check that `git status` is clean. + +4. **Repair the oracle, when authorized.** + - Extend the oracle with the missing history, observation, or law. + - Show that the extension rejects the motivating mutant and a nearby + plausible wrong design, and still accepts every allowed alternative. + - For a bug on the baseline, show RED on production before its repair. + - Then run the gap check again. + +**Return:** for each chosen mutant: + +- its pinned revision and exact edit +- the observed run outcome, using ORC-006's categories in the + [checklist](../oracle-checklist.md) +- which step 3 class it showed, with the distinguishing + history + +Keep raw logs task-local unless the task asks to publish them. + +**Control:** public observations decide product claims. Check that each probe +reaches the path and the premise it names. A green suite does not prove that +no gap exists. Use a positive control when instrumentation reports that a +branch was never reached. Recheck types after a tool rewrites code. + +**Distortion:** + +- mutants chosen to satisfy the existing tests +- syntax failures counted as semantic kills +- private-state differences treated as product bugs +- an oracle extension that still accepts the motivating mutant +- a negative witness that accepts any failure +- a decision checked in one build mode while another mode can observe it + +**Stop when:** each selected mutant has an executed distinction, a removed +piece of dead code, a law question with the user, or a framework bug +reported. Do not infer broad coverage from a kill count. diff --git a/packages/oracles/docs/instruments/tension-scan.md b/packages/oracles/docs/instruments/tension-scan.md new file mode 100644 index 0000000..a51b62b --- /dev/null +++ b/packages/oracles/docs/instruments/tension-scan.md @@ -0,0 +1,37 @@ +# Tension scan + +**Use when:** individually sensible demands appear to collide, or new evidence +may have changed an earlier conflict. + +**Inputs:** the obligations and their sources, the condition in which they +interact, and any earlier formulation. Record which statements are accepted +contracts, user goals, current behavior, or your own inferences. + +## Operation + +1. Establish why each demand matters independently. If only one has support, + return that asymmetry rather than manufacturing two equally sound sides. +2. State the collision concretely: we need A and B, but under C, satisfying A + undermines B through mechanism M. Work through a legal history or design + consequence; a pair of opposing slogans is insufficient. +3. Look for a missing distinction in scope, authority, ordering, ownership, or + checkpoint. Recompute the consequences with that distinction explicit. +4. Check whether both original obligations survive. Removing a guarantee can + be a proposed tradeoff, but does not demonstrate that the tension dissolved. +5. On a recheck, identify the evidence that makes the tension remain live, + sharpen, move, weaken, or dissolve. Read it against the original purpose; + recent or vivid material alone does not justify changing the question. + +**Return:** the sourced demands, concrete collision and mechanism, possible +missing distinction, and current status; or no supported tension, with the +missing support identified. Preserve materially different interpretations when +the available authority cannot choose among them. + +**Control:** restate the case without an A-versus-B framing. Does the conflict +still exist? Keep a generated collision separate from an observed one. + +**Distortion:** invented symmetry, apparent resolution by deleting an obligation, +or using an ordinary clarification to claim a major design discovery. + +**Stop when:** the collision or its dissolution is demonstrated at the stated +scope, or a specific missing fact or design decision is identified. diff --git a/packages/oracles/docs/oracle-checklist.md b/packages/oracles/docs/oracle-checklist.md new file mode 100644 index 0000000..c228ecc --- /dev/null +++ b/packages/oracles/docs/oracle-checklist.md @@ -0,0 +1,321 @@ +# Oracle checklist + +Version 1 (2026-11-10). Cite requirements as `ORC-NNN (checklist v1)`. + +This is the complete list of requirements for an oracle that uses +`@tanstack/oracles`. The review skill reads it in full. The design skill +cites it by ID. Other guides explain techniques; they do not add +requirements unless they cite an ID here. + +It was ported from TanStack DB's oracle rubric. The IDs are the same, so +existing citations keep working. [Changes from the rubric](#changes-from-tanstack-dbs-rubric) +lists every difference. + +## How to read a requirement + +`MUST` is required when its trigger applies. `SHOULD` is expected, unless a +review accepts an explicit, technically grounded reason to do otherwise. +`MAY` is optional. + +Only these uppercase words inside a requirement block create a requirement. +Lowercase words such as "must" in explanations do not. + +Each requirement says what **the framework enforces** and what is left for +the author and the reviewer. Do not work around an enforced part. A +workaround is a finding, even when the tests pass. + +An **important generated property** is an `oracleProperty` that protects a +reusable law, claims coverage beyond named examples, or demonstrates a +bug-class repair. A fixed regression or a bounded enumeration is not one +merely because it uses test data. + +Score the stated obligation. Do not score the presence of a heading, comment +template, or helper. + +## There are no exits + +No requirement here can be met by declaring something out of reach. + +- There are no equivalent mutants. A surviving mutant shows a missing + history, observation, or law, or dead code. +- An effect that no check observes means that the driver does not record it. + Record it, or remove the code that causes it. +- The only exclusions are probe-only code and the `super()` region of a + derived constructor. +- A gap that nobody can kill or remove is a framework bug. Stop and report it + to the maintainer with its key and evidence. A mutant that no oracle could + possibly detect is invalid, and the framework must not generate it. The + author does not decide that a mutant is invalid, and the label does not + close a gap. + +--- + +### ORC-001: Contract authority and limits + +- **Trigger:** A test computes or constrains an expected product result. +- **Obligation:** The oracle MUST identify the promised law, its authority, + and the limits of the claimed result. +- **Framework:** `law.statement`, `law.authority`, and `law.limits` are + required fields. +- **Acceptance evidence:** A reviewer can trace the expected result to an + approved API, contract document and section, or dated maintainer decision, + and can say what the oracle does not establish. Production output is + evidence; it is never authority. +- **Not required:** A complete list of every behavior outside the subsystem. + +### ORC-002: Independent judgment + +- **Trigger:** An oracle compares production with an expected result. +- **Obligation:** The expected result MUST be derived independently of the + production semantic machinery that it judges. +- **Author:** The model receives the plan and the trace. It may read the + inputs that the driver recorded, never production's outputs. The framework + does not enforce this; a review checks it. +- **Acceptance evidence:** The model, alternate implementation, or relation + does not import or reproduce the relevant production classifier, + transition, comparator, or state machine without a separately justified + trusted base. +- **Not required:** Independence of the driver. The driver is expected to call + the real production entry point. + +### ORC-003: Distinguishable oracle responsibilities + +- **Trigger:** Any oracle. +- **Obligation:** The oracle MUST be a literate program. + - Opening prose MUST state the law and its limits before the mechanics. + - Prose beside the model MUST explain why its independent rule predicts the + promised result. + - The oracle MUST explain its legal histories, production path, public + observations, and the checkpoint where the check compares, beside the + relevant code. + - The contract, model, history grammar, driver, and check MUST remain + distinguishable. + - A change to the executable law MUST update its prose in the same change. +- **Framework:** `oracleProperty` takes these as separate parts: `law`, + `histories`, `model`, `driver`, and `check`. Only the check can report a + violation. +- **Acceptance evidence:** A reader learns the contract from the prose and + code together. A compact oracle may use one opening comment and short notes + beside its code. +- **Does not satisfy:** Code alone, headings without explanations, or + explanations that exist only in a pull request. + +### ORC-004: Generated-history grammar controls + +- **Trigger:** A generated property claims coverage of a legal history or + input grammar. +- **Obligation:** The change MUST account for reconstruction, ablation, range, + and exclusion for that grammar. +- **Acceptance evidence:** + - **Reconstruction:** every known valid witness within the claimed scope can + be generated. + - **Ablation:** each declared axis, rule, or overlap has a stated semantic + contribution. + - **Range:** the bounded domains and marginal cases are stated. + - **Exclusion:** a nearby invalid history or state is named, and the grammar + rejects it. +- **Contract changes:** A change that narrows the grammar, a filter, or a + validity predicate removes histories from the claim. It is a contract + change, and it MUST say which previously legal histories it removes and why. +- **Evidence location:** Executable checks are preferred when they are cheap + to keep. Comments or pull-request evidence MAY carry the rest. +- **Not required:** One test per control, or a Cartesian product of unrelated + axes. + +### ORC-005: Production path and observation + +- **Trigger:** Any oracle used as product evidence. +- **Obligation:** The driver MUST exercise the named production entry point + and MUST record the promised public observation at the named checkpoint. +- **Framework:** The driver records observations with + `observe(checkpoint, value)`. Observations must be data; anything else fails + its history with an `OracleDataError`. +- **Acceptance evidence:** The intended path ran, and the check ran at the checkpoint. The observations + can represent the violations relevant to the law, including duplicates, + omissions, order, or partial output when those are contractual. +- **Information sufficiency:** When a concrete hypothesis says a public result + is owed at a checkpoint before the production path has the facts it needs, + the evidence MUST show whether those facts arrive by then. If they do not, + the claim is unresolved. A green comparison after later delivery MUST NOT be + cited as proof of the earlier result. +- **Not required:** Observing internal facts that the contract does not + expose. + +### ORC-006: Checker calibration + +- **Trigger:** An important generated property, or a claimed oracle repair. +- **Obligation:** The evidence MUST name at least one plausible wrong answer + or design and show that the relevant law rejects it. +- **Framework:** A `calibration` names a wrong answer, as a `corrupt` trace or + a mutant `driver`, and the law that must reject it. The calibration test + fails if that law does not report it. +- **Acceptance evidence:** The wrong answer reaches the intended checkpoint + and the law reports the distinguishing failure. +- **Conditional obligation:** When a production mutant is run, the evidence + MUST classify its outcome as one of: assertion failure, timeout, setup + failure, runtime failure outside the intended comparison, unreached path, or + survival. A survivor is a gap (see [There are no exits](#there-are-no-exits)). +- **Not required:** Checking in a mutant. + +### ORC-007: Fixed and random campaigns with direct replay + +- **Trigger:** An important generated property. +- **Obligation:** The same property, generators, driver, check, and budget + MUST run with fixed seeds and without a seed, and a failure MUST be + replayable directly. +- **Framework:** Enforced. A generated `oracleProperty` runs one test per + fixed seed and one random test with the same arbitrary, check, and budget. + A failure prints how to replay it. +- **Author:** Do not bypass this with a raw `fc.assert`, `fc.check`, or `.prop` + in an oracle file. The `oracle/oracle-property-only` lint rule reports them. + +### ORC-008: Stateful-model minimality + +- **Trigger:** A change introduces, removes, combines, or splits state in a + stateful model. +- **Obligation:** The change MUST explain whether a legal next action can + distinguish states that the model proposes to treat as equal. +- **Acceptance evidence:** A distinguishing history justifies retained state. + Removed state needs a reasoned argument that it cannot change a promised + observation or the legality of an action. +- **Not required:** An executable mutant for every model field. + +### ORC-009: Vocabulary mapping + +- **Trigger:** A model combines or splits production concepts, or introduces + a model-only term that could be mistaken for a production concept. +- **Obligation:** The oracle MUST map the differing concepts explicitly. It + MUST use the project's term for a shared domain concept, and the + [package glossary](glossary.md)'s term for an oracle concept. +- **Acceptance evidence:** A reviewer can translate model actions, states, and + checkpoints to production boundaries without guessing. +- **Not required:** A mapping for local data holders whose meaning is + unambiguous. + +### ORC-010: Failure fidelity and cleanup + +- **Trigger:** Shrinking, capture, or cleanup can replace or erase an oracle + failure. +- **Obligation:** The harness MUST preserve the original violated law and + checkpoint, keep cleanup diagnostics distinguishable, and release its + resources. +- **Framework:** Enforced for shrinking. Once a history fails one way, a + smaller history that fails another way counts as passing, so the reported + counterexample still shows the first failure. +- **Author:** The driver releases what it created before it returns or + throws. An `AggregateError` whose `cause` and `errors` keep the distinctions + satisfies this requirement. + +### ORC-011: Independent second formulation + +- **Trigger:** A reviewer names a plausible semantic fault that production and + the model could share, and a meaningfully different formulation could + distinguish it. +- **Obligation:** The change SHOULD add that formulation. If it does not, the + evidence MUST say why it is impractical and keep the risk in the oracle's + limits. +- **Acceptance evidence:** The second formulation states its equivalence or + metamorphic relation, including differences in ordering, projection, + duplicates, and empty results. +- **Does not satisfy:** Agreement between two structurally identical paths. + +### ORC-012: Review evidence + +- **Trigger:** A change claims that an oracle is new, repaired, or fully + audited against this checklist. +- **Obligation:** The review evidence MUST record the outcome of every other + applicable requirement and MUST name the requirements that do not apply, + each with a reason tied to its trigger. +- **Bug-class closure:** A claim that a change eliminates a bug class MUST name + its contract × history × production path × observation boundary, the + original and nearby distinguishing witnesses, and a wrong design rejected at + the intended checkpoint. Unresolved in-scope combinations of contract, history, production path, + and observation boundary go in the oracle's + `limits`. A known reachable counterexample prevents a closure claim. +- **Evidence location:** Evidence that decides the verdict and lives outside + the oracle MUST be in a versioned record tied to the reviewed commit. An + editable pull-request description alone does not satisfy this requirement. + +### ORC-013: Distinguishing witness for a reusable boundary law + +- **Trigger:** An oracle is cited as protecting a reusable conditional, + threshold, or range law beyond its named examples. +- **Obligation:** The evidence MUST show a legal witness that reaches the + law's premise, and a nearby in-domain witness whose observation + distinguishes the law from a plausible wrong boundary or consequence. If + those witnesses are unavailable, the claim MUST be limited to the tested + cases, and the gap MUST be stated in `limits`. +- **Acceptance evidence:** The evidence names the wrong rule and the case that + rejects it. For a numeric threshold, cases on opposite sides are not enough + if a misplaced threshold still produces every expected result. + +### ORC-014: Controlled-premise handoff + +- **Trigger:** A controlled provider or host supplies a premise that matters to + an oracle result, and that result is cited as evidence for behavior with the + real provider or host. +- **Obligation:** The evidence MUST identify that premise. It MUST either + point to a receiving witness in which the real provider supplies the same + premise, or state the cross-boundary claim as unresolved in `limits`. +- **Shared law:** When both environments check the same law, the receiving + witness SHOULD be another driver for the same oracle. It reuses the law, + model, histories, and check, and supplies only its entry point, premise + control, and observations. +- **Not required:** An end-to-end run inside every component oracle. An oracle + whose claim is limited to the controlled boundary needs no real-provider + witness. + +### ORC-015: Declared reach + +- **Trigger:** A change adds or changes an important generated property, its + histories, or its driver, or cites one as evidence for a bug-class repair. +- **Obligation:** The property MUST declare the states that its laws depend + on, and a history MUST be constructed to enter each one. +- **Framework:** Enforced. Every state in `states` must be named by a + constructed history. Every campaign runs each constructed history, and one + that does not enter its named state fails. Half of the generated histories + are also constructed, so the histories after each state are explored. A + state that random histories reach only by chance goes in `measured`, which + is reported but not required. +- **Author:** Reach is a predicate over the plan or trace, or a production + probe. A literal example is not reach. A `fixed` history is a regression, + not reach. A law that needs a state entered at least N times in each + campaign needs N constructed histories that enter it. +- **Acceptance evidence:** The run reports each declared state and how many + histories entered it. + +### ORC-016: Law-only oracle files + +- **Trigger:** A file defines an oracle, its histories, or a campaign, + including a compile-time type oracle. +- **Obligation:** The file MUST NOT contain ordinary example-based unit tests + or standalone concrete type assertions. Fixed regressions MUST be inputs to + the same model, driver, and check as the campaign, and MUST NOT state their + own expected answers. A type oracle MUST apply a reusable independent type + law to generated legal inputs. +- **Framework:** The `oracle/no-unit-test` lint rule reports unit tests in + oracle files. The gap check counts only kills through a registered + `oracleProperty` route; a plain test in an oracle file is reported as an + unregistered kill. +- **Does not satisfy:** Renaming a unit test, adding contract prose, or + wrapping fixed expectations in a property runner. + +--- + +## Changes from TanStack DB's rubric + +Each change below keeps the requirement's purpose. A citation of the +rubric's ID still names the same requirement. + +| ID | Change | Reason | +| ------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | +| All | "Framework" and "Author" notes added | Say what `@tanstack/oracles` enforces, so nobody re-implements or bypasses it. | +| All | Coverage-map owner entries replaced by the oracle's `limits` | Adopters have no coverage map; code ownership belongs to tools such as CODEOWNERS (maintainer, 2026-11-10). | +| ORC-004 | Narrowing a grammar, filter, or validity predicate is a contract change | Maintainer, 2026-11-10. | +| ORC-006 | "Equivalence within the tested domain" removed from mutant outcomes | There are no equivalent mutants (maintainer, 2026-11-10). | +| ORC-007 | Rewritten as an enforced requirement | `oracleProperty` runs the campaigns and prints the replay command. | +| ORC-009 | Oracle terms come from the package glossary | The project glossary keeps domain terms. | +| ORC-015 | Reach comes from constructed histories, not `createOracleReach` | `oracleProperty` is the only campaign API (maintainer, 2026-11-09). | + +TanStack DB adds its own requirements, such as its coverage map, on top of +this checklist in its repository. diff --git a/packages/oracles/package.json b/packages/oracles/package.json index 540ae55..d14f411 100644 --- a/packages/oracles/package.json +++ b/packages/oracles/package.json @@ -48,6 +48,8 @@ "files": [ "dist", "src", + "skills", + "docs", "oracle-gaps", "!oracle-gaps/**/*.property.test.mjs", "!oracle-gaps/gap-report-model-oracle.mjs", diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md new file mode 100644 index 0000000..6886820 --- /dev/null +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -0,0 +1,251 @@ +--- +name: oracles-design +description: Design, write, or change an oracle with @tanstack/oracles - write a law sheet first, choose a model, build histories that reach the states the law needs, show RED on unchanged production, change production, and close every gap. Use for a new oracle, an oracle repair, any production change in code that an oracle owns, and any change to a model, history grammar, driver, or check. +--- + +# Design an oracle + +An oracle states a law, generates the histories where the law matters, and +checks production against an independent model. Make the law clear enough +that someone could predict an observation without reading production. The +code comes after that. + +Read this skill in full, then read the [glossary](../../docs/glossary.md). +The [checklist](../../docs/oracle-checklist.md) is the complete list of +requirements. This skill cites it by ID, such as ORC-004. + +**Work in this order.** Reviewers check the order as well as the result. A +shortcut moves the work into review, where each finding costs a fix, a new +witness, and another round. The common shortcuts are changing production +first, editing the model to match production, narrowing histories until a +failure goes away, and pinning one example. `oracles-review` §2 lists them all. + +## 1. Write the law sheet, then stop + +Before any code, write one row per law and give the sheet to the user. + +| Field | What to write | +| ---------- | --------------------------------------------------------------------------------------------------------------------- | +| Law | The promise, stated so that a reader could predict an observation. | +| Authority | The API or contract document and section, or a dated maintainer decision. | +| Status | `adopted` if it has an authority, `observed` if production does it but nobody adopted it, `open` if it is a question. | +| Rival | A plausible different law. | +| Witness | A legal history where the law and its rival predict different observations. | +| Checkpoint | Where the contract owes the observation. | + +- **Production output is evidence, never authority.** A rule that you write + only because production does it is model hacking. Mark that rule + `observed`, and ask the user. +- Only `adopted` laws become expected results. An `observed` or `open` law + goes to the user as a question. +- When the promise is unclear, or two readings predict different behavior, use + the [law discovery](../../docs/instruments/law-discovery.md) card. + +**Stop here.** Do not write the oracle until the user has reviewed the sheet. + +## 2. Freeze what the oracle must preserve + +Write these down. They stay fixed while you build the oracle. + +- The laws and their checkpoints, from the sheet. +- The owned production code: the files or functions the oracle answers for. +- The freedoms the contract leaves open, such as the order of independent + events, scheduling, or representation. The oracle must accept every + allowed choice. + +Do not weaken a check, history, or law in the same step that clears a +failure. Make each weakening its own decision, with its own reason. + +## 3. Choose the model + +Pick by the shape of the law, in this order, and use the first style that +can state it: + +1. **The observation is a function of the inputs or the whole history.** + Write a pure function from the plan to the expected observation. This is + the default. +2. **A data structure with a simple abstract view.** Write a naive reference + structure, such as an array, a Map, or a sorted copy, and compare + production's observable state with it at each checkpoint. +3. **A lifecycle or protocol with named phases.** Write a transition table + whose rows are `{ from, event, guard, to, effect }`. +4. **A stateful API without natural phases.** Write a pure step function, + `(state, action) → { state, expected }`, that is defined for every legal + action. +5. **The exact output is hard to compute.** State relations between runs, + such as a round trip, incremental equals recompute, or order independence. + Each relation needs a calibration that shows it can fail. +6. **The contract allows several outcomes.** Predict the **allowed set**, and + cite the contract clause for each choice. Never pick one outcome that the + contract leaves open. + +Mixing styles is fine. + +**Structure rules:** + +- One dispatch per role. The resolver, model, and driver may each branch on + the action kind once. +- The model is pure. It takes the plan and returns expectations. It never + reads production or driver values, never mutates its input, and keeps no + hidden state. +- Reach is a predicate over the plan or trace, or a production probe. It is + not a side effect inside the model. +- Every model rule cites its authority. +- Use the project's terms for domain concepts. For anything the model + combines, splits, or invents, apply the glossary's model alignment rule + (ORC-009). + +**Banned patterns** (each is a finding): + +| Pattern | Enforced by | +| ------------------------------------------------------------------------------- | ---------------------------------- | +| B1: expected values captured from production (`toMatchSnapshot` and similar) | review (lint pending) | +| B2: a raw campaign (`fc.assert`, `fc.check`, `.prop`) in an oracle file | lint `oracle/oracle-property-only` | +| B3: `fc.pre` in an oracle file; legality belongs in the grammar or `resolve` | review (lint pending) | +| B4: a model that imports runtime code from production | review (lint pending) | +| B6: a check whose only assertions are "does not throw", "defined", or "settles" | review (warning) | +| B7: the same expression on both sides of a comparison | review (lint pending) | +| B8: a model that branches on a case name or a literal id from the grammar | review (warning) | +| Unit tests or fixed expected answers in an oracle file | lint `oracle/no-unit-test` | + +B1, B3, B4, and B7 are review checks until lint ships them. + +Techniques: [guide § The model](../../docs/guide.md#the-model). + +## 4. Account for every distinction + +Show what each law, model field, and model branch contributes. + +- **A law:** give a legal history that the other laws allow and this law + rejects. +- **A model state or field:** give a legal next action where two states that + differ only in it must behave differently (ORC-008). +- **A branch or exception:** give the history that needs it. If none exists, + remove it. + +If you cannot find a witness, look for a simpler representation before +adding more machinery. For example, three disjoint sets of ids that move +between each other may be one map from id to status. Count complexity that +you move into helpers, adapters, or mappings. Moving it does not remove it. + +**Laws stay simple; history generators may be elaborate.** Reaching a hard +state can need a sophisticated generator. Put that work in the histories, not +in the model. + +## 5. Build histories that reach the law + +- **Grammar.** The fast-check arbitrary and `resolve` produce only legal + histories. Legality belongs here, never in a filter after generation. +- **Changing a filter or validity predicate is a contract change.** Narrowing + the grammar removes histories from the claim. Say which previously legal + histories it removes and why (ORC-004). +- **Reach.** Declare in `states` each state that a law needs, and give each + one a constructed history that enters it (ORC-015). Put a state that you + only count in `measured`. A literal example or a `fixed` history is not + reach. +- **Probes.** Where production decides to enter a state, bind the state to a + production probe, and predict entry from the plan alone. +- **ORC-004 controls.** For a claim about generated coverage, account for: + - reconstruction: every known valid example can be generated + - ablation: what each axis or rule contributes + - range: the bounded domains and marginal cases + - exclusion: a nearby invalid history that the grammar rejects + +Techniques: [guide § Histories](../../docs/guide.md#histories). + +## 6. Write the driver and the check + +- **The driver** calls the real production entry point and records public + observations with `observe(checkpoint, value)`. + - It records production errors, unhandled rejections, and warnings that + the law cares about. + - It observes the contract's return values. + - It awaits the work it starts. + - It bounds its own run time and work, and throws a structured error when + a bound is passed. + - It releases what it created. +- **The check** compares observations with the model at each checkpoint. It + is the only place that reports a violation. For exact values, use + `differs(...) ?? pass`. For an allowed set, accept any member. +- **Calibration.** Give at least one `calibration`: a wrong answer, as a + corrupted trace or a mutant driver, and the law that must reject it + (ORC-006). +- **Ownership.** Put the owned files or functions on the property's `owns` + list. The gap check judges only owned code. + +Keep the law, its authority, and its limits in `law`, and prose beside each +part that says why it predicts the promised result (ORC-003). + +Techniques: [guide § The driver and the check](../../docs/guide.md#the-driver-and-the-check). + +## 7. RED, change, GREEN, hostile mutant + +When the task changes production: + +1. Run the extended oracle file on **unchanged** production. It must fail on + the defect. This is RED. A setup failure is not RED: fix the oracle first. + (Until the gap check reports a failing baseline, run the oracle file with + Vitest for this step.) +2. Change production, and rerun the oracle after each attempt. It also shows + when an attempt breaks a neighboring law. +3. Run the oracle again. It passes. This is GREEN. +4. Show that a hostile mutant, such as the original defect, fails the + oracle. + +Do not change production before step 1. A later run does not make up for a +skipped RED. + +Techniques: [guide § Evidence](../../docs/guide.md#evidence). + +## 8. Close every gap + +Run the gap check on the owned code, then work from its report. Every gap +comes with one instruction. Do what it says, within the rules below. + +- **There are no equivalent mutants.** A surviving mutant shows a missing + history, observation, or law, or it shows dead code. +- **An effect that no check sees is one that the driver does not record.** + Record it. +- **Code whose removal no legal history can observe is dead.** Remove it. +- **The only exclusions are probe-only code and the `super()` region of a + derived constructor.** +- **Never propose an exit.** If a gap can be neither killed nor removed, it is + a framework bug. Stop work on that gap, report its key and evidence to the + user, and keep working on the other gaps. +- **A mutant that no oracle could possibly detect is an invalid mutant.** The + framework must not generate it, so it is a framework bug: report it the same + way. You do not decide that a mutant is invalid, and the label does not close + its gap. +- **Temporary, until the report's messages change:** if a message calls a + mutant "equivalent", asks for a "maintainer's approval", or offers a limit + that "needs a maintainer's decision", do not follow that part. Find the + history and the law. + +Use the [mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) card for +a survivor whose missing history is not obvious. + +## 9. Report the evidence + +The evidence counts only as a set. One item alone proves little. + +- **Law:** the law and its authority. +- **Reach:** each declared state and how many histories entered it. +- **RED:** the command and the failure on unchanged production. +- **GREEN:** the command and the result after the change. +- **Mutant:** the hostile mutant and the law that rejected it. +- **Gaps:** the gap check's totals, and any gap reported as a framework bug. +- **Limits:** the states, paths, and observations that the evidence does not + cover. + +## Instruments + +Use a card when one of these uncertainties is real. There is no required +sequence. + +| Uncertainty | Card | +| ------------------------------------------------------------------ | ------------------------------------------------------------------ | +| The promise is unclear, or two readings predict different behavior | [Law discovery](../../docs/instruments/law-discovery.md) | +| Production and the model may share a mistake | [Adversarial review](../../docs/instruments/adversarial-review.md) | +| Two obligations collide under some condition | [Tension scan](../../docs/instruments/tension-scan.md) | +| Laws overlap, or their concepts seem wrong | [Law restructuring](../../docs/instruments/law-restructuring.md) | +| The oracle is green but may accept a wrong implementation | [Mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) | diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md new file mode 100644 index 0000000..6c1d7df --- /dev/null +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -0,0 +1,163 @@ +--- +name: oracles-review +description: Review oracle work done with @tanstack/oracles - challenge whether an oracle enforces its law, whether it is the right law, and whether the work took a shortcut. Use for reviewing a new or changed oracle, a production change in code that an oracle owns, or a proposed change to what an oracle promises. +--- + +# Review oracle work + +Ask two questions: does the oracle enforce its stated law, and is that the +right law to promise? A passing oracle can coexist with a wrong law. A +failing one can coexist with correct production. + +Read the [checklist](../../docs/oracle-checklist.md) and the +[glossary](../../docs/glossary.md) in full before you judge anything. Judge +conformance by the checklist's requirements, not by headings or templates. +After a handoff or a context reset, read them again: another agent's claim +that it read them does not give you their definitions. + +The repository may add its own review rules on top of this skill, in its +`AGENTS.md` or its own review skill. Apply both. + +## 1. Rebuild one observation yourself + +- Follow the law's authority to its source. Read the neighboring promises + there. The oracle's summary can omit an obligation that it should check. +- From the contract alone, without production, predict one promised + observation for one history. +- Then trace what the oracle actually does with that history: + - the model's rule + - the legal histories that reach it + - the production path the driver calls + - what the driver records + - the check at the checkpoint + +Every claimed model output must reach the check at its promised +checkpoint. A final state can hide a wrong intermediate observation. + +## 2. Check the order of the work + +These are the shortcuts. Report each one as a finding. A change that +contains one is not ready. + +| Shortcut | What to look for | +| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Law after code** | No law sheet before the oracle was written, or a law with no authority. | +| **Production first** | A production change in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | +| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | +| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. | +| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | +| **Pinned only** | A fixed example where the law needs a generator dimension. | +| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | +| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | +| **No mutant** | A claim that a defect is closed, without a hostile mutant that fails the oracle. | +| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | + +The evidence counts only as a set: law, reach, RED, GREEN, mutant, gaps, +limits. A missing item is a finding, even when the rest is strong. + +## 3. Hunt what the tools cannot see + +Lint, reach counts, the gap check, and calibrations miss the weaknesses +below. Hunt for each one across the whole oracle, not only the diff. This +hunt is your main job. + +- **Run but not checked.** A history executes the code, but the check + never observes its effect. Reach passes anyway. Find a line that the oracle + claims, and name the mutant on it that would survive. +- **A model that copies production.** The model mirrors production's + branches, classifications, or helpers, so both agree on the same wrong + answer. Compare the model's structure with production, not only its + results. +- **A grammar narrower than the law.** The law claims a value domain, action, + or case that the grammar never generates. Examples: + - a type that the generator never produces + - an exclusion stated in prose but missing from `limits` + - a size setting that makes part of the claimed range unreachable + + Check each claim in the law against a generator dimension. + +- **Easy declared states and probes.** A declared state or probe sits where + it is always true, or uses a weaker condition than the state it names. Its + count rises without showing that histories enter the interesting state. + Check each definition against the state that the law needs. +- **A late or coarse checkpoint.** The check runs only at the end of a + history, or observes only a summary, so a wrong intermediate observation + disappears. Ask whether a user could see the wrong value before the + checkpoint. +- **A limit that hides in-scope work.** A stated limit excludes a legal case + that the law promises. Check that each limit excludes exactly what its + reason covers. +- **A dismissed survivor.** A surviving mutant is called equivalent, + unobservable, or out of scope, or is called invalid and treated as closed. + That is an exit, and a finding. The survivor shows a missing history, + observation, or law, dead code, or a framework bug. Reporting a suspected + invalid mutant as a framework bug, with its key and evidence, is correct and + is not an exit. +- **An invariant guard without a witness.** A throw for a contradictory + collaborator signal or an impossible state has no history that reaches it. + Require a fault-injected step that supplies the contradictory signal. + Expect production to fail at the boundary, before it releases established + state or publishes success. + +## 4. Choose attacks for the real uncertainty + +Use the [instrument cards](../../docs/instruments/) when they fit. Do not run +every card, and do not require a finding at every level. + +- **[Adversarial review](../../docs/instruments/adversarial-review.md):** + attacks enforcement, the law's statement, its suitability, or its + concepts. Name the level each attack addresses. +- **[Law discovery](../../docs/instruments/law-discovery.md):** for a law with + no clear authority, or two readings that predict different behavior. +- **[Tension scan](../../docs/instruments/tension-scan.md):** for demands that + collide under a concrete condition. +- **[Law restructuring](../../docs/instruments/law-restructuring.md):** for + whether merging, splitting, or replacing laws gives a better account. +- **[Mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md):** for whether + a plausible wrong implementation can pass. + +Ask these questions of every model and grammar: + +- **What does each distinction contribute?** For each law, model field, and + branch, is there a legal history that needs it? If not, it is a candidate + for removal or for a simpler representation (ORC-008). +- **Is the complexity in the right place?** Laws stay simple; generators may + be elaborate (`oracles-design` §4). +- **Did the domain shrink?** A changed filter, validity predicate, or grammar + bound removes histories from the claim. That is a contract change, and it + needs its own reason (ORC-004). +- **Does the oracle keep the contract's freedoms?** Where the contract allows + several outcomes, the model must predict the allowed set. An oracle that + accepts only the current implementation's choice is a finding. + +When you can, use a fresh reviewer for an independent challenge. Give it the +work, the sources, and the task, but not the author's preferred verdict. +Label a self-review as one. + +Techniques: [guide](../../docs/guide.md), especially [§ Evidence](../../docs/guide.md#evidence) and [§ Reusable boundary laws](../../docs/guide.md#reusable-boundary-laws). + +## 5. Report findings + +For each finding, give: + +1. The claim you challenge. +2. Its source or authority. +3. The smallest legal witness, or a concrete design consequence. +4. The evidence status: run, read from source, or a proposed experiment. +5. The repair, or the next experiment. + +- **A confirmed bug:** say why the existing oracle missed it, and name the + smallest general improvement: a generator dimension, an observation, or a + law. A pinned example alone is not the repair. +- **Defects and proposals stay separate.** A defect breaks an accepted + contract. A proposal changes the contract. A proposal must name the + external standard or engineering reason behind it, and its cost. Existing + behavior does not prove itself correct, and a preference does not prove a + violation. +- **Setup failures and timeouts are not kills.** Nor are they evidence that a + law rejected a mutant. + +Report the supported findings and the remaining uncertainty. Do not invent findings, retire laws, or rewrite expectations to look +decisive. + +If the review leads to oracle edits, load `oracles-design` before you edit. diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md new file mode 100644 index 0000000..6920a63 --- /dev/null +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -0,0 +1,195 @@ +--- +name: oracles-setup +description: Set up @tanstack/oracles in a repository - install the pinned packages, add the lint preset, write the AGENTS.md block, make the oracle skills findable by coding agents, and run a first oracle through the gap check. Use when a repository adopts @tanstack/oracles or upgrades its pinned commit. +--- + +# Set up @tanstack/oracles + +You are setting up oracle testing in this repository. Do the steps in order. +Ask the user before each step that writes outside the repository. + +Setup ends when one oracle passes on production and the gap check has judged +it with no gap left open, unless the user decides otherwise. It does not end +when the packages are installed. + +## 1. Read the repository + +Find out, and tell the user: + +- The package manager, and whether the repository is one package or a + workspace. For a workspace, ask which package to start in. +- The test runner. The framework runs on Vitest 4 or later, with fast-check 4. + If the package uses another runner, oracles run in a separate Vitest config + beside it; do not change the existing suite. +- Where production source lives (for example `src/`), and whether ESLint + parses TypeScript already. +- Which coding agents the user works with. Look for `.claude/`, `.agents/`, + `.codex/`, `AGENTS.md`, and `CLAUDE.md`, and ask. + +## 2. Install two pinned packages + +The packages are not on npm yet. Install both from pkg.pr.new, pinned to the +same commit of TanStack/oracles. Ask the user for the commit if you were not +given one. + +```sh +pnpm add -D \ + https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@ \ + https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@ \ + fast-check vitest +``` + +The only peers to add are `fast-check` and `vitest`. Never add, upgrade, or +downgrade the repository's `typescript` for oracles: the gap check uses its +own TypeScript and never reads the project's, from commit `d8adfee` on. Pin a +commit at or after `d8adfee`. + +Use the repository's own package manager. To upgrade, change both URLs to +the new commit at once, then do step 4 again and rerun the gap check (step 7). + +## 3. Configure + +**Framework.** The defaults use the `ORACLE` environment-variable prefix and +accept every property id. Call `configureOracles` only to change that, in a +Vitest setup file that runs before any oracle module is imported. + +**Lint.** Add the preset to the ESLint flat config: + +```js +import oracles from '@tanstack/eslint-plugin-oracles' + +export default [ + // ...the repository's existing config + oracles.configs.recommended, + { + // Oracle files: the rules must not be switched off inline. + files: ['**/*oracle*.{ts,tsx,js,jsx,mjs}'], + linterOptions: { noInlineConfig: true }, + }, + { + // The probe form, on this repository's production source. + files: ['src/**/*.{ts,tsx,js,jsx,mjs}'], + plugins: { oracle: oracles }, + rules: { 'oracle/probe-form': 'error' }, + }, +] +``` + +- Change the `src/**` glob to the production source you found in step 1. The + preset's own `production` config matches only `packages/*/src/**`. +- If ESLint does not already parse TypeScript, add `@typescript-eslint/parser` + as a dev dependency and set it as `languageOptions.parser` for the oracle + files. +- Do not add `legacyCampaigns` or any other exemption. Adopters have none. + +Run the linter once and confirm that it loads the rules without an error. + +## 4. Write the AGENTS.md block + +Add this block to `AGENTS.md` (create the file if needed). On an upgrade, +replace everything between the markers and keep the rest of the file. + +```md + + +## Oracle tests + +This repository tests its laws with `@tanstack/oracles`. + +- Before you create or change an oracle, or change production code that an + oracle owns, read `node_modules/@tanstack/oracles/skills/oracles-design/SKILL.md` + in full and follow it. +- Before you review oracle work, read + `node_modules/@tanstack/oracles/skills/oracles-review/SKILL.md` in full. +- Oracle files: ``. Gap check: ``. + + +``` + +Fill in the two placeholders in step 7, once their values exist. If the repository has a +`CLAUDE.md` and no `AGENTS.md`, ask the user which file gets the block. + +## 5. Make the skills findable + +The skills ship inside the package, so agents find them only through a +pointer. Offer the user these choices. They can pick more than one. + +| Choice | Writes | Use when | +| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | +| **AGENTS.md only** | Step 4's block | The user's agents read `AGENTS.md`. This is always done. | +| **Repository pointers** | A pointer skill per skill in `.agents/skills/` and `.claude/skills/` | Everyone who clones the repository gets the skills. Commit them. | +| **Global pointers** | A pointer skill per skill in `~/.agents/skills/` and `~/.claude/skills/` (and any other skills directory the user's agents read) | The user wants the skills in every repository that uses `@tanstack/oracles`. | + +Write a pointer skill as `//SKILL.md` for each of +`oracles-design`, `oracles-review`, and `oracles-setup`. + +- Copy `name` and `description` exactly from the package's `SKILL.md`, so the + agent selects it for the same requests. +- The body says only where the real skill is: + +```md +--- +name: oracles-design +description: +--- + +Read `node_modules/@tanstack/oracles/skills/oracles-design/SKILL.md` from the +root of the current repository in full, and follow it. It matches the +installed version of `@tanstack/oracles`. If that file does not exist, this +repository does not have `@tanstack/oracles` installed: say so and stop. +``` + +Use pointers, not copies or symbolic links: a pointer always reads the +installed version, and never breaks before install. + +Before writing a global pointer: + +- Show the user the exact paths. +- If a skill with the same name exists, show it and ask. Never overwrite it + silently. + +## 6. Write the first oracle + +Choose one small, owned production function with a clear promise. Ask the +user which one if it is not obvious. + +Then load `oracles-design` and follow it from its first step. The law sheet +comes first, and the user reviews it before you write code. Setup does not +skip that step to finish sooner. + +Oracle files must match the glob `*oracle*`, for example +`tests/clamp.oracle.property.test.ts`. Put the oracle's `owns` list on its +`oracleProperty` call. If the gap check counts **0 oracle files**, the +oracle is not declared. That is a setup failure, even when the report shows +0 gaps and the command exits 0. + +In an oracle file, only `oracleProperty` registers tests. A plain Vitest test +there is not a kill, and the gap check reports it. + +## 7. Run the gap check + +Run the gap check on the owned source: + +```sh +pnpm exec oracle-gaps --package --source +``` + +Run it after the oracle passes on production. Each gap in the report comes +with one instruction. Follow +that instruction, and follow `oracles-design` for what a gap means. There +are no equivalent mutants and no exits. If a gap can be neither killed nor +removed, stop work on that gap and report its key and evidence to the user. + +Now fill in the two placeholders in the `AGENTS.md` block: the oracle file +glob and the gap check command you ran. + +## 8. Report + +Tell the user: + +- what was installed (both pinned URLs) and where configuration changed +- which pointers were written, and where +- the first oracle's law and its result +- the gap check's totals, and every gap left with its instruction + +Do not call setup done while a gap is open, unless the user decides otherwise. From 25b8bbe91cea2ff81399f1c3a69ce7fa2966bdef Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 13:33:07 -0700 Subject: [PATCH 02/76] chore: add changeset for the oracle skills and docs Co-authored-by: Isaac --- .changeset/ship-oracle-skills.md | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 .changeset/ship-oracle-skills.md diff --git a/.changeset/ship-oracle-skills.md b/.changeset/ship-oracle-skills.md new file mode 100644 index 0000000..06372e5 --- /dev/null +++ b/.changeset/ship-oracle-skills.md @@ -0,0 +1,6 @@ +--- +'@tanstack/oracles': patch +'@tanstack/eslint-plugin-oracles': patch +--- + +Ship agent skills and docs inside `@tanstack/oracles`. The package now includes `skills/` with `oracles-setup`, `oracles-design`, and `oracles-review`. It also includes `docs/`: the oracle checklist (version 1, ORC-001 to ORC-016), a glossary, a techniques guide, and five instrument cards. A coding agent reads them from `node_modules`. `@tanstack/eslint-plugin-oracles` gains a README that lists its configs and rules. From 27a9ad7d13fc11512dc5377611bec34cce588740 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 14:18:03 -0700 Subject: [PATCH 03/76] docs(oracles): add one worked example per model style `oracles-design` step 3 named six model styles but showed no example of any but a pure function. docs/model-styles.md gives a small, generic, type-checked example of each: a pure function, a naive reference structure, a transition table with derived reach, a pure step function, a relation between runs with its calibration, and an allowed set. It also states two rules from the model prototypes: each model rule is one sentence of the law, and legality lives in a separate pure guard. Co-authored-by: Isaac --- packages/oracles/README.md | 1 + packages/oracles/docs/model-styles.md | 289 ++++++++++++++++++ .../oracles/skills/oracles-design/SKILL.md | 3 +- 3 files changed, 292 insertions(+), 1 deletion(-) create mode 100644 packages/oracles/docs/model-styles.md diff --git a/packages/oracles/README.md b/packages/oracles/README.md index ae2a197..c68ea8e 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -33,6 +33,7 @@ coding agents, and walks through a first oracle and gap check. | `skills/oracles-review/` | How to review oracle work. | | `docs/oracle-checklist.md` | The complete requirements, by ID (ORC-001 to ORC-016). | | `docs/guide.md` | Techniques for each design and review step. | +| `docs/model-styles.md` | One small example of each model style. | | `docs/glossary.md` | The oracle terms. | | `docs/instruments/` | Short reasoning cards that the skills route to. | | `oracle-gaps/` | The gap check (`oracle-gaps` binary). | diff --git a/packages/oracles/docs/model-styles.md b/packages/oracles/docs/model-styles.md new file mode 100644 index 0000000..b37e59c --- /dev/null +++ b/packages/oracles/docs/model-styles.md @@ -0,0 +1,289 @@ +# Model styles, by example + +`oracles-design` step 3 lists six model styles in order. Use the first style +that can state the law. This page gives one small example of each. The +examples use made-up domains. Copy the shape, not the domain. + +Two rules hold for every style: + +- **Each model rule is one sentence of the law.** A reviewer can point from a + sentence of `law.statement` to the rule that enforces it, and back. +- **Legality is not the model.** If the grammar or `resolve` needs state to + pick legal actions (counts, phases, open handles), compute it in a separate + pure `legal(prefix, action)` guard. Keep the model a pure function from the + plan to the expected result. + +## 1. A pure function of the inputs or the whole history + +Use this by default. The expected observation is a formula over the plan. + +Law: "A page holds the rows that match the filter, sorted by `id`, from +`offset`, at most `limit` rows." + +```ts +type Row = { id: number; tag: string } +type PagePlan = { rows: Array; tag: string; offset: number; limit: number } + +function expectedPage(plan: PagePlan): Array { + return plan.rows + .filter((row) => row.tag === plan.tag) + .map((row) => row.id) + .sort((a, b) => a - b) + .slice(plan.offset, plan.offset + plan.limit) +} +``` + +The model ignores how production finds the rows (an index, a cursor, a cache). +A formula like this also forces you to state edge cases: an `offset` past the +end gives an empty page, and a `limit` of 0 gives an empty page. + +## 2. A naive reference structure and an abstraction + +Use this for a data structure with a simple abstract view. Keep a plain copy +of that view, and compare it with what production shows at each checkpoint. + +Law: "A cache of capacity `n` keeps the `n` most recently used keys. `get` +returns the stored value of a kept key, and `undefined` otherwise." + +```ts +type CacheAction = + { kind: 'set'; key: string; value: number } | { kind: 'get'; key: string } + +function expectedReads( + capacity: number, + actions: Array, +): Array { + // The reference: keys from least to most recently used. + const order: Array = [] + const values = new Map() + const reads: Array = [] + const touch = (key: string) => { + order.splice(order.indexOf(key), 1) + order.push(key) + } + for (const action of actions) { + if (action.kind === 'set') { + if (!values.has(action.key)) order.push(action.key) + values.set(action.key, action.value) + touch(action.key) + if (order.length > capacity) values.delete(order.shift()!) + } else { + const value = values.get(action.key) + if (value !== undefined) touch(action.key) + reads.push(value) + } + } + return reads +} +``` + +The driver records only what the contract exposes: each `get` result. If the +contract also promises iteration order, the driver records the key order +(the abstraction of production's state) at the same checkpoint. + +## 3. A transition table for a lifecycle + +Use this when the law names phases. Each row is one rule, with an id and its +authority. Reach is the set of rows that fired, so transition coverage is +measured, not guessed. + +Law: "A connection opens only from `idle`. A failure while connecting returns +it to `idle`. Closing is final." + +```ts +type Phase = 'idle' | 'connecting' | 'open' | 'closed' +type ConnEvent = 'connect' | 'connected' | 'fail' | 'close' +type TableRow = { + id: string + from: ReadonlyArray + event: ConnEvent + to: Phase + authority: string +} + +const rows: ReadonlyArray = [ + { + id: 'connect', + from: ['idle'], + event: 'connect', + to: 'connecting', + authority: 'README, Lifecycle', + }, + { + id: 'connected', + from: ['connecting'], + event: 'connected', + to: 'open', + authority: 'README, Lifecycle', + }, + { + id: 'fail', + from: ['connecting'], + event: 'fail', + to: 'idle', + authority: 'README, Lifecycle', + }, + { + id: 'close', + from: ['idle', 'connecting', 'open'], + event: 'close', + to: 'closed', + authority: 'README, Lifecycle', + }, +] + +function expectedPhases(events: Array): { + phases: Array + fired: Array +} { + let phase: Phase = 'idle' + const phases: Array = [] + const fired: Array = [] + for (const event of events) { + const row = rows.find((r) => r.event === event && r.from.includes(phase)) + // No row means the event has no effect in this phase. + if (row) { + phase = row.to + fired.push(row.id) + } + phases.push(phase) + } + return { phases, fired } +} +``` + +Return `fired` from `reached`, so each row id is a reach state. A row that +never fires shows a missing history, or a rule that the law does not need. + +A table costs more lines than a formula. Use it when the phases are part of +the contract, or when you need to show which transitions the histories cover. +A table may also call a formula for one rule, such as a timeout guard. + +## 4. A pure step function + +Use this for a stateful API whose answers depend on what came before, without +named phases. The step function handles every legal action. + +Law: "A withdrawal succeeds only when it does not exceed the balance. A +deposit always succeeds. Each call returns the new balance, or `rejected`." + +```ts +type Account = { balance: number } +type AccountAction = + { kind: 'deposit'; amount: number } | { kind: 'withdraw'; amount: number } +type AccountResult = number | 'rejected' + +function step( + state: Account, + action: AccountAction, +): { state: Account; expected: AccountResult } { + switch (action.kind) { + case 'deposit': + return { + state: { balance: state.balance + action.amount }, + expected: state.balance + action.amount, + } + case 'withdraw': + return action.amount <= state.balance + ? { + state: { balance: state.balance - action.amount }, + expected: state.balance - action.amount, + } + : { state, expected: 'rejected' } + } +} + +function expectedResults(actions: Array): Array { + let state: Account = { balance: 0 } + return actions.map((action) => { + const next = step(state, action) + state = next.state + return next.expected + }) +} +``` + +There is one `switch` on the action kind. A rejected withdrawal is an +expected result, not an illegal history, so the grammar generates it. + +## 5. Relations between runs + +Use this when the exact output is hard to compute. State a relation that must +hold between two runs, or between a run and its inverse. + +Law: "`parse(serialize(value))` equals `value` for every value in the domain." + +```ts +type Value = { name: string; tags: Array } + +function roundTripHolds( + serialize: (value: Value) => string, + parse: (text: string) => Value, + value: Value, +): boolean { + const back = parse(serialize(value)) + return ( + back.name === value.name && + back.tags.join('\u0000') === value.tags.join('\u0000') + ) +} +``` + +In an oracle, the driver records `value` and `parse(serialize(value))`, and the +check compares them. A relation can hold for a useless implementation: a +`serialize` that returns `""` and a `parse` that returns a fixed value +satisfy many symmetries. So each relation needs a calibration that shows it +fails, such as a trace whose parsed value drops a tag. + +Other useful relations: an incremental result equals a recompute from scratch, +and the order of independent updates does not change the final state. + +## 6. An allowed set of outcomes + +Use this when the contract leaves a choice open. Predict every allowed +outcome, cite the clause for each, and accept any member. + +Law: "After `save`, the store notifies `written` and `indexed`, in either +order (README, Events: the two listeners are independent)." + +```ts +type Notice = 'written' | 'indexed' + +function allowedNotices(): Array> { + return [ + ['written', 'indexed'], + ['indexed', 'written'], + ] +} + +function noticesAllowed(observed: Array): boolean { + return allowedNotices().some( + (allowed) => + allowed.length === observed.length && + allowed.every((notice, i) => notice === observed[i]), + ) +} +``` + +Never pick the order that production happens to use: that would reject a +valid implementation. A mutant that only switches production to the other +allowed order survives a sound oracle. Until the framework recognizes +allowed sets in the gap check, treat such a survivor as a law question for +the user (see the [mutant gap hunt](instruments/mutant-gap-hunt.md) card). + +## Mixing styles + +Mix styles when one law has parts of different shapes. For example, a +lifecycle table can call a formula for its timeout rule: + +```ts +// A row's guard that is a formula, not more table rows. +const retiresNow = ( + unheldSince: number, + now: number, + gcTime: number, +): boolean => now > unheldSince + gcTime +``` + +The formula states the timing rule in one line. The table keeps the phases +and makes each transition a row. diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 6886820..2b63161 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -79,7 +79,8 @@ can state it: cite the contract clause for each choice. Never pick one outcome that the contract leaves open. -Mixing styles is fine. +Mixing styles is fine. [Model styles](../../docs/model-styles.md) shows a +small example of each style. **Structure rules:** From 045347fab22ae012448c826621558f8084b78a58 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 14:29:42 -0700 Subject: [PATCH 04/76] docs(oracles): use review and testing terms, not security terms Oracle reviews were sometimes read as security work. The skills and docs now avoid words that suggest it: the adversarial-review card is the challenge-review card, attacks are challenges, a hostile mutant is a mutant (as TanStack DB's glossary already asks), a fault-injected step is a step that simulates a signal, and "bypass", "defeat", and "payload" are reworded. Links follow the renamed card. Co-authored-by: Isaac --- packages/oracles/docs/glossary.md | 3 +-- packages/oracles/docs/guide.md | 2 +- ...ersarial-review.md => challenge-review.md} | 18 ++++++++-------- packages/oracles/docs/instruments/index.md | 20 ++++++++---------- .../docs/instruments/law-restructuring.md | 4 ++-- packages/oracles/docs/oracle-checklist.md | 4 ++-- .../oracles/skills/oracles-design/SKILL.md | 21 ++++++++++--------- .../oracles/skills/oracles-review/SKILL.md | 12 +++++------ 8 files changed, 41 insertions(+), 43 deletions(-) rename packages/oracles/docs/instruments/{adversarial-review.md => challenge-review.md} (88%) diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md index b0c7c53..f7c96f3 100644 --- a/packages/oracles/docs/glossary.md +++ b/packages/oracles/docs/glossary.md @@ -58,7 +58,7 @@ term for one concept. Do not rotate synonyms for style. | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | owns | The production files or functions an oracle answers for, listed in the `owns` option of `oracleProperty`. The gap check judges owned code only. | | gap check | The loop that runs the oracle files against production and against mutants of the owned code. It reports every gap with one instruction. | -| mutant | A small deliberate change to owned production code. | +| mutant | A small deliberate change to owned production code. Every mutant is deliberately wrong, so do not qualify the word. The original defect is a common mutant to check. | | kill | A mutant that a law rejects through a registered `oracleProperty` route. A plain test in an oracle file is an unregistered kill, and it does not count. | | survivor | A mutant that no law rejects. Every survivor is a gap. | | gap | Owned behavior that the oracles do not judge. It closes only when a law kills it or the code is removed. | @@ -70,7 +70,6 @@ term for one concept. Do not rotate synonyms for style. | unowned | Production code that no oracle owns. The gap check reports it as a warning, not as judged. | | RED | The extended oracle failing on unchanged production. It shows that the oracle can see the defect. | | GREEN | The same oracle passing after the change. | -| hostile mutant | A deliberate wrong implementation, often the original defect, that the oracle must reject before a repair is called closed. | There is no "equivalent mutant" and no "unobservable effect". A survivor shows a missing history, observation, or law, or dead code. An effect that diff --git a/packages/oracles/docs/guide.md b/packages/oracles/docs/guide.md index 410084b..e38cb0d 100644 --- a/packages/oracles/docs/guide.md +++ b/packages/oracles/docs/guide.md @@ -158,7 +158,7 @@ count is not a substitute for a constructed history. ### Generate histories that reach the problem Once the model can tell the relevant states apart, the histories must put -production into them. Random payloads alone do not create reuse, failed +production into them. Random values alone do not create reuse, failed startup, or work after recovery. Look for relationships and transitions: diff --git a/packages/oracles/docs/instruments/adversarial-review.md b/packages/oracles/docs/instruments/challenge-review.md similarity index 88% rename from packages/oracles/docs/instruments/adversarial-review.md rename to packages/oracles/docs/instruments/challenge-review.md index 992e37d..e5757e1 100644 --- a/packages/oracles/docs/instruments/adversarial-review.md +++ b/packages/oracles/docs/instruments/challenge-review.md @@ -1,4 +1,4 @@ -# Adversarial review +# Challenge review **Use when:** a model, oracle, or proposed law needs a concrete challenge. @@ -12,7 +12,7 @@ validation. Start with a plausible way this account could be wrong. Select the relevant levels; they are different questions, not a mandatory sequence: -| Level | Attack | +| Level | Challenge | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Enforcement | Find a legal history or wrong output that the model, generator, driver, or recorder cannot distinguish. Can the intended production path remain unexercised while the test passes? | | Formulation | Find an omitted precondition, excessive guarantee, wrong checkpoint, or ambiguous observation. Try a correct alternative implementation as well as a wrong one. | @@ -25,7 +25,7 @@ step 5. ### Locate the repair -The attack level names what to challenge. Once evidence shows a problem, use +The level names what to challenge. Once evidence shows a problem, use this table to locate the repair: | Finding or question | Possible repair location | @@ -46,14 +46,14 @@ information needed to meet it. A repair might require additional provider data. Moving the promised checkpoint or narrowing the guarantee needs an authorized contract decision; changing an assertion supplies no data. -For each material attack, name the broken claim and construct a small legal +For each material challenge, name the broken claim and construct a small legal history or concrete failure scene. State the competing expected observations, the evidence that could decide between them, and a possible repair condition. -Check that the attack is within scope and that its standard actually applies. +Check that the challenge is within scope and that its standard actually applies. An outside-domain case may motivate a scope change; label that proposal. When execution is authorized and practical, run the discriminating experiment. -A hostile control must reach the intended comparison to demonstrate rejection. +A wrong-answer control must reach the intended comparison to demonstrate rejection. Use the oracle guide's calibration and mutant-outcome rules. A source-level argument can remain useful when execution is unavailable; label it accurately. For a systematic search across plausible wrong implementations, use the @@ -64,13 +64,13 @@ survivors. separate kinds of finding, with sources, witnesses, outcomes, and repair conditions. For bugs, identify the test gap that permitted the class of failure. -**Control:** explain what would defeat the criticism. Check whether the attack +**Control:** explain what would refute the criticism. Check whether the challenge would also reject a valid implementation or depend on a condition the contract -excludes. Do not presume a defect merely because the stance is adversarial. +excludes. Do not presume a defect merely because the stance is critical. **Distortion:** generic pessimism, irrelevant external standards, author capture, or treating a preferred design as evidence of a current correctness bug. -**Stop when:** the material attacks have evidence or an explicit unresolved +**Stop when:** the material challenges have evidence or an explicit unresolved experiment. No supported finding is a valid result. Challenging a law does not silently replace it. diff --git a/packages/oracles/docs/instruments/index.md b/packages/oracles/docs/instruments/index.md index 5e6b2c5..da1ce96 100644 --- a/packages/oracles/docs/instruments/index.md +++ b/packages/oracles/docs/instruments/index.md @@ -5,13 +5,13 @@ review insufficient. Read only the relevant card. Choose and use it within the user's authorized task; these cards do not require a separate approval for each reasoning step or a fixed workflow. -| Uncertainty | Instrument | Result | -| --------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -| What behavior should this subsystem promise? | [Law discovery](law-discovery.md) | A sourced candidate law, competing predictions, and a distinguishing witness. | -| Could the implementation, model, or law itself be wrong? | [Adversarial review](adversarial-review.md) | Concrete attacks at named levels, their evidence, and repair conditions. | -| Would a plausible wrong implementation still pass the oracle? | [Mutant gap hunt](mutant-gap-hunt.md) | A semantic mutant and what its survival shows: a missing history, observation, or law, dead code, a law question, or a framework bug. | -| Do sound obligations conflict under the same conditions? | [Tension scan](tension-scan.md) | A traced collision, missing distinction, or no supported tension. | -| Should these laws or concepts be merged, split, replaced, or removed? | [Law restructuring](law-restructuring.md) | An alternative organization and a check of what it preserves or loses. | +| Uncertainty | Instrument | Result | +| --------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| What behavior should this subsystem promise? | [Law discovery](law-discovery.md) | A sourced candidate law, competing predictions, and a distinguishing witness. | +| Could the implementation, model, or law itself be wrong? | [Challenge review](challenge-review.md) | Concrete challenges at named levels, their evidence, and repair conditions. | +| Would a plausible wrong implementation still pass the oracle? | [Mutant gap hunt](mutant-gap-hunt.md) | A semantic mutant and what its survival shows: a missing history, observation, or law, dead code, a law question, or a framework bug. | +| Do sound obligations conflict under the same conditions? | [Tension scan](tension-scan.md) | A traced collision, missing distinction, or no supported tension. | +| Should these laws or concepts be merged, split, replaced, or removed? | [Law restructuring](law-restructuring.md) | An alternative organization and a check of what it preserves or loses. | These are reasoning aids, not additional oracle conformance requirements. The [checklist](../oracle-checklist.md) owns testing requirements. Use the @@ -31,7 +31,5 @@ supported tension, and a failed simplification are useful results. A reasoning witness is not an executed test. Passing sampled histories is not proof. -Adapted from Field Lab's assumption and boundary probes, hostile failure -assay, tension statement, structural recombinator, and position-preservation -assay. The adversarial review card also permits criticism of a candidate's -standard and concepts. No Field Lab install is needed. +Adapted from Field Lab instruments. The challenge review card also permits +criticism of a candidate's standard and concepts. No Field Lab install is needed. diff --git a/packages/oracles/docs/instruments/law-restructuring.md b/packages/oracles/docs/instruments/law-restructuring.md index e93b4d0..2dfd76f 100644 --- a/packages/oracles/docs/instruments/law-restructuring.md +++ b/packages/oracles/docs/instruments/law-restructuring.md @@ -20,7 +20,7 @@ recovering the obligations they represent. alone do not establish improvement. 3. Map every original obligation to where it is preserved, narrowed, strengthened, rejected, or still unresolved in the candidate. State what insight originally - made the obligation necessary. A claim may be defeated, but needs a reason + made the obligation necessary. A claim may be overturned, but needs a reason and a design decision rather than disappearing from the mapping. 4. Derive old and proposed predictions for a witness exercising each distinction that the reorganization could erase. Compare both allowed and forbidden @@ -29,7 +29,7 @@ recovering the obligations they represent. 5. When an independent preservation check is useful and authorized, give a fresh agent the original obligation, its sources, and one candidate. Hide sibling candidates and the author's preferred outcome. Ask what insight survives, - what is defeated, and what smallest repair is needed. Without that separation, + what is overturned, and what smallest repair is needed. Without that separation, describe the result as a trace audit, not independent validation. **Return:** the proposed organization, the old-to-new obligation mapping, diff --git a/packages/oracles/docs/oracle-checklist.md b/packages/oracles/docs/oracle-checklist.md index c228ecc..7f8e893 100644 --- a/packages/oracles/docs/oracle-checklist.md +++ b/packages/oracles/docs/oracle-checklist.md @@ -166,7 +166,7 @@ No requirement here can be met by declaring something out of reach. - **Framework:** Enforced. A generated `oracleProperty` runs one test per fixed seed and one random test with the same arbitrary, check, and budget. A failure prints how to replay it. -- **Author:** Do not bypass this with a raw `fc.assert`, `fc.check`, or `.prop` +- **Author:** Do not work around this with a raw `fc.assert`, `fc.check`, or `.prop` in an oracle file. The `oracle/oracle-property-only` lint rule reports them. ### ORC-008: Stateful-model minimality @@ -309,7 +309,7 @@ rubric's ID still names the same requirement. | ID | Change | Reason | | ------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -| All | "Framework" and "Author" notes added | Say what `@tanstack/oracles` enforces, so nobody re-implements or bypasses it. | +| All | "Framework" and "Author" notes added | Say what `@tanstack/oracles` enforces, so nobody re-implements or works around it. | | All | Coverage-map owner entries replaced by the oracle's `limits` | Adopters have no coverage map; code ownership belongs to tools such as CODEOWNERS (maintainer, 2026-11-10). | | ORC-004 | Narrowing a grammar, filter, or validity predicate is a contract change | Maintainer, 2026-11-10. | | ORC-006 | "Equivalence within the tested domain" removed from mutant outcomes | There are no equivalent mutants (maintainer, 2026-11-10). | diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 2b63161..8e1f14f 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -179,7 +179,7 @@ part that says why it predicts the promised result (ORC-003). Techniques: [guide § The driver and the check](../../docs/guide.md#the-driver-and-the-check). -## 7. RED, change, GREEN, hostile mutant +## 7. RED, change, GREEN, mutant When the task changes production: @@ -190,7 +190,7 @@ When the task changes production: 2. Change production, and rerun the oracle after each attempt. It also shows when an attempt breaks a neighboring law. 3. Run the oracle again. It passes. This is GREEN. -4. Show that a hostile mutant, such as the original defect, fails the +4. Show that a mutant, such as the original defect, fails the oracle. Do not change production before step 1. A later run does not make up for a @@ -233,7 +233,8 @@ The evidence counts only as a set. One item alone proves little. - **Reach:** each declared state and how many histories entered it. - **RED:** the command and the failure on unchanged production. - **GREEN:** the command and the result after the change. -- **Mutant:** the hostile mutant and the law that rejected it. +- **Mutant:** the mutant, such as the original defect, and the law that + rejected it. - **Gaps:** the gap check's totals, and any gap reported as a framework bug. - **Limits:** the states, paths, and observations that the evidence does not cover. @@ -243,10 +244,10 @@ The evidence counts only as a set. One item alone proves little. Use a card when one of these uncertainties is real. There is no required sequence. -| Uncertainty | Card | -| ------------------------------------------------------------------ | ------------------------------------------------------------------ | -| The promise is unclear, or two readings predict different behavior | [Law discovery](../../docs/instruments/law-discovery.md) | -| Production and the model may share a mistake | [Adversarial review](../../docs/instruments/adversarial-review.md) | -| Two obligations collide under some condition | [Tension scan](../../docs/instruments/tension-scan.md) | -| Laws overlap, or their concepts seem wrong | [Law restructuring](../../docs/instruments/law-restructuring.md) | -| The oracle is green but may accept a wrong implementation | [Mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) | +| Uncertainty | Card | +| ------------------------------------------------------------------ | ---------------------------------------------------------------- | +| The promise is unclear, or two readings predict different behavior | [Law discovery](../../docs/instruments/law-discovery.md) | +| Production and the model may share a mistake | [Challenge review](../../docs/instruments/challenge-review.md) | +| Two obligations collide under some condition | [Tension scan](../../docs/instruments/tension-scan.md) | +| Laws overlap, or their concepts seem wrong | [Law restructuring](../../docs/instruments/law-restructuring.md) | +| The oracle is green but may accept a wrong implementation | [Mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) | diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index 6c1d7df..e63c5c9 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -49,7 +49,7 @@ contains one is not ready. | **Pinned only** | A fixed example where the law needs a generator dimension. | | **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | | **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | -| **No mutant** | A claim that a defect is closed, without a hostile mutant that fails the oracle. | +| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | | **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | The evidence counts only as a set: law, reach, RED, GREEN, mutant, gaps, @@ -95,18 +95,18 @@ hunt is your main job. is not an exit. - **An invariant guard without a witness.** A throw for a contradictory collaborator signal or an impossible state has no history that reaches it. - Require a fault-injected step that supplies the contradictory signal. + Require a history step that simulates the contradictory signal. Expect production to fail at the boundary, before it releases established state or publishes success. -## 4. Choose attacks for the real uncertainty +## 4. Choose challenges for the real uncertainty Use the [instrument cards](../../docs/instruments/) when they fit. Do not run every card, and do not require a finding at every level. -- **[Adversarial review](../../docs/instruments/adversarial-review.md):** - attacks enforcement, the law's statement, its suitability, or its - concepts. Name the level each attack addresses. +- **[Challenge review](../../docs/instruments/challenge-review.md):** + challenges enforcement, the law's statement, its suitability, or its + concepts. Name the level each challenge addresses. - **[Law discovery](../../docs/instruments/law-discovery.md):** for a law with no clear authority, or two readings that predict different behavior. - **[Tension scan](../../docs/instruments/tension-scan.md):** for demands that From db3eb66f4990f0b232a47a4520cb62029e93f88b Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 14:31:56 -0700 Subject: [PATCH 05/76] docs(oracles): restore the design-grammar step and add its card Design step 2 now asks the agent to name the model's parts and rules, map each rule to a sentence of the law, rebuild the known examples before trusting the model, and record rule conflicts as open questions. docs/instruments/design-grammar.md carries the heavier method for a hard law or a growing model: a stated preservation relation, parts admitted only for a job, overlaps, reconstruction and simplification by a different representation, a two-way comparison for every simplification with refactor, refine, and revise kept apart, and an optional TLA+ check with a weakened-configuration control. The design and review skills and the card index route to it. Co-authored-by: Isaac --- .changeset/ship-oracle-skills.md | 2 +- .../docs/instruments/design-grammar.md | 129 ++++++++++++++++++ packages/oracles/docs/instruments/index.md | 15 +- .../oracles/skills/oracles-design/SKILL.md | 32 ++++- .../oracles/skills/oracles-review/SKILL.md | 3 + 5 files changed, 166 insertions(+), 15 deletions(-) create mode 100644 packages/oracles/docs/instruments/design-grammar.md diff --git a/.changeset/ship-oracle-skills.md b/.changeset/ship-oracle-skills.md index 06372e5..53558ce 100644 --- a/.changeset/ship-oracle-skills.md +++ b/.changeset/ship-oracle-skills.md @@ -3,4 +3,4 @@ '@tanstack/eslint-plugin-oracles': patch --- -Ship agent skills and docs inside `@tanstack/oracles`. The package now includes `skills/` with `oracles-setup`, `oracles-design`, and `oracles-review`. It also includes `docs/`: the oracle checklist (version 1, ORC-001 to ORC-016), a glossary, a techniques guide, and five instrument cards. A coding agent reads them from `node_modules`. `@tanstack/eslint-plugin-oracles` gains a README that lists its configs and rules. +Ship agent skills and docs inside `@tanstack/oracles`. The package now includes `skills/` with `oracles-setup`, `oracles-design`, and `oracles-review`. It also includes `docs/`: the oracle checklist (version 1, ORC-001 to ORC-016), a glossary, a techniques guide, model-style examples, and six instrument cards. A coding agent reads them from `node_modules`. `@tanstack/eslint-plugin-oracles` gains a README that lists its configs and rules. diff --git a/packages/oracles/docs/instruments/design-grammar.md b/packages/oracles/docs/instruments/design-grammar.md new file mode 100644 index 0000000..185f2f1 --- /dev/null +++ b/packages/oracles/docs/instruments/design-grammar.md @@ -0,0 +1,129 @@ +# Design grammar + +**Use when:** the law has many interacting rules, the model keeps growing, a +simplification is proposed, or the law makes claims about state, concurrency, +retries, or ownership. For an ordinary oracle, design step 2 is enough. + +**Inputs:** + +- the law sheet and the frozen list from design step 2 +- the owned production code +- the known examples: docs, issues, and existing tests +- one nearby invalid history that the oracle must reject + +Keep the invalid history and the examples fixed before you simplify anything. +A case that shaped the design is a regression case afterwards, not evidence +of range. + +## Operation + +1. **State the preservation relation.** Say which one the work keeps: + - **Reconstruction:** the model accounts for the documented behavior. + - **Named laws:** the model keeps each adopted law. Behavior that no law + names may change. + - **Same observations:** under fixed assumptions, the old and new models + allow the same observations at the same checkpoints. + + Name the checkpoints, the assumptions, and the freedoms the contract leaves + open. The user's review of the law sheet fixes the intended contract. It is + not evidence that a model meets it. + +2. **Separate what you found from what you added.** Each rule cites its + authority, or you mark it as added. Production output is evidence, never + authority. A rule that only reproduces production's current behavior can + freeze a defect into the law. + +3. **Admit a part only for a job.** A part is an action, a state field, or an + observation. Admit it only when it rebuilds a sourced behavior, decides + legality, or keeps a named law. A renamed production component is not a + part. + +4. **Record rules and conflicts.** Write each rule as one sentence of the law. + When two rules conflict, keep both and mark the conflict as an open question + for the user. Record a priority only when the authority gives one. + +5. **Check overlaps.** Where two parts share state or timing, ask whether the + interaction itself carries a law, such as a cleanup that races a late + completion. Keep the function of that interaction. Its original shape can + change. + +6. **Reconstruct, then simplify.** + - Rebuild every known example with the histories and the model. Revise the + model if it cannot. + - Remove each part and each rule in turn. If nothing breaks (no example, no + law, no rejected invalid history), remove it or demote it to a comment. + - Then try a different representation. Removing one part at a time does not + find a joint replacement. Examples: + - Three disjoint sets of ids become one map from id to status. + - A stored value becomes a derived one. + - Two states with the same futures merge. + - A case table becomes one general rule. + - Count the complexity that moves into helpers, adapters, or mappings. + Moving it does not remove it. + + Call the result a reduced model, not a minimal one. Local removal shows only + that no tried simplification succeeded. + +7. **Compare every simplification in both directions.** Under the same + assumptions and checkpoints, find: + - histories the new model allows and the old one rejects (a weakening) + - histories the new model rejects and the old one allows (a strengthening) + + Report one of: weakening, strengthening, both, or no difference found within + the stated scope. Name the scope, such as "sampled, 2,000 histories" or + "every history up to 6 steps". No difference found in a sample is not + equivalence. A mapping between the two models is part of what a reviewer + checks: it must not hide an observation to make the models agree. + + Keep three kinds of change apart: + - **Refactor:** the allowed behavior stays the same. + - **Refine:** the model adds a guarantee or narrows what it allows. + - **Revise:** the contract itself changes. That is a law question for the + user. + + Do not change the law, a history filter, and an observation in one step. + Each needs its own comparison. + +8. **Optionally check a formal model.** For a consequential claim about state, + concurrency, retries, or ownership, translate the reduced model into TLA+ + or a similar model checker. + - Add state only where it changes allowed future behavior. Write derived + values as definitions. + - Classify each obligation: a state invariant, a property of transitions or + histories, or a progress property. Progress needs a stated fairness + assumption. Do not add fairness only to make a check pass. + - Mark a property that the transition relation enforces by construction. It + shows a choice in the model, not a fact about production. + - Run a passing configuration and a deliberately weakened one that must + produce the expected counterexample. Without the second run, the check + has not shown that it can find the failure. + - Report the exact mode: an exhaustive finite model, a depth-limited run, + simulation, or an incomplete run. Record the constraints and overrides. + + A passing model check does not show that the model matches production. The + oracle's histories and checks do that. + +**Return:** + +- the preservation relation, with its checkpoints, assumptions, and freedoms +- the parts, each with its job and a witness +- the rules with their sources, and the open conflicts +- the reduced model, and the comparison for each simplification +- formal-check results with their mode, when run +- the limits: what the work does not establish + +**Control:** keep the known examples and the invalid history fixed while you +simplify. A simplification that rejects an allowed outcome narrows the contract. + +**Distortion:** + +- forcing elegance: an awkward rule may be earned, so keep it with its witness +- calling a reduced model minimal +- calling two models equivalent because a sample found no difference +- writing production's current behavior into the law +- a mapping that hides an observation + +**Stop when:** every part has a job and a witness, every simplification has +its comparison, and each open conflict is with the user. + +Adapted from Field Lab's design grammar instrument. diff --git a/packages/oracles/docs/instruments/index.md b/packages/oracles/docs/instruments/index.md index da1ce96..7f6a141 100644 --- a/packages/oracles/docs/instruments/index.md +++ b/packages/oracles/docs/instruments/index.md @@ -5,13 +5,14 @@ review insufficient. Read only the relevant card. Choose and use it within the user's authorized task; these cards do not require a separate approval for each reasoning step or a fixed workflow. -| Uncertainty | Instrument | Result | -| --------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -| What behavior should this subsystem promise? | [Law discovery](law-discovery.md) | A sourced candidate law, competing predictions, and a distinguishing witness. | -| Could the implementation, model, or law itself be wrong? | [Challenge review](challenge-review.md) | Concrete challenges at named levels, their evidence, and repair conditions. | -| Would a plausible wrong implementation still pass the oracle? | [Mutant gap hunt](mutant-gap-hunt.md) | A semantic mutant and what its survival shows: a missing history, observation, or law, dead code, a law question, or a framework bug. | -| Do sound obligations conflict under the same conditions? | [Tension scan](tension-scan.md) | A traced collision, missing distinction, or no supported tension. | -| Should these laws or concepts be merged, split, replaced, or removed? | [Law restructuring](law-restructuring.md) | An alternative organization and a check of what it preserves or loses. | +| Uncertainty | Instrument | Result | +| ----------------------------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| What behavior should this subsystem promise? | [Law discovery](law-discovery.md) | A sourced candidate law, competing predictions, and a distinguishing witness. | +| Could the implementation, model, or law itself be wrong? | [Challenge review](challenge-review.md) | Concrete challenges at named levels, their evidence, and repair conditions. | +| Would a plausible wrong implementation still pass the oracle? | [Mutant gap hunt](mutant-gap-hunt.md) | A semantic mutant and what its survival shows: a missing history, observation, or law, dead code, a law question, or a framework bug. | +| Do sound obligations conflict under the same conditions? | [Tension scan](tension-scan.md) | A traced collision, missing distinction, or no supported tension. | +| Should these laws or concepts be merged, split, replaced, or removed? | [Law restructuring](law-restructuring.md) | An alternative organization and a check of what it preserves or loses. | +| Is this model the smallest one that keeps the law, and what does a simplification change? | [Design grammar](design-grammar.md) | Parts with jobs and witnesses, open rule conflicts, a reduced model, and a two-way comparison for each simplification. | These are reasoning aids, not additional oracle conformance requirements. The [checklist](../oracle-checklist.md) owns testing requirements. Use the diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 8e1f14f..25d4bd7 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -56,6 +56,21 @@ Write these down. They stay fixed while you build the oracle. Do not weaken a check, history, or law in the same step that clears a failure. Make each weakening its own decision, with its own reason. +**Name the parts, then reconstruct.** Before you trust a model: + +- List its actions, its state, and its observations. Give each one its job: + the law sentence it serves, or the legality it decides. +- Write each model rule as one sentence of the law. Mark any rule that you + added rather than found in the authority. +- Rebuild the known examples (docs, issues, existing tests) with the + histories and the model. A known example that the model cannot reproduce + shows a missing part or a wrong rule. +- When two rules conflict, record the conflict as an open question for the + user. Do not pick a winner. + +For a hard law or a model that keeps growing, use the +[design grammar](../../docs/instruments/design-grammar.md) card. + ## 3. Choose the model Pick by the shape of the law, in this order, and use the first style that @@ -128,6 +143,8 @@ If you cannot find a witness, look for a simpler representation before adding more machinery. For example, three disjoint sets of ids that move between each other may be one map from id to status. Count complexity that you move into helpers, adapters, or mappings. Moving it does not remove it. +The [design grammar](../../docs/instruments/design-grammar.md) card shows how +to compare a simplification with the model it replaces. **Laws stay simple; history generators may be elaborate.** Reaching a hard state can need a sophisticated generator. Put that work in the histories, not @@ -244,10 +261,11 @@ The evidence counts only as a set. One item alone proves little. Use a card when one of these uncertainties is real. There is no required sequence. -| Uncertainty | Card | -| ------------------------------------------------------------------ | ---------------------------------------------------------------- | -| The promise is unclear, or two readings predict different behavior | [Law discovery](../../docs/instruments/law-discovery.md) | -| Production and the model may share a mistake | [Challenge review](../../docs/instruments/challenge-review.md) | -| Two obligations collide under some condition | [Tension scan](../../docs/instruments/tension-scan.md) | -| Laws overlap, or their concepts seem wrong | [Law restructuring](../../docs/instruments/law-restructuring.md) | -| The oracle is green but may accept a wrong implementation | [Mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) | +| Uncertainty | Card | +| -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | +| The promise is unclear, or two readings predict different behavior | [Law discovery](../../docs/instruments/law-discovery.md) | +| Production and the model may share a mistake | [Challenge review](../../docs/instruments/challenge-review.md) | +| Two obligations collide under some condition | [Tension scan](../../docs/instruments/tension-scan.md) | +| Laws overlap, or their concepts seem wrong | [Law restructuring](../../docs/instruments/law-restructuring.md) | +| The model keeps growing, a simplification is proposed, or the law has many interacting rules | [Design grammar](../../docs/instruments/design-grammar.md) | +| The oracle is green but may accept a wrong implementation | [Mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) | diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index e63c5c9..ee3c4fa 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -113,6 +113,9 @@ every card, and do not require a finding at every level. collide under a concrete condition. - **[Law restructuring](../../docs/instruments/law-restructuring.md):** for whether merging, splitting, or replacing laws gives a better account. +- **[Design grammar](../../docs/instruments/design-grammar.md):** for a + simplification or a growing model: what each part does, and what the change + newly allows or rejects. - **[Mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md):** for whether a plausible wrong implementation can pass. From 15f03f67e84a866d5bd4b80d0bc4ca7bfd5fdd4a Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 14:46:38 -0700 Subject: [PATCH 06/76] docs(oracles): add oracles-evaluate-review and tighten the design loop - New skill oracles-evaluate-review: a lossless ledger of review findings, enforcement gaps fixed and law changes brought to the user, one oracle plan for the whole review run against unchanged production, each bug turned into a family of histories with a nearby passing case, and a loss audit. - oracles-design: the approved law sheet becomes the law block at the top of the oracle file; a run-budget heuristic (critical states by construction and probes, starting runs, fast files, longer sweeps by the run multiplier, a lowered budget is a weakening); lint after each edit; a production cleanup step after GREEN; the evidence block goes in the pull request description. - oracles-review: findings for a lowered budget and for inserted rather than refactored code; a law block that disagrees with the law option. - oracles-setup: the AGENTS.md block names the new skill and asks for lint after each edit; pointer skills include the new skill. - Skill descriptions and a few sentences follow the STE structural rules. Co-authored-by: Isaac --- .changeset/ship-oracle-skills.md | 2 +- packages/oracles/README.md | 23 +-- .../oracles/skills/oracles-design/SKILL.md | 53 +++++- .../skills/oracles-evaluate-review/SKILL.md | 177 ++++++++++++++++++ .../oracles/skills/oracles-review/SKILL.md | 9 +- .../oracles/skills/oracles-setup/SKILL.md | 12 +- 6 files changed, 255 insertions(+), 21 deletions(-) create mode 100644 packages/oracles/skills/oracles-evaluate-review/SKILL.md diff --git a/.changeset/ship-oracle-skills.md b/.changeset/ship-oracle-skills.md index 53558ce..8a7904b 100644 --- a/.changeset/ship-oracle-skills.md +++ b/.changeset/ship-oracle-skills.md @@ -3,4 +3,4 @@ '@tanstack/eslint-plugin-oracles': patch --- -Ship agent skills and docs inside `@tanstack/oracles`. The package now includes `skills/` with `oracles-setup`, `oracles-design`, and `oracles-review`. It also includes `docs/`: the oracle checklist (version 1, ORC-001 to ORC-016), a glossary, a techniques guide, model-style examples, and six instrument cards. A coding agent reads them from `node_modules`. `@tanstack/eslint-plugin-oracles` gains a README that lists its configs and rules. +Ship agent skills and docs inside `@tanstack/oracles`. The package now includes `skills/` with `oracles-setup`, `oracles-design`, `oracles-review`, and `oracles-evaluate-review`. It also includes `docs/`: the oracle checklist (version 1, ORC-001 to ORC-016), a glossary, a techniques guide, model-style examples, and six instrument cards. A coding agent reads them from `node_modules`. `@tanstack/eslint-plugin-oracles` gains a README that lists its configs and rules. diff --git a/packages/oracles/README.md b/packages/oracles/README.md index c68ea8e..6b8c25a 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -26,17 +26,18 @@ coding agents, and walks through a first oracle and gap check. ## What ships in this package -| Path | What it is | -| -------------------------- | ----------------------------------------------------------------------------------------------- | -| `skills/oracles-setup/` | Setup, upgrade, and making the skills findable. | -| `skills/oracles-design/` | How to design, write, or change an oracle: law sheet first, model, histories, RED, GREEN, gaps. | -| `skills/oracles-review/` | How to review oracle work. | -| `docs/oracle-checklist.md` | The complete requirements, by ID (ORC-001 to ORC-016). | -| `docs/guide.md` | Techniques for each design and review step. | -| `docs/model-styles.md` | One small example of each model style. | -| `docs/glossary.md` | The oracle terms. | -| `docs/instruments/` | Short reasoning cards that the skills route to. | -| `oracle-gaps/` | The gap check (`oracle-gaps` binary). | +| Path | What it is | +| --------------------------------- | -------------------------------------------------------------------------------------------------- | +| `skills/oracles-setup/` | Setup, upgrade, and making the skills findable. | +| `skills/oracles-design/` | How to design, write, or change an oracle: law sheet first, model, histories, RED, GREEN, gaps. | +| `skills/oracles-evaluate-review/` | How to act on a review: a lossless ledger, RED on unchanged production, and closing the bug class. | +| `skills/oracles-review/` | How to review oracle work. | +| `docs/oracle-checklist.md` | The complete requirements, by ID (ORC-001 to ORC-016). | +| `docs/guide.md` | Techniques for each design and review step. | +| `docs/model-styles.md` | One small example of each model style. | +| `docs/glossary.md` | The oracle terms. | +| `docs/instruments/` | Short reasoning cards that the skills route to. | +| `oracle-gaps/` | The gap check (`oracle-gaps` binary). | ## An oracle diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 25d4bd7..efa945c 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -1,6 +1,6 @@ --- name: oracles-design -description: Design, write, or change an oracle with @tanstack/oracles - write a law sheet first, choose a model, build histories that reach the states the law needs, show RED on unchanged production, change production, and close every gap. Use for a new oracle, an oracle repair, any production change in code that an oracle owns, and any change to a model, history grammar, driver, or check. +description: Design, write, or change an oracle with @tanstack/oracles. Write a law sheet first, choose a model, and build histories that reach the states the law needs. Show RED on unchanged production, change production, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. --- # Design an oracle @@ -43,6 +43,12 @@ Before any code, write one row per law and give the sheet to the user. **Stop here.** Do not write the oracle until the user has reviewed the sheet. +Once the user approves it, the sheet becomes the **law block**: the opening +comment of the oracle file, beside the model that enacts it. Each row's law, +authority, and limits also go in that property's `law` option. Keep the +block, the `law` option, and the model in step: a change to one updates the +others in the same change (ORC-003). + ## 2. Freeze what the oracle must preserve Write these down. They stay fixed while you build the oracle. @@ -117,7 +123,7 @@ small example of each style. | ------------------------------------------------------------------------------- | ---------------------------------- | | B1: expected values captured from production (`toMatchSnapshot` and similar) | review (lint pending) | | B2: a raw campaign (`fc.assert`, `fc.check`, `.prop`) in an oracle file | lint `oracle/oracle-property-only` | -| B3: `fc.pre` in an oracle file; legality belongs in the grammar or `resolve` | review (lint pending) | +| B3: `fc.pre` in an oracle file. Legality belongs in the grammar or `resolve`. | review (lint pending) | | B4: a model that imports runtime code from production | review (lint pending) | | B6: a check whose only assertions are "does not throw", "defined", or "settles" | review (warning) | | B7: the same expression on both sides of a comparison | review (lint pending) | @@ -146,7 +152,7 @@ you move into helpers, adapters, or mappings. Moving it does not remove it. The [design grammar](../../docs/instruments/design-grammar.md) card shows how to compare a simplification with the model it replaces. -**Laws stay simple; history generators may be elaborate.** Reaching a hard +**Laws stay simple. History generators may be elaborate.** Reaching a hard state can need a sophisticated generator. Put that work in the histories, not in the model. @@ -169,6 +175,26 @@ in the model. - range: the bounded domains and marginal cases - exclusion: a nearby invalid history that the grammar rejects +**Choose the run budget.** Random runs explore around the critical states. +They do not reach them. + +- **Cover every critical state by construction.** Each state that a law + needs is a declared state with a constructed history, and a probe where + production decides to enter it. Never raise `runs` to reach a state. +- **Need a state at least N times?** Give it N constructed histories. +- **Start with one fixed seed** in `fixedSeeds`, beside the random campaign. +- **Start `runs` at 100** for a fast, synchronous driver, and at 25 to 50 + for a driver that awaits timers or I/O. +- **Keep it fast.** The gap check runs every owning oracle once for each + mutant, so oracle time multiplies. Aim for each oracle file to finish in a + few seconds in a normal run. Lower `runs` before you remove a dimension. +- **Read the reach counts.** If a measured state is rare, add a constructed + history or change the generator's weights. Do not only add runs. +- **Run longer sweeps with the run multiplier** (`_RUNS_MULTIPLIER`), + locally or on a schedule, not by raising `runs` in the file. +- **Lowering `runs`, removing a fixed seed, or removing a constructed history + is a weakening.** Give its reason in the change. + Techniques: [guide § Histories](../../docs/guide.md#histories). ## 6. Write the driver and the check @@ -194,6 +220,10 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). Keep the law, its authority, and its limits in `law`, and prose beside each part that says why it predicts the promised result (ORC-003). +**Lint as you go.** Run the repository's lint on each oracle file and each +changed production file after you edit it, and again before you finish. Fix +every report. Do not disable a rule. + Techniques: [guide § The driver and the check](../../docs/guide.md#the-driver-and-the-check). ## 7. RED, change, GREEN, mutant @@ -213,6 +243,17 @@ When the task changes production: Do not change production before step 1. A later run does not make up for a skipped RED. +**Clean up production once GREEN.** The oracle now protects the law, so use +it to simplify: + +- Refactor the code around the change. Do not insert a new branch beside an + old one that does almost the same thing. +- Remove states, flags, branches, and recovery paths that no law needs. The + gap check shows which code no history can observe. +- Prefer a change that shrinks the code, or leaves its size the same. If the + repository sets a size rule, such as a net-negative diff, follow it. +- Rerun the oracle and the gap check after each cleanup. + Techniques: [guide § Evidence](../../docs/guide.md#evidence). ## 8. Close every gap @@ -256,6 +297,12 @@ The evidence counts only as a set. One item alone proves little. - **Limits:** the states, paths, and observations that the evidence does not cover. +Put the evidence block in the pull request description. When a change claims +that an oracle is new, repaired, or fully audited (ORC-012), also keep the +evidence in a versioned record tied to the reviewed commit. + +When review findings arrive, load `oracles-evaluate-review`. + ## Instruments Use a card when one of these uncertainties is real. There is no required diff --git a/packages/oracles/skills/oracles-evaluate-review/SKILL.md b/packages/oracles/skills/oracles-evaluate-review/SKILL.md new file mode 100644 index 0000000..1069135 --- /dev/null +++ b/packages/oracles/skills/oracles-evaluate-review/SKILL.md @@ -0,0 +1,177 @@ +--- +name: oracles-evaluate-review +description: Evaluate a review of oracle work done with @tanstack/oracles. Keep a lossless ledger of every finding, and check each claim against the code. Extend the oracle so it shows each confirmed bug on unchanged production, then fix production and close the bug class. Use when review findings arrive for a branch that touches an oracle or owned code. +--- + +# Evaluate a review + +Use a review to close the class of bug it reveals, not only the reported +case. A passing reproduction or a complete ledger is a diagnosis. Closure +needs an oracle that rejects the original behavior and its neighbors. + +Read this skill in full. Before you edit any oracle, load `oracles-design`. +Use the [glossary](../../docs/glossary.md) terms in the ledger and the report. + +## 1. Write the ledger first + +Before you test or edit anything, write an append-only ledger of the raw +review in a task-local scratch file. Do not commit it unless the user asks. + +- Give every item a stable ID in source order: each finding, footnote, capped + item, non-blocking comment, and testing idea. +- Keep the original claim and the proposed fix. Do not merge overlapping + items. Cross-reference them. +- Record the file and line, the severity, and the commit you check against. + +For every item, keep three judgments apart: + +1. **Is it true** on the reviewed or the current commit? +2. **What does this change do about it:** fix now, already fixed, ask for a + decision, or no action? +3. **What lasting value does it have:** a test idea, a law, a cleanup, or a + later investigation? + +Use only these dispositions: `fixed-now`, `confirmed-open`, `stale` (with the +surviving idea recorded), `refuted` (with evidence), `accepted-design` (only +for behavior an adopted law or decision covers, cited), `design-decision` +(with the exact question), and `duplicate` (linked, with its unique details +kept). There is no `skip`. + +Deferral is rare. A confirmed, in-scope defect stays `confirmed-open` until it +is fixed, unless the user agrees to postpone it. A deferral records the open +work, why it cannot be done now, the remaining risk, an owner, and the +condition for resuming. + +## 2. Sort the findings + +For each confirmed behavioral finding, decide which kind it is. + +- **An enforcement gap: fix it, do not ask.** The adopted law is right, but + the oracle does not enforce it or production breaks it. The cause is a + missing generator dimension, a missing history or timing, a missing + observation, an unreached path, a driver seam, a check that passes when it + should fail, or a production defect against an adopted law. +- **A law change: present it and wait.** The finding questions what is + promised: a new law, a narrower or revised law, a new checkpoint, an + obligation on a collaborator, two laws in conflict, or a rule with no + authority. Give the user the candidate law and its rival, the evidence, + what each choice changes for users, and your recommendation. + +A finding can contain both. Fix the enforcement part, and present the law +question. + +## 3. Plan the oracle changes for the whole review + +Before you work on single findings, look at all of them together. Findings +that look separate often share one missing history dimension, schedule, +driver path, or observation. + +1. Group the findings by the state, boundary, history dimension, or + observation they share. +2. For each finding, name the oracle change that would have generated it, + and why the current oracle missed it. +3. Choose the smallest set of oracle changes that generates every finding. + Prefer one dimension that reaches several findings and their neighbors + over one pinned example per finding. +4. Make those changes first. Run the extended oracle on **unchanged** + production. Record which findings it reproduces (RED), which it misses, and + any new counterexamples. A miss is a gap in the plan: revise the plan. +5. Then fix production, with the extended oracle as feedback on all findings + at once. + +## 4. Turn each bug into a family + +A reviewer's bug shows one history. The oracle must cover the family it +belongs to. + +- **State the causal condition:** the relationship that makes the history + fail, such as "a completion outlives the operation that started it". +- **Add it to the grammar:** a generator dimension or a constructed history + that varies the incidental details (counts, order, values, timing) and + keeps the condition. A pinned example alone is not the fix. Keep the + original history as a `fixed` regression. +- **Search the neighbors:** run the extended oracle and look for related + failures that the reviewer did not report. Record each one as a separate + ledger item. +- **Keep a nearby passing case:** a history close to the bug that must still + pass, such as the same operations in a legal order. It stops the repair from + rejecting valid behavior. + +## 5. Use the right evidence + +Do not accept or reject a behavioral claim from reading code alone. + +| Claim | Evidence | +| --------------------------------------- | --------------------------------------------------------------------- | +| Wrong product behavior | A failing oracle history or focused probe on the same path. | +| A check that passes when it should fail | A wrong implementation or corrupted trace that the check accepts. | +| Missing generator coverage | Reach counts or a sample showing that the regime never occurs. | +| Async or lifecycle | Controlled promises, events, or a controlled clock, not timing alone. | +| Work or performance | Deterministic counts, not elapsed time. | +| Docs or maintainability | Direct source evidence and a focused check. | + +If the current code passes, check whether the review targeted an older +commit, missed a precondition, or found a bug that a later change turned into +another one. Mark a claim `refuted` only when a probe reaches the same path +and every stated precondition and disproves it. Otherwise ask the reviewer for +the smallest failing history. + +For each confirmed bug: + +1. Show RED: the extended oracle fails on unchanged production. +2. Make the simplest production change that enforces the law across its + histories. +3. Show GREEN: the same oracle and its neighbors pass. +4. Show that a mutant, such as the original defect, fails the oracle. +5. Run the gap check on the owned code, and close every gap + (`oracles-design` step 8). + +Mark the item `fixed-now` only when the repair and the oracle coverage are +both complete. If only the reported history is fixed, keep the class open. + +## 6. Account for the whole law + +For each law the review touched, answer: + +- Is it fully in the model and in reachable histories? +- Is it checked at every checkpoint the contract owes? +- Which mutant shows that the check is sensitive? +- Which neighboring laws does the repair keep? + +A missing link is open work. A neighboring green test does not supply it. + +If the branch moves during the evaluation, mark earlier evidence stale and +rerun the affected items on the new commit. + +## 7. Audit for losses + +Before the final report: + +1. Return to the raw review, not a summary. +2. Account for every item, including footnotes and items you dismissed. +3. Compare each item with the final diff. +4. Recheck every law and sibling failure against the final model, histories, + driver, checkpoints, and calibration. +5. Check the totals: raw items = fixed-now + confirmed-open + stale + + refuted + accepted-design + design-decision + duplicate (+ any agreed + deferral). + +## 8. Report + +Lead with the law changes that need a decision. Then: + +- **Laws:** for each law, what the oracle proves, the decisive experiment + (RED, the neighbor, the mutant), why the suite missed it, and what remains + open. +- **The ledger:** every item, with its verdict, action, and lasting value. +- **The reviewer:** accuracy, depth, signal-to-noise, and fix quality, with a + hire recommendation and evidence. +- **Changes:** what you changed and the exact verification. Keep RED, a + mutant's result, and the repaired branch's GREEN separate. +- **Losses:** the totals by disposition, and every open evidence or coverage + gap. + +If the user asks whether the pull request is ready, give a separate verdict: +`ready`, `not ready` (a confirmed in-scope defect or an open decision blocks +it), `readiness unproven` (a decisive witness is missing), or `not assessed`. +A complete ledger and a green run do not make a pull request ready. diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index ee3c4fa..d940a2f 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -1,6 +1,6 @@ --- name: oracles-review -description: Review oracle work done with @tanstack/oracles - challenge whether an oracle enforces its law, whether it is the right law, and whether the work took a shortcut. Use for reviewing a new or changed oracle, a production change in code that an oracle owns, or a proposed change to what an oracle promises. +description: Review oracle work done with @tanstack/oracles. Challenge whether an oracle enforces its law, whether it is the right law, and whether the work took a shortcut. Use for a new or changed oracle, or a production change in owned code. Also use for a proposed change to what an oracle promises. --- # Review oracle work @@ -41,12 +41,14 @@ contains one is not ready. | Shortcut | What to look for | | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Law after code** | No law sheet before the oracle was written, or a law with no authority. | +| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | | **Production first** | A production change in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | | **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | | **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. | | **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | | **Pinned only** | A fixed example where the law needs a generator dimension. | +| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | +| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | | **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | | **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | | **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | @@ -124,7 +126,7 @@ Ask these questions of every model and grammar: - **What does each distinction contribute?** For each law, model field, and branch, is there a legal history that needs it? If not, it is a candidate for removal or for a simpler representation (ORC-008). -- **Is the complexity in the right place?** Laws stay simple; generators may +- **Is the complexity in the right place?** Laws stay simple, and generators may be elaborate (`oracles-design` §4). - **Did the domain shrink?** A changed filter, validity predicate, or grammar bound removes histories from the claim. That is a contract change, and it @@ -164,3 +166,4 @@ Report the supported findings and the remaining uncertainty. Do not invent findi decisive. If the review leads to oracle edits, load `oracles-design` before you edit. +To act on the findings, load `oracles-evaluate-review`. diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 6920a63..d405d3d 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -1,6 +1,6 @@ --- name: oracles-setup -description: Set up @tanstack/oracles in a repository - install the pinned packages, add the lint preset, write the AGENTS.md block, make the oracle skills findable by coding agents, and run a first oracle through the gap check. Use when a repository adopts @tanstack/oracles or upgrades its pinned commit. +description: Set up @tanstack/oracles in a repository. Install the pinned packages, add the lint preset, and write the AGENTS.md block. Make the oracle skills findable by coding agents, and run a first oracle through the gap check. Use when a repository adopts @tanstack/oracles or upgrades its pinned commit. --- # Set up @tanstack/oracles @@ -20,7 +20,7 @@ Find out, and tell the user: workspace. For a workspace, ask which package to start in. - The test runner. The framework runs on Vitest 4 or later, with fast-check 4. If the package uses another runner, oracles run in a separate Vitest config - beside it; do not change the existing suite. + beside it. Do not change the existing suite. - Where production source lives (for example `src/`), and whether ESLint parses TypeScript already. - Which coding agents the user works with. Look for `.claude/`, `.agents/`, @@ -101,6 +101,11 @@ This repository tests its laws with `@tanstack/oracles`. in full and follow it. - Before you review oracle work, read `node_modules/@tanstack/oracles/skills/oracles-review/SKILL.md` in full. +- Before you act on review findings, read + `node_modules/@tanstack/oracles/skills/oracles-evaluate-review/SKILL.md` in + full. +- Lint each oracle file and each changed production file after you edit it. + Fix every report. Do not disable a rule. - Oracle files: ``. Gap check: ``. @@ -121,7 +126,8 @@ pointer. Offer the user these choices. They can pick more than one. | **Global pointers** | A pointer skill per skill in `~/.agents/skills/` and `~/.claude/skills/` (and any other skills directory the user's agents read) | The user wants the skills in every repository that uses `@tanstack/oracles`. | Write a pointer skill as `//SKILL.md` for each of -`oracles-design`, `oracles-review`, and `oracles-setup`. +`oracles-design`, `oracles-review`, `oracles-evaluate-review`, and +`oracles-setup`. - Copy `name` and `description` exactly from the package's `SKILL.md`, so the agent selects it for the same requests. From f5011b0af19a23e547de232dc5f96a5057699b95 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 15:00:16 -0700 Subject: [PATCH 07/76] docs(oracles): follow #6 in setup, and require a fixed seed #6 made `recommended` turn off inline directives and widened the `production` glob to `src/**`, so setup now adds both presets as they are, without the workaround blocks. The plugin README lists no-oracle-disable and the new glob. The design skill now asks for at least one fixed seed (an empty list will be refused) and treats a random-only kill as a gap until its history is kept. Co-authored-by: Isaac --- packages/eslint-plugin-oracles/README.md | 39 ++++++------------- .../oracles/skills/oracles-design/SKILL.md | 4 +- .../oracles/skills/oracles-setup/SKILL.md | 21 ++++------ 3 files changed, 22 insertions(+), 42 deletions(-) diff --git a/packages/eslint-plugin-oracles/README.md b/packages/eslint-plugin-oracles/README.md index cc83f3d..94ee2af 100644 --- a/packages/eslint-plugin-oracles/README.md +++ b/packages/eslint-plugin-oracles/README.md @@ -10,33 +10,17 @@ Install it at the same pkg.pr.new commit as `@tanstack/oracles`. The ## Configs -| Config | Files | Rules | -| --------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -| `configs.recommended` | Files whose names contain `oracle` (`.ts`, `.tsx`, `.js`, `.jsx`, `.mjs`) | `no-disabled`, `no-unit-test`, `oracle-property-only`, `pure-model`, `replayable-seed` | -| `configs.production` | `packages/*/src/**` | `probe-form` | - -The `production` config matches only a workspace layout. For a single package -with `src/`, add the rule with your own glob: - -```js -import oracles from '@tanstack/eslint-plugin-oracles' - -export default [ - oracles.configs.recommended, - { - files: ['**/*oracle*.{ts,tsx,js,jsx,mjs}'], - linterOptions: { noInlineConfig: true }, - }, - { - files: ['src/**/*.{ts,tsx,js,jsx,mjs}'], - plugins: { oracle: oracles }, - rules: { 'oracle/probe-form': 'error' }, - }, -] -``` - -`noInlineConfig` stops a directive comment from switching a rule off in an -oracle file. Fix a report; do not disable the rule. +| Config | Files | Rules | +| --------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | +| `configs.recommended` | Files whose names contain `oracle` (`.ts`, `.tsx`, `.js`, `.jsx`, `.mjs`) | `no-disabled`, `no-oracle-disable`, `no-unit-test`, `oracle-property-only`, `pure-model`, `replayable-seed` | +| `configs.production` | `src/**` and `packages/*/src/**` | `probe-form` | + +The `recommended` config also sets `linterOptions.noInlineConfig` for oracle +files, so a directive comment cannot switch a rule off there. Fix a report; +do not disable the rule. + +The `production` config covers `src/**` and `packages/*/src/**`. For source +elsewhere, add `oracle/probe-form` with your own glob. ## Rules @@ -44,6 +28,7 @@ oracle file. Fix a report; do not disable the rule. | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `oracle/no-disabled` | An oracle test that stops running: `.only`, a skipped body, a switch-off condition, or one run with no fixed history. | | `oracle/no-unit-test` | A test in an oracle file that reaches no declared check or model: an ordinary unit test. | +| `oracle/no-oracle-disable` | A directive comment that would silence an oracle rule. Oracle rules cannot be disabled inline. | | `oracle/oracle-property-only` | A raw campaign (`fc.assert`, `fc.check`, or a `.prop` registration) or a legacy campaign API in an oracle file. Only `oracleProperty` declares a generated-history law. | | `oracle/pure-model` | A model, check, or reach classifier that reads a clock, ambient randomness, or a timer. | | `oracle/probe-form` | A state probe in production code that is not in the one form the production build strips. | diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index efa945c..c33d6c7 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -182,7 +182,9 @@ They do not reach them. needs is a declared state with a constructed history, and a probe where production decides to enter it. Never raise `runs` to reach a state. - **Need a state at least N times?** Give it N constructed histories. -- **Start with one fixed seed** in `fixedSeeds`, beside the random campaign. +- **Give at least one fixed seed** in `fixedSeeds`. The random campaign always + runs beside it. A kill that only the random campaign finds is a gap until you + keep its history as a fixed or constructed history. - **Start `runs` at 100** for a fast, synchronous driver, and at 25 to 50 for a driver that awaits timers or I/O. - **Keep it fast.** The gap check runs every owning oracle once for each diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index d405d3d..a7dee73 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -53,7 +53,7 @@ the new commit at once, then do step 4 again and rerun the gap check (step 7). accept every property id. Call `configureOracles` only to change that, in a Vitest setup file that runs before any oracle module is imported. -**Lint.** Add the preset to the ESLint flat config: +**Lint.** Add both presets to the ESLint flat config: ```js import oracles from '@tanstack/eslint-plugin-oracles' @@ -61,22 +61,15 @@ import oracles from '@tanstack/eslint-plugin-oracles' export default [ // ...the repository's existing config oracles.configs.recommended, - { - // Oracle files: the rules must not be switched off inline. - files: ['**/*oracle*.{ts,tsx,js,jsx,mjs}'], - linterOptions: { noInlineConfig: true }, - }, - { - // The probe form, on this repository's production source. - files: ['src/**/*.{ts,tsx,js,jsx,mjs}'], - plugins: { oracle: oracles }, - rules: { 'oracle/probe-form': 'error' }, - }, + oracles.configs.production, ] ``` -- Change the `src/**` glob to the production source you found in step 1. The - preset's own `production` config matches only `packages/*/src/**`. +- `recommended` applies to oracle files and turns off inline directives there, + so a disable comment has no effect. +- `production` checks the probe form in `src/**` and `packages/*/src/**`. If + production source lives elsewhere, add the `oracle/probe-form` rule with that + glob. - If ESLint does not already parse TypeScript, add `@typescript-eslint/parser` as a dev dependency and set it as `languageOptions.parser` for the oracle files. From 8baf6e29728d3851fd5135d4f17e753d8e67b879 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 15:33:40 -0700 Subject: [PATCH 08/76] docs(oracles): fix what two fresh-agent validation runs hit Two fresh agents set up the package in vercel/ms (Jest and Biome, no ESLint) and developit/mitt (mocha, ESLint 7), using only the packed packages and the skills. mitt reached zero gaps. ms stopped correctly on a law that nobody can decide for a library the adopter does not own. Setup now covers what they hit: - the Node version against CI, the lint setup, and the command runner - `eslint@^9` when the repository has none, a stable parser major, and TypeScript 5 or newer for the oracle files, kept out of an older `tsc` - where oracle files go, and a Vitest config beside another runner - a separate ESLint 9 run for an ESLint 7 or 8 repository, with a check that the rules are live - `owns` entries that name one function - `--source` selecting files, an explicit `--seed`, `--out` inside the repository, and `.oracle-gaps/` in `.gitignore` - dead code that the gap check finds counts as a production change The design skill teaches function-level `owns`, dead-code removal as a production change, and what a law that kills nothing means. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 14 +++- .../oracles/skills/oracles-setup/SKILL.md | 83 ++++++++++++++++--- 2 files changed, 81 insertions(+), 16 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index c33d6c7..6623bce 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -216,8 +216,11 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). - **Calibration.** Give at least one `calibration`: a wrong answer, as a corrupted trace or a mutant driver, and the law that must reject it (ORC-006). -- **Ownership.** Put the owned files or functions on the property's `owns` - list. The gap check judges only owned code. +- **Ownership.** Put the owned code on the property's `owns` list. An entry is + a file (`src/x.ts`) or one function and its static call tree + (`src/x.ts#run`, `src/x.ts#Class.method`). Own only what the laws cover. The + gap check judges only owned code, and the gap check's `--source` selects + files. Keep the law, its authority, and its limits in `law`, and prose beside each part that says why it predicts the promised result (ORC-003). @@ -267,7 +270,12 @@ comes with one instruction. Do what it says, within the rules below. history, observation, or law, or it shows dead code. - **An effect that no check sees is one that the driver does not record.** Record it. -- **Code whose removal no legal history can observe is dead.** Remove it. +- **Code whose removal no legal history can observe is dead.** Remove it in + the same change as the oracle. That is a production change, so say so in + the evidence, and rerun the oracle and the gap check. +- **A law that kills nothing** owns no mutated code, or its witness never + varies in the histories. Check its witness and its constructed history. If + the law still kills nothing, it may restate another law: compare the two. - **The only exclusions are probe-only code and the `super()` region of a derived constructor.** - **Never propose an exit.** If a gap can be neither killed nor removed, it is diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index a7dee73..38c3e1a 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -21,8 +21,14 @@ Find out, and tell the user: - The test runner. The framework runs on Vitest 4 or later, with fast-check 4. If the package uses another runner, oracles run in a separate Vitest config beside it. Do not change the existing suite. -- Where production source lives (for example `src/`), and whether ESLint - parses TypeScript already. +- Where production source lives (for example `src/`). +- The lint setup: ESLint 9 or later with a flat config, an older ESLint with + `.eslintrc`, or no ESLint (for example Biome only). +- The Node version that CI and the `engines` field use. The package needs + Node 20 or later. If CI runs an older Node, tell the user. Do not change CI. +- The repository's TypeScript version. Do not change it (step 2). +- The command runner for package binaries: `pnpm exec`, `npx`, or + `yarn`. Use it wherever this skill shows `pnpm exec`. - Which coding agents the user works with. Look for `.claude/`, `.agents/`, `.codex/`, `AGENTS.md`, and `CLAUDE.md`, and ask. @@ -39,10 +45,17 @@ pnpm add -D \ fast-check vitest ``` -The only peers to add are `fast-check` and `vitest`. Never add, upgrade, or -downgrade the repository's `typescript` for oracles: the gap check uses its -own TypeScript and never reads the project's, from commit `d8adfee` on. Pin a -commit at or after `d8adfee`. +Add `fast-check` and `vitest` beside them. For lint, also add `eslint@^9` +if the repository has no ESLint, and `@typescript-eslint/parser@^8` if +nothing parses TypeScript yet. Name those majors: a package manager can pick +a prerelease otherwise. For an older ESLint, see the fallback in step 3. + +Never add, upgrade, or downgrade the repository's `typescript` for oracles. +The gap check uses its own TypeScript and never reads the project's, from +commit `d8adfee` on. Pin a commit at or after `d8adfee`. Oracle files need +TypeScript 5 or newer. If the repository's `tsc` is older, keep the oracle +files and the Vitest config out of its run (its `include` or `exclude`). Report any typecheck errors +that existed before setup as pre-existing. Use the repository's own package manager. To upgrade, change both URLs to the new commit at once, then do step 4 again and rerun the gap check (step 7). @@ -53,6 +66,23 @@ the new commit at once, then do step 4 again and rerun the gap check (step 7). accept every property id. Call `configureOracles` only to change that, in a Vitest setup file that runs before any oracle module is imported. +**Where oracle files go.** Put oracle files in a test directory, named +`*oracle*.test.ts`, for example `tests/parse.oracle.property.test.ts`. If the +repository's main runner is not Vitest, check that its test glob does not +also match the oracle files, and run them with their own Vitest config: + +```ts +// vitest.oracle.config.ts +import { defineConfig } from 'vitest/config' + +export default defineConfig({ + test: { include: ['tests/**/*oracle*.test.ts'] }, +}) +``` + +Add a script for it, such as `test:oracles`. Leave the existing suite +unchanged. + **Lint.** Add both presets to the ESLint flat config: ```js @@ -75,7 +105,22 @@ export default [ files. - Do not add `legacyCampaigns` or any other exemption. Adopters have none. -Run the linter once and confirm that it loads the rules without an error. +**An older ESLint (7 or 8, `.eslintrc`).** Do not upgrade the repository's +lint. Lint the oracle files with a separate ESLint 9: + +1. Install the plugin with `--legacy-peer-deps` (npm), or the package + manager's equivalent. +2. Write a flat `eslint.oracle.config.mjs` with both presets and a TypeScript + parser for the oracle files. +3. Run it as `npx -y -p eslint@9 eslint -c eslint.oracle.config.mjs `, and add a `lint:oracles` script. + +The config's parser import resolves from the repository, so it uses the +repository's own `@typescript-eslint/parser`. + +Run the linter once and confirm that it loads the rules. Then check that the +rules are live: a deliberately wrong line in an oracle file (for example +`fc.assert(...)`) must produce an error. Remove that line again. ## 4. Write the AGENTS.md block @@ -156,9 +201,10 @@ Then load `oracles-design` and follow it from its first step. The law sheet comes first, and the user reviews it before you write code. Setup does not skip that step to finish sooner. -Oracle files must match the glob `*oracle*`, for example -`tests/clamp.oracle.property.test.ts`. Put the oracle's `owns` list on its -`oracleProperty` call. If the gap check counts **0 oracle files**, the +Put the oracle's `owns` list on its `oracleProperty` call. An entry is a file +(`src/index.ts`) or one function in it (`src/index.ts#parse`, or +`src/x.ts#Class.method`). A function entry owns that function and its static +call tree. For a first oracle, own only the function it covers. If the gap check counts **0 oracle files**, the oracle is not declared. That is a setup failure, even when the report shows 0 gaps and the command exits 0. @@ -170,14 +216,25 @@ there is not a kill, and the gap check reports it. Run the gap check on the owned source: ```sh -pnpm exec oracle-gaps --package --source +pnpm exec oracle-gaps --package --source \ + --seed --out /.oracle-gaps/out ``` +- `--source` selects files. `owns` can name a single function in a file, so + the report judges only the owned functions. Code in the file that no oracle + owns appears as unowned warnings. +- Pass `--seed`, so the baseline and every mutant run the same histories. + Pass `--out` inside the repository. (Both become defaults later.) +- Add `.oracle-gaps/` to `.gitignore`. + Run it after the oracle passes on production. Each gap in the report comes with one instruction. Follow that instruction, and follow `oracles-design` for what a gap means. There -are no equivalent mutants and no exits. If a gap can be neither killed nor -removed, stop work on that gap and report its key and evidence to the user. +are no equivalent mutants and no exits. If the gap check shows dead code +in an owned function, remove it: that is a production change, so tell the +user, and rerun the oracle and the gap check. If a gap can be neither killed +nor removed, stop work on that gap and report its key and evidence to the +user. Now fill in the two placeholders in the `AGENTS.md` block: the oracle file glob and the gap check command you ran. From 12ddccde55602b5fb5b7688b007ade07e8bbe7f4 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 16:39:18 -0700 Subject: [PATCH 09/76] docs(oracles): address the external review of the skills - Evidence by kind of change: a defect fix shows RED, GREEN, and the defect as a mutant; a behavior-preserving refactor shows the same oracle passing on old and new production with kill sets compared; a new oracle over correct code shows production passing and calibrations failing. Review no longer treats a refactor without RED as a shortcut. - Approval follows the semantics: strengthening an adopted law reuses it without a new sheet; a law change needs a law delta; a new law or missing authority needs the full sheet; an approval the task already gave stands. - A boundary-selection step before the law sheet, with an example. - Model state is justified by a legal continuation (one or more actions), not only the next action, in the design skill, ORC-008, the guide, and the law-discovery card. - Dead code needs a contract reason before removal; a surviving mutant only shows what the current histories did not expose. - A survivor that the adopted law already allows is reported as an allowed alternative, not sent back as a law question. - Setup: pointers and the AGENTS.md block read the chosen package's installed copy in a workspace; the TypeScript parser covers production source as well as oracle files. - The round-trip example compares tags element by element and names the two corruptions the joined comparison accepted as calibrations. Co-authored-by: Isaac --- packages/oracles/docs/guide.md | 5 +- .../oracles/docs/instruments/law-discovery.md | 2 +- .../docs/instruments/mutant-gap-hunt.md | 14 ++- packages/oracles/docs/model-styles.md | 17 ++- packages/oracles/docs/oracle-checklist.md | 24 +++-- .../oracles/skills/oracles-design/SKILL.md | 100 ++++++++++++++---- .../skills/oracles-evaluate-review/SKILL.md | 3 + .../oracles/skills/oracles-review/SKILL.md | 7 +- .../oracles/skills/oracles-setup/SKILL.md | 42 ++++++-- 9 files changed, 156 insertions(+), 58 deletions(-) diff --git a/packages/oracles/docs/guide.md b/packages/oracles/docs/guide.md index e38cb0d..52a5845 100644 --- a/packages/oracles/docs/guide.md +++ b/packages/oracles/docs/guide.md @@ -40,8 +40,9 @@ When the contract has independent laws, treat the oracle as a small graph: A failure then names the law that diverged, and a reviewer can inspect each rule alone. Plain functions and Maps are usually enough. -Do not split state that a legal next action can observe only as a whole. If two -states look equal to the proposed nodes, try one legal next action. If it can +Do not split state that a legal continuation can observe only as a whole. If +two states look equal to the proposed nodes, try a legal continuation of one +or more actions. If it can produce different promised results, add the missing edge, or keep that state in one model. Decomposition does not mean one model per assertion. Group facts that form one state machine, and keep independent policies separate. diff --git a/packages/oracles/docs/instruments/law-discovery.md b/packages/oracles/docs/instruments/law-discovery.md index d15ed8f..5ac9cc4 100644 --- a/packages/oracles/docs/instruments/law-discovery.md +++ b/packages/oracles/docs/instruments/law-discovery.md @@ -43,7 +43,7 @@ lead, not evidence. from step 1 and makes the rival and boundary predictions from steps 3–4. Retain the source facts needed to examine any information gap from step 5, even when the production path has not received them. - For any proposed merge of model states, ask whether a legal next action can + For any proposed merge of model states, ask whether a legal continuation can distinguish them. Map model abstractions to the project vocabulary. The information check asks what production can receive. The model-state test diff --git a/packages/oracles/docs/instruments/mutant-gap-hunt.md b/packages/oracles/docs/instruments/mutant-gap-hunt.md index 0ee5740..cbc26ff 100644 --- a/packages/oracles/docs/instruments/mutant-gap-hunt.md +++ b/packages/oracles/docs/instruments/mutant-gap-hunt.md @@ -3,7 +3,7 @@ **Use when:** a recent refactor, a suspicious green oracle, a survivor whose missing history is not obvious, or an open review question leaves doubt that the oracle would reject a plausible wrong behavior. One carefully chosen -mutant can answer a useful question; a campaign is optional. +mutant can answer a useful question. A campaign is optional. **Inputs:** @@ -23,7 +23,7 @@ Keep test bodies and assertions out of the design brief. a changed order, or a skipped callback. - Turn each open review question into a candidate. - If code is called dead, try removing it. - - Spread candidates across the affected files and boundaries; don't + - Spread candidates across the affected files and boundaries. Do not cluster them where edits are easy. - Hand-write candidates at boundaries a designer may miss, such as wrappers, lazy getters, and caches. @@ -51,8 +51,14 @@ Keep test bodies and assertions out of the design brief. it. State the law and find its authority (use [law discovery](law-discovery.md)). - **Dead code:** no legal history can observe the decision. Remove the code. - - **A law question:** the observations differ, but the contract seems to - allow both. Do not record that as acceptable. Give the user the two + - **An allowed alternative:** the observations differ, and the adopted law + already allows both, as an allowed set with its clause. The mutant is a + valid implementation, so do not strengthen the law or change production + to kill it. Report it with the mutant's key, the clause, and the history, + separately from the kills. Until the gap check recognizes allowed sets, + it stays open in the report. Only the framework closes it. + - **A law question:** the observations differ, and the contract only seems + to allow both. Do not record that as acceptable. Give the user the two observations and the history that shows them, and ask whether the law should decide the choice. - **A framework bug:** none of the above fits, such as a mutant that no diff --git a/packages/oracles/docs/model-styles.md b/packages/oracles/docs/model-styles.md index b37e59c..08d2b31 100644 --- a/packages/oracles/docs/model-styles.md +++ b/packages/oracles/docs/model-styles.md @@ -222,9 +222,12 @@ function roundTripHolds( value: Value, ): boolean { const back = parse(serialize(value)) + // Compare element by element. Joining the tags would accept [''] -> [] and + // ['a\u0000b'] -> ['a', 'b'], which lose information. return ( back.name === value.name && - back.tags.join('\u0000') === value.tags.join('\u0000') + back.tags.length === value.tags.length && + back.tags.every((tag, i) => tag === value.tags[i]) ) } ``` @@ -233,7 +236,9 @@ In an oracle, the driver records `value` and `parse(serialize(value))`, and the check compares them. A relation can hold for a useless implementation: a `serialize` that returns `""` and a `parse` that returns a fixed value satisfy many symmetries. So each relation needs a calibration that shows it -fails, such as a trace whose parsed value drops a tag. +fails. Good calibrations here are traces whose parsed tags lose an empty +string (`['']` becomes `[]`) or split one tag in two (`['a\u0000b']` becomes +`['a', 'b']`). Other useful relations: an incremental result equals a recompute from scratch, and the order of independent updates does not change the final state. @@ -267,9 +272,11 @@ function noticesAllowed(observed: Array): boolean { Never pick the order that production happens to use: that would reject a valid implementation. A mutant that only switches production to the other -allowed order survives a sound oracle. Until the framework recognizes -allowed sets in the gap check, treat such a survivor as a law question for -the user (see the [mutant gap hunt](instruments/mutant-gap-hunt.md) card). +allowed order survives a sound oracle. The law already answers the question, +so report it as an allowed alternative: its key, the clause, and the history. +Do not strengthen the law to kill it. Until the gap check recognizes allowed +sets, it stays open in the report (see the +[mutant gap hunt](instruments/mutant-gap-hunt.md) card). ## Mixing styles diff --git a/packages/oracles/docs/oracle-checklist.md b/packages/oracles/docs/oracle-checklist.md index 7f8e893..5bbec3f 100644 --- a/packages/oracles/docs/oracle-checklist.md +++ b/packages/oracles/docs/oracle-checklist.md @@ -173,8 +173,9 @@ No requirement here can be met by declaring something out of reach. - **Trigger:** A change introduces, removes, combines, or splits state in a stateful model. -- **Obligation:** The change MUST explain whether a legal next action can - distinguish states that the model proposes to treat as equal. +- **Obligation:** The change MUST explain whether a legal continuation (one + or more actions) can distinguish states that the model proposes to treat as + equal, by a later observation or by which actions are legal. - **Acceptance evidence:** A distinguishing history justifies retained state. Removed state needs a reasoned argument that it cannot change a promised observation or the legality of an action. @@ -307,15 +308,16 @@ No requirement here can be met by declaring something out of reach. Each change below keeps the requirement's purpose. A citation of the rubric's ID still names the same requirement. -| ID | Change | Reason | -| ------- | ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -| All | "Framework" and "Author" notes added | Say what `@tanstack/oracles` enforces, so nobody re-implements or works around it. | -| All | Coverage-map owner entries replaced by the oracle's `limits` | Adopters have no coverage map; code ownership belongs to tools such as CODEOWNERS (maintainer, 2026-11-10). | -| ORC-004 | Narrowing a grammar, filter, or validity predicate is a contract change | Maintainer, 2026-11-10. | -| ORC-006 | "Equivalence within the tested domain" removed from mutant outcomes | There are no equivalent mutants (maintainer, 2026-11-10). | -| ORC-007 | Rewritten as an enforced requirement | `oracleProperty` runs the campaigns and prints the replay command. | -| ORC-009 | Oracle terms come from the package glossary | The project glossary keeps domain terms. | -| ORC-015 | Reach comes from constructed histories, not `createOracleReach` | `oracleProperty` is the only campaign API (maintainer, 2026-11-09). | +| ID | Change | Reason | +| ------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | +| All | "Framework" and "Author" notes added | Say what `@tanstack/oracles` enforces, so nobody re-implements or works around it. | +| All | Coverage-map owner entries replaced by the oracle's `limits` | Adopters have no coverage map; code ownership belongs to tools such as CODEOWNERS (maintainer, 2026-11-10). | +| ORC-004 | Narrowing a grammar, filter, or validity predicate is a contract change | Maintainer, 2026-11-10. | +| ORC-006 | "Equivalence within the tested domain" removed from mutant outcomes | There are no equivalent mutants (maintainer, 2026-11-10). | +| ORC-007 | Rewritten as an enforced requirement | `oracleProperty` runs the campaigns and prints the replay command. | +| ORC-008 | "A legal next action" becomes "a legal continuation (one or more actions)" | Two states can agree after one action and differ after two, such as a FIFO queue's tail. | +| ORC-009 | Oracle terms come from the package glossary | The project glossary keeps domain terms. | +| ORC-015 | Reach comes from constructed histories, not `createOracleReach` | `oracleProperty` is the only campaign API (maintainer, 2026-11-09). | TanStack DB adds its own requirements, such as its coverage map, on top of this checklist in its repository. diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 6623bce..49479a0 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -1,6 +1,6 @@ --- name: oracles-design -description: Design, write, or change an oracle with @tanstack/oracles. Write a law sheet first, choose a model, and build histories that reach the states the law needs. Show RED on unchanged production, change production, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. +description: Design, write, or change an oracle with @tanstack/oracles. Choose the boundary, write a law sheet first, choose a model, and build histories that reach the states the law needs. Show the evidence the change needs, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. --- # Design an oracle @@ -20,9 +20,40 @@ witness, and another round. The common shortcuts are changing production first, editing the model to match production, narrowing histories until a failure goes away, and pinning one example. `oracles-review` §2 lists them all. -## 1. Write the law sheet, then stop - -Before any code, write one row per law and give the sheet to the user. +## 1. Choose the boundary, then write the law sheet + +**Choose the boundary first.** An oracle maintains one caller-visible +promise, not one file. Before the law sheet, establish: + +| Question | What to establish | +| ---------------------------------- | ------------------------------------------------------------------------------ | +| What promise belongs together? | One caller-visible behavioral contract. | +| Where is it observed? | The public entry points and the checkpoints where it is owed. | +| What does it assume? | What providers, the environment, and neighboring components must supply. | +| What else witnesses it? | Other drivers for the same law: component, integration, end-to-end, or replay. | +| Where does another contract begin? | A different authority, lifecycle, observation boundary, or composition rule. | +| Which source implements it? | The `owns` list that the gap check uses. | + +Example: an oracle for optimistic publication can hold several related laws +(an optimistic write is visible at once, a confirmation keeps it, and a +rollback removes it) and several drivers (an in-memory provider and a real one). It +keeps its identity if the code moves between files. One source file can also +serve several contracts. A new failure belongs to the oracle whose law covers +its promise. + +**Decide whether the user must approve.** + +- **Strengthening an adopted law** (a missing history, observation, driver + path, or check) reuses the law as it is. Do not write a new sheet, and do + not stop. Name the law in the evidence. +- **Changing a law** (its promise, domain, checkpoint, assumptions, or the + freedoms it allows) needs a law delta: the old and new row for each changed + law. Stop for the user's review. +- **A new law, or missing or contradictory authority,** needs the full law + sheet below. Stop for the user's review. +- **The task already approved these laws?** Do not ask again. + +Write one row per law and give the sheet to the user. | Field | What to write | | ---------- | --------------------------------------------------------------------------------------------------------------------- | @@ -41,7 +72,8 @@ Before any code, write one row per law and give the sheet to the user. - When the promise is unclear, or two readings predict different behavior, use the [law discovery](../../docs/instruments/law-discovery.md) card. -**Stop here.** Do not write the oracle until the user has reviewed the sheet. +**Stop here** when the sheet or a law delta is needed. Do not write the oracle +until the user has reviewed it. Once the user approves it, the sheet becomes the **law block**: the opening comment of the oracle file, beside the model that enacts it. Each row's law, @@ -140,8 +172,11 @@ Show what each law, model field, and model branch contributes. - **A law:** give a legal history that the other laws allow and this law rejects. -- **A model state or field:** give a legal next action where two states that - differ only in it must behave differently (ORC-008). +- **A model state or field:** give a legal continuation (one or more + actions) after which two states that differ only in it must give different + observations, or allow different actions (ORC-008). A FIFO queue holding + `[1, 2]` and one holding `[1, 3]` agree after one dequeue and differ after + two. - **A branch or exception:** give the history that needs it. If none exists, remove it. @@ -231,9 +266,20 @@ every report. Do not disable a rule. Techniques: [guide § The driver and the check](../../docs/guide.md#the-driver-and-the-check). -## 7. RED, change, GREEN, mutant +## 7. Show the evidence that fits the change + +The evidence depends on the kind of work: + +| Work | Evidence | +| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| **Fix a defect** | The extended oracle fails on unchanged production (RED). The repaired production passes (GREEN). Neighboring laws still pass. The original defect, as a mutant, fails the oracle. | +| **Refactor without changing behavior** | Old and new production pass the same oracle: the same laws, histories, and checkpoints. The gap check on both kills at least the same mutants, compared by key with the same seed. No RED is expected. | +| **Add or strengthen an oracle over correct production** | Production passes. A calibration (a wrong answer or a corrupted trace) and a mutant fail the intended law. No RED is expected. | + +A refactor that needs a changed law, check, or history is not a pure +refactor: treat the law change first (step 1). -When the task changes production: +**To fix a defect:** 1. Run the extended oracle file on **unchanged** production. It must fail on the defect. This is RED. A setup failure is not RED: fix the oracle first. @@ -245,16 +291,20 @@ When the task changes production: 4. Show that a mutant, such as the original defect, fails the oracle. -Do not change production before step 1. A later run does not make up for a -skipped RED. +For a defect fix, do not change production before step 1. A later run does +not make up for a skipped RED. **Clean up production once GREEN.** The oracle now protects the law, so use it to simplify: - Refactor the code around the change. Do not insert a new branch beside an old one that does almost the same thing. -- Remove states, flags, branches, and recovery paths that no law needs. The - gap check shows which code no history can observe. +- Remove states, flags, branches, and recovery paths that no law needs. A + surviving mutant shows only that the current histories did not expose the + code. Before you remove it, use the mutant gap hunt to rule out a missing + history, observation, or law, and give the contract's reason that no legal + history can observe it. A green rerun of the same oracle is not that + reason. - Prefer a change that shrinks the code, or leaves its size the same. If the repository sets a size rule, such as a net-negative diff, follow it. - Rerun the oracle and the gap check after each cleanup. @@ -270,9 +320,11 @@ comes with one instruction. Do what it says, within the rules below. history, observation, or law, or it shows dead code. - **An effect that no check sees is one that the driver does not record.** Record it. -- **Code whose removal no legal history can observe is dead.** Remove it in - the same change as the oracle. That is a production change, so say so in - the evidence, and rerun the oracle and the gap check. +- **Code whose removal no legal history can observe is dead.** The gap check + alone cannot show this, because its histories are a sample. First rule out + a missing history, observation, or law. Then remove the code in the same + change, give the contract's reason in the evidence (it is a production + change), and rerun the oracle and the gap check. - **A law that kills nothing** owns no mutated code, or its witness never varies in the histories. Check its witness and its constructed history. If the law still kills nothing, it may restate another law: compare the two. @@ -295,14 +347,18 @@ a survivor whose missing history is not obvious. ## 9. Report the evidence -The evidence counts only as a set. One item alone proves little. +The evidence counts only as a set. One item alone proves little. Give the +items that fit the kind of change (step 7). -- **Law:** the law and its authority. +- **Law:** the law and its authority, and whether it was reused, changed, or + new. - **Reach:** each declared state and how many histories entered it. -- **RED:** the command and the failure on unchanged production. -- **GREEN:** the command and the result after the change. -- **Mutant:** the mutant, such as the original defect, and the law that - rejected it. +- **Before and after:** for a defect fix, RED on unchanged production and + GREEN after the change. For a refactor, the same oracle passing on old and + new production. For a new oracle over correct code, production passing. +- **Mutant:** a mutant, such as the original defect, and the law that + rejected it, plus the calibrations that fail. For a refactor, the kill + sets before and after. - **Gaps:** the gap check's totals, and any gap reported as a framework bug. - **Limits:** the states, paths, and observations that the evidence does not cover. diff --git a/packages/oracles/skills/oracles-evaluate-review/SKILL.md b/packages/oracles/skills/oracles-evaluate-review/SKILL.md index 1069135..1228a7a 100644 --- a/packages/oracles/skills/oracles-evaluate-review/SKILL.md +++ b/packages/oracles/skills/oracles-evaluate-review/SKILL.md @@ -60,6 +60,9 @@ For each confirmed behavioral finding, decide which kind it is. A finding can contain both. Fix the enforcement part, and present the law question. +An enforcement gap reuses the adopted law, so `oracles-design` step 1 does not +stop for it. Name the law in the ledger and the evidence. + ## 3. Plan the oracle changes for the whole review Before you work on single findings, look at all of them together. Findings diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index d940a2f..347e05c 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -42,7 +42,9 @@ contains one is not ready. | Shortcut | What to look for | | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | -| **Production first** | A production change in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | +| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | +| **Unproven refactor** | A refactor of owned code without the same oracle passing on old and new production, and kill sets compared before and after. A refactor needs no RED. | +| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | | **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | | **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. | | **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | @@ -54,7 +56,8 @@ contains one is not ready. | **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | | **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | -The evidence counts only as a set: law, reach, RED, GREEN, mutant, gaps, +The evidence counts only as a set, chosen by the kind of change +(`oracles-design` step 7): law, reach, before-and-after runs, mutant, gaps, limits. A missing item is a finding, even when the rest is strong. ## 3. Hunt what the tools cannot see diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 38c3e1a..4e40c51 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -101,8 +101,10 @@ export default [ production source lives elsewhere, add the `oracle/probe-form` rule with that glob. - If ESLint does not already parse TypeScript, add `@typescript-eslint/parser` - as a dev dependency and set it as `languageOptions.parser` for the oracle - files. + as a dev dependency. Set it as `languageOptions.parser` for every TypeScript + file that the presets cover: the oracle files and the production source. + Otherwise a typed production file fails to parse before the probe rule + runs. - Do not add `legacyCampaigns` or any other exemption. Adopters have none. **An older ESLint (7 or 8, `.eslintrc`).** Do not upgrade the repository's @@ -111,7 +113,7 @@ lint. Lint the oracle files with a separate ESLint 9: 1. Install the plugin with `--legacy-peer-deps` (npm), or the package manager's equivalent. 2. Write a flat `eslint.oracle.config.mjs` with both presets and a TypeScript - parser for the oracle files. + parser for the oracle files and the production source. 3. Run it as `npx -y -p eslint@9 eslint -c eslint.oracle.config.mjs `, and add a `lint:oracles` script. @@ -155,7 +157,16 @@ Fill in the two placeholders in step 7, once their values exist. If the reposito ## 5. Make the skills findable The skills ship inside the package, so agents find them only through a -pointer. Offer the user these choices. They can pick more than one. +pointer. A pointer must read the copy that the chosen package installed: + +- **One package, or a workspace with a root install:** + `node_modules/@tanstack/oracles/skills//SKILL.md`. +- **A package-local install in a workspace:** + `/node_modules/@tanstack/oracles/skills//SKILL.md`, + for the package chosen in step 1. Check that the file exists there. + +Use that path in the `AGENTS.md` block (step 4) too. Offer the user these +choices. They can pick more than one. | Choice | Writes | Use when | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | @@ -177,12 +188,19 @@ name: oracles-design description: --- -Read `node_modules/@tanstack/oracles/skills/oracles-design/SKILL.md` from the -root of the current repository in full, and follow it. It matches the -installed version of `@tanstack/oracles`. If that file does not exist, this -repository does not have `@tanstack/oracles` installed: say so and stop. +Read `/skills/oracles-design/SKILL.md` in full, and follow it. +It matches the installed version of `@tanstack/oracles`. If that file does not +exist, `@tanstack/oracles` is not installed there: say so and stop. ``` +- In a **repository pointer**, `` is the path above, relative + to the repository root, such as `packages/app/node_modules/@tanstack/oracles`. +- In a **global pointer**, write instead: "Find the nearest `package.json` at + or above the files you are working on whose dependencies include + `@tanstack/oracles`. Use the `node_modules/@tanstack/oracles` in that + package's directory." Then the pointer loads the right version in every + workspace package. + Use pointers, not copies or symbolic links: a pointer always reads the installed version, and never breaks before install. @@ -230,9 +248,11 @@ pnpm exec oracle-gaps --package --source \ Run it after the oracle passes on production. Each gap in the report comes with one instruction. Follow that instruction, and follow `oracles-design` for what a gap means. There -are no equivalent mutants and no exits. If the gap check shows dead code -in an owned function, remove it: that is a production change, so tell the -user, and rerun the oracle and the gap check. If a gap can be neither killed +are no equivalent mutants and no exits. A surviving mutant shows only that +the current histories did not expose that code. If you believe the code is +dead, first follow `oracles-design` step 8 to rule out a missing history, +observation, or law. Removing it is a production change, so tell the user, +and rerun the oracle and the gap check. If a gap can be neither killed nor removed, stop work on that gap and report its key and evidence to the user. From 62e8c423f37d5026b73444c21aaef184b8376897 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 16:40:59 -0700 Subject: [PATCH 10/76] docs(oracles): mutant validity follows the TypeScript types The framework judges a mutant over the domain the operand types admit, so a narrower domain stated only in prose does not make a mutant invalid. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 49479a0..98b84c5 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -337,6 +337,11 @@ comes with one instruction. Do what it says, within the rules below. framework must not generate it, so it is a framework bug: report it the same way. You do not decide that a mutant is invalid, and the label does not close its gap. +- **Validity follows the types.** The framework judges a mutant over the + domain that the TypeScript types admit, so `number` includes `-0` and `NaN`. + A narrower domain stated only in the law's prose, such as "positive + integers", does not make a mutant invalid. Cover the type's whole domain, + or narrow the type. - **Temporary, until the report's messages change:** if a message calls a mutant "equivalent", asks for a "maintainer's approval", or offers a limit that "needs a maintainer's decision", do not follow that part. Find the From 76057d9db3dc60f86a30ae29bcd01549a2cbe24a Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 17:28:53 -0700 Subject: [PATCH 11/76] docs(oracles): fix what round-2 validation hit Four fresh agents ran the branch's skills on four workflow branches: a behavior-preserving refactor, a review with two findings, a planted defect, and a package-local workspace. Each took the right branch. Their findings, plus the framework session's answers: - Refactor evidence follows the gate's rule: oracle files unchanged or only strengthened and passing on old and new code with the same seed; shared killed keys still killed; new keys in changed functions killed or acted on. - Design: an allowed alternative is its own outcome; a gap that existed before a change still counts in a changed function; widening histories inside the stated domain is strengthening; a grammar that covers part of the law's domain says so; constructed histories must fit the arbitrary; the driver records production errors; own the whole file when an owned function reads module data; several oracles may own one function; public state needs a law or a limit; an exhaustiveness-guard survivor is a known CLI bug, recorded without stopping or removing the guard. - Review: a law-dependent state in `measured` is an unreachable claim. - Setup: name the oracle config `vitest.config.ts` (the gap check loads only that name), pnpm 11's allowBuilds placeholder, refresh pointers on upgrade, the AGENTS.md block in a package-local workspace, keep oracle files out of the repository's own lint or match its style, install before a pointer gives up, and a path for runs with no user. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 69 +++++++++++++------ .../oracles/skills/oracles-review/SKILL.md | 32 ++++----- .../oracles/skills/oracles-setup/SKILL.md | 38 ++++++++-- 3 files changed, 96 insertions(+), 43 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 98b84c5..44fc865 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -25,14 +25,15 @@ failure goes away, and pinning one example. `oracles-review` §2 lists them all. **Choose the boundary first.** An oracle maintains one caller-visible promise, not one file. Before the law sheet, establish: -| Question | What to establish | -| ---------------------------------- | ------------------------------------------------------------------------------ | -| What promise belongs together? | One caller-visible behavioral contract. | -| Where is it observed? | The public entry points and the checkpoints where it is owed. | -| What does it assume? | What providers, the environment, and neighboring components must supply. | -| What else witnesses it? | Other drivers for the same law: component, integration, end-to-end, or replay. | -| Where does another contract begin? | A different authority, lifecycle, observation boundary, or composition rule. | -| Which source implements it? | The `owns` list that the gap check uses. | +| Question | What to establish | +| ---------------------------------- | --------------------------------------------------------------------------------------------- | +| What promise belongs together? | One caller-visible behavioral contract. | +| Where is it observed? | The public entry points and the checkpoints where it is owed. | +| What does it assume? | What providers, the environment, and neighboring components must supply. | +| What else witnesses it? | Other drivers for the same law: component, integration, end-to-end, or replay. | +| Where does another contract begin? | A different authority, lifecycle, observation boundary, or composition rule. | +| Which source implements it? | The `owns` list that the gap check uses. | +| What public state does it expose? | Each readable field or getter needs a law or a stated limit. A limit leaves its mutants open. | Example: an oracle for optimistic publication can hold several related laws (an optimistic write is visible at once, a confirmation keeps it, and a @@ -45,7 +46,9 @@ its promise. - **Strengthening an adopted law** (a missing history, observation, driver path, or check) reuses the law as it is. Do not write a new sheet, and do - not stop. Name the law in the evidence. + not stop. Name the law in the evidence. Widening the histories inside the + law's stated domain is strengthening. Changing the stated domain, a + checkpoint, or `limits` is a law change. - **Changing a law** (its promise, domain, checkpoint, assumptions, or the freedoms it allows) needs a law delta: the old and new row for each changed law. Stop for the user's review. @@ -201,7 +204,11 @@ in the model. - **Reach.** Declare in `states` each state that a law needs, and give each one a constructed history that enters it (ORC-015). Put a state that you only count in `measured`. A literal example or a `fixed` history is not - reach. + reach. A constructed history must be one the arbitrary could generate, + within its bounds. +- **Cover the law's whole domain.** A mutant can only show what the histories + vary. If the grammar covers part of the domain the law states (for example, + fractions in quarter steps only), say so in `limits`. Better, widen it. - **Probes.** Where production decides to enter a state, bind the state to a production probe, and predict entry from the plan alone. - **ORC-004 controls.** For a claim about generated coverage, account for: @@ -238,8 +245,9 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). - **The driver** calls the real production entry point and records public observations with `observe(checkpoint, value)`. - - It records production errors, unhandled rejections, and warnings that - the law cares about. + - It records production errors, unhandled rejections, and warnings as + observations, so the check can reject them. A mutant that crashes + production is not a kill. - It observes the contract's return values. - It awaits the work it starts. - It bounds its own run time and work, and throws a structured error when @@ -256,6 +264,11 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). (`src/x.ts#run`, `src/x.ts#Class.method`). Own only what the laws cover. The gap check judges only owned code, and the gap check's `--source` selects files. + - If an owned function reads module-level data (a lookup table, a constant + object), own the whole file for now. Data outside a function is not yet + part of a function entry's tree. + - Several oracles may own the same function. A mutant counts as killed when + any of its owners kills it. Keep the law, its authority, and its limits in `law`, and prose beside each part that says why it predicts the promised result (ORC-003). @@ -270,11 +283,11 @@ Techniques: [guide § The driver and the check](../../docs/guide.md#the-driver-a The evidence depends on the kind of work: -| Work | Evidence | -| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| **Fix a defect** | The extended oracle fails on unchanged production (RED). The repaired production passes (GREEN). Neighboring laws still pass. The original defect, as a mutant, fails the oracle. | -| **Refactor without changing behavior** | Old and new production pass the same oracle: the same laws, histories, and checkpoints. The gap check on both kills at least the same mutants, compared by key with the same seed. No RED is expected. | -| **Add or strengthen an oracle over correct production** | Production passes. A calibration (a wrong answer or a corrupted trace) and a mutant fail the intended law. No RED is expected. | +| Work | Evidence | +| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Fix a defect** | The extended oracle fails on unchanged production (RED). The repaired production passes (GREEN). Neighboring laws still pass. The original defect, as a mutant, fails the oracle. | +| **Refactor without changing behavior** | The oracle files are unchanged, or only strengthened, and pass on old and new production with the same seed. Every mutant key present in both gap runs that was killed on the old code is still killed. Every new key in a changed function is killed, or its survivor directive is acted on. Keys that disappear need nothing. No RED is expected. | +| **Add or strengthen an oracle over correct production** | Production passes. A calibration (a wrong answer or a corrupted trace) and a mutant fail the intended law. No RED is expected. | A refactor that needs a changed law, check, or history is not a pure refactor: treat the law change first (step 1). @@ -330,6 +343,14 @@ comes with one instruction. Do what it says, within the rules below. the law still kills nothing, it may restate another law: compare the two. - **The only exclusions are probe-only code and the `super()` region of a derived constructor.** +- **A survivor that the adopted law allows is an allowed alternative.** If a + mutant only moves production to another member of an allowed set, report it + with its key, the clause, and the history, apart from the kills. Do not + strengthen the law or change production to kill it. It stays open until the + framework recognizes allowed sets. The user decides whether that blocks the + work. +- **A gap that existed before your change still counts** when it is in a + function you changed. Report it with its law question. Do not take an exit. - **Never propose an exit.** If a gap can be neither killed nor removed, it is a framework bug. Stop work on that gap, report its key and evidence to the user, and keep working on the other gaps. @@ -346,6 +367,13 @@ comes with one instruction. Do what it says, within the rules below. mutant "equivalent", asks for a "maintainer's approval", or offers a limit that "needs a maintainer's decision", do not follow that part. Find the history and the law. +- **Temporary, a known CLI bug:** a survivor inside an exhaustiveness guard + (a branch with `x satisfies never`) should never have been generated. + Record it in your report as "CLI bug: exhaustiveness guard mutated", and + keep working on the other gaps. Do not stop, do not count it as an oracle + limit, and do not remove the guard: it is a compile-time check. +- **Temporary:** in a package without `"type": "module"`, Vite may warn + while loading the gap check's own config copies. Ignore that warning. Use the [mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) card for a survivor whose missing history is not obvious. @@ -360,10 +388,11 @@ items that fit the kind of change (step 7). - **Reach:** each declared state and how many histories entered it. - **Before and after:** for a defect fix, RED on unchanged production and GREEN after the change. For a refactor, the same oracle passing on old and - new production. For a new oracle over correct code, production passing. + new production with the same seed. For a new oracle over correct code, + production passing. - **Mutant:** a mutant, such as the original defect, and the law that - rejected it, plus the calibrations that fail. For a refactor, the kill - sets before and after. + rejected it, plus the calibrations that fail. For a refactor, the shared + keys still killed, and each new key killed or acted on. - **Gaps:** the gap check's totals, and any gap reported as a framework bug. - **Limits:** the states, paths, and observations that the evidence does not cover. diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index 347e05c..8e1b6a1 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -39,22 +39,22 @@ checkpoint. A final state can hide a wrong intermediate observation. These are the shortcuts. Report each one as a finding. A change that contains one is not ready. -| Shortcut | What to look for | -| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | -| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | -| **Unproven refactor** | A refactor of owned code without the same oracle passing on old and new production, and kill sets compared before and after. A refactor needs no RED. | -| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | -| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | -| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. | -| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | -| **Pinned only** | A fixed example where the law needs a generator dimension. | -| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | -| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | -| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | -| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | -| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | -| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | +| Shortcut | What to look for | +| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | +| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | +| **Unproven refactor** | A refactor of owned code that lacks one of these: the oracle passing on old and new production with the same seed, every shared killed key still killed, and every new key in a changed function killed or acted on. A refactor needs no RED. | +| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | +| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | +| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. Also a state that a law depends on but that sits in `measured`, where nothing requires it. | +| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | +| **Pinned only** | A fixed example where the law needs a generator dimension. | +| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | +| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | +| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | +| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | +| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | +| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | The evidence counts only as a set, chosen by the kind of change (`oracles-design` step 7): law, reach, before-and-after runs, mutant, gaps, diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 4e40c51..0974d9d 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -6,7 +6,9 @@ description: Set up @tanstack/oracles in a repository. Install the pinned packag # Set up @tanstack/oracles You are setting up oracle testing in this repository. Do the steps in order. -Ask the user before each step that writes outside the repository. +Ask the user before each step that writes outside the repository. When no +user is available, choose the default this skill gives, record each choice in +your report, and continue. Setup ends when one oracle passes on production and the gap check has judged it with no gap left open, unless the user decides otherwise. It does not end @@ -57,8 +59,14 @@ TypeScript 5 or newer. If the repository's `tsc` is older, keep the oracle files and the Vitest config out of its run (its `include` or `exclude`). Report any typecheck errors that existed before setup as pre-existing. -Use the repository's own package manager. To upgrade, change both URLs to -the new commit at once, then do step 4 again and rerun the gap check (step 7). +Use the repository's own package manager. With pnpm 11, an install that skips +a dependency's build script stops every later pnpm command +(`ERR_PNPM_IGNORED_BUILDS`), and pnpm writes an `allowBuilds` placeholder into +`pnpm-workspace.yaml`. Answer it, usually `false`, and tell the user. + +To upgrade, change both URLs to the new commit at once. Then do step 4 and +step 5 again (copy each pointer's `name` and `description` again, since they +can change), and rerun the gap check (step 7). ## 3. Configure @@ -69,10 +77,15 @@ Vitest setup file that runs before any oracle module is imported. **Where oracle files go.** Put oracle files in a test directory, named `*oracle*.test.ts`, for example `tests/parse.oracle.property.test.ts`. If the repository's main runner is not Vitest, check that its test glob does not -also match the oracle files, and run them with their own Vitest config: +also match the oracle files. The gap check loads only `vitest.config.*` or +`vite.config.*`, so: + +- If the repository has no Vitest config, create `vitest.config.ts` for the + oracle files. Jest and mocha ignore that name. +- If it has one, add the oracle files to its `include`. ```ts -// vitest.oracle.config.ts +// vitest.config.ts import { defineConfig } from 'vitest/config' export default defineConfig({ @@ -83,6 +96,10 @@ export default defineConfig({ Add a script for it, such as `test:oracles`. Leave the existing suite unchanged. +If the repository's own lint or formatter covers the oracle files (for +example, an older ESLint with Prettier over `test/`), keep the oracle files +out of its globs, or match its style. Say which in your report. + **Lint.** Add both presets to the ESLint flat config: ```js @@ -154,6 +171,10 @@ This repository tests its laws with `@tanstack/oracles`. Fill in the two placeholders in step 7, once their values exist. If the repository has a `CLAUDE.md` and no `AGENTS.md`, ask the user which file gets the block. +In a workspace with a package-local install, put the block in the package's +`AGENTS.md`, where its `node_modules` paths resolve. A root copy must use the +full paths, such as `packages//node_modules/@tanstack/oracles/...`. + ## 5. Make the skills findable The skills ship inside the package, so agents find them only through a @@ -190,7 +211,8 @@ description: /skills/oracles-design/SKILL.md` in full, and follow it. It matches the installed version of `@tanstack/oracles`. If that file does not -exist, `@tanstack/oracles` is not installed there: say so and stop. +exist, run the repository's install, then look again. If it is still missing, +`@tanstack/oracles` is not installed there: say so and stop. ``` - In a **repository pointer**, `` is the path above, relative @@ -227,7 +249,9 @@ oracle is not declared. That is a setup failure, even when the report shows 0 gaps and the command exits 0. In an oracle file, only `oracleProperty` registers tests. A plain Vitest test -there is not a kill, and the gap check reports it. +there is not a kill, and the gap check reports it. The driver records +production errors as observations: a mutant that crashes production is not a +kill until the check rejects the recorded error. ## 7. Run the gap check From 5f573b98526ed8a3ddc5792f740b2baae225fa9d Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 17:29:09 -0700 Subject: [PATCH 12/76] docs(oracles): point the refactor finding at design step 7 Co-authored-by: Isaac --- .../oracles/skills/oracles-review/SKILL.md | 32 +++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index 8e1b6a1..2f7f654 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -39,22 +39,22 @@ checkpoint. A final state can hide a wrong intermediate observation. These are the shortcuts. Report each one as a finding. A change that contains one is not ready. -| Shortcut | What to look for | -| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | -| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | -| **Unproven refactor** | A refactor of owned code that lacks one of these: the oracle passing on old and new production with the same seed, every shared killed key still killed, and every new key in a changed function killed or acted on. A refactor needs no RED. | -| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | -| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | -| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. Also a state that a law depends on but that sits in `measured`, where nothing requires it. | -| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | -| **Pinned only** | A fixed example where the law needs a generator dimension. | -| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | -| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | -| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | -| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | -| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | -| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | +| Shortcut | What to look for | +| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | +| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | +| **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | +| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | +| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | +| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. Also a state that a law depends on but that sits in `measured`, where nothing requires it. | +| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | +| **Pinned only** | A fixed example where the law needs a generator dimension. | +| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | +| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | +| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | +| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | +| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | +| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | The evidence counts only as a set, chosen by the kind of change (`oracles-design` step 7): law, reach, before-and-after runs, mutant, gaps, From 6bf40d8eb2f2af9ee7b19c8b3ceb28d6e2d7f6dc Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 17:56:05 -0700 Subject: [PATCH 13/76] docs(oracles): the README example covers -0 and reaches zero gaps With #11 merged, the README sample type-checks and the gap check counts it as an oracle file. It still left one gap: `>` to `>=` differs at clamp(-0, 0), and the histories never generated -0. The example now covers its type's domain, as the design skill teaches: -0 is in the grammar, a constructed history enters "negative zero at max", and the gap check kills all three mutants. Co-authored-by: Isaac --- packages/oracles/README.md | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/packages/oracles/README.md b/packages/oracles/README.md index 6b8c25a..7f61898 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -50,8 +50,9 @@ import { clamp } from '../src/clamp' type History = { n: number; max: number } +// The type is number, so the domain includes -0. const histories = fc.record({ - n: fc.integer({ min: -100, max: 100 }), + n: fc.oneof(fc.integer({ min: -100, max: 100 }), fc.constant(-0)), max: fc.integer({ min: 0, max: 50 }), }) @@ -62,13 +63,14 @@ oracleProperty({ statement: 'clamp(n, max) returns max when n is above max, and n otherwise.', authority: 'README, "clamp"', - limits: 'Integers in [-100, 100] for n and [0, 50] for max.', + limits: + 'Integers in [-100, 100] and -0 for n, and [0, 50] for max. No NaN or infinities.', }, histories: { arbitrary: histories, runs: 100, fixedSeeds: [1], - states: ['above max', 'at max'], + states: ['above max', 'at max', 'negative zero at max'], measured: ['below max'], constructed: [ { @@ -81,6 +83,12 @@ oracleProperty({ enters: ['at max'], build: ({ max }) => ({ n: max, max }), }, + { + // -0 is not above 0, so the result is n (-0), not max (0). + name: '-0 at max 0', + enters: ['negative zero at max'], + build: () => ({ n: -0, max: 0 }), + }, ], }, // The law restated without production. @@ -95,11 +103,13 @@ oracleProperty({ expected.result, ) ?? pass, reached: (plan) => [ - plan.n > plan.max - ? 'above max' - : plan.n === plan.max - ? 'at max' - : 'below max', + Object.is(plan.n, -0) && plan.max === 0 + ? 'negative zero at max' + : plan.n > plan.max + ? 'above max' + : plan.n === plan.max + ? 'at max' + : 'below max', ], calibrations: [ { From d6d445574fad597971c72872f580ee43ae35caa1 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 18:34:31 -0700 Subject: [PATCH 14/76] docs(oracles): fix what round-3 validation hit Three fresh agents adopted the package in p-limit (async JavaScript, xo, AVA), dequal (uvu with esm), and nanostores (TypeScript 7, oxlint). All three followed the method, and dequal's agent found real library defects. Their findings, plus the framework session's answers: - Setup: offer a separate Node 20 job when CI is older; use the repository's package manager for install commands; place oracle files where the existing runner will not load them (their own directory, or a `*.oracle.ts` name); probe-form for root-level source; xo repositories; the TypeScript 7 plugin bug as a temporary note; the live-rule check through stdin before any oracle file exists; choose a function with a documented promise; a first oracle that is RED found a defect (report it, never weaken the oracle, run the gap check on a scratch fix until the CLI judges a failing baseline); group the open gaps in the report. - Design: an analogy is not authority; laws people forget (no throw on valid input, the algebraic laws of the kind of function); constructed values derived from the generated history; whole-file entries that pull in another contract; owned code that serves another module; where `oracleProperty`'s options are typed; do not copy exit wording into the agent's own report; a synchronous-spin timeout is a CLI limitation. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 35 +++++++++++- .../oracles/skills/oracles-setup/SKILL.md | 55 +++++++++++++++---- 2 files changed, 76 insertions(+), 14 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 44fc865..f6d5232 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -53,7 +53,18 @@ its promise. freedoms it allows) needs a law delta: the old and new row for each changed law. Stop for the user's review. - **A new law, or missing or contradictory authority,** needs the full law - sheet below. Stop for the user's review. + sheet below. Stop for the user's review. An analogy to another documented + rule is not authority: mark that law `open`. + +**Laws people forget to write down:** + +- **No throw on valid input.** When the contract says a call returns a value, + it promises not to throw for valid input. State that as a law, with the same + authority, so a mutant that throws is a kill, not a crash. +- **The laws of the kind of function.** An equality check is reflexive and + symmetric. A parser and a serializer round-trip. A setter is idempotent when + set to the same value. Give these as rows, with the function's documented + purpose as their authority. - **The task already approved these laws?** Do not ask again. Write one row per law and give the sheet to the user. @@ -205,7 +216,9 @@ in the model. one a constructed history that enters it (ORC-015). Put a state that you only count in `measured`. A literal example or a `fixed` history is not reach. A constructed history must be one the arbitrary could generate, - within its bounds. + within its bounds. Build its values from the generated history (for example + "the initial value plus one"), not from constants that the generated values + might equal. - **Cover the law's whole domain.** A mutant can only show what the histories vary. If the grammar covers part of the domain the law states (for example, fractions in quarter steps only), say so in `limits`. Better, widen it. @@ -267,12 +280,22 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). - If an owned function reads module-level data (a lookup table, a constant object), own the whole file for now. Data outside a function is not yet part of a function entry's tree. + - When a whole-file entry pulls in another contract (for example a batching + API in the same file), write that contract's own oracle, or report its + gaps as outside this oracle's scope. + - Owned code that serves another module's contract belongs to that module's + oracle. Owned code with no behavior anyone observes is possible dead code: + investigate it as step 8 says. - Several oracles may own the same function. A mutant counts as killed when any of its owners kills it. Keep the law, its authority, and its limits in `law`, and prose beside each part that says why it predicts the promised result (ORC-003). +The options of `oracleProperty` are typed in +`node_modules/@tanstack/oracles/src/oracle-property.ts` +(`OraclePropertyOptions`). Read them there. + **Lint as you go.** Run the repository's lint on each oracle file and each changed production file after you edit it, and again before you finish. Fix every report. Do not disable a rule. @@ -366,7 +389,13 @@ comes with one instruction. Do what it says, within the rules below. - **Temporary, until the report's messages change:** if a message calls a mutant "equivalent", asks for a "maintainer's approval", or offers a limit that "needs a maintainer's decision", do not follow that part. Find the - history and the law. + history and the law. Do not copy those words into your own report either: + for a survivor you cannot kill, give the histories you tried and what they + showed. +- **Temporary, a CLI limitation:** a mutant that makes production loop forever + synchronously shows as a timeout. The framework's breakage laws will judge + it, and they are not built yet. Report it as a known CLI limitation, not an + oracle gap. Do not add driver timeouts for it, and keep working. - **Temporary, a known CLI bug:** a survivor inside an exhaustiveness guard (a branch with `x satisfies never`) should never have been generated. Record it in your report as "CLI bug: exhaustiveness guard mutated", and diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 0974d9d..77df71b 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -27,10 +27,12 @@ Find out, and tell the user: - The lint setup: ESLint 9 or later with a flat config, an older ESLint with `.eslintrc`, or no ESLint (for example Biome only). - The Node version that CI and the `engines` field use. The package needs - Node 20 or later. If CI runs an older Node, tell the user. Do not change CI. + Node 20 or later. If CI runs an older Node, tell the user, and offer a + separate Node 20 job for the oracles. Do not change the existing jobs. - The repository's TypeScript version. Do not change it (step 2). -- The command runner for package binaries: `pnpm exec`, `npx`, or - `yarn`. Use it wherever this skill shows `pnpm exec`. +- The package manager and its runner for package binaries: `pnpm exec`, + `npx`, or `yarn`. Use them wherever this skill shows `pnpm add` or + `pnpm exec`. - Which coding agents the user works with. Look for `.claude/`, `.agents/`, `.codex/`, `AGENTS.md`, and `CLAUDE.md`, and ask. @@ -76,8 +78,16 @@ Vitest setup file that runs before any oracle module is imported. **Where oracle files go.** Put oracle files in a test directory, named `*oracle*.test.ts`, for example `tests/parse.oracle.property.test.ts`. If the -repository's main runner is not Vitest, check that its test glob does not -also match the oracle files. The gap check loads only `vitest.config.*` or +repository's main runner is not Vitest, check that it does not also pick up +the oracle files: + +- If the runner loads every file in its test directory (uvu, mocha), put the + oracle files in their own directory, such as `oracles/`. +- If its default glob matches `*.test.ts` (better-node-test, Jest), name the + oracle files `*.oracle.ts`, and include that name in the Vitest config. + +The gap check finds oracle files by their `oracleProperty` call, so the name +only has to match the Vitest config and the lint glob. The gap check loads only `vitest.config.*` or `vite.config.*`, so: - If the repository has no Vitest config, create `vitest.config.ts` for the @@ -115,8 +125,15 @@ export default [ - `recommended` applies to oracle files and turns off inline directives there, so a disable comment has no effect. - `production` checks the probe form in `src/**` and `packages/*/src/**`. If - production source lives elsewhere, add the `oracle/probe-form` rule with that - glob. + production source lives elsewhere (for example a root-level `index.js`), add + the `oracle/probe-form` rule with that glob. +- If the repository lints with xo, keep the oracle files and the new config + files out of xo (its `ignores`), and lint them with the oracle ESLint + config. +- **Temporary, a known plugin bug:** under TypeScript 7 the plugin cannot load. + Skip the oracle lint, and report "known plugin bug: plugin cannot load under + TypeScript 7 (fix in progress)". This is not an oracle limit, and not a + reason to stop. - If ESLint does not already parse TypeScript, add `@typescript-eslint/parser` as a dev dependency. Set it as `languageOptions.parser` for every TypeScript file that the presets cover: the oracle files and the production source. @@ -139,7 +156,9 @@ repository's own `@typescript-eslint/parser`. Run the linter once and confirm that it loads the rules. Then check that the rules are live: a deliberately wrong line in an oracle file (for example -`fc.assert(...)`) must produce an error. Remove that line again. +`fc.assert(...)`) must produce an error. Remove that line again. Before the +first oracle file exists, do this check through the linter's `--stdin` with +an oracle file name, since a glob with no files fails. ## 4. Write the AGENTS.md block @@ -234,8 +253,17 @@ Before writing a global pointer: ## 6. Write the first oracle -Choose one small, owned production function with a clear promise. Ask the -user which one if it is not obvious. +Choose one small, owned production function with a clear, documented +promise. If most of its laws would have no authority (no docs, only tests or +code), choose another one. Ask the user which one if it is not obvious. + +If the first oracle fails on unchanged production under an adopted law, it +found a defect. The RED is the finding: report it with its replay line, and +tell the user. Never weaken the oracle to make it pass. The gap check cannot +judge a failing baseline yet, so make the smallest local fix on a scratch +branch, run the gap check there, and say in the report that it ran on the +patched code. With the user's approval, follow `oracles-design` step 7's +defect path for the real fix. Then load `oracles-design` and follow it from its first step. The law sheet comes first, and the user reviews it before you write code. Setup does not @@ -290,6 +318,11 @@ Tell the user: - what was installed (both pinned URLs) and where configuration changed - which pointers were written, and where - the first oracle's law and its result -- the gap check's totals, and every gap left with its instruction +- the gap check's totals, and every gap left, grouped as: + - inside a stated limit, with the limit + - framework or CLI limitations (for example a timeout from a synchronous + infinite loop, or an exhaustiveness-guard mutant) + - every other gap, with its instruction +- any defect the oracle found, with its witness Do not call setup done while a gap is open, unless the user decides otherwise. From 573c671b118327278889ddadb6ad465fd3df34a7 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 19:40:49 -0700 Subject: [PATCH 15/76] docs(oracles): write the general law down, and trigger the design grammar Agents go from examples to a general law only when made to write it down, and no agent in three validation rounds opened the design-grammar card, because it triggered on judgment ("a hard law"). - Design step 1: before the law sheet, each law gets a written block: the examples with sources, what varies and what stays, the law as one "For every ..." sentence with its edges, a new case it predicts, and every example rebuilt from the sentence alone. - Five concrete triggers send the agent to the design-grammar card; the evidence says which fired. - Review: "Law as examples" and "Skipped design grammar" findings. - Evaluate-review: the causal condition is written as a sentence over histories, with a second history it predicts. - Setup: TypeScript 7 is an environment limit (typescript-eslint), and the plugin's `unparsedModule` report for a `.js` oracle file that imports TypeScript (#13). - Design: the interim exit-wording bullet is gone with #12; the rule not to write those words in a report stays. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 47 +++++++++++++++---- .../skills/oracles-evaluate-review/SKILL.md | 7 ++- .../oracles/skills/oracles-review/SKILL.md | 2 + .../oracles/skills/oracles-setup/SKILL.md | 12 +++-- 4 files changed, 54 insertions(+), 14 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index f6d5232..1f0bb3c 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -67,6 +67,41 @@ its promise. purpose as their authority. - **The task already approved these laws?** Do not ask again. +**Write the general law down, from the examples.** Docs, tests, and bug +reports give examples. A law is a statement over the whole domain. Before the +table, write this block for each law, in the sheet itself: + +1. **Examples:** each concrete case, with its source. For example: + `ms('2 hrs')` is `7200000` (README, Examples). +2. **What varies, what stays:** for each example, name the parts that could + change without changing the expected answer (the number, the spelling, + the spacing). Then name the part that decides it (the unit's length). +3. **The law:** one sentence over the domain, in the form "For every ... , + ...". Name the domain's edges. For example: "For every finite number `n` + and every unit name `u` in the table, in any case and with or without a + space, `parse(n + u)` is `n` times the length of `u`." +4. **A new case the law predicts:** one input that is in none of the sources, + with its predicted result. If you cannot write one, the law is still a list + of examples. Write it again. +5. **Rebuild the examples:** derive every example from item 1 using only the + law's sentence. Name any example that the law gets wrong. A wrong one + means a missing condition, a second law, or a source that disagrees. Say + which. + +A law written only as examples, or one that cannot rebuild its examples, is +not ready for the sheet. + +**Use the [design grammar](../../docs/instruments/design-grammar.md) card +when any of these is true.** Do not judge whether the law is hard: + +- the sheet has more than five laws for one boundary +- a law has no new case that it predicts (item 4 above) +- the model has more than one branch per law +- the work changes or simplifies an existing model +- the gap check leaves three or more gaps in one function + +Say in the evidence which trigger fired, or that none did. + Write one row per law and give the sheet to the user. | Field | What to write | @@ -120,8 +155,7 @@ failure. Make each weakening its own decision, with its own reason. - When two rules conflict, record the conflict as an open question for the user. Do not pick a winner. -For a hard law or a model that keeps growing, use the -[design grammar](../../docs/instruments/design-grammar.md) card. +For the design grammar card, see the triggers in step 1. ## 3. Choose the model @@ -386,12 +420,9 @@ comes with one instruction. Do what it says, within the rules below. A narrower domain stated only in the law's prose, such as "positive integers", does not make a mutant invalid. Cover the type's whole domain, or narrow the type. -- **Temporary, until the report's messages change:** if a message calls a - mutant "equivalent", asks for a "maintainer's approval", or offers a limit - that "needs a maintainer's decision", do not follow that part. Find the - history and the law. Do not copy those words into your own report either: - for a survivor you cannot kill, give the histories you tried and what they - showed. +- **Never write "equivalent" or "needs a maintainer's decision" in your + report.** For a survivor you cannot kill, give the histories you tried and + what they showed, and follow its directive. - **Temporary, a CLI limitation:** a mutant that makes production loop forever synchronously shows as a timeout. The framework's breakage laws will judge it, and they are not built yet. Report it as a known CLI limitation, not an diff --git a/packages/oracles/skills/oracles-evaluate-review/SKILL.md b/packages/oracles/skills/oracles-evaluate-review/SKILL.md index 1228a7a..a258721 100644 --- a/packages/oracles/skills/oracles-evaluate-review/SKILL.md +++ b/packages/oracles/skills/oracles-evaluate-review/SKILL.md @@ -87,8 +87,11 @@ driver path, or observation. A reviewer's bug shows one history. The oracle must cover the family it belongs to. -- **State the causal condition:** the relationship that makes the history - fail, such as "a completion outlives the operation that started it". +- **State the causal condition, in writing:** the relationship that makes + the history fail, as one sentence over histories: "For every history where + a completion outlives the operation that started it, ...". Then write one + history, different from the reported one, that the sentence says also + fails, and run it. - **Add it to the grammar:** a generator dimension or a constructed history that varies the incidental details (counts, order, values, timing) and keeps the condition. A pinned example alone is not the fix. Keep the diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index 2f7f654..f72e6be 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -42,6 +42,8 @@ contains one is not ready. | Shortcut | What to look for | | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | +| **Law as examples** | A law written as a list of cases, not one sentence over the domain. Or a sheet with no new case that the law predicts. | +| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired, and the design grammar card was not used. | | **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | | **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | | **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 77df71b..5819938 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -130,10 +130,14 @@ export default [ - If the repository lints with xo, keep the oracle files and the new config files out of xo (its `ignores`), and lint them with the oracle ESLint config. -- **Temporary, a known plugin bug:** under TypeScript 7 the plugin cannot load. - Skip the oracle lint, and report "known plugin bug: plugin cannot load under - TypeScript 7 (fix in progress)". This is not an oracle limit, and not a - reason to stop. +- **TypeScript 7:** typescript-eslint does not support TypeScript 7 yet. If the + repository's ESLint config cannot parse TypeScript, the oracle lint cannot + run. Report it as an environment limit, with that reason. Do not change the + repository's TypeScript. +- The plugin parses the modules an oracle file imports with that file's + parser. A `.js` oracle file that imports a TypeScript module gets an + `unparsedModule` report: write the oracle file in TypeScript, with a + TypeScript parser. - If ESLint does not already parse TypeScript, add `@typescript-eslint/parser` as a dev dependency. Set it as `languageOptions.parser` for every TypeScript file that the presets cover: the oracle files and the production source. From 3ac23b96375f7d5ad661fbb7e2d85c0ed402b2c3 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 19:44:09 -0700 Subject: [PATCH 16/76] docs(oracles): boundary-condition thinking and adjacent forms - Design step 1, item 6: sort each law into dynamics (how things change, stated by the model's rules), constraints (what holds in every history), and boundary conditions (the edges, where real behavior and most defects live). Every boundary gets a constructed history or an open question; it is never left out of the grammar. Every defect the validation agents found was at a boundary. - Adjacent forms keep design diversity: the design-grammar card's new step 8 sketches two to four forms that keep every adopted law, for the user to pick; design step 3 names a second fitting model style; design step 7 sketches alternative cleanup designs instead of keeping the first. - Glossary: dynamics, constraint, boundary condition, adjacent form. Co-authored-by: Isaac --- packages/oracles/docs/glossary.md | 30 +++++++++++-------- .../docs/instruments/design-grammar.md | 12 +++++++- .../oracles/skills/oracles-design/SKILL.md | 21 +++++++++++++ 3 files changed, 49 insertions(+), 14 deletions(-) diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md index f7c96f3..99ae460 100644 --- a/packages/oracles/docs/glossary.md +++ b/packages/oracles/docs/glossary.md @@ -6,19 +6,23 @@ term for one concept. Do not rotate synonyms for style. ## Laws and evidence -| Term | Meaning | -| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | -| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | -| limits | What a property does not establish. Unresolved parts of a law go here. | -| law sheet | The first output of oracle design: each law with its authority, status, rival, distinguishing witness, and checkpoint. A human reviews it before any code. | -| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | -| rival | A plausible different law that predicts a different observation for some history. | -| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | -| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | -| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | -| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | -| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | +| Term | Meaning | +| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | +| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | +| limits | What a property does not establish. Unresolved parts of a law go here. | +| law sheet | The first output of oracle design: each law with its authority, status, rival, distinguishing witness, and checkpoint. A human reviews it before any code. | +| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | +| rival | A plausible different law that predicts a different observation for some history. | +| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | +| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | +| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | +| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | +| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | +| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | +| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the user can pick. | +| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | +| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | ## The parts of an `oracleProperty` diff --git a/packages/oracles/docs/instruments/design-grammar.md b/packages/oracles/docs/instruments/design-grammar.md index 185f2f1..143be9f 100644 --- a/packages/oracles/docs/instruments/design-grammar.md +++ b/packages/oracles/docs/instruments/design-grammar.md @@ -84,7 +84,16 @@ of range. Do not change the law, a history filter, and an observation in one step. Each needs its own comparison. -8. **Optionally check a formal model.** For a consequential claim about state, +8. **Offer adjacent forms.** Keep design diversity. From the parts and rules, + sketch two to four forms that keep every adopted law but differ in a real + way: another representation, another allowed-set choice, another split of + laws across oracles, or another production structure that the same oracle + accepts. Give each form's route, what it keeps, what it changes, and its + cost. Merge forms that differ only in naming or detail. Do not rank them + unless asked. When the choice is the user's, present the forms and let the + user pick. + +9. **Optionally check a formal model.** For a consequential claim about state, concurrency, retries, or ownership, translate the reduced model into TLA+ or a similar model checker. - Add state only where it changes allowed future behavior. Write derived @@ -109,6 +118,7 @@ of range. - the parts, each with its job and a witness - the rules with their sources, and the open conflicts - the reduced model, and the comparison for each simplification +- the adjacent forms, each with what it keeps, changes, and costs - formal-check results with their mode, when run - the limits: what the work does not establish diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 1f0bb3c..5a5e90c 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -87,6 +87,18 @@ table, write this block for each law, in the sheet itself: law's sentence. Name any example that the law gets wrong. A wrong one means a missing condition, a second law, or a source that disagrees. Say which. +6. **Dynamics, constraints, and boundary conditions:** sort what the law + says into three kinds. + - **Dynamics:** how the result or the state changes with each action or + input. These are the model's rules, and the histories vary them. + - **Constraints:** what holds in every history, whatever happens: a count + never above its limit, an equality that is symmetric, a read that returns + the last write. The check tests them at every checkpoint. + - **Boundary conditions:** the edges, where real behavior and most defects + live: empty, zero, `-0`, the size limit, at capacity, the first and the + last item, keys that shadow prototype names, cleanup. List them. Give + each one a constructed history. A boundary that the law does not decide + is an open question, not something to leave out of the grammar. A law written only as examples, or one that cannot rebuild its examples, is not ready for the sheet. @@ -183,6 +195,10 @@ can state it: Mixing styles is fine. [Model styles](../../docs/model-styles.md) shows a small example of each style. +When two styles fit the law, sketch both briefly, keep the one that states +the law most directly, and name the other in the law block, so a reader can +choose it later. + **Structure rules:** - One dispatch per role. The resolver, model, and driver may each branch on @@ -377,6 +393,11 @@ it to simplify: reason. - Prefer a change that shrinks the code, or leaves its size the same. If the repository sets a size rule, such as a net-negative diff, follow it. +- For a larger cleanup, sketch two or three designs that pass the same oracle + (a table instead of branches, one state field instead of three, a + recompute instead of a cache). Give each one's trade-off, and let the user + choose when the choice is theirs. Do not keep only the first design you + find. - Rerun the oracle and the gap check after each cleanup. Techniques: [guide § Evidence](../../docs/guide.md#evidence). From e50349ec8bd23fc50281c6954dddd8f92b1bc707 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 20:13:40 -0700 Subject: [PATCH 17/76] docs(oracles): fill-in law sheet and round-4 validation fixes The law sheet is now a template that the agent fills in for each law. Its examples, law sentence, new case, and dynamics, constraints, and boundary conditions are fields, and the summary table is only an index. In round 4, one of four agents skipped the written block because it was prose before a table. Other fixes from round 4 (validation findings V64 to V96): - design: arbitraries must take histories back (`.map` needs its inverse); string units; observations are data; brand checks, not instanceof; methods of a returned object are owned through the enclosing function; the whole-file entry only when module data affects a law; authored mutants when operators are sparse; triggers checked again after each gap check, counting every row. - setup: the no-user default covers setup questions only; the end condition is the first oracle's own contract; keep an existing Vitest include; register the plugin in a custom probe-form block; xo and TS 7 installs; the live-rule probe needs the fast-check import; placeholders on a stopped run; scratch copy wording. - review: the shortcut rows match the template and the later trigger check. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 200 +++++++++++------- .../oracles/skills/oracles-review/SKILL.md | 36 ++-- .../oracles/skills/oracles-setup/SKILL.md | 63 ++++-- 3 files changed, 183 insertions(+), 116 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 5a5e90c..2a2c282 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -10,7 +10,8 @@ checks production against an independent model. Make the law clear enough that someone could predict an observation without reading production. The code comes after that. -Read this skill in full, then read the [glossary](../../docs/glossary.md). +Read this skill in full, then read the [glossary](../../docs/glossary.md) +in full. It defines the terms that the sheet and the evidence use. The [checklist](../../docs/oracle-checklist.md) is the complete list of requirements. This skill cites it by ID, such as ORC-004. @@ -50,97 +51,110 @@ its promise. law's stated domain is strengthening. Changing the stated domain, a checkpoint, or `limits` is a law change. - **Changing a law** (its promise, domain, checkpoint, assumptions, or the - freedoms it allows) needs a law delta: the old and new row for each changed - law. Stop for the user's review. + freedoms it allows) needs a law delta: the old and new block for each + changed law. Stop for the user's review. - **A new law, or missing or contradictory authority,** needs the full law sheet below. Stop for the user's review. An analogy to another documented rule is not authority: mark that law `open`. -**Laws people forget to write down:** +**Write the law sheet from this template.** Docs, tests, and bug reports give +examples. A law is a statement over the whole domain. The template makes you +write the step from one to the other. Copy it once for each law, and fill in +every field. The template is the sheet. Do not replace it with a table. + +```markdown +### L1 + +- Examples: +- What varies, what stays: +- Law: For every , . +- Domain edges: +- New case: +- Rebuilt examples: +- Dynamics: +- Constraints: +- Boundary conditions: +- Authority: +- Status: adopted | observed | open +- Rival: +- Witness: +- Checkpoint: +``` + +How to fill in the fields: + +- **Examples** and **What varies, what stays:** for example, `ms('2 hrs')` + is `7200000` (README, Examples). The number, the spelling, and the spacing + can change. The unit's length decides the answer. +- **Law:** one sentence over the domain. For example: "For every finite + number `n` and every unit name `u` in the table, in any case and with or + without a space, `parse(n + u)` is `n` times the length of `u`." +- **New case:** if you cannot write one, the law is still a list of + examples. Write it again. +- **Rebuilt examples:** an example that the law gets wrong shows a missing + condition, a second law, or a source that disagrees. Say which. +- **Dynamics** are the model's rules, and the histories vary them. For a pure + function, write "none: one call", and put the variation into the domain. +- **Constraints** hold whatever happens: a count never above its limit, an + equality that is symmetric, a read that returns the last write. The check + tests them at every checkpoint. +- **Boundary conditions** are the edges, where real behavior and most + defects live: empty, zero, `-0`, the size limit, at capacity, the first and + the last item, keys that shadow prototype names, cleanup. A boundary that + the law does not decide goes under open questions. Do not leave it out of + the grammar. +- **Status:** `adopted` if it has an authority, `observed` if production does + it but nobody adopted it, `open` if it is a question. Only `adopted` laws + become expected results. An `observed` or `open` law goes to the user as a + question. +- **Production output is evidence, never authority.** A rule that you write + only because production does it is model hacking. Mark that rule + `observed`, and ask the user. An analogy to another documented rule is not + authority either: mark that law `open`. + +**Required blocks: the laws people forget.** The sheet has a block for each +of these, or one line that says why it does not apply: - **No throw on valid input.** When the contract says a call returns a value, it promises not to throw for valid input. State that as a law, with the same authority, so a mutant that throws is a kill, not a crash. - **The laws of the kind of function.** An equality check is reflexive and symmetric. A parser and a serializer round-trip. A setter is idempotent when - set to the same value. Give these as rows, with the function's documented - purpose as their authority. -- **The task already approved these laws?** Do not ask again. - -**Write the general law down, from the examples.** Docs, tests, and bug -reports give examples. A law is a statement over the whole domain. Before the -table, write this block for each law, in the sheet itself: - -1. **Examples:** each concrete case, with its source. For example: - `ms('2 hrs')` is `7200000` (README, Examples). -2. **What varies, what stays:** for each example, name the parts that could - change without changing the expected answer (the number, the spelling, - the spacing). Then name the part that decides it (the unit's length). -3. **The law:** one sentence over the domain, in the form "For every ... , - ...". Name the domain's edges. For example: "For every finite number `n` - and every unit name `u` in the table, in any case and with or without a - space, `parse(n + u)` is `n` times the length of `u`." -4. **A new case the law predicts:** one input that is in none of the sources, - with its predicted result. If you cannot write one, the law is still a list - of examples. Write it again. -5. **Rebuild the examples:** derive every example from item 1 using only the - law's sentence. Name any example that the law gets wrong. A wrong one - means a missing condition, a second law, or a source that disagrees. Say - which. -6. **Dynamics, constraints, and boundary conditions:** sort what the law - says into three kinds. - - **Dynamics:** how the result or the state changes with each action or - input. These are the model's rules, and the histories vary them. - - **Constraints:** what holds in every history, whatever happens: a count - never above its limit, an equality that is symmetric, a read that returns - the last write. The check tests them at every checkpoint. - - **Boundary conditions:** the edges, where real behavior and most defects - live: empty, zero, `-0`, the size limit, at capacity, the first and the - last item, keys that shadow prototype names, cleanup. List them. Give - each one a constructed history. A boundary that the law does not decide - is an open question, not something to leave out of the grammar. - -A law written only as examples, or one that cannot rebuild its examples, is -not ready for the sheet. + set to the same value. Their authority is the function's documented purpose. + +The task already approved these laws? Do not ask again. + +After the blocks, list the **open questions**. Then add a summary table with +one row per block: law, status, authority, and checkpoint. The table is only +an index to the blocks. + +When the promise is unclear, or two readings predict different behavior, use +the [law discovery](../../docs/instruments/law-discovery.md) card. **Use the [design grammar](../../docs/instruments/design-grammar.md) card -when any of these is true.** Do not judge whether the law is hard: +when any of these is true.** Do not judge whether the law is hard. Check the +list at the sheet, and again after each gap check: -- the sheet has more than five laws for one boundary -- a law has no new case that it predicts (item 4 above) +- the sheet has more than five rows for one boundary (adopted, observed, and + open rows all count) +- a law has no new case that it predicts - the model has more than one branch per law - the work changes or simplifies an existing model - the gap check leaves three or more gaps in one function Say in the evidence which trigger fired, or that none did. -Write one row per law and give the sheet to the user. - -| Field | What to write | -| ---------- | --------------------------------------------------------------------------------------------------------------------- | -| Law | The promise, stated so that a reader could predict an observation. | -| Authority | The API or contract document and section, or a dated maintainer decision. | -| Status | `adopted` if it has an authority, `observed` if production does it but nobody adopted it, `open` if it is a question. | -| Rival | A plausible different law. | -| Witness | A legal history where the law and its rival predict different observations. | -| Checkpoint | Where the contract owes the observation. | - -- **Production output is evidence, never authority.** A rule that you write - only because production does it is model hacking. Mark that rule - `observed`, and ask the user. -- Only `adopted` laws become expected results. An `observed` or `open` law - goes to the user as a question. -- When the promise is unclear, or two readings predict different behavior, use - the [law discovery](../../docs/instruments/law-discovery.md) card. - -**Stop here** when the sheet or a law delta is needed. Do not write the oracle -until the user has reviewed it. +**Stop here** when the sheet or a law delta is needed. Do not stop with an +empty field: each law has a filled block first. Do not write the oracle until +the user has reviewed the sheet. With no user present, write the sheet and +stop. A law sheet has no default. Once the user approves it, the sheet becomes the **law block**: the opening -comment of the oracle file, beside the model that enacts it. Each row's law, -authority, and limits also go in that property's `law` option. Keep the -block, the `law` option, and the model in step: a change to one updates the -others in the same change (ORC-003). +comment of the oracle file, beside the model that enacts it. The blocks can +be shorter there, but keep each law sentence, its authority, and its limits. +Each law's sentence, authority, and limits also go in that property's `law` +option. Keep the block, the `law` option, and the model in step: a change to +one updates the others in the same change (ORC-003). ## 2. Freeze what the oracle must preserve @@ -266,9 +280,21 @@ in the model. one a constructed history that enters it (ORC-015). Put a state that you only count in `measured`. A literal example or a `fixed` history is not reach. A constructed history must be one the arbitrary could generate, - within its bounds. Build its values from the generated history (for example - "the initial value plus one"), not from constants that the generated values - might equal. + within its bounds. Use a constant when the arbitrary can generate it and it + cannot collide with a generated value (a fixed boundary shape). Otherwise + derive it from the generated history (for example "the initial value plus + one"). +- **Every arbitrary must take its histories back.** fast-check replays each + `fixed` and constructed history through the arbitrary. A `.map` with no + inverse cannot do this, and every such history fails at run time with + `canShrinkWithoutContext`. Give each `.map` its inverse + (`.map(f, unmap)`, where `unmap` throws on a value it cannot map back), or build from `fc.record`, `fc.constantFrom`, and plain + values. Run the oracle once right after you write each constructed history. +- **Choose the string unit on purpose.** fast-check's unicode strings include + lone surrogates, and slicing by code unit splits astral characters. Use + `fc.string({ unit: 'grapheme' })` or code points when the law is about + characters. When the law is about any JavaScript string, lone surrogates + are a boundary condition that the law must decide. - **Cover the law's whole domain.** A mutant can only show what the histories vary. If the grammar covers part of the domain the law states (for example, fractions in quarter steps only), say so in `limits`. Better, widen it. @@ -312,6 +338,13 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). observations, so the check can reject them. A mutant that crashes production is not a kill. - It observes the contract's return values. + - It records only data: primitives other than symbols, plain arrays and + objects, Maps, Sets, and Errors. A symbol, a Date, a function, or a class + instance throws `OracleDataError`. Convert it first, for example to its + description or its time value. + - It identifies kinds with brand checks (`node:util` `types`, or + `Object.prototype.toString`), not `instanceof`, when a law is about + prototypes. `instanceof` follows a corrupted prototype chain. - It awaits the work it starts. - It bounds its own run time and work, and throws a structured error when a bound is passed. @@ -327,9 +360,14 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). (`src/x.ts#run`, `src/x.ts#Class.method`). Own only what the laws cover. The gap check judges only owned code, and the gap check's `--source` selects files. - - If an owned function reads module-level data (a lookup table, a constant - object), own the whole file for now. Data outside a function is not yet - part of a function entry's tree. + - A method of an object that a function returns (`mitt().emit`) has no name + of its own. Own it through the enclosing function (`src/index.ts#mitt`). + That entry also brings in the sibling methods: cover them, or report their + gaps as outside this oracle's scope. + - Temporary: data outside a function (a lookup table, a constant object) is + not yet part of a function entry's tree. Own the whole file only when + that data affects an adopted law, such as a unit table that decides the + result. A constant that feeds no law does not need it. - When a whole-file entry pulls in another contract (for example a batching API in the same file), write that contract's own oracle, or report its gaps as outside this oracle's scope. @@ -456,6 +494,14 @@ comes with one instruction. Do what it says, within the rules below. - **Temporary:** in a package without `"type": "module"`, Vite may warn while loading the gap check's own config copies. Ignore that warning. +- **Few mutants are weak evidence.** If the gap check generates few mutants + for an owned function, or none on a law's decisions (such as a return + expression), write authored mutants for the laws it misses. Put them in a + directory and pass it with `--mutants `. The form is in the + [mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) card. +- **Three or more gaps in one function** fire the design grammar trigger + (step 1). Apply the card before you add histories. + Use the [mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) card for a survivor whose missing history is not obvious. diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index f72e6be..215c811 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -39,24 +39,24 @@ checkpoint. A final state can hide a wrong intermediate observation. These are the shortcuts. Report each one as a finding. A change that contains one is not ready. -| Shortcut | What to look for | -| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | -| **Law as examples** | A law written as a list of cases, not one sentence over the domain. Or a sheet with no new case that the law predicts. | -| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired, and the design grammar card was not used. | -| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | -| **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | -| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | -| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | -| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. Also a state that a law depends on but that sits in `measured`, where nothing requires it. | -| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | -| **Pinned only** | A fixed example where the law needs a generator dimension. | -| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | -| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | -| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | -| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | -| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | -| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | +| Shortcut | What to look for | +| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | +| **Law as examples** | A law written as a list of cases, not one sentence over the domain. A sheet with no new case that the law predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | +| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired at the sheet or after a gap check, and the design grammar card was not used. | +| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | +| **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | +| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | +| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | +| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. Also a state that a law depends on but that sits in `measured`, where nothing requires it. | +| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | +| **Pinned only** | A fixed example where the law needs a generator dimension. | +| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | +| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | +| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | +| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | +| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | +| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | The evidence counts only as a set, chosen by the kind of change (`oracles-design` step 7): law, reach, before-and-after runs, mutant, gaps, diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 5819938..0658cae 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -8,11 +8,12 @@ description: Set up @tanstack/oracles in a repository. Install the pinned packag You are setting up oracle testing in this repository. Do the steps in order. Ask the user before each step that writes outside the repository. When no user is available, choose the default this skill gives, record each choice in -your report, and continue. +your report, and continue. That default covers setup questions only. A law +sheet has no default: write it, report it, and stop. -Setup ends when one oracle passes on production and the gap check has judged -it with no gap left open, unless the user decides otherwise. It does not end -when the packages are installed. +Setup ends when one oracle passes on production and the gap check leaves no +gap open in that oracle's own contract, outside its stated limits, unless the +user decides otherwise. It does not end when the packages are installed. ## 1. Read the repository @@ -28,7 +29,8 @@ Find out, and tell the user: `.eslintrc`, or no ESLint (for example Biome only). - The Node version that CI and the `engines` field use. The package needs Node 20 or later. If CI runs an older Node, tell the user, and offer a - separate Node 20 job for the oracles. Do not change the existing jobs. + separate Node 20 job for the oracles. Do not change the existing jobs. With + no user present, do not change CI: report the mismatch. - The repository's TypeScript version. Do not change it (step 2). - The package manager and its runner for package binaries: `pnpm exec`, `npx`, or `yarn`. Use them wherever this skill shows `pnpm add` or @@ -92,7 +94,9 @@ only has to match the Vitest config and the lint glob. The gap check loads only - If the repository has no Vitest config, create `vitest.config.ts` for the oracle files. Jest and mocha ignore that name. -- If it has one, add the oracle files to its `include`. +- If it has one, add the oracle glob to its existing `include`. Keep the + globs that are there: replacing them drops the existing suite, since an + `include` replaces Vitest's default. ```ts // vitest.config.ts @@ -126,23 +130,33 @@ export default [ so a disable comment has no effect. - `production` checks the probe form in `src/**` and `packages/*/src/**`. If production source lives elsewhere (for example a root-level `index.js`), add - the `oracle/probe-form` rule with that glob. + the rule with that glob. The config object must register the plugin itself: + + ```js + { files: ['index.js'], plugins: { oracle: oracles }, rules: { 'oracle/probe-form': 'error' } } + ``` + - If the repository lints with xo, keep the oracle files and the new config files out of xo (its `ignores`), and lint them with the oracle ESLint - config. + config. Do not install `eslint@^9` as a dev dependency: xo loads ESLint + from the hoisted copy, and a second major breaks it. Use the `npx -p +eslint@9` fallback below instead. - **TypeScript 7:** typescript-eslint does not support TypeScript 7 yet. If the repository's ESLint config cannot parse TypeScript, the oracle lint cannot - run. Report it as an environment limit, with that reason. Do not change the - repository's TypeScript. + run. Installing `@typescript-eslint/parser` fails with `ERESOLVE`, since its + TypeScript peer range ends before 7. Do not force the install. Report it as + an environment limit, with that reason. Do not change the repository's + TypeScript. - The plugin parses the modules an oracle file imports with that file's parser. A `.js` oracle file that imports a TypeScript module gets an `unparsedModule` report: write the oracle file in TypeScript, with a TypeScript parser. - If ESLint does not already parse TypeScript, add `@typescript-eslint/parser` as a dev dependency. Set it as `languageOptions.parser` for every TypeScript - file that the presets cover: the oracle files and the production source. - Otherwise a typed production file fails to parse before the probe rule - runs. + file that the presets cover: the oracle files, and the production source if + it is TypeScript. Otherwise a typed production file fails to parse before + the probe rule runs. For JavaScript production source, set the parser for + the oracle files only. - Do not add `legacyCampaigns` or any other exemption. Adopters have none. **An older ESLint (7 or 8, `.eslintrc`).** Do not upgrade the repository's @@ -159,9 +173,10 @@ The config's parser import resolves from the repository, so it uses the repository's own `@typescript-eslint/parser`. Run the linter once and confirm that it loads the rules. Then check that the -rules are live: a deliberately wrong line in an oracle file (for example -`fc.assert(...)`) must produce an error. Remove that line again. Before the -first oracle file exists, do this check through the linter's `--stdin` with +rules are live: a deliberately wrong line in an oracle file must produce an +error. Use `import fc from 'fast-check'` and then `fc.assert(...)`. The rule +follows the fast-check import, so an `fc` that is not imported passes. +Remove the probe again. Before the first oracle file exists, do this check through the linter's `--stdin` with an oracle file name, since a glob with no files fails. ## 4. Write the AGENTS.md block @@ -191,7 +206,8 @@ This repository tests its laws with `@tanstack/oracles`. ``` -Fill in the two placeholders in step 7, once their values exist. If the repository has a +Fill in the two placeholders in step 7, once their values exist. A run that +stops before step 7 leaves them in place, and its report says so. If the repository has a `CLAUDE.md` and no `AGENTS.md`, ask the user which file gets the block. In a workspace with a package-local install, put the block in the package's @@ -229,7 +245,7 @@ Write a pointer skill as `//SKILL.md` for each of ```md --- name: oracles-design -description: +description: --- Read `/skills/oracles-design/SKILL.md` in full, and follow it. @@ -265,7 +281,8 @@ If the first oracle fails on unchanged production under an adopted law, it found a defect. The RED is the finding: report it with its replay line, and tell the user. Never weaken the oracle to make it pass. The gap check cannot judge a failing baseline yet, so make the smallest local fix on a scratch -branch, run the gap check there, and say in the report that it ran on the +copy or a local branch (never pushed or committed to the user's branch), run +the gap check there, and say in the report that it ran on the patched code. With the user's approval, follow `oracles-design` step 7's defect path for the real fix. @@ -276,7 +293,8 @@ skip that step to finish sooner. Put the oracle's `owns` list on its `oracleProperty` call. An entry is a file (`src/index.ts`) or one function in it (`src/index.ts#parse`, or `src/x.ts#Class.method`). A function entry owns that function and its static -call tree. For a first oracle, own only the function it covers. If the gap check counts **0 oracle files**, the +call tree. For a first oracle, own only the function it covers. Own the whole file only +when module data affects an adopted law (see `oracles-design` step 6). If the gap check counts **0 oracle files**, the oracle is not declared. That is a setup failure, even when the report shows 0 gaps and the command exits 0. @@ -326,7 +344,10 @@ Tell the user: - inside a stated limit, with the limit - framework or CLI limitations (for example a timeout from a synchronous infinite loop, or an exhaustiveness-guard mutant) + - other contracts that a whole-file entry pulled in, listed as outside + this oracle's scope - every other gap, with its instruction - any defect the oracle found, with its witness -Do not call setup done while a gap is open, unless the user decides otherwise. +Do not call setup done while a gap is open in the oracle's own contract, +outside its stated limits, unless the user decides otherwise. From 23de7372b94d940c79ac9c1af9364c3f90f7bdc8 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 20:15:38 -0700 Subject: [PATCH 18/76] docs(oracles): interim rules for per-law kill credit and hoisted TypeScript The framework credits a kill to the first law that a history violates, so a later law can look idle. The design skill now says to aim hand mutants at that law before restating or deleting it. The setup skill reports a TypeScript copy that npm hoisted from @tanstack/oracles as a known packaging bug, so it is not taken as the repository's own TypeScript. Both are temporary: the framework fixes are queued. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 3 +++ packages/oracles/skills/oracles-setup/SKILL.md | 10 ++++++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 2a2c282..f48917e 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -457,6 +457,9 @@ comes with one instruction. Do what it says, within the rules below. - **A law that kills nothing** owns no mutated code, or its witness never varies in the histories. Check its witness and its constructed history. If the law still kills nothing, it may restate another law: compare the two. + Temporary: a kill is credited only to the first law that a history + violates, so a later law can look idle. Before you restate or delete it, + run hand mutants aimed at that law alone. - **The only exclusions are probe-only code and the `super()` region of a derived constructor.** - **A survivor that the adopted law allows is an allowed alternative.** If a diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 0658cae..7faf99c 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -60,8 +60,14 @@ Never add, upgrade, or downgrade the repository's `typescript` for oracles. The gap check uses its own TypeScript and never reads the project's, from commit `d8adfee` on. Pin a commit at or after `d8adfee`. Oracle files need TypeScript 5 or newer. If the repository's `tsc` is older, keep the oracle -files and the Vitest config out of its run (its `include` or `exclude`). Report any typecheck errors -that existed before setup as pre-existing. +files and the Vitest config out of its run (its `include` or `exclude`). +Report any typecheck errors that existed before setup as pre-existing. + +Temporary, a known packaging bug: `@tanstack/oracles` depends on +`typescript`, and npm can hoist that copy to the repository root. A +repository that declares no `typescript` then gets one, and `npx tsc` +resolves to it. Report it as this bug. Do not treat that copy as the +repository's TypeScript, and do not add a typecheck that uses it. Use the repository's own package manager. With pnpm 11, an install that skips a dependency's build script stops every later pnpm command From 949c4f7293d20e49e0a8645d449937d66809ff31 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 22:40:22 -0700 Subject: [PATCH 19/76] docs(oracles): round-5 fixes, prose law blocks, and the ownership split The law block in an oracle file is now prose for a reader who has not seen the sheet: a question for a title, how the code works, the documented promises, how the model states them, and what the driver observes. The sheet's fields stay in the review record. The AGENTS.md block (template 2) says who owns what. The maintainers own the laws. The agent owns the production code that the oracles cover, and makes it the smallest, clearest code that passes them. Round-5 validation fixes: - A stated limit no longer closes a gap. Setup's end condition, its report groups, design step 8, and the review's exit row now say so. - A block whose domain is its own example is a fixed history, not a law. - The string-unit advice names `binary` (every code point except lone surrogates), since `grapheme` has no control characters. - A guard that TypeScript narrowing makes constant (the last member of an exhaustive union) is a known CLI bug. Other mutants on its line still count. - A `not-observed` survivor can need a history where the branches diverge. - The design-grammar trigger after a gap check is a stop, not a note. - An idle law that another law covers is merged. No empty tests in oracle files. `vitest.config.mts` in a CommonJS package. The AGENTS.md lint line when lint cannot run. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 77 ++++++++++++++----- .../oracles/skills/oracles-review/SKILL.md | 36 ++++----- .../oracles/skills/oracles-setup/SKILL.md | 34 +++++--- 3 files changed, 100 insertions(+), 47 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index f48917e..74d2fcb 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -90,7 +90,11 @@ How to fill in the fields: number `n` and every unit name `u` in the table, in any case and with or without a space, `parse(n + u)` is `n` times the length of `u`." - **New case:** if you cannot write one, the law is still a list of - examples. Write it again. + examples. Write it again. A new case that only renames a value in the + example (`/:id` for `/:foo`) does not count. +- **Domain edges:** wider than the examples. A block whose domain is only its + own examples is not a law. Keep those examples as `fixed` histories, and as + rebuilt examples of the general law that explains them. - **Rebuilt examples:** an example that the law gets wrong shows a missing condition, a second law, or a source that disagrees. Say which. - **Dynamics** are the model's rules, and the histories vary them. For a pure @@ -150,10 +154,24 @@ the user has reviewed the sheet. With no user present, write the sheet and stop. A law sheet has no default. Once the user approves it, the sheet becomes the **law block**: the opening -comment of the oracle file, beside the model that enacts it. The blocks can -be shorter there, but keep each law sentence, its authority, and its limits. -Each law's sentence, authority, and limits also go in that property's `law` -option. Keep the block, the `law` option, and the model in step: a change to +comment of the oracle file. The oracle file is a literate program (ORC-003), +so write the block as prose for a reader who has not seen the sheet. Do not +paste the template's fields or a list of law IDs. In this order: + +1. A title that asks the question the oracle answers, for example "Does a + copy keep every value, and share nothing with its source?" +2. How the code works at this boundary, and what goes wrong there, in plain + sentences. +3. What the documentation promises, in its own words where you can, with the + source of each promise. These are the laws. +4. How the model states those promises, and what it reads to predict them. +5. What the driver does, what it observes, and what it does not observe and + why. Then the limits, each with its reason. + +The prose beside the model, the histories, the driver, and the check says why +each one predicts or tests the promised result. The sheet's examples, rivals, +and witnesses stay in the review record, not the file. Each law's sentence, +authority, and limits also go in that property's `law` option. Keep the block, the `law` option, and the model in step: a change to one updates the others in the same change (ORC-003). ## 2. Freeze what the oracle must preserve @@ -290,11 +308,14 @@ in the model. `canShrinkWithoutContext`. Give each `.map` its inverse (`.map(f, unmap)`, where `unmap` throws on a value it cannot map back), or build from `fc.record`, `fc.constantFrom`, and plain values. Run the oracle once right after you write each constructed history. -- **Choose the string unit on purpose.** fast-check's unicode strings include - lone surrogates, and slicing by code unit splits astral characters. Use - `fc.string({ unit: 'grapheme' })` or code points when the law is about - characters. When the law is about any JavaScript string, lone surrogates - are a boundary condition that the law must decide. +- **Choose the string unit that covers the law's domain.** fast-check's + default strings include lone surrogates, and slicing by code unit splits + astral characters. `fc.string({ unit: 'binary' })` gives every code point + except lone surrogates, control characters included. `unit: 'grapheme'` + gives printable graphemes only, so it leaves out control characters, and + a `fixed` history that uses one is refused. When the law is about any + JavaScript string, lone surrogates are a boundary condition that the law + must decide. - **Cover the law's whole domain.** A mutant can only show what the histories vary. If the grammar covers part of the domain the law states (for example, fractions in quarter steps only), say so in `limits`. Better, widen it. @@ -447,8 +468,11 @@ comes with one instruction. Do what it says, within the rules below. - **There are no equivalent mutants.** A surviving mutant shows a missing history, observation, or law, or it shows dead code. -- **An effect that no check sees is one that the driver does not record.** - Record it. +- **An effect that no check sees** is one that the driver does not record, + or one that no history makes different. A changed condition can still lead + to the same result when both branches agree on every generated input. Then + add a history where they diverge. Otherwise, record the effect. + (Temporary: the `not-observed` directive names only the second case.) - **Code whose removal no legal history can observe is dead.** The gap check alone cannot show this, because its histories are a sample. First rule out a missing history, observation, or law. Then remove the code in the same @@ -457,8 +481,9 @@ comes with one instruction. Do what it says, within the rules below. - **A law that kills nothing** owns no mutated code, or its witness never varies in the histories. Check its witness and its constructed history. If the law still kills nothing, it may restate another law: compare the two. - Temporary: a kill is credited only to the first law that a history - violates, so a later law can look idle. Before you restate or delete it, + If every change that breaks it also breaks the other law, it restates + that law: merge them, as a law change. Temporary: a kill is credited only + to the first law that a history violates, so a later law can look idle. Before you restate or delete it, run hand mutants aimed at that law alone. - **The only exclusions are probe-only code and the `super()` region of a derived constructor.** @@ -468,6 +493,13 @@ comes with one instruction. Do what it says, within the rules below. strengthen the law or change production to kill it. It stays open until the framework recognizes allowed sets. The user decides whether that blocks the work. +- **A stated limit does not close a gap.** If a survivor differs from the + original only on inputs that the law's limits leave out (a null-prototype + object, say), the gap is open. Report it with that reason, and recommend + widening the grammar and the law to say what the code promises there. If + that behavior is truly not part of the contract, the code that the mutant + changes is a production question for the owner. Either way, the user + decides. - **A gap that existed before your change still counts** when it is in a function you changed. Report it with its law question. Do not take an exit. - **Never propose an exit.** If a gap can be neither killed nor removed, it is @@ -489,11 +521,14 @@ comes with one instruction. Do what it says, within the rules below. synchronously shows as a timeout. The framework's breakage laws will judge it, and they are not built yet. Report it as a known CLI limitation, not an oracle gap. Do not add driver timeouts for it, and keep working. -- **Temporary, a known CLI bug:** a survivor inside an exhaustiveness guard - (a branch with `x satisfies never`) should never have been generated. - Record it in your report as "CLI bug: exhaustiveness guard mutated", and - keep working on the other gaps. Do not stop, do not count it as an oracle - limit, and do not remove the guard: it is a compile-time check. +- **Temporary, a known CLI bug:** a gap on a guard that is constant under + TypeScript's narrowing is an invalid mutant that the CLI should not have + generated. Two forms: a branch after `x satisfies never`, and the test for + the last member of an exhaustive union (after the other members' branches + return or continue). Record it in your report under CLI limitations, with + its line, and keep working. Do not remove the guard and do not change + production: the guard protects untyped callers. Every other mutant on that + line must still be killed. - **Temporary:** in a package without `"type": "module"`, Vite may warn while loading the gap check's own config copies. Ignore that warning. @@ -503,7 +538,9 @@ comes with one instruction. Do what it says, within the rules below. directory and pass it with `--mutants `. The form is in the [mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) card. - **Three or more gaps in one function** fire the design grammar trigger - (step 1). Apply the card before you add histories. + (step 1). Stop, read the card, and write its parts list for that function + before you add a history. Adding histories first is the shortcut "Skipped + design grammar". Use the [mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) card for a survivor whose missing history is not obvious. diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index 215c811..da81a8b 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -39,24 +39,24 @@ checkpoint. A final state can hide a wrong intermediate observation. These are the shortcuts. Report each one as a finding. A change that contains one is not ready. -| Shortcut | What to look for | -| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | -| **Law as examples** | A law written as a list of cases, not one sentence over the domain. A sheet with no new case that the law predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | -| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired at the sheet or after a gap check, and the design grammar card was not used. | -| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | -| **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | -| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | -| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | -| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. Also a state that a law depends on but that sits in `measured`, where nothing requires it. | -| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | -| **Pinned only** | A fixed example where the law needs a generator dimension. | -| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | -| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | -| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | -| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | -| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | -| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | +| Shortcut | What to look for | +| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | +| **Law as examples** | A law written as a list of cases, not one sentence over the domain. A sheet with no new case that the law predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | +| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired at the sheet or after a gap check, and the design grammar card was not used. | +| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | +| **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | +| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | +| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | +| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. Also a state that a law depends on but that sits in `measured`, where nothing requires it. | +| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | +| **Pinned only** | A fixed example where the law needs a generator dimension. | +| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | +| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | +| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | +| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | +| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | +| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable, or that counts a gap as closed because a stated limit covers it. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | The evidence counts only as a set, chosen by the kind of change (`oracles-design` step 7): law, reach, before-and-after runs, mutant, gaps, diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 7faf99c..6a1ad42 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -12,8 +12,8 @@ your report, and continue. That default covers setup questions only. A law sheet has no default: write it, report it, and stop. Setup ends when one oracle passes on production and the gap check leaves no -gap open in that oracle's own contract, outside its stated limits, unless the -user decides otherwise. It does not end when the packages are installed. +gap open in that oracle's own contract, unless the user decides otherwise. A +stated limit does not close a gap. It does not end when the packages are installed. ## 1. Read the repository @@ -99,7 +99,9 @@ only has to match the Vitest config and the lint glob. The gap check loads only `vite.config.*`, so: - If the repository has no Vitest config, create `vitest.config.ts` for the - oracle files. Jest and mocha ignore that name. + oracle files. Jest and mocha ignore that name. In a package without + `"type": "module"`, name it `vitest.config.mts`, or Vite warns on every + run. - If it has one, add the oracle glob to its existing `include`. Keep the globs that are there: replacing them drops the existing suite, since an `include` replaces Vitest's default. @@ -191,12 +193,20 @@ Add this block to `AGENTS.md` (create the file if needed). On an upgrade, replace everything between the markers and keep the rest of the file. ```md - + ## Oracle tests This repository tests its laws with `@tanstack/oracles`. +**Who owns what.** The laws belong to the people who maintain this +repository: what the code promises, the authority for each promise, and its +limits. Propose a law, but do not adopt or change one without their +approval. The production code that the oracles own is yours. How it is built +is your call, and you will keep building on it. Make it the smallest, clearest +code that satisfies the oracles: merge repeated code, remove code that no law +needs (after the checks in `oracles-design` step 8), and restructure freely while the oracles pass. + - Before you create or change an oracle, or change production code that an oracle owns, read `node_modules/@tanstack/oracles/skills/oracles-design/SKILL.md` in full and follow it. @@ -212,7 +222,9 @@ This repository tests its laws with `@tanstack/oracles`. ``` -Fill in the two placeholders in step 7, once their values exist. A run that +If lint is an environment limit (step 3), replace the lint line with "Lint: +not available here ()." Fill in the two placeholders in step 7, +once their values exist. A run that stops before step 7 leaves them in place, and its report says so. If the repository has a `CLAUDE.md` and no `AGENTS.md`, ask the user which file gets the block. @@ -305,7 +317,8 @@ oracle is not declared. That is a setup failure, even when the report shows 0 gaps and the command exits 0. In an oracle file, only `oracleProperty` registers tests. A plain Vitest test -there is not a kill, and the gap check reports it. The driver records +there is not a kill, and the gap check reports it. Do not leave an empty test +there either: it checks nothing and looks like coverage. The driver records production errors as observations: a mutant that crashes production is not a kill until the check rejects the recorded error. @@ -347,13 +360,16 @@ Tell the user: - which pointers were written, and where - the first oracle's law and its result - the gap check's totals, and every gap left, grouped as: - - inside a stated limit, with the limit + - open because the inputs that expose it lie outside the law's stated + limits: name those inputs, and recommend widening the grammar and the + law. The user decides. Do not count these as closed. + - allowed alternatives (`oracles-design` step 8), apart from the kills - framework or CLI limitations (for example a timeout from a synchronous - infinite loop, or an exhaustiveness-guard mutant) + infinite loop, or a guard that TypeScript narrowing makes constant) - other contracts that a whole-file entry pulled in, listed as outside this oracle's scope - every other gap, with its instruction - any defect the oracle found, with its witness Do not call setup done while a gap is open in the oracle's own contract, -outside its stated limits, unless the user decides otherwise. +unless the user decides otherwise. From c0a7e71111a37b7eb0ec18580793734cafd03fe7 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 22:45:58 -0700 Subject: [PATCH 20/76] docs(oracles): write the oracle file's prose laws first, then the code The law sheet is now the new oracle file before it has any code: an opening comment that states the laws as prose (ORC-003), and a review-notes comment with the per-law template that turns examples into a general law. After approval, the review notes are deleted, and the code is added under the prose. A law delta edits the prose in place. This removes the step that copied an approved sheet into the file. Co-authored-by: Isaac --- packages/oracles/README.md | 2 +- packages/oracles/docs/glossary.md | 34 +++---- .../oracles/skills/oracles-design/SKILL.md | 90 +++++++++++-------- .../oracles/skills/oracles-review/SKILL.md | 4 +- .../oracles/skills/oracles-setup/SKILL.md | 6 +- 5 files changed, 75 insertions(+), 61 deletions(-) diff --git a/packages/oracles/README.md b/packages/oracles/README.md index 7f61898..c831c26 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -29,7 +29,7 @@ coding agents, and walks through a first oracle and gap check. | Path | What it is | | --------------------------------- | -------------------------------------------------------------------------------------------------- | | `skills/oracles-setup/` | Setup, upgrade, and making the skills findable. | -| `skills/oracles-design/` | How to design, write, or change an oracle: law sheet first, model, histories, RED, GREEN, gaps. | +| `skills/oracles-design/` | How to design, write, or change an oracle: prose laws first, model, histories, RED, GREEN, gaps. | | `skills/oracles-evaluate-review/` | How to act on a review: a lossless ledger, RED on unchanged production, and closing the bug class. | | `skills/oracles-review/` | How to review oracle work. | | `docs/oracle-checklist.md` | The complete requirements, by ID (ORC-001 to ORC-016). | diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md index 99ae460..62ba200 100644 --- a/packages/oracles/docs/glossary.md +++ b/packages/oracles/docs/glossary.md @@ -6,23 +6,23 @@ term for one concept. Do not rotate synonyms for style. ## Laws and evidence -| Term | Meaning | -| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | -| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | -| limits | What a property does not establish. Unresolved parts of a law go here. | -| law sheet | The first output of oracle design: each law with its authority, status, rival, distinguishing witness, and checkpoint. A human reviews it before any code. | -| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | -| rival | A plausible different law that predicts a different observation for some history. | -| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | -| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | -| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | -| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | -| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | -| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | -| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the user can pick. | -| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | -| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | +| Term | Meaning | +| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | +| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | +| limits | What a property does not establish. Unresolved parts of a law go here. | +| law sheet | The first output of oracle design: the new oracle file with no code yet. Its opening comment states the laws in prose, and its review notes give each law's examples, general sentence, authority, status, rival, witness, and checkpoint. A human reviews it before any code is added. | +| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | +| rival | A plausible different law that predicts a different observation for some history. | +| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | +| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | +| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | +| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | +| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | +| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | +| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the user can pick. | +| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | +| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | ## The parts of an `oracleProperty` diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 74d2fcb..67b7191 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -1,6 +1,6 @@ --- name: oracles-design -description: Design, write, or change an oracle with @tanstack/oracles. Choose the boundary, write a law sheet first, choose a model, and build histories that reach the states the law needs. Show the evidence the change needs, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. +description: Design, write, or change an oracle with @tanstack/oracles. Choose the boundary, write the oracle file's prose laws first, choose a model, and build histories that reach the states the law needs. Show the evidence the change needs, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. --- # Design an oracle @@ -51,16 +51,37 @@ its promise. law's stated domain is strengthening. Changing the stated domain, a checkpoint, or `limits` is a law change. - **Changing a law** (its promise, domain, checkpoint, assumptions, or the - freedoms it allows) needs a law delta: the old and new block for each + freedoms it allows) needs a law delta. Edit the law block's prose in the + oracle file, and add review notes with the old and the new block for each changed law. Stop for the user's review. - **A new law, or missing or contradictory authority,** needs the full law sheet below. Stop for the user's review. An analogy to another documented rule is not authority: mark that law `open`. -**Write the law sheet from this template.** Docs, tests, and bug reports give -examples. A law is a statement over the whole domain. The template makes you -write the step from one to the other. Copy it once for each law, and fill in -every field. The template is the sheet. Do not replace it with a table. +**Write the oracle file's prose first.** The law sheet is the new oracle +file before it has any code. Create it at its final path, with two comments +and nothing else: + +1. **The law block**, the file's opening comment. The oracle file is a + literate program (ORC-003), so write it as prose for a reader who has not + seen the review. Do not paste the template's fields or a list of law IDs. + In this order: + 1. A title that asks the question the oracle answers, for example "Does a + copy keep every value, and share nothing with its source?" + 2. How the code works at this boundary, and what goes wrong there, in + plain sentences. + 3. What the documentation promises, in its own words where you can, with + the source of each promise. State each law as one sentence over its + domain ("For every ..."). These are the laws. + 4. How the model will state those promises, and what it reads to predict + them. + 5. What the driver will do, what it observes, and what it does not + observe and why. Then the limits, each with its reason. +2. **Review notes**, a second comment headed "Review notes (delete when + approved)". Docs, tests, and bug reports give examples. A law is a + statement over the whole domain. The template below makes you write the + step from one to the other. Copy it once for each law, and fill in every + field. Then list the open questions. ```markdown ### L1 @@ -116,8 +137,8 @@ How to fill in the fields: `observed`, and ask the user. An analogy to another documented rule is not authority either: mark that law `open`. -**Required blocks: the laws people forget.** The sheet has a block for each -of these, or one line that says why it does not apply: +**Required blocks: the laws people forget.** The review notes have a block +for each of these, or one line that says why it does not apply: - **No throw on valid input.** When the contract says a call returns a value, it promises not to throw for valid input. State that as a law, with the same @@ -128,9 +149,8 @@ of these, or one line that says why it does not apply: The task already approved these laws? Do not ask again. -After the blocks, list the **open questions**. Then add a summary table with -one row per block: law, status, authority, and checkpoint. The table is only -an index to the blocks. +The law block and the review notes must agree: each law in the prose has its +block, with the same sentence. When the promise is unclear, or two readings predict different behavior, use the [law discovery](../../docs/instruments/law-discovery.md) card. @@ -139,8 +159,8 @@ the [law discovery](../../docs/instruments/law-discovery.md) card. when any of these is true.** Do not judge whether the law is hard. Check the list at the sheet, and again after each gap check: -- the sheet has more than five rows for one boundary (adopted, observed, and - open rows all count) +- the sheet has more than five laws for one boundary (adopted, observed, and + open ones all count) - a law has no new case that it predicts - the model has more than one branch per law - the work changes or simplifies an existing model @@ -149,36 +169,30 @@ list at the sheet, and again after each gap check: Say in the evidence which trigger fired, or that none did. **Stop here** when the sheet or a law delta is needed. Do not stop with an -empty field: each law has a filled block first. Do not write the oracle until -the user has reviewed the sheet. With no user present, write the sheet and -stop. A law sheet has no default. - -Once the user approves it, the sheet becomes the **law block**: the opening -comment of the oracle file. The oracle file is a literate program (ORC-003), -so write the block as prose for a reader who has not seen the sheet. Do not -paste the template's fields or a list of law IDs. In this order: - -1. A title that asks the question the oracle answers, for example "Does a - copy keep every value, and share nothing with its source?" -2. How the code works at this boundary, and what goes wrong there, in plain - sentences. -3. What the documentation promises, in its own words where you can, with the - source of each promise. These are the laws. -4. How the model states those promises, and what it reads to predict them. -5. What the driver does, what it observes, and what it does not observe and - why. Then the limits, each with its reason. - -The prose beside the model, the histories, the driver, and the check says why -each one predicts or tests the promised result. The sheet's examples, rivals, -and witnesses stay in the review record, not the file. Each law's sentence, -authority, and limits also go in that property's `law` option. Keep the block, the `law` option, and the model in step: a change to -one updates the others in the same change (ORC-003). +empty field: each law has a filled block first. Do not add code to the file +until the user has reviewed it. With no user present, write the file and +stop. A law sheet has no default. Until it has code, the file fails Vitest +with "No test suite found", so keep it off a branch whose CI must pass, or +say in the review request that it will fail. + +**Once the user approves it:** + +- Apply the review's changes to the law block first. +- Delete the review notes. They stay in the review record. +- Add the code under the law block. Each boundary condition becomes a + constructed history, and each law's sentence, authority, and limits go in + that property's `law` option. +- Put prose beside the model, the histories, the driver, and the check that + says why each one predicts or tests the promised result. + +Keep the law block, the `law` option, and the model in step: a change to one +updates the others in the same change (ORC-003). ## 2. Freeze what the oracle must preserve Write these down. They stay fixed while you build the oracle. -- The laws and their checkpoints, from the sheet. +- The laws and their checkpoints, from the approved law block. - The owned production code: the files or functions the oracle answers for. - The freedoms the contract leaves open, such as the order of independent events, scheduling, or representation. The oracle must accept every diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index da81a8b..493e174 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -41,8 +41,8 @@ contains one is not ready. | Shortcut | What to look for | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Law after code** | No law sheet before the oracle was written, a law with no authority, or a law block that disagrees with the `law` option or the model. | -| **Law as examples** | A law written as a list of cases, not one sentence over the domain. A sheet with no new case that the law predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | +| **Law after code** | Oracle code written before the user approved the file's prose laws, a law with no authority, or a law block that disagrees with the `law` option or the model. Also a law block that is a list of fields or law IDs, not prose (ORC-003). | +| **Law as examples** | A law written as a list of cases, not one sentence over the domain. Review notes with no new case that the law predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | | **Skipped design grammar** | A trigger from `oracles-design` step 1 fired at the sheet or after a gap check, and the design grammar card was not used. | | **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | | **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 6a1ad42..a55798d 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -304,9 +304,9 @@ the gap check there, and say in the report that it ran on the patched code. With the user's approval, follow `oracles-design` step 7's defect path for the real fix. -Then load `oracles-design` and follow it from its first step. The law sheet -comes first, and the user reviews it before you write code. Setup does not -skip that step to finish sooner. +Then load `oracles-design` and follow it from its first step. The oracle +file's prose laws come first, and the user reviews them before you add code. +Setup does not skip that step to finish sooner. Put the oracle's `owns` list on its `oracleProperty` call. An entry is a file (`src/index.ts`) or one function in it (`src/index.ts#parse`, or From 3dec7d5dd5bd0e68622874cb6739c285bd15acee Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 23:01:40 -0700 Subject: [PATCH 21/76] docs(oracles): teach the gap CLI's exit codes, and round-5 setup fixes After #14, a gap check run exits 0 (completed), 2 (refused, with one `fix:` line to follow), or 3 (a CLI bug, with error.log). The setup and design skills now say what to do with each, and "no oracle file" is a refusal. Setup also covers: `--legacy-peer-deps` for every install when the repository's ESLint is older than 9, and checking Vitest's `vite` peer; TypeScript older than 4.8 as an environment limit for the parser; and keeping the new files out of a published package. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 4 ++- .../oracles/skills/oracles-setup/SKILL.md | 25 ++++++++++++++----- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 67b7191..4fd0b3a 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -477,7 +477,9 @@ Techniques: [guide § Evidence](../../docs/guide.md#evidence). ## 8. Close every gap -Run the gap check on the owned code, then work from its report. Every gap +Run the gap check on the owned code, then work from its report. Exit 2 is a +refusal: do what its `fix:` line says. Exit 3 is a CLI bug: report it with +`/error.log`, and do not change the oracle. Every gap comes with one instruction. Do what it says, within the rules below. - **There are no equivalent mutants.** A surviving mutant shows a missing diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index a55798d..9c0d56d 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -54,7 +54,10 @@ pnpm add -D \ Add `fast-check` and `vitest` beside them. For lint, also add `eslint@^9` if the repository has no ESLint, and `@typescript-eslint/parser@^8` if nothing parses TypeScript yet. Name those majors: a package manager can pick -a prerelease otherwise. For an older ESLint, see the fallback in step 3. +a prerelease otherwise. If the repository's ESLint is older than 9 (xo +included), install every package here with `--legacy-peer-deps` (npm), and see +the fallback in step 3. With that flag, npm skips peers, so check that +Vitest's own peer `vite` is installed, and add it if it is not. Never add, upgrade, or downgrade the repository's `typescript` for oracles. The gap check uses its own TypeScript and never reads the project's, from @@ -116,7 +119,9 @@ export default defineConfig({ ``` Add a script for it, such as `test:oracles`. Leave the existing suite -unchanged. +unchanged. In a published package, keep the new files out of the published +package (its `files` field or `.npmignore`): the configs, `AGENTS.md`, +`.agents/`, and `.claude/`. If the repository's own lint or formatter covers the oracle files (for example, an older ESLint with Prettier over `test/`), keep the oracle files @@ -155,6 +160,9 @@ eslint@9` fallback below instead. TypeScript peer range ends before 7. Do not force the install. Report it as an environment limit, with that reason. Do not change the repository's TypeScript. +- **TypeScript older than 4.8:** `@typescript-eslint/parser` 8 fails to + load, and the failure takes down the whole flat config. Treat it the same + way: an environment limit, reported, with no parser in the config. - The plugin parses the modules an oracle file imports with that file's parser. A `.js` oracle file that imports a TypeScript module gets an `unparsedModule` report: write the oracle file in TypeScript, with a @@ -311,10 +319,10 @@ Setup does not skip that step to finish sooner. Put the oracle's `owns` list on its `oracleProperty` call. An entry is a file (`src/index.ts`) or one function in it (`src/index.ts#parse`, or `src/x.ts#Class.method`). A function entry owns that function and its static -call tree. For a first oracle, own only the function it covers. Own the whole file only -when module data affects an adopted law (see `oracles-design` step 6). If the gap check counts **0 oracle files**, the -oracle is not declared. That is a setup failure, even when the report shows -0 gaps and the command exits 0. +call tree. For a first oracle, own only the function it covers. Own the whole +file only when module data affects an adopted law (see `oracles-design` step +6). If the gap check refuses with "no oracle file", the oracle is not +declared: that is a setup failure. In an oracle file, only `oracleProperty` registers tests. A plain Vitest test there is not a kill, and the gap check reports it. Do not leave an empty test @@ -337,6 +345,11 @@ pnpm exec oracle-gaps --package --source \ - Pass `--seed`, so the baseline and every mutant run the same histories. Pass `--out` inside the repository. (Both become defaults later.) - Add `.oracle-gaps/` to `.gitignore`. +- Read the exit code. **0:** the run completed; work from its report. + **2:** refused. It prints `refused: ` and one `fix: `. + Do what the `fix:` line says, then run again. **3:** an internal error, "a + bug in oracle-gaps", with the stack in `/error.log`. Report it as a CLI + bug with that file, and do not change the oracle to get around it. Run it after the oracle passes on production. Each gap in the report comes with one instruction. Follow From bd5b900756a91c01882385e98475189257c9c82f Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 23:02:36 -0700 Subject: [PATCH 22/76] docs(oracles): interim rule for an unnamed default export in owns Until a function entry can name the default export, own the whole file and report the gaps it pulls in as caused by that CLI limitation. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 4fd0b3a..13231b6 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -399,6 +399,10 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). of its own. Own it through the enclosing function (`src/index.ts#mitt`). That entry also brings in the sibling methods: cover them, or report their gaps as outside this oracle's scope. + - Temporary: an unnamed default export (`module.exports = function ...`, + `export default () => ...`) cannot be named yet. Own the whole file, and + report the gaps that this pulls in from other code in the file as caused + by that CLI limitation, not as gaps in this oracle. - Temporary: data outside a function (a lookup table, a constant object) is not yet part of a function entry's tree. Own the whole file only when that data affects an adopted law, such as a unit table that decides the From 222a780eed06670114b7b1e6052359bf35d46455 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 23:04:02 -0700 Subject: [PATCH 23/76] docs(oracles): xo inline disables, other source dirs, and calls typed by an interface From the p-queue validation run: lint the oracle config with --no-inline-config under xo, name source/ as a production location that needs its own glob, and own the implementation of a method that the checker reaches only through an interface or a type parameter. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 5 +++++ packages/oracles/skills/oracles-setup/SKILL.md | 10 +++++++--- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 13231b6..87be99b 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -399,6 +399,11 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). of its own. Own it through the enclosing function (`src/index.ts#mitt`). That entry also brings in the sibling methods: cover them, or report their gaps as outside this oracle's scope. + - The tree follows what the TypeScript checker resolves. A call whose + target is typed only by an interface or a type parameter (`this.#queue.filter` + where `#queue: QueueType`) reaches no function body, so it is not followed. + If the law depends on that method, own the implementation too + (`source/priority-queue.ts#PriorityQueue.filter`). - Temporary: an unnamed default export (`module.exports = function ...`, `export default () => ...`) cannot be named yet. Own the whole file, and report the gaps that this pulls in from other code in the file as caused diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 9c0d56d..a5ec385 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -142,7 +142,8 @@ export default [ - `recommended` applies to oracle files and turns off inline directives there, so a disable comment has no effect. - `production` checks the probe form in `src/**` and `packages/*/src/**`. If - production source lives elsewhere (for example a root-level `index.js`), add + production source lives elsewhere (for example `source/`, or a root-level + `index.js`), add the rule with that glob. The config object must register the plugin itself: ```js @@ -152,8 +153,11 @@ export default [ - If the repository lints with xo, keep the oracle files and the new config files out of xo (its `ignores`), and lint them with the oracle ESLint config. Do not install `eslint@^9` as a dev dependency: xo loads ESLint - from the hoisted copy, and a second major breaks it. Use the `npx -p -eslint@9` fallback below instead. + from the hoisted copy, and a second major breaks it. Use the + `npx -p eslint@9` fallback below instead, and run it with + `--no-inline-config`: + xo's own disable comments in production files name rules that this config + does not load, and ESLint reports each one. - **TypeScript 7:** typescript-eslint does not support TypeScript 7 yet. If the repository's ESLint config cannot parse TypeScript, the oracle lint cannot run. Installing `@typescript-eslint/parser` fails with `ERESOLVE`, since its From 7ff006c808da65c200edbaeaaa9644c53677cde9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Tue, 10 Nov 2026 23:04:43 -0700 Subject: [PATCH 24/76] docs(oracles): put everything on oracleProperty, and do not call declareOracle declareOracle is legacy, kept only for TanStack DB's migration. Existing uses in a repository stay as they are. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 87be99b..666b1c4 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -424,6 +424,11 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). Keep the law, its authority, and its limits in `law`, and prose beside each part that says why it predicts the promised result (ORC-003). +Put `owns`, the law, the model, the histories, the driver, and the check on +`oracleProperty`. Do not call `declareOracle`: it is legacy, kept only for +TanStack DB's migration. If a repository's existing oracles already use it, +leave them as they are. + The options of `oracleProperty` are typed in `node_modules/@tanstack/oracles/src/oracle-property.ts` (`OraclePropertyOptions`). Read them there. From 9f255db16f2d79812f8b1adb996219bf9c201175 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 01:23:27 -0700 Subject: [PATCH 25/76] docs(oracles): follow main 1bb905a (#15 to #19) Replace interim rules with what landed: the take-back check now fails at load (#15); lint reports empty tests, and the presets ignore tool configs, so a custom ignores must merge the preset's (#16); the gap check's TypeScript is an npm alias, with only a .bin/tsc residue (#18); enumerated sources declare states, and the reach test enforces them (#19). Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 19 ++++++++++++------- .../oracles/skills/oracles-setup/SKILL.md | 15 +++++++++------ 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 666b1c4..f4c5f6a 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -316,12 +316,13 @@ in the model. cannot collide with a generated value (a fixed boundary shape). Otherwise derive it from the generated history (for example "the initial value plus one"). -- **Every arbitrary must take its histories back.** fast-check replays each - `fixed` and constructed history through the arbitrary. A `.map` with no - inverse cannot do this, and every such history fails at run time with - `canShrinkWithoutContext`. Give each `.map` its inverse - (`.map(f, unmap)`, where `unmap` throws on a value it cannot map back), or build from `fc.record`, `fc.constantFrom`, and plain - values. Run the oracle once right after you write each constructed history. +- **Every arbitrary must take its histories back.** Each `fixed` and + constructed history goes back through the arbitrary, and one that cannot + fails when the file loads, naming the history. Give each `.map` its inverse + (`.map(f, unmap)`, where `unmap` throws on a value it cannot map back), and + avoid `.chain`. Build objects with `fc.record` or `fc.tuple` over primitive + leaves. `fc.constant` and `fc.constantFrom` take back only primitives, or + the same object reference. - **Choose the string unit that covers the law's domain.** fast-check's default strings include lone surrogates, and slicing by code unit splits astral characters. `fc.string({ unit: 'binary' })` gives every code point @@ -346,7 +347,11 @@ They do not reach them. - **Cover every critical state by construction.** Each state that a law needs is a declared state with a constructed history, and a probe where - production decides to enter it. Never raise `runs` to reach a state. + production decides to enter it. Never raise `runs` to reach a state. With + an enumerated source (`histories: { enumerated, states, measured }`), every + history runs, so a required state needs no constructed history: the `reach` + test fails if no history entered it. No enumerated history's label may be + `reach` or `determinism`: those name the framework's own tests. - **Need a state at least N times?** Give it N constructed histories. - **Give at least one fixed seed** in `fixedSeeds`. The random campaign always runs beside it. A kill that only the random campaign finds is a gap until you diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index a5ec385..a02f2ab 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -66,10 +66,10 @@ TypeScript 5 or newer. If the repository's `tsc` is older, keep the oracle files and the Vitest config out of its run (its `include` or `exclude`). Report any typecheck errors that existed before setup as pre-existing. -Temporary, a known packaging bug: `@tanstack/oracles` depends on -`typescript`, and npm can hoist that copy to the repository root. A -repository that declares no `typescript` then gets one, and `npx tsc` -resolves to it. Report it as this bug. Do not treat that copy as the +The gap check's own TypeScript installs as `typescript-for-tanstack-oracles`, +so it does not replace the repository's. In a repository with no TypeScript, +or after an incremental npm install, npm can still link its `tsc` into +`node_modules/.bin` until the next `npm ci`. Do not treat that `tsc` as the repository's TypeScript, and do not add a typecheck that uses it. Use the repository's own package manager. With pnpm 11, an install that skips @@ -139,6 +139,9 @@ export default [ ] ``` +- If you spread a preset and set your own `ignores` on it, merge + `...preset.ignores` into them. The presets ignore tool configs + (`eslint.config.*`, `vite.config.*`, `vitest.config.*`, `*.oracle.config.*`). - `recommended` applies to oracle files and turns off inline directives there, so a disable comment has no effect. - `production` checks the probe form in `src/**` and `packages/*/src/**`. If @@ -329,8 +332,8 @@ file only when module data affects an adopted law (see `oracles-design` step declared: that is a setup failure. In an oracle file, only `oracleProperty` registers tests. A plain Vitest test -there is not a kill, and the gap check reports it. Do not leave an empty test -there either: it checks nothing and looks like coverage. The driver records +there is not a kill, and the gap check reports it. Lint also reports an empty test +there: it checks nothing and looks like coverage. Remove it, or use `it.todo`. The driver records production errors as observations: a mutant that crashes production is not a kill until the check rejects the recorded error. From 27f5367f96d7800a282cdca48d34c3a17600ee8e Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 04:03:09 -0700 Subject: [PATCH 26/76] docs(oracles): pass --config for a non-default Vitest config name (#26) The gap check now takes --config, so a repository whose other tools would pick up vitest.config.ts can use vitest.oracle.config.ts. The Vite warning in CommonJS packages is gone, and so is its interim note. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 2 -- .../oracles/skills/oracles-setup/SKILL.md | 19 +++++++++++-------- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 92fc8d6..7a458b9 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -566,8 +566,6 @@ comes with one instruction. Do what it says, within the rules below. its line, and keep working. Do not remove the guard and do not change production: the guard protects untyped callers. Every other mutant on that line must still be killed. -- **Temporary:** in a package without `"type": "module"`, Vite may warn - while loading the gap check's own config copies. Ignore that warning. - **Few mutants are weak evidence.** If the gap check generates few mutants for an owned function, or none on a law's decisions (such as a return diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index d1bc972..68c8022 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -98,16 +98,16 @@ the oracle files: oracle files `*.oracle.ts`, and include that name in the Vitest config. The gap check finds oracle files by their `oracleProperty` call, so the name -only has to match the Vitest config and the lint glob. The gap check loads only `vitest.config.*` or -`vite.config.*`, so: +only has to match the Vitest config and the lint glob. - If the repository has no Vitest config, create `vitest.config.ts` for the - oracle files. Jest and mocha ignore that name. In a package without - `"type": "module"`, name it `vitest.config.mts`, or Vite warns on every - run. -- If it has one, add the oracle glob to its existing `include`. Keep the - globs that are there: replacing them drops the existing suite, since an - `include` replaces Vitest's default. + oracle files. Jest and mocha ignore that name. +- If another tool would pick up `vitest.config.ts`, name it + `vitest.oracle.config.ts`, and pass `--config vitest.oracle.config.ts` to + the gap check (step 7) and to the Vitest script. +- If the repository has a Vitest config, add the oracle glob to its existing + `include`. Keep the globs that are there: replacing them drops the + existing suite, since an `include` replaces Vitest's default. ```ts // vitest.config.ts @@ -347,6 +347,9 @@ pnpm exec oracle-gaps --package --source \ --seed --out /.oracle-gaps/out ``` +Add `--config ` (relative to the package) when the oracle Vitest config +is not Vite's default name. The baseline prints which config it used. + - `--source` selects files. `owns` can name a single function in a file, so the report judges only the owned functions. Code in the file that no oracle owns appears as unowned warnings. From 87cdf12c90205900d876e0e79c472fe371473e5a Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 07:35:37 -0700 Subject: [PATCH 27/76] docs(oracles): a copy-and-paste prompt to try the package The package README's Try it section has a prompt that an adopter pastes into a coding agent as is: it finds the current commit with git ls-remote, installs that commit from pkg.pr.new, and follows the setup skill, stopping at the law review. The setup skill uses the same lookup when no commit is given. Co-authored-by: Isaac --- README.md | 4 +- packages/oracles/README.md | 44 ++++++++++++++----- .../oracles/skills/oracles-setup/SKILL.md | 6 ++- 3 files changed, 40 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index e153ce3..27f00f1 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,9 @@ every commit with [pkg.pr.new](https://pkg.pr.new). ## Packages -To adopt it, start with [`packages/oracles/README.md`](packages/oracles/README.md): +To try it, paste the prompt in [Try it](packages/oracles/README.md#try-it) +into your coding agent. To adopt it, start with +[`packages/oracles/README.md`](packages/oracles/README.md): the package ships setup, design, and review skills for coding agents, the oracle checklist, and a glossary. diff --git a/packages/oracles/README.md b/packages/oracles/README.md index c831c26..b1470c9 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -8,21 +8,43 @@ Oracle testing for TypeScript on Vitest and fast-check: The gap check then lists the owned production behavior that no law judges. -Not on npm yet. Install a commit from pkg.pr.new (see [Start](#start)). +Not on npm yet. Install a commit from pkg.pr.new (see [Try it](#try-it)). -## Start +## Try it -Give your coding agent this prompt in the repository: +Paste this prompt into your coding agent at the root of the repository you +want to test. Any agent that can run shell commands works. -> Install `https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@` as a -> dev dependency, then read -> `node_modules/@tanstack/oracles/skills/oracles-setup/SKILL.md` in full and -> follow it. +```text +Set up oracle testing in this repository with @tanstack/oracles. -The setup skill installs this package and `@tanstack/eslint-plugin-oracles` -at the same commit, adds the lint preset, and writes a block in `AGENTS.md`. -It offers to make the skills findable, in the repository or globally for your -coding agents, and walks through a first oracle and gap check. +1. Find the current commit of TanStack/oracles. Run + `git ls-remote https://github.com/TanStack/oracles refs/heads/main`, and use + the first 7 characters of the hash as . +2. Install https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@ as a dev + dependency, with this repository's package manager. +3. Read node_modules/@tanstack/oracles/skills/oracles-setup/SKILL.md in full, + and follow it. Use wherever it asks for the commit to pin. +4. When the skill asks me to review the laws for the first oracle, stop and + show me the oracle file. +``` + +What happens next: + +1. The agent installs this package and `@tanstack/eslint-plugin-oracles` at + the same commit. It adds the lint preset, writes a block in `AGENTS.md`, + and offers to make the skills findable for your coding agents. +2. It chooses one documented function and writes the first oracle file as + prose only: what the code promises, with the source of each promise. Then + it stops. +3. You review the laws. This is the one step that needs you: you decide what + the code promises, and the agent does the rest. +4. The agent adds the model, the histories, the driver, and the check. It runs + the oracle on your unchanged code and reports any defect with a replay + line. Then it runs the gap check and reports what is left. + +The package needs Node 20 or later. The setup skill installs Vitest and +fast-check if the repository does not have them. ## What ships in this package diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 68c8022..8398e1b 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -41,8 +41,10 @@ Find out, and tell the user: ## 2. Install two pinned packages The packages are not on npm yet. Install both from pkg.pr.new, pinned to the -same commit of TanStack/oracles. Ask the user for the commit if you were not -given one. +same commit of TanStack/oracles. If you were not given one, use the current +commit on main: the first 7 characters of the hash that +`git ls-remote https://github.com/TanStack/oracles refs/heads/main` prints. +Tell the user which commit you pinned. ```sh pnpm add -D \ From 833f7caf2d60d6447d7018b04eec5231db743dd0 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 08:24:48 -0700 Subject: [PATCH 28/76] docs(oracles): open the law block with what the oracle owns, then a hook The law block now opens with two or three plain sentences: the promise the oracle maintains, where it is observed, and the owned code. Then comes a hook chosen by the oracle's shape (a surprising rule, a documented example, or a plausible wrong implementation), the mechanism with short names that the code reuses, the laws, how the oracle knows, and the limits. It names the two generic openings to avoid, adds three checks before the agent stops for review, and treats the approved review notes as the anchor for later prose edits. Setup now asks for a public, documented promise whose law says more than its code, so a first oracle is not an internal helper that a model would copy. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 65 ++++++++++++++----- .../oracles/skills/oracles-setup/SKILL.md | 10 ++- 2 files changed, 55 insertions(+), 20 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 7a458b9..5452a6e 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -66,17 +66,40 @@ and nothing else: literate program (ORC-003), so write it as prose for a reader who has not seen the review. Do not paste the template's fields or a list of law IDs. In this order: - 1. A title that asks the question the oracle answers, for example "Does a - copy keep every value, and share nothing with its source?" - 2. How the code works at this boundary, and what goes wrong there, in - plain sentences. - 3. What the documentation promises, in its own words where you can, with - the source of each promise. State each law as one sentence over its - domain ("For every ..."). These are the laws. - 4. How the model will state those promises, and what it reads to predict - them. - 5. What the driver will do, what it observes, and what it does not - observe and why. Then the limits, each with its reason. + 1. **What this oracle owns**, in two or three plain sentences: the + promise it maintains, the public entry point and checkpoints where the + promise is owed, and the owned code. A reader who stops here knows what + the file is for. For example: "This oracle owns `klona`'s deep-copy + promise for the default mode: the copy has the source's kind, + prototype, and values at every depth, and shares no object with it. It + checks the value that `klona(x)` returns. It owns `src/index.js#klona`." + 2. **The hook:** one concrete case where the obvious reading of the + promise is wrong. Choose it by the oracle's shape: + + | Shape | Hook | + | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------- | + | State, timing, or order with a surprising rule | A question that names the surprise: "Why does a ready transaction wait?" | + | A pure function with documented examples | The documented example as code, then what in it can vary and what decides the answer | + | The obvious implementation is wrong | The plausible implementation and the one input where it fails: why a deep copy is not a JSON round trip | + + 3. **The mechanism:** the two or three moving parts that make the + behavior necessary, in plain sentences. Give each a short name (the + window, the head, the applied receipt), and use the same name in the + law names, the `states`, and the calibrations. + 4. **The laws:** each one sentence over its domain ("For every ..."), with + the source of its promise in the documentation's own words where you + can. Then the edges where it is easy to get wrong, and what a caller + sees when the law breaks. + 5. **How the oracle knows:** what the model reads to predict each law, + which histories reach the edges, what the driver observes and does not + observe and why, and where the check compares. + 6. **Limits and open questions**, last, each with its reason. + + Avoid the two generic openings: a title or summary that asks whether the + code does what its documentation promises, and a law that restates the + implementation. A model of that law would be the same code, so it could + not catch a defect. + 2. **Review notes**, a second comment headed "Review notes (delete when approved)". Docs, tests, and bug reports give examples. A law is a statement over the whole domain. The template below makes you write the @@ -169,15 +192,23 @@ list at the sheet, and again after each gap check: Say in the evidence which trigger fired, or that none did. **Stop here** when the sheet or a law delta is needed. Do not stop with an -empty field: each law has a filled block first. Do not add code to the file -until the user has reviewed it. With no user present, write the file and -stop. A law sheet has no default. Until it has code, the file fails Vitest -with "No test suite found", so keep it off a branch whose CI must pass, or -say in the review request that it will fail. +empty field: each law has a filled block first. Before you stop, check the +law block: + +- From the prose alone, a reader can predict one observation of each law. +- Each name from the mechanism appears in the law names, the states, and the + calibrations. +- The opening says what the oracle owns, and no law restates the code. Do not add code to the file + until the user has reviewed it. With no user present, write the file and + stop. A law sheet has no default. Until it has code, the file fails Vitest + with "No test suite found", so keep it off a branch whose CI must pass, or + say in the review request that it will fail. **Once the user approves it:** -- Apply the review's changes to the law block first. +- Apply the review's changes to the law block first. The approved review + notes are the anchor: a later edit to the prose that changes a law's + domain, checkpoint, or strength ("may" to "does") is a law change. - Delete the review notes. They stay in the review record. - Add the code under the law block. Each boundary condition becomes a constructed history, and each law's sentence, authority, and limits go in diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 8398e1b..a01c768 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -309,9 +309,13 @@ Before writing a global pointer: ## 6. Write the first oracle -Choose one small, owned production function with a clear, documented -promise. If most of its laws would have no authority (no docs, only tests or -code), choose another one. Ask the user which one if it is not obvious. +Choose one public, documented promise with a small owned surface: an +exported function or method that callers use, with a contract in the README +or the API docs. Its law must say more than its code does: if a model of the +law would repeat the implementation (an internal helper that only wraps or +escapes a string), choose the caller-visible promise that uses it. If most +of its laws would have no authority (no docs, only tests or code), choose +another one. Ask the user which one if it is not obvious. If the first oracle fails on unchanged production under an adopted law, it found a defect. The RED is the finding: report it with its replay line, and From e08ad18253e0b58c64a04b60a7389a4f6b9a972e Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 08:30:49 -0700 Subject: [PATCH 29/76] docs(oracles): sharper law block guidance from the round-6 rewrites The opening must name the concrete promise, with a bad and a good example. The prose does not label its parts with the skill's step names. A pure function's hook is the documented example plus the nearby case that a careless reading gets wrong, and a tie between forms goes to the case a reader would most likely miss. The checks before stopping are their own list, cover a no-throw law, and say that the review notes decide when the prose disagrees, and that a new heading is not a law change. Names in states and calibrations wait for the code. An entry whose call tree reaches another contract reports those gaps as outside this oracle's scope. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 81 +++++++++++-------- 1 file changed, 48 insertions(+), 33 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 5452a6e..508c83e 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -65,27 +65,38 @@ and nothing else: 1. **The law block**, the file's opening comment. The oracle file is a literate program (ORC-003), so write it as prose for a reader who has not seen the review. Do not paste the template's fields or a list of law IDs. - In this order: + Write it as continuous prose in this order. Do not label the parts with + the names below ("The hook", "The mechanism"): they are for you, not the + reader. 1. **What this oracle owns**, in two or three plain sentences: the - promise it maintains, the public entry point and checkpoints where the - promise is owed, and the owned code. A reader who stops here knows what - the file is for. For example: "This oracle owns `klona`'s deep-copy - promise for the default mode: the copy has the source's kind, - prototype, and values at every depth, and shares no object with it. It - checks the value that `klona(x)` returns. It owns `src/index.js#klona`." - 2. **The hook:** one concrete case where the obvious reading of the - promise is wrong. Choose it by the oracle's shape: - - | Shape | Hook | - | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------- | - | State, timing, or order with a surprising rule | A question that names the surprise: "Why does a ready transaction wait?" | - | A pure function with documented examples | The documented example as code, then what in it can vary and what decides the answer | - | The obvious implementation is wrong | The plausible implementation and the one input where it fails: why a deep copy is not a JSON round trip | + concrete promise it maintains, the public entry point and checkpoints + where the promise is owed, and the owned code. A reader who stops here + knows what the file is for. Name the promise itself. "The stylesheet + matches the options the README documents" names no promise. "`klona` + returns a copy with the source's kind, prototype, and values at every + depth, sharing no object with it" does. For example: "This oracle owns + `klona`'s deep-copy promise for the default mode: the copy has the + source's kind, prototype, and values at every depth, and shares no + object with it. It checks the value that `klona(x)` returns. It owns + `src/index.js#klona`." + 2. **The hook:** the one concrete case that shows where the promise is + easy to get wrong. Write it as code where you can. Choose its form by + the oracle's shape: + + | Shape | Hook | + | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | + | State, timing, or order with a surprising rule | A question that names the surprise: "Why does a ready transaction wait?" | + | A pure function with documented examples | The documented example, then the nearby case that a careless reading gets wrong: ratio 2 and base 16px give 32px, so what does base 15.5px give? | + | The obvious implementation is wrong | The plausible implementation and the one input where it fails: why a deep copy is not a JSON round trip | + + When two forms fit, choose the one whose case a reader is most likely + to get wrong. 3. **The mechanism:** the two or three moving parts that make the behavior necessary, in plain sentences. Give each a short name (the window, the head, the applied receipt), and use the same name in the - law names, the `states`, and the calibrations. + law names. When you write the code, use it in the `states` and the + calibrations too. 4. **The laws:** each one sentence over its domain ("For every ..."), with the source of its promise in the documentation's own words where you can. Then the edges where it is easy to get wrong, and what a caller @@ -95,10 +106,9 @@ and nothing else: observe and why, and where the check compares. 6. **Limits and open questions**, last, each with its reason. - Avoid the two generic openings: a title or summary that asks whether the - code does what its documentation promises, and a law that restates the - implementation. A model of that law would be the same code, so it could - not catch a defect. + Avoid the two generic openings: a summary that says the code does what + its documentation promises, and a law that restates the implementation. A + model of that law would be the same code, so it could not catch a defect. 2. **Review notes**, a second comment headed "Review notes (delete when approved)". Docs, tests, and bug reports give examples. A law is a @@ -191,18 +201,22 @@ list at the sheet, and again after each gap check: Say in the evidence which trigger fired, or that none did. -**Stop here** when the sheet or a law delta is needed. Do not stop with an -empty field: each law has a filled block first. Before you stop, check the -law block: +**Check the law block before you stop:** - From the prose alone, a reader can predict one observation of each law. -- Each name from the mechanism appears in the law names, the states, and the - calibrations. -- The opening says what the oracle owns, and no law restates the code. Do not add code to the file - until the user has reviewed it. With no user present, write the file and - stop. A law sheet has no default. Until it has code, the file fails Vitest - with "No test suite found", so keep it off a branch whose CI must pass, or - say in the review request that it will fail. + For a no-throw law, the observation is that the call returns. +- The opening names the promise that the oracle owns. +- No law restates the code. +- Each law sentence matches its block in the review notes. If they disagree, + the review notes decide: fix the prose, and tell the user. A new heading + for a law is not a law change. A changed sentence, domain, or checkpoint is. + +**Stop here** when the sheet or a law delta is needed. Do not stop with an +empty field: each law has a filled block first. Do not add code to the file +until the user has reviewed it. With no user present, write the file and +stop. A law sheet has no default. Until it has code, the file fails Vitest +with "No test suite found", so keep it off a branch whose CI must pass, or +say in the review request that it will fail. **Once the user approves it:** @@ -448,9 +462,10 @@ function`, `export default () => ...`, `module.exports = fn`): initializer refers to. An entry can also name it: `src/index.ts#UNIT_MS`. A destructured declaration (`const { a, b } = ...`) is not a unit: if a law depends on one, own the whole file. - - When a whole-file entry pulls in another contract (for example a batching - API in the same file), write that contract's own oracle, or report its - gaps as outside this oracle's scope. + - When an entry pulls in another contract (a batching API in the same + file, or a helper in a default export's call tree that serves another + promise), write that contract's own oracle, or report its gaps as + outside this oracle's scope. - Owned code that serves another module's contract belongs to that module's oracle. Owned code with no behavior anyone observes is possible dead code: investigate it as step 8 says. From d36633093e565f1f8d73f7c40f001e3422a84c94 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 08:44:54 -0700 Subject: [PATCH 30/76] docs(oracles): start the law block with the contract as a short code block A reader takes in a few lines of real calls faster than dense prose. The law block now opens with one or two lines per main law: the call to the public entry point, the checkpoint as an expression, and the relation that must hold, with open laws marked. The ownership sentences follow, shorter. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 38 +++++++++++++------ 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 508c83e..2c1979d 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -68,18 +68,30 @@ and nothing else: Write it as continuous prose in this order. Do not label the parts with the names below ("The hook", "The mechanism"): they are for you, not the reader. - 1. **What this oracle owns**, in two or three plain sentences: the + 1. **The contract at a glance:** a short code block, first, that a reader + takes in at one look. One line, or two, for each main law: the real + call to the public entry point, the checkpoint as an expression, and + the relation that must hold. Use plain operators (`===`, `<=`) or a + standard assertion. A helper name (`sharesNoObject`) is fine when the + next sentence says what it means. Mark an open law `// open`. The + block states the promise, not the implementation: it never shows how + production computes the answer. For example: + + ```ts + // For every x in the domain below: + assert.deepStrictEqual(klona(x), x) // same kind, prototype, and values + sharesNoObject(klona(x), x) // no object of x is reachable from the copy + ``` + + 2. **What this oracle owns**, in one or two plain sentences: the concrete promise it maintains, the public entry point and checkpoints where the promise is owed, and the owned code. A reader who stops here knows what the file is for. Name the promise itself. "The stylesheet - matches the options the README documents" names no promise. "`klona` - returns a copy with the source's kind, prototype, and values at every - depth, sharing no object with it" does. For example: "This oracle owns - `klona`'s deep-copy promise for the default mode: the copy has the - source's kind, prototype, and values at every depth, and shares no - object with it. It checks the value that `klona(x)` returns. It owns + matches the options the README documents" names no promise. For + example: "This oracle owns `klona`'s deep-copy promise for the default + mode, checked on the value that `klona(x)` returns. It owns `src/index.js#klona`." - 2. **The hook:** the one concrete case that shows where the promise is + 3. **The hook:** the one concrete case that shows where the promise is easy to get wrong. Write it as code where you can. Choose its form by the oracle's shape: @@ -92,19 +104,19 @@ and nothing else: When two forms fit, choose the one whose case a reader is most likely to get wrong. - 3. **The mechanism:** the two or three moving parts that make the + 4. **The mechanism:** the two or three moving parts that make the behavior necessary, in plain sentences. Give each a short name (the window, the head, the applied receipt), and use the same name in the law names. When you write the code, use it in the `states` and the calibrations too. - 4. **The laws:** each one sentence over its domain ("For every ..."), with + 5. **The laws:** each one sentence over its domain ("For every ..."), with the source of its promise in the documentation's own words where you can. Then the edges where it is easy to get wrong, and what a caller sees when the law breaks. - 5. **How the oracle knows:** what the model reads to predict each law, + 6. **How the oracle knows:** what the model reads to predict each law, which histories reach the edges, what the driver observes and does not observe and why, and where the check compares. - 6. **Limits and open questions**, last, each with its reason. + 7. **Limits and open questions**, last, each with its reason. Avoid the two generic openings: a summary that says the code does what its documentation promises, and a law that restates the implementation. A @@ -205,6 +217,8 @@ Say in the evidence which trigger fired, or that none did. - From the prose alone, a reader can predict one observation of each law. For a no-throw law, the observation is that the call returns. +- The law block starts with the contract code block, and each line in it + states the same law as its sentence. - The opening names the promise that the oracle owns. - No law restates the code. - Each law sentence matches its block in the review notes. If they disagree, From 1c3a2f0b43d0581a4913b844049a8088c472bc30 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 08:48:18 -0700 Subject: [PATCH 31/76] docs(oracles): write the contract block as public calls and their results Agents read the assertion example as a cue to write test code with helpers, which a reader has to decode. Each line is now a public API call, the way a user writes it, and what the user gets back, with conditions as "when" and open laws marked. No title, no assertion, no helper. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 32 +++++++++++-------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 2c1979d..d2d1553 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -68,19 +68,24 @@ and nothing else: Write it as continuous prose in this order. Do not label the parts with the names below ("The hook", "The mechanism"): they are for you, not the reader. - 1. **The contract at a glance:** a short code block, first, that a reader - takes in at one look. One line, or two, for each main law: the real - call to the public entry point, the checkpoint as an expression, and - the relation that must hold. Use plain operators (`===`, `<=`) or a - standard assertion. A helper name (`sharesNoObject`) is fine when the - next sentence says what it means. Mark an open law `// open`. The - block states the promise, not the implementation: it never shows how - production computes the answer. For example: + 1. **The contract at a glance:** a short code block, first, that shows the public API promises this oracle maintains. A reader takes + it in at one look. Give it no title, and no sentence that introduces + it. Write each line as `call // → result`: + - the call is the public API, written the way a user writes it, with + concrete values where they make the promise clear. + - the result is what the user gets back or sees: a value, or a short + phrase. + - a condition goes at the end of the line, as "when ...". + - an open law ends with `(open)`. + + Keep it to about five lines. Never write an assertion, a test helper, + or a checkpoint expression there: those belong in the code. The block + never shows how production computes the answer. For example: ```ts - // For every x in the domain below: - assert.deepStrictEqual(klona(x), x) // same kind, prototype, and values - sharesNoObject(klona(x), x) // no object of x is reachable from the copy + klona(x) // → a deep copy: same kind, prototype, and values as x, sharing no object with it + typography({ scaleRatio: 2, baseFontSize: '16px' }) // → h1 font size 32px (ratio × base) + typography({ scaleRatio: 2, baseFontSize: '15.5px' }) // → h1 font size 31px (open) ``` 2. **What this oracle owns**, in one or two plain sentences: the @@ -217,8 +222,9 @@ Say in the evidence which trigger fired, or that none did. - From the prose alone, a reader can predict one observation of each law. For a no-throw law, the observation is that the call returns. -- The law block starts with the contract code block, and each line in it - states the same law as its sentence. +- The law block starts with the contract code block: public API calls and + their results, with no assertion or helper. Each line states the same law + as its sentence. - The opening names the promise that the oracle owns. - No law restates the code. - Each law sentence matches its block in the review notes. If they disagree, From 86fd7c1ee895f46b0a067bdf176561af4b4b3a0a Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 08:50:53 -0700 Subject: [PATCH 32/76] docs(oracles): write a shared call once in the contract block Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index d2d1553..c4b583b 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -68,8 +68,9 @@ and nothing else: Write it as continuous prose in this order. Do not label the parts with the names below ("The hook", "The mechanism"): they are for you, not the reader. - 1. **The contract at a glance:** a short code block, first, that shows the public API promises this oracle maintains. A reader takes - it in at one look. Give it no title, and no sentence that introduces + 1. **The contract at a glance:** a short code block, first, that shows the + public API promises this oracle maintains. A reader takes it in at one + look. Give it no title, and no sentence that introduces it. Write each line as `call // → result`: - the call is the public API, written the way a user writes it, with concrete values where they make the promise clear. @@ -77,6 +78,8 @@ and nothing else: phrase. - a condition goes at the end of the line, as "when ...". - an open law ends with `(open)`. + - when several laws share one call, write the call once, and put one + `// → result` line under it for each law. Keep it to about five lines. Never write an assertion, a test helper, or a checkpoint expression there: those belong in the code. The block From 4284ba774685fc2454a8a337e1652f3efa0114f8 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 08:52:25 -0700 Subject: [PATCH 33/76] docs(oracles): count results, not lines, in the contract block Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index c4b583b..2db1f02 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -81,7 +81,7 @@ and nothing else: - when several laws share one call, write the call once, and put one `// → result` line under it for each law. - Keep it to about five lines. Never write an assertion, a test helper, + Keep it to about five results. A call may span several lines. Never write an assertion, a test helper, or a checkpoint expression there: those belong in the code. The block never shows how production computes the answer. For example: From 714b3d65124a30b722142237507fbb1c8d5f0b67 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 08:56:00 -0700 Subject: [PATCH 34/76] docs(oracles): open the law block with what the oracle owns, then the code block One or two sentences first, so the reader knows what the file is for, then the contract as public calls and results. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 37 ++++++++++--------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 2db1f02..7a5c031 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -68,10 +68,19 @@ and nothing else: Write it as continuous prose in this order. Do not label the parts with the names below ("The hook", "The mechanism"): they are for you, not the reader. - 1. **The contract at a glance:** a short code block, first, that shows the - public API promises this oracle maintains. A reader takes it in at one - look. Give it no title, and no sentence that introduces - it. Write each line as `call // → result`: + 1. **What this oracle owns**, in one or two plain sentences: the + concrete promise it maintains, the public entry point and checkpoints + where the promise is owed, and the owned code. A reader who stops here + knows what the file is for. Name the promise itself. "The stylesheet + matches the options the README documents" names no promise. For + example: "This oracle owns `klona`'s deep-copy promise for the default + mode, checked on the value that `klona(x)` returns. It owns + `src/index.js#klona`." + + 2. **The contract at a glance:** a short code block, right after those + sentences, that shows the public API promises this oracle maintains. A + reader takes it in at one look. Give it no title, and no sentence that + introduces it. Write each line as `call // → result`: - the call is the public API, written the way a user writes it, with concrete values where they make the promise clear. - the result is what the user gets back or sees: a value, or a short @@ -81,9 +90,10 @@ and nothing else: - when several laws share one call, write the call once, and put one `// → result` line under it for each law. - Keep it to about five results. A call may span several lines. Never write an assertion, a test helper, - or a checkpoint expression there: those belong in the code. The block - never shows how production computes the answer. For example: + Keep it to about five results. A call may span several lines. Never write + an assertion, a test helper, or a checkpoint expression there: those + belong in the code. The block never shows how production computes the + answer. For example: ```ts klona(x) // → a deep copy: same kind, prototype, and values as x, sharing no object with it @@ -91,14 +101,6 @@ and nothing else: typography({ scaleRatio: 2, baseFontSize: '15.5px' }) // → h1 font size 31px (open) ``` - 2. **What this oracle owns**, in one or two plain sentences: the - concrete promise it maintains, the public entry point and checkpoints - where the promise is owed, and the owned code. A reader who stops here - knows what the file is for. Name the promise itself. "The stylesheet - matches the options the README documents" names no promise. For - example: "This oracle owns `klona`'s deep-copy promise for the default - mode, checked on the value that `klona(x)` returns. It owns - `src/index.js#klona`." 3. **The hook:** the one concrete case that shows where the promise is easy to get wrong. Write it as code where you can. Choose its form by the oracle's shape: @@ -225,8 +227,9 @@ Say in the evidence which trigger fired, or that none did. - From the prose alone, a reader can predict one observation of each law. For a no-throw law, the observation is that the call returns. -- The law block starts with the contract code block: public API calls and - their results, with no assertion or helper. Each line states the same law +- The law block starts with one or two sentences on what the oracle owns, + then the contract code block: public API calls and their results, with no + assertion or helper. Each line states the same law as its sentence. - The opening names the promise that the oracle owns. - No law restates the code. From a05cbba777b27975caded33ef7d73c0766c29436 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 08:58:19 -0700 Subject: [PATCH 35/76] docs(oracles): keep the opening sentences to about 40 words Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 7a5c031..39e7b6e 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -68,7 +68,8 @@ and nothing else: Write it as continuous prose in this order. Do not label the parts with the names below ("The hook", "The mechanism"): they are for you, not the reader. - 1. **What this oracle owns**, in one or two plain sentences: the + 1. **What this oracle owns**, in one or two plain sentences, about 40 + words in all: the concrete promise it maintains, the public entry point and checkpoints where the promise is owed, and the owned code. A reader who stops here knows what the file is for. Name the promise itself. "The stylesheet From 9c5a3e260fecf3f5e85fa9d5d29fe2c491d57d96 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 08:58:40 -0700 Subject: [PATCH 36/76] docs(oracles): rewrap the opening-sentences item Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 39e7b6e..597c6d1 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -68,12 +68,11 @@ and nothing else: Write it as continuous prose in this order. Do not label the parts with the names below ("The hook", "The mechanism"): they are for you, not the reader. - 1. **What this oracle owns**, in one or two plain sentences, about 40 - words in all: the - concrete promise it maintains, the public entry point and checkpoints - where the promise is owed, and the owned code. A reader who stops here - knows what the file is for. Name the promise itself. "The stylesheet - matches the options the README documents" names no promise. For + 1. **What this oracle owns**, in one or two plain sentences, about 40 words + in all: the concrete promise it maintains, the public entry point and + checkpoints where the promise is owed, and the owned code. A reader who + stops here knows what the file is for. Name the promise itself. "The + stylesheet matches the options the README documents" names no promise. For example: "This oracle owns `klona`'s deep-copy promise for the default mode, checked on the value that `klona(x)` returns. It owns `src/index.js#klona`." From b5836fba64aa97ec251e6c3c30e12e81acfd4242 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 09:00:22 -0700 Subject: [PATCH 37/76] docs(oracles): write the law block for maintainers and serious users The readers know the codebase. The prose names internal parts without introducing them, skips explaining basics, and spends its words on what such a reader could get wrong. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 597c6d1..9a4b06e 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -64,7 +64,11 @@ and nothing else: 1. **The law block**, the file's opening comment. The oracle file is a literate program (ORC-003), so write it as prose for a reader who has not - seen the review. Do not paste the template's fields or a list of law IDs. + seen the review. That reader is a maintainer or a serious user who knows + the codebase. Name internal parts without introducing them, and do not + explain the language, the library's basic ideas, or what a reader of the + code can see at once. Spend the words on what such a reader could get + wrong. Do not write about "a careless reader": state the case. Do not paste the template's fields or a list of law IDs. Write it as continuous prose in this order. Do not label the parts with the names below ("The hook", "The mechanism"): they are for you, not the reader. From cf644917f2d70224e1ced77eaea063205383fc88 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 10:02:58 -0700 Subject: [PATCH 38/76] docs(oracles): a guided first session, and the oracle file written once Onboarding now follows what an engineer trying the tool does. A new oracles-start skill gives a short introduction (docs/intro.md), looks around a named subsystem or the whole repository, explores one spot with probes, and writes sketches of candidate oracles in oracles/SKETCHES.md. Every finding needs a probe that calls the real API, because an unproven claim is often wrong. Probes also find the constraints, dynamics, boundaries, and inconsistencies that become the oracle's laws and histories. The law sheet and its review notes move out of the oracle file and into the sketch. oracles-design starts from an approved sketch, and writes the oracle file once, with its law block (about 40 lines, no review material) and its code together. oracles-setup becomes configuration before the first build, with fixes from the round-6 and round-7 runs: the ESLint 9 fallback that the plugin can resolve, a separate Vitest config when the main run would pick up oracle files, no silent Vitest upgrade, Flow syntax, ignored pointer directories, and local tarball installs. The README prompt starts oracles-start. Co-authored-by: Isaac --- packages/oracles/README.md | 40 ++- packages/oracles/docs/glossary.md | 35 +- .../docs/instruments/design-grammar.md | 4 +- packages/oracles/docs/intro.md | 54 +++ .../oracles/skills/oracles-design/SKILL.md | 329 ++++++------------ .../oracles/skills/oracles-review/SKILL.md | 6 +- .../oracles/skills/oracles-setup/SKILL.md | 65 ++-- .../oracles/skills/oracles-start/SKILL.md | 223 ++++++++++++ 8 files changed, 481 insertions(+), 275 deletions(-) create mode 100644 packages/oracles/docs/intro.md create mode 100644 packages/oracles/skills/oracles-start/SKILL.md diff --git a/packages/oracles/README.md b/packages/oracles/README.md index b1470c9..2f9843a 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -16,32 +16,34 @@ Paste this prompt into your coding agent at the root of the repository you want to test. Any agent that can run shell commands works. ```text -Set up oracle testing in this repository with @tanstack/oracles. +Show me what oracle testing with @tanstack/oracles can find in this repository. 1. Find the current commit of TanStack/oracles. Run `git ls-remote https://github.com/TanStack/oracles refs/heads/main`, and use the first 7 characters of the hash as . 2. Install https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@ as a dev dependency, with this repository's package manager. -3. Read node_modules/@tanstack/oracles/skills/oracles-setup/SKILL.md in full, - and follow it. Use wherever it asks for the commit to pin. -4. When the skill asks me to review the laws for the first oracle, stop and - show me the oracle file. +3. Read node_modules/@tanstack/oracles/skills/oracles-start/SKILL.md in full, + and follow it. Use wherever a skill asks for the commit to pin. +4. Stop after each step of the skill, show me what you found, and wait. ``` What happens next: -1. The agent installs this package and `@tanstack/eslint-plugin-oracles` at - the same commit. It adds the lint preset, writes a block in `AGENTS.md`, - and offers to make the skills findable for your coding agents. -2. It chooses one documented function and writes the first oracle file as - prose only: what the code promises, with the source of each promise. Then - it stops. -3. You review the laws. This is the one step that needs you: you decide what - the code promises, and the agent does the rest. -4. The agent adds the model, the histories, the driver, and the check. It runs - the oracle on your unchanged code and reports any defect with a replay - line. Then it runs the gap check and reports what is left. +1. The agent asks whether you have something specific in mind, introduces + oracles in a few sentences, and offers to look at a subsystem you name or + around the whole repository. +2. It reports two or three spots where an oracle would help, and you pick + one. +3. It explores that spot with probes: small scripts that call your real API. + It reports what it found, each with a command you can rerun: bugs, + places where the docs and the code disagree, behavior no test checks, and + design questions nobody has decided. +4. It writes sketches of candidate oracles in `oracles/SKETCHES.md`. You + choose one and answer its questions. This is the step that needs you: you + decide what the code promises. +5. The agent configures the repository, writes the oracle, runs it on your + unchanged code, and runs the gap check. The package needs Node 20 or later. The setup skill installs Vitest and fast-check if the repository does not have them. @@ -50,10 +52,12 @@ fast-check if the repository does not have them. | Path | What it is | | --------------------------------- | -------------------------------------------------------------------------------------------------- | -| `skills/oracles-setup/` | Setup, upgrade, and making the skills findable. | -| `skills/oracles-design/` | How to design, write, or change an oracle: prose laws first, model, histories, RED, GREEN, gaps. | +| `skills/oracles-start/` | A first session: an introduction, a look around, probes, and sketches of candidate oracles. | +| `skills/oracles-setup/` | Configuration, upgrade, and making the skills findable. | +| `skills/oracles-design/` | How to design, write, or change an oracle: from an approved sketch to model, histories, and gaps. | | `skills/oracles-evaluate-review/` | How to act on a review: a lossless ledger, RED on unchanged production, and closing the bug class. | | `skills/oracles-review/` | How to review oracle work. | +| `docs/intro.md` | What an oracle is, in a few minutes. | | `docs/oracle-checklist.md` | The complete requirements, by ID (ORC-001 to ORC-016). | | `docs/guide.md` | Techniques for each design and review step. | | `docs/model-styles.md` | One small example of each model style. | diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md index 62ba200..c3bcd2f 100644 --- a/packages/oracles/docs/glossary.md +++ b/packages/oracles/docs/glossary.md @@ -6,23 +6,24 @@ term for one concept. Do not rotate synonyms for style. ## Laws and evidence -| Term | Meaning | -| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | -| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | -| limits | What a property does not establish. Unresolved parts of a law go here. | -| law sheet | The first output of oracle design: the new oracle file with no code yet. Its opening comment states the laws in prose, and its review notes give each law's examples, general sentence, authority, status, rival, witness, and checkpoint. A human reviews it before any code is added. | -| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | -| rival | A plausible different law that predicts a different observation for some history. | -| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | -| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | -| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | -| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | -| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | -| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | -| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the user can pick. | -| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | -| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | +| Term | Meaning | +| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | +| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | +| limits | What a property does not establish. Unresolved parts of a law go here. | +| sketch | A candidate oracle in `oracles/SKETCHES.md`: its promise, its contract as calls and results, a block for each law (examples, general sentence, authority, status, rival, witness, checkpoint), what the probes found, and its open questions. A human chooses one and answers its questions before the oracle file is written. | +| probe | A small script that calls the real public API with chosen inputs and records what happens. A finding needs one. Probes find the constraints, dynamics, and boundaries that become an oracle's laws and histories. | +| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | +| rival | A plausible different law that predicts a different observation for some history. | +| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | +| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | +| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | +| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | +| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | +| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | +| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the user can pick. | +| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | +| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | ## The parts of an `oracleProperty` diff --git a/packages/oracles/docs/instruments/design-grammar.md b/packages/oracles/docs/instruments/design-grammar.md index 143be9f..099b3be 100644 --- a/packages/oracles/docs/instruments/design-grammar.md +++ b/packages/oracles/docs/instruments/design-grammar.md @@ -6,7 +6,7 @@ retries, or ownership. For an ordinary oracle, design step 2 is enough. **Inputs:** -- the law sheet and the frozen list from design step 2 +- the approved sketch and the frozen list from design step 2 - the owned production code - the known examples: docs, issues, and existing tests - one nearby invalid history that the oracle must reject @@ -25,7 +25,7 @@ of range. allow the same observations at the same checkpoints. Name the checkpoints, the assumptions, and the freedoms the contract leaves - open. The user's review of the law sheet fixes the intended contract. It is + open. The user's review of the sketch fixes the intended contract. It is not evidence that a model meets it. 2. **Separate what you found from what you added.** Each rule cites its diff --git a/packages/oracles/docs/intro.md b/packages/oracles/docs/intro.md new file mode 100644 index 0000000..f2931ab --- /dev/null +++ b/packages/oracles/docs/intro.md @@ -0,0 +1,54 @@ +# What an oracle is + +Types check that a value has the right shape. An oracle checks that the code +does what it promises, for every input and sequence of calls the promise +covers. It is engineering assurance for software: the way a bridge is +checked against its load rating, not against one truck. + +## From a test to an oracle + +A unit test records one answer. Give a sort function `[3, 1, 2]`, and check +that it returns `[1, 2, 3]`. You chose the input, and you supplied the +answer. + +A property test generalizes the input. A generator makes thousands of +lists, including empty ones, repeated values, and negative numbers. The test +checks one rule for each: the result holds exactly the original elements, in +ascending order. You still write the rule by hand. + +An oracle supplies the answer for any input. It is a small, independent +model that computes the expected result the simple way, even when production +uses complicated machinery to compute it fast. For a live query that keeps +the ten highest-priority open tasks up to date, the model keeps the tasks in +a plain `Map`, filters, sorts, and takes ten. The oracle gives the same +generated changes to the model and to the real code, and compares what a +user would see after each one. + +An isolated test preserves an answer. An oracle preserves the understanding +that produces the answers. + +## What oracles find + +Oracles find bugs that example tests miss, because the generator tries the +combinations nobody wrote down. They also find design questions: places +where the docs, the code, and the tests disagree, or behavior that nobody +decided. Writing the oracle forces each decision, and the oracle then holds +it. + +With an oracle in place, an agent can replace the machinery behind a promise +(a faster algorithm, less code, a different data structure) while the oracle +checks that the promise still holds. + +The gap check then shows what is left: production behavior that no law +judges yet. + +## How a first session goes + +1. Look around the code, or one subsystem you name. +2. Probe one area: small scripts that call the real API, to find what holds, + what changes, where the edges are, and where the sources disagree. +3. Sketch candidate oracles from what the probes found. +4. Build the one you choose. + +For the story of how TanStack DB uses oracles, read +[TanStack DB 1.0 RC1: Going All In on Oracles](https://tanstack.com/blog/tanstack-db-1.0-rc1-going-all-in-on-oracles). diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 9a4b06e..bff0cf7 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -1,6 +1,6 @@ --- name: oracles-design -description: Design, write, or change an oracle with @tanstack/oracles. Choose the boundary, write the oracle file's prose laws first, choose a model, and build histories that reach the states the law needs. Show the evidence the change needs, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. +description: Design, write, or change an oracle with @tanstack/oracles. Start from an approved sketch, write the oracle file with its law block and code, choose a model, and build histories that reach the states the law needs. Show the evidence the change needs, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. --- # Design an oracle @@ -11,7 +11,7 @@ that someone could predict an observation without reading production. The code comes after that. Read this skill in full, then read the [glossary](../../docs/glossary.md) -in full. It defines the terms that the sheet and the evidence use. +in full. It defines the terms that the sketch and the evidence use. The [checklist](../../docs/oracle-checklist.md) is the complete list of requirements. This skill cites it by ID, such as ORC-004. @@ -21,205 +21,43 @@ witness, and another round. The common shortcuts are changing production first, editing the model to match production, narrowing histories until a failure goes away, and pinning one example. `oracles-review` §2 lists them all. -## 1. Choose the boundary, then write the law sheet +## 1. Start from an approved sketch, then write the oracle file once -**Choose the boundary first.** An oracle maintains one caller-visible -promise, not one file. Before the law sheet, establish: - -| Question | What to establish | -| ---------------------------------- | --------------------------------------------------------------------------------------------- | -| What promise belongs together? | One caller-visible behavioral contract. | -| Where is it observed? | The public entry points and the checkpoints where it is owed. | -| What does it assume? | What providers, the environment, and neighboring components must supply. | -| What else witnesses it? | Other drivers for the same law: component, integration, end-to-end, or replay. | -| Where does another contract begin? | A different authority, lifecycle, observation boundary, or composition rule. | -| Which source implements it? | The `owns` list that the gap check uses. | -| What public state does it expose? | Each readable field or getter needs a law or a stated limit. A limit leaves its mutants open. | - -Example: an oracle for optimistic publication can hold several related laws -(an optimistic write is visible at once, a confirmation keeps it, and a -rollback removes it) and several drivers (an in-memory provider and a real one). It -keeps its identity if the code moves between files. One source file can also -serve several contracts. A new failure belongs to the oracle whose law covers -its promise. +**A new oracle starts from an approved sketch** in `oracles/SKETCHES.md` +(`oracles-start` step 4): the user chose it, and no open question is left. +If there is none, do `oracles-start` steps 3 and 4 for this boundary first: +explore with probes, write the sketch, and stop for review. Exploration and +review never go in the oracle file. The file is written once, when the +direction is settled, with its law block and its code together. **Decide whether the user must approve.** - **Strengthening an adopted law** (a missing history, observation, driver - path, or check) reuses the law as it is. Do not write a new sheet, and do + path, or check) reuses the law as it is. Do not write a new sketch, and do not stop. Name the law in the evidence. Widening the histories inside the law's stated domain is strengthening. Changing the stated domain, a checkpoint, or `limits` is a law change. - **Changing a law** (its promise, domain, checkpoint, assumptions, or the - freedoms it allows) needs a law delta. Edit the law block's prose in the - oracle file, and add review notes with the old and the new block for each - changed law. Stop for the user's review. -- **A new law, or missing or contradictory authority,** needs the full law - sheet below. Stop for the user's review. An analogy to another documented - rule is not authority: mark that law `open`. - -**Write the oracle file's prose first.** The law sheet is the new oracle -file before it has any code. Create it at its final path, with two comments -and nothing else: - -1. **The law block**, the file's opening comment. The oracle file is a - literate program (ORC-003), so write it as prose for a reader who has not - seen the review. That reader is a maintainer or a serious user who knows - the codebase. Name internal parts without introducing them, and do not - explain the language, the library's basic ideas, or what a reader of the - code can see at once. Spend the words on what such a reader could get - wrong. Do not write about "a careless reader": state the case. Do not paste the template's fields or a list of law IDs. - Write it as continuous prose in this order. Do not label the parts with - the names below ("The hook", "The mechanism"): they are for you, not the - reader. - 1. **What this oracle owns**, in one or two plain sentences, about 40 words - in all: the concrete promise it maintains, the public entry point and - checkpoints where the promise is owed, and the owned code. A reader who - stops here knows what the file is for. Name the promise itself. "The - stylesheet matches the options the README documents" names no promise. For - example: "This oracle owns `klona`'s deep-copy promise for the default - mode, checked on the value that `klona(x)` returns. It owns - `src/index.js#klona`." - - 2. **The contract at a glance:** a short code block, right after those - sentences, that shows the public API promises this oracle maintains. A - reader takes it in at one look. Give it no title, and no sentence that - introduces it. Write each line as `call // → result`: - - the call is the public API, written the way a user writes it, with - concrete values where they make the promise clear. - - the result is what the user gets back or sees: a value, or a short - phrase. - - a condition goes at the end of the line, as "when ...". - - an open law ends with `(open)`. - - when several laws share one call, write the call once, and put one - `// → result` line under it for each law. - - Keep it to about five results. A call may span several lines. Never write - an assertion, a test helper, or a checkpoint expression there: those - belong in the code. The block never shows how production computes the - answer. For example: - - ```ts - klona(x) // → a deep copy: same kind, prototype, and values as x, sharing no object with it - typography({ scaleRatio: 2, baseFontSize: '16px' }) // → h1 font size 32px (ratio × base) - typography({ scaleRatio: 2, baseFontSize: '15.5px' }) // → h1 font size 31px (open) - ``` - - 3. **The hook:** the one concrete case that shows where the promise is - easy to get wrong. Write it as code where you can. Choose its form by - the oracle's shape: - - | Shape | Hook | - | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | - | State, timing, or order with a surprising rule | A question that names the surprise: "Why does a ready transaction wait?" | - | A pure function with documented examples | The documented example, then the nearby case that a careless reading gets wrong: ratio 2 and base 16px give 32px, so what does base 15.5px give? | - | The obvious implementation is wrong | The plausible implementation and the one input where it fails: why a deep copy is not a JSON round trip | - - When two forms fit, choose the one whose case a reader is most likely - to get wrong. - - 4. **The mechanism:** the two or three moving parts that make the - behavior necessary, in plain sentences. Give each a short name (the - window, the head, the applied receipt), and use the same name in the - law names. When you write the code, use it in the `states` and the - calibrations too. - 5. **The laws:** each one sentence over its domain ("For every ..."), with - the source of its promise in the documentation's own words where you - can. Then the edges where it is easy to get wrong, and what a caller - sees when the law breaks. - 6. **How the oracle knows:** what the model reads to predict each law, - which histories reach the edges, what the driver observes and does not - observe and why, and where the check compares. - 7. **Limits and open questions**, last, each with its reason. - - Avoid the two generic openings: a summary that says the code does what - its documentation promises, and a law that restates the implementation. A - model of that law would be the same code, so it could not catch a defect. - -2. **Review notes**, a second comment headed "Review notes (delete when - approved)". Docs, tests, and bug reports give examples. A law is a - statement over the whole domain. The template below makes you write the - step from one to the other. Copy it once for each law, and fill in every - field. Then list the open questions. - -```markdown -### L1 - -- Examples: -- What varies, what stays: -- Law: For every , . -- Domain edges: -- New case: -- Rebuilt examples: -- Dynamics: -- Constraints: -- Boundary conditions: -- Authority: -- Status: adopted | observed | open -- Rival: -- Witness: -- Checkpoint: -``` - -How to fill in the fields: - -- **Examples** and **What varies, what stays:** for example, `ms('2 hrs')` - is `7200000` (README, Examples). The number, the spelling, and the spacing - can change. The unit's length decides the answer. -- **Law:** one sentence over the domain. For example: "For every finite - number `n` and every unit name `u` in the table, in any case and with or - without a space, `parse(n + u)` is `n` times the length of `u`." -- **New case:** if you cannot write one, the law is still a list of - examples. Write it again. A new case that only renames a value in the - example (`/:id` for `/:foo`) does not count. -- **Domain edges:** wider than the examples. A block whose domain is only its - own examples is not a law. Keep those examples as `fixed` histories, and as - rebuilt examples of the general law that explains them. -- **Rebuilt examples:** an example that the law gets wrong shows a missing - condition, a second law, or a source that disagrees. Say which. -- **Dynamics** are the model's rules, and the histories vary them. For a pure - function, write "none: one call", and put the variation into the domain. -- **Constraints** hold whatever happens: a count never above its limit, an - equality that is symmetric, a read that returns the last write. The check - tests them at every checkpoint. -- **Boundary conditions** are the edges, where real behavior and most - defects live: empty, zero, `-0`, the size limit, at capacity, the first and - the last item, keys that shadow prototype names, cleanup. A boundary that - the law does not decide goes under open questions. Do not leave it out of - the grammar. -- **Status:** `adopted` if it has an authority, `observed` if production does - it but nobody adopted it, `open` if it is a question. Only `adopted` laws - become expected results. An `observed` or `open` law goes to the user as a - question. -- **Production output is evidence, never authority.** A rule that you write - only because production does it is model hacking. Mark that rule - `observed`, and ask the user. An analogy to another documented rule is not - authority either: mark that law `open`. - -**Required blocks: the laws people forget.** The review notes have a block -for each of these, or one line that says why it does not apply: - -- **No throw on valid input.** When the contract says a call returns a value, - it promises not to throw for valid input. State that as a law, with the same - authority, so a mutant that throws is a kill, not a crash. -- **The laws of the kind of function.** An equality check is reflexive and - symmetric. A parser and a serializer round-trip. A setter is idempotent when - set to the same value. Their authority is the function's documented purpose. - -The task already approved these laws? Do not ask again. - -The law block and the review notes must agree: each law in the prose has its -block, with the same sentence. + freedoms it allows) needs a law delta. Add it to the sketch: the old and + the new block for each changed law, with the probe that shows why. Stop + for the user's review, then update the oracle file. +- **A new law, or missing or contradictory authority,** needs an approved + sketch. An analogy to another documented rule is not authority. +- **The task already approved these laws?** Do not ask again. + +**Production output is evidence, never authority.** A rule that exists only +because production does it is model hacking. When the law and the code +disagree, the sketch's answer decides, not the code. When the promise is unclear, or two readings predict different behavior, use the [law discovery](../../docs/instruments/law-discovery.md) card. **Use the [design grammar](../../docs/instruments/design-grammar.md) card when any of these is true.** Do not judge whether the law is hard. Check the -list at the sheet, and again after each gap check: +list at the sketch, and again after each gap check: -- the sheet has more than five laws for one boundary (adopted, observed, and - open ones all count) +- the sketch has more than five laws for one boundary (adopted, observed, + and open ones all count) - a law has no new case that it predicts - the model has more than one branch per law - the work changes or simplifies an existing model @@ -227,38 +65,101 @@ list at the sheet, and again after each gap check: Say in the evidence which trigger fired, or that none did. -**Check the law block before you stop:** +**Write the law block.** The law block is the file's opening comment. The oracle file is a literate +program (ORC-003), so write it as prose. That reader is a maintainer or a serious user who knows +the codebase. Name internal parts without introducing them, and do not +explain the language, the library's basic ideas, or what a reader of the +code can see at once. Spend the words on what such a reader could get +wrong. Do not write about "a careless reader": state the case. Do not paste the template's fields or a list of law IDs. +Write it as continuous prose in this order. Do not label the parts with +the names below ("The hook", "The mechanism"): they are for you, not the +reader. + +1. **What this oracle owns**, in one or two plain sentences, about 40 words + in all: the concrete promise it maintains, the public entry point and + checkpoints where the promise is owed, and the owned code. A reader who + stops here knows what the file is for. Name the promise itself. "The + stylesheet matches the options the README documents" names no promise. For + example: "This oracle owns `klona`'s deep-copy promise for the default + mode, checked on the value that `klona(x)` returns. It owns + `src/index.js#klona`." + +2. **The contract at a glance:** a short code block, right after those + sentences, that shows the public API promises this oracle maintains. A + reader takes it in at one look. Give it no title, and no sentence that + introduces it. Write each line as `call // → result`: + - the call is the public API, written the way a user writes it, with + concrete values where they make the promise clear. + - the result is what the user gets back or sees: a value, or a short + phrase. + - a condition goes at the end of the line, as "when ...". + - an open law ends with `(open)`. + - when several laws share one call, write the call once, and put one + `// → result` line under it for each law. + + Keep it to about five results. A call may span several lines. Never write + an assertion, a test helper, or a checkpoint expression there: those + belong in the code. The block never shows how production computes the + answer. For example: + + ```ts + klona(x) // → a deep copy: same kind, prototype, and values as x, sharing no object with it + typography({ scaleRatio: 2, baseFontSize: '16px' }) // → h1 font size 32px (ratio × base) + typography({ scaleRatio: 2, baseFontSize: '15.5px' }) // → h1 font size 31px (open) + ``` + +3. **The hook:** the one concrete case that shows where the promise is + easy to get wrong. Write it as code where you can. Choose its form by + the oracle's shape: + + | Shape | Hook | + | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | + | State, timing, or order with a surprising rule | A question that names the surprise: "Why does a ready transaction wait?" | + | A pure function with documented examples | The documented example, then the nearby case that a careless reading gets wrong: ratio 2 and base 16px give 32px, so what does base 15.5px give? | + | The obvious implementation is wrong | The plausible implementation and the one input where it fails: why a deep copy is not a JSON round trip | + + When two forms fit, choose the one whose case a reader is most likely + to get wrong. + +4. **The mechanism:** the two or three moving parts that make the + behavior necessary, in plain sentences. Give each a short name (the + window, the head, the applied receipt), and use the same name in the + law names. When you write the code, use it in the `states` and the + calibrations too. +5. **The laws:** each one sentence over its domain ("For every ..."), with + the source of its promise in the documentation's own words where you + can. Then the edges where it is easy to get wrong, and what a caller + sees when the law breaks. +6. **How the oracle knows:** what the model reads to predict each law, + which histories reach the edges, what the driver observes and does not + observe and why, and where the check compares. +7. **Limits and open questions**, last, each with its reason. + +Avoid the two generic openings: a summary that says the code does what +its documentation promises, and a law that restates the implementation. A +model of that law would be the same code, so it could not catch a defect. + +Keep the law block to about 40 lines of prose, plus the contract code block. +It never mentions the sketch, a review, or an open question: a point the +sketch left undecided appears only as a limit, with its reason. + +**Then write the code under it.** Each boundary condition in the sketch +becomes a constructed history. Each law's sentence, authority, and limits go +in that property's `law` option. Put prose beside the model, the histories, +the driver, and the check that says why each one predicts or tests the +promised result. + +**Check the law block:** - From the prose alone, a reader can predict one observation of each law. For a no-throw law, the observation is that the call returns. -- The law block starts with one or two sentences on what the oracle owns, - then the contract code block: public API calls and their results, with no - assertion or helper. Each line states the same law - as its sentence. -- The opening names the promise that the oracle owns. +- It starts with one or two sentences on what the oracle owns, then the + contract code block: public API calls and their results, with no assertion + or helper. Each line states the same law as its sentence. - No law restates the code. -- Each law sentence matches its block in the review notes. If they disagree, - the review notes decide: fix the prose, and tell the user. A new heading - for a law is not a law change. A changed sentence, domain, or checkpoint is. - -**Stop here** when the sheet or a law delta is needed. Do not stop with an -empty field: each law has a filled block first. Do not add code to the file -until the user has reviewed it. With no user present, write the file and -stop. A law sheet has no default. Until it has code, the file fails Vitest -with "No test suite found", so keep it off a branch whose CI must pass, or -say in the review request that it will fail. - -**Once the user approves it:** - -- Apply the review's changes to the law block first. The approved review - notes are the anchor: a later edit to the prose that changes a law's - domain, checkpoint, or strength ("may" to "does") is a law change. -- Delete the review notes. They stay in the review record. -- Add the code under the law block. Each boundary condition becomes a - constructed history, and each law's sentence, authority, and limits go in - that property's `law` option. -- Put prose beside the model, the histories, the driver, and the check that - says why each one predicts or tests the promised result. +- Each law sentence matches its block in the sketch. If they disagree, the + sketch decides: fix the prose, and tell the user. A new heading for a law + is not a law change. A changed sentence, domain, or checkpoint is. Keep the law block, the `law` option, and the model in step: a change to one updates the others in the same change (ORC-003). @@ -267,7 +168,7 @@ updates the others in the same change (ORC-003). Write these down. They stay fixed while you build the oracle. -- The laws and their checkpoints, from the approved law block. +- The laws and their checkpoints, from the approved sketch. - The owned production code: the files or functions the oracle answers for. - The freedoms the contract leaves open, such as the order of independent events, scheduling, or representation. The oracle must accept every diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index 493e174..bed6e80 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -41,9 +41,9 @@ contains one is not ready. | Shortcut | What to look for | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Law after code** | Oracle code written before the user approved the file's prose laws, a law with no authority, or a law block that disagrees with the `law` option or the model. Also a law block that is a list of fields or law IDs, not prose (ORC-003). | -| **Law as examples** | A law written as a list of cases, not one sentence over the domain. Review notes with no new case that the law predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | -| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired at the sheet or after a gap check, and the design grammar card was not used. | +| **Law after code** | An oracle written without an approved sketch, a law with no authority, or a law block that disagrees with the `law` option or the model. Also a law block that is a list of fields or law IDs, not prose (ORC-003), or one that carries review material or open questions. | +| **Law as examples** | A law written as a list of cases, not one sentence over the domain. A sketch with no new case that the law predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | +| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired at the sketch or after a gap check, and the design grammar card was not used. | | **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | | **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | | **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index a01c768..b9e3477 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -1,6 +1,6 @@ --- name: oracles-setup -description: Set up @tanstack/oracles in a repository. Install the pinned packages, add the lint preset, and write the AGENTS.md block. Make the oracle skills findable by coding agents, and run a first oracle through the gap check. Use when a repository adopts @tanstack/oracles or upgrades its pinned commit. +description: Configure a repository for @tanstack/oracles. Install the pinned packages, add the lint preset and the Vitest config, write the AGENTS.md block, and make the oracle skills findable by coding agents. Then build the first oracle from an approved sketch and run it through the gap check. Use once oracles-start has an approved sketch, or when a repository upgrades its pinned commit. --- # Set up @tanstack/oracles @@ -8,8 +8,11 @@ description: Set up @tanstack/oracles in a repository. Install the pinned packag You are setting up oracle testing in this repository. Do the steps in order. Ask the user before each step that writes outside the repository. When no user is available, choose the default this skill gives, record each choice in -your report, and continue. That default covers setup questions only. A law -sheet has no default: write it, report it, and stop. +your report, and continue. That default covers setup questions only. A +sketch has no default: write it, report it, and stop. + +Setup starts once the engineer has chosen a sketch in `oracles-start`. If +there is none yet, load `oracles-start` first. Setup ends when one oracle passes on production and the gap check leaves no gap open in that oracle's own contract, unless the user decides otherwise. A @@ -53,7 +56,10 @@ pnpm add -D \ fast-check vitest ``` -Add `fast-check` and `vitest` beside them. For lint, also add `eslint@^9` +Add `fast-check` and `vitest` beside them. The framework needs Vitest 4.1 or +later. If the repository already runs an older Vitest, upgrading it is the +user's decision: ask, and with no user present, do not upgrade. Report it as +a blocker. For lint, also add `eslint@^9` if the repository has no ESLint, and `@typescript-eslint/parser@^8` if nothing parses TypeScript yet. Name those majors: a package manager can pick a prerelease otherwise. If the repository's ESLint is older than 9 (xo @@ -79,6 +85,10 @@ a dependency's build script stops every later pnpm command (`ERR_PNPM_IGNORED_BUILDS`), and pnpm writes an `allowBuilds` placeholder into `pnpm-workspace.yaml`. Answer it, usually `false`, and tell the user. +A run that installs from a local tarball has no commit to pin or check. Say +which tarball you installed, and note that `package.json` now points at a +local path that must be replaced before anything is committed. + To upgrade, change both URLs to the new commit at once. Then do step 4 and step 5 again (copy each pointer's `name` and `description` again, since they can change), and rerun the gap check (step 7). @@ -98,6 +108,9 @@ the oracle files: oracle files in their own directory, such as `oracles/`. - If its default glob matches `*.test.ts` (better-node-test, Jest), name the oracle files `*.oracle.ts`, and include that name in the Vitest config. +- If the repository's own Vitest run would also pick them up (a config with + no `include`, or one whose setup needs a server), name the oracle files + `*.oracle.ts`, and give them their own `vitest.oracle.config.ts`. The gap check finds oracle files by their `oracleProperty` call, so the name only has to match the Vitest config and the lint glob. @@ -120,6 +133,10 @@ export default defineConfig({ }) ``` +If the production source uses syntax that Vite cannot parse (Flow types, +for example), add a Vite plugin to the oracle config that strips it, such as +one built on `flow-remove-types`. + Add a script for it, such as `test:oracles`. Leave the existing suite unchanged. In a published package, keep the new files out of the published package (its `files` field or `.npmignore`): the configs, `AGENTS.md`, @@ -192,8 +209,14 @@ lint. Lint the oracle files with a separate ESLint 9: manager's equivalent. 2. Write a flat `eslint.oracle.config.mjs` with both presets and a TypeScript parser for the oracle files and the production source. -3. Run it as `npx -y -p eslint@9 eslint -c eslint.oracle.config.mjs `, and add a `lint:oracles` script. +3. Run it with an ESLint 9 that the plugin can find: the plugin loads + `eslint` at run time, from where the plugin is installed. `npx -p eslint@9` + is not enough, because npx runs the repository's local ESLint when one + exists. With pnpm, use `pnpm dlx eslint@9 -c eslint.oracle.config.mjs + `. Otherwise, install `eslint@^9` in the + workspace package that holds the plugin, if that does not replace the + ESLint that the repository's own lint uses. If neither works, lint is an + environment limit: report it. Add a `lint:oracles` script. The config's parser import resolves from the repository, so it uses the repository's own `@typescript-eslint/parser`. @@ -211,7 +234,7 @@ Add this block to `AGENTS.md` (create the file if needed). On an upgrade, replace everything between the markers and keep the rest of the file. ```md - + ## Oracle tests @@ -225,6 +248,9 @@ is your call, and you will keep building on it. Make it the smallest, clearest code that satisfies the oracles: merge repeated code, remove code that no law needs (after the checks in `oracles-design` step 8), and restructure freely while the oracles pass. +- To find where an oracle would help, or to sketch a new one, read + `node_modules/@tanstack/oracles/skills/oracles-start/SKILL.md` in full and + follow it. - Before you create or change an oracle, or change production code that an oracle owns, read `node_modules/@tanstack/oracles/skills/oracles-design/SKILL.md` in full and follow it. @@ -271,8 +297,13 @@ choices. They can pick more than one. | **Global pointers** | A pointer skill per skill in `~/.agents/skills/` and `~/.claude/skills/` (and any other skills directory the user's agents read) | The user wants the skills in every repository that uses `@tanstack/oracles`. | Write a pointer skill as `//SKILL.md` for each of -`oracles-design`, `oracles-review`, `oracles-evaluate-review`, and -`oracles-setup`. +`oracles-start`, `oracles-design`, `oracles-review`, +`oracles-evaluate-review`, and `oracles-setup`. + +Before you write repository pointers, check that git does not ignore their +directories (`git check-ignore .claude/skills .agents/skills`). If one is +ignored, tell the user: they can commit the pointers in the other +directory, or add an exception to `.gitignore`. - Copy `name` and `description` exactly from the package's `SKILL.md`, so the agent selects it for the same requests. @@ -307,15 +338,11 @@ Before writing a global pointer: - If a skill with the same name exists, show it and ask. Never overwrite it silently. -## 6. Write the first oracle +## 6. Build the first oracle -Choose one public, documented promise with a small owned surface: an -exported function or method that callers use, with a contract in the README -or the API docs. Its law must say more than its code does: if a model of the -law would repeat the implementation (an internal helper that only wraps or -escapes a string), choose the caller-visible promise that uses it. If most -of its laws would have no authority (no docs, only tests or code), choose -another one. Ask the user which one if it is not obvious. +The first oracle is the sketch that the engineer chose in `oracles-start`, +with its questions answered. Load `oracles-design`, and follow it from its +first step. If the first oracle fails on unchanged production under an adopted law, it found a defect. The RED is the finding: report it with its replay line, and @@ -326,10 +353,6 @@ the gap check there, and say in the report that it ran on the patched code. With the user's approval, follow `oracles-design` step 7's defect path for the real fix. -Then load `oracles-design` and follow it from its first step. The oracle -file's prose laws come first, and the user reviews them before you add code. -Setup does not skip that step to finish sooner. - Put the oracle's `owns` list on its `oracleProperty` call. An entry is a file (`src/index.ts`) or one function in it (`src/index.ts#parse`, or `src/x.ts#Class.method`). A function entry owns that function and its static diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md new file mode 100644 index 0000000..d254a52 --- /dev/null +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -0,0 +1,223 @@ +--- +name: oracles-start +description: Introduce @tanstack/oracles to an engineer and find a first oracle in their codebase. Give a short introduction, look around a subsystem they name or the whole repository, explore one area with probes that prove each finding, and write sketches of candidate oracles for the engineer to choose from. Use when someone is trying @tanstack/oracles, asks where an oracle would help, or wants a new oracle and has no approved sketch yet. +--- + +# Start with oracles + +The engineer is trying this tool, often in the mood of "impress me". Earn it +with evidence about their own code: a probe that shows a bug, an +inconsistency between the docs and the code, or a behavior that no test +checks. An agent that only says things is often wrong, and one wrong claim +costs the tool its credibility. So never claim what a probe has not shown. + +Probes are the small first instrument. They bootstrap the oracle: each probe +finds a constraint (or its absence), a dynamic, or a boundary, and those +become the oracle's laws and histories. The oracle then explores the whole +design space and checks the production code far more thoroughly than any +probe can. + +Work in these four steps. Stop after each one, show the result, and wait for +the engineer. Then `oracles-design` builds the oracle that the engineer +chooses. + +## 1. Ask, then introduce + +Ask one question first: is there something specific they want to try this +on? Then introduce the tool in a few sentences and one example, from the +[introduction](../../docs/intro.md). Keep it short: the engineer came to see +it work. + +Offer two ways to start: + +- **A subsystem they name:** look closely at that part of the code. +- **No preference:** look around the repository for a good first oracle. + +Say what you will do: read the repository, run small scripts against its +code from a scratch directory, and change no file in the repository until +the engineer chooses an oracle to build. + +## 2. Look around + +Make it quick. Read the public API and its docs (the README, the API docs, +the JSDoc), the existing tests, and the git history of fixes (for example +`git log --oneline -i --grep fix`, and the files that fixes change most). + +Look for behavior that is easy to break, and whose right answer is simpler +to state than the code that produces it. Good signs: + +- a round trip (parse and serialize, encode and decode, copy and compare) +- documented examples and defaults +- a cache, a queue, or other state that changes over a sequence of calls +- an ordering, timing, or normalization rule +- an invariant stated in a comment +- code that git history shows was fixed more than once + +Avoid an internal helper whose only law would restate its code. Avoid +behavior that nothing documents, unless the engineer wants to decide it. + +Report two or three spots, a short paragraph each: what the code seems to +promise and where that is stated, and why it makes a good first oracle. +Recommend one. The engineer picks. + +## 3. Explore with probes + +Read the chosen spot's docs, code, tests, and fix history. Then probe it. + +**A probe** is a small script that calls the real public API with chosen +inputs and prints what happens. Run it from a scratch directory, outside the +repository or in an ignored path, against the unchanged code. Keep each +probe and its output, so the engineer can run it again. + +**Every finding needs a probe.** A claim without one is a suspicion: label +it as a suspicion, or leave it out. + +**Probe to find the design, not only bugs.** Look for: + +- **Constraints:** what holds whatever the input or the sequence (a round + trip, an order, uniqueness, a bound), and where it stops holding. +- **Dynamics:** how the result or the state changes with each call or + input. +- **Boundaries:** empty, zero, `-0`, limits, the first and the last item, + capacity, time, and odd input (digits, unicode, lone surrogates, keys that + shadow prototype names). +- **Inconsistencies:** places where the docs, the code, the tests, and the + examples disagree, and behavior that nobody decided, where two readings + predict different results. + +When a probe is cheap, run it over a grid or a random sample of inputs, +count the failures, and show the smallest one. + +**Report each finding with its kind:** + +| Kind | What it means | +| --------------- | ----------------------------------------------------------------------- | +| Bug | The code breaks a promise that a document states. | +| Inconsistency | Two sources disagree: say which ones, and what each says. | +| Test gap | A probe shows behavior that no existing test checks. | +| Design question | Nothing decides the behavior: give two readings and what each predicts. | + +For each finding, give the promise and its source, the input, what +happened, the command that reproduces it, and why the existing tests miss +it. For a bug, ask the engineer two questions: is it unintended, and did +they already know? + +Then say why an oracle would help here: it would generate the whole domain +instead of a few probes, state the law that the probes point at, settle the +open questions, and hold the answer for every later change. For example: + +> Column mapping between snake_case and camelCase. The JSDoc promises a +> round trip. The tests cover `user_id` and `created_at`. A probe over +> 21,844 names found 3,518 that do not round-trip, for example +> `address2_line → address2Line → address2line`. The JSDoc also shows +> `userHTTPSURL → user_https_url`, but the code gives `user_httpsurl`. An +> oracle would generate every name shape, state the round trip as a law, +> settle the acronym rule, and hold it. + +If the probes find nothing, say so plainly: "No confirmed defect in the +paths examined." Then report what they did find: the constraints that held, +and the design questions. + +## 4. Sketch candidate oracles + +Write one to three candidate oracles for the chosen spot in +`oracles/SKETCHES.md`. Create the file if it does not exist, and add to it +if it does: it is the repository's list of oracle ideas. Mark a sketch +"built" once its oracle exists. + +Each sketch establishes the boundary: + +| Question | What to establish | +| ---------------------------------- | --------------------------------------------------------------------------------------------- | +| What promise belongs together? | One caller-visible behavioral contract. | +| Where is it observed? | The public entry points and the checkpoints where it is owed. | +| What does it assume? | What providers, the environment, and neighboring components must supply. | +| What else witnesses it? | Other drivers for the same law: component, integration, end-to-end, or replay. | +| Where does another contract begin? | A different authority, lifecycle, observation boundary, or composition rule. | +| Which source implements it? | The `owns` list that the gap check uses. | +| What public state does it expose? | Each readable field or getter needs a law or a stated limit. A limit leaves its mutants open. | + +Then each sketch gives: + +- **The promise,** in one or two sentences, and what the oracle would own. +- **The contract,** as `call // → result` lines (see `oracles-design` step 1). +- **One block for each law,** from this template. It is where examples + become a general law. + +```markdown +### L1 + +- Examples: +- What varies, what stays: +- Law: For every , . +- Domain edges: +- New case: +- Rebuilt examples: +- Dynamics: +- Constraints: +- Boundary conditions: +- Authority: +- Status: adopted | observed | open +- Rival: +- Witness: +- Checkpoint: +``` + +How to fill in the fields: + +- **Examples** and **What varies, what stays:** for example, `ms('2 hrs')` + is `7200000` (README, Examples). The number, the spelling, and the spacing + can change. The unit's length decides the answer. +- **Law:** one sentence over the domain. For example: "For every finite + number `n` and every unit name `u` in the table, in any case and with or + without a space, `parse(n + u)` is `n` times the length of `u`." +- **New case:** if you cannot write one, the law is still a list of + examples. Write it again. A new case that only renames a value in the + example (`/:id` for `/:foo`) does not count. +- **Domain edges:** wider than the examples. A block whose domain is only its + own examples is not a law. Keep those examples as `fixed` histories, and as + rebuilt examples of the general law that explains them. +- **Rebuilt examples:** an example that the law gets wrong shows a missing + condition, a second law, or a source that disagrees. Say which. +- **Dynamics** are the model's rules, and the histories vary them. For a pure + function, write "none: one call", and put the variation into the domain. +- **Constraints** hold whatever happens: a count never above its limit, an + equality that is symmetric, a read that returns the last write. The check + tests them at every checkpoint. +- **Boundary conditions** are the edges, where real behavior and most + defects live: empty, zero, `-0`, the size limit, at capacity, the first and + the last item, keys that shadow prototype names, cleanup. A boundary that + the law does not decide goes under open questions. Do not leave it out of + the grammar. +- **Status:** `adopted` if it has an authority, `observed` if production does + it but nobody adopted it, `open` if it is a question. Only `adopted` laws + become expected results. An `observed` or `open` law goes to the user as a + question. +- **Production output is evidence, never authority.** A rule that you write + only because production does it is model hacking. Mark that rule + `observed`, and ask the user. An analogy to another documented rule is not + authority either: mark that law `open`. + +The sketch also gives: + +- **What the probes found,** sorted into constraints, dynamics, and + boundaries. Each one becomes a law, a history dimension, or a constructed + history. Link each probe. +- **The laws people forget,** as blocks, or one line that says why each does + not apply: + - **No throw on valid input.** When the contract says a call returns a + value, it promises not to throw for valid input. + - **The laws of the kind of function.** An equality check is reflexive and + symmetric. A parser and a serializer round-trip. A setter is idempotent + when set to the same value. +- **Open design questions,** each with its two readings and what the probe + showed. + +Check each sketch: from its contract and laws alone, a reader can predict +one observation of each law, and no law restates the code. + +**Stop here.** The engineer picks a sketch and answers its questions. Record +the answers in the sketch. A sketch has no default: with no engineer +present, write it and stop. When the chosen sketch has no open question +left, its direction is settled. Then load `oracles-setup` if the repository +is not configured yet, and then `oracles-design` to build it. From 8c8311eea987d866c349aca498a60ed36640c288 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 12:58:18 -0700 Subject: [PATCH 39/76] docs(oracles): rewrite the first session as a guided, probe-first walk to the aha From the user's own onboarding tests and a design-grammar pass over them. The first session now opens by saying what it will do, shows instead of explaining (the introduction becomes a reference for questions), and stops only where the engineer must decide. It is written as an engaging teacher: it narrates what it does and why, tied to what it just showed. Findings follow the shape of a maintainer report (possible bug, smallest case, the authority quoted with file and line, why the tests miss it, how often), and the authority found for each promise carries into the law and the law block. Potential oracles and potential laws stay in the chat, in consistent terms. Nothing is written to the repository but the minimal installs and the oracle file. The 14-field law template becomes the agent's private worksheet. The session builds and runs the oracle until it finds a bug and the engineer has settled some questions, keeps the prose current as laws change, and then presents, offers a walk-through of the model, histories, and driver, and explains the path to production. With no bug, it offers the next oracle. oracles-setup becomes taking an oracle to production, and checks CLAUDE.md as well as AGENTS.md. oracles-design starts from approved potential laws, and adds two rules: no wall-clock waits, and a test-free companion module for a model that something else imports. Co-authored-by: Isaac --- packages/oracles/README.md | 45 +-- packages/oracles/docs/glossary.md | 36 +- .../docs/instruments/design-grammar.md | 4 +- packages/oracles/docs/intro.md | 16 +- .../oracles/skills/oracles-design/SKILL.md | 66 ++-- .../oracles/skills/oracles-review/SKILL.md | 6 +- .../oracles/skills/oracles-setup/SKILL.md | 36 +- .../oracles/skills/oracles-start/SKILL.md | 331 ++++++++++-------- 8 files changed, 303 insertions(+), 237 deletions(-) diff --git a/packages/oracles/README.md b/packages/oracles/README.md index 2f9843a..a469e51 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -24,26 +24,31 @@ Show me what oracle testing with @tanstack/oracles can find in this repository. 2. Install https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@ as a dev dependency, with this repository's package manager. 3. Read node_modules/@tanstack/oracles/skills/oracles-start/SKILL.md in full, - and follow it. Use wherever a skill asks for the commit to pin. -4. Stop after each step of the skill, show me what you found, and wait. + and follow it. ``` What happens next: -1. The agent asks whether you have something specific in mind, introduces - oracles in a few sentences, and offers to look at a subsystem you name or - around the whole repository. -2. It reports two or three spots where an oracle would help, and you pick - one. -3. It explores that spot with probes: small scripts that call your real API. - It reports what it found, each with a command you can rerun: bugs, - places where the docs and the code disagree, behavior no test checks, and - design questions nobody has decided. -4. It writes sketches of candidate oracles in `oracles/SKETCHES.md`. You - choose one and answer its questions. This is the step that needs you: you - decide what the code promises. -5. The agent configures the repository, writes the oracle, runs it on your - unchanged code, and runs the gap check. +1. The agent says what the session will do, and asks whether you have a + part of the code in mind. +2. It looks around, and suggests two or three places where an oracle would + help. You pick one. +3. It tries that code with probes: small scripts that call your real API. + Each finding comes with a snippet you can rerun, and the doc or test that + the behavior seems to break: possible bugs, places where the docs and the + code disagree, behavior no test checks, and design questions nobody has + decided. +4. It proposes a potential oracle for that behavior, with its potential laws. + You approve the laws. This is the step that needs you: you decide what the + code promises. +5. It writes the oracle file and runs it on your unchanged code, telling you + what each part does as it goes, until the oracle has found a bug and you + have settled some design questions. +6. It shows you what it built, offers to walk you through how it works, and + explains how to take the oracle to production. + +Until you choose to ship it, the agent changes nothing in your repository +except the minimal installs and the oracle file. The package needs Node 20 or later. The setup skill installs Vitest and fast-check if the repository does not have them. @@ -52,12 +57,12 @@ fast-check if the repository does not have them. | Path | What it is | | --------------------------------- | -------------------------------------------------------------------------------------------------- | -| `skills/oracles-start/` | A first session: an introduction, a look around, probes, and sketches of candidate oracles. | -| `skills/oracles-setup/` | Configuration, upgrade, and making the skills findable. | -| `skills/oracles-design/` | How to design, write, or change an oracle: from an approved sketch to model, histories, and gaps. | +| `skills/oracles-start/` | A first session: a look around, probes, a potential oracle, and the oracle built and run. | +| `skills/oracles-setup/` | Taking an oracle to production: setup, the gap check, upgrades, making the skills findable. | +| `skills/oracles-design/` | How to design, write, or change an oracle: from approved laws to model, histories, and gaps. | | `skills/oracles-evaluate-review/` | How to act on a review: a lossless ledger, RED on unchanged production, and closing the bug class. | | `skills/oracles-review/` | How to review oracle work. | -| `docs/intro.md` | What an oracle is, in a few minutes. | +| `docs/intro.md` | What an oracle is, for when someone asks. | | `docs/oracle-checklist.md` | The complete requirements, by ID (ORC-001 to ORC-016). | | `docs/guide.md` | Techniques for each design and review step. | | `docs/model-styles.md` | One small example of each model style. | diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md index c3bcd2f..11c8823 100644 --- a/packages/oracles/docs/glossary.md +++ b/packages/oracles/docs/glossary.md @@ -6,24 +6,24 @@ term for one concept. Do not rotate synonyms for style. ## Laws and evidence -| Term | Meaning | -| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | -| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | -| limits | What a property does not establish. Unresolved parts of a law go here. | -| sketch | A candidate oracle in `oracles/SKETCHES.md`: its promise, its contract as calls and results, a block for each law (examples, general sentence, authority, status, rival, witness, checkpoint), what the probes found, and its open questions. A human chooses one and answers its questions before the oracle file is written. | -| probe | A small script that calls the real public API with chosen inputs and records what happens. A finding needs one. Probes find the constraints, dynamics, and boundaries that become an oracle's laws and histories. | -| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | -| rival | A plausible different law that predicts a different observation for some history. | -| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | -| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | -| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | -| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | -| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | -| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | -| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the user can pick. | -| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | -| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | +| Term | Meaning | +| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | +| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | +| limits | What a property does not establish. Unresolved parts of a law go here. | +| potential oracle | A candidate oracle that the agent proposes in the conversation: the public behavior it would own, its contract as calls and results, and its potential laws (each a "For every" sentence with its authority, and what the probes showed), plus its open questions. The engineer approves the laws that the authority decides before the oracle file is written. | +| probe | A small script that calls the real public API with chosen inputs and records what happens. A finding needs one. Probes find the constraints, dynamics, and boundaries that become an oracle's laws and histories. | +| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | +| rival | A plausible different law that predicts a different observation for some history. | +| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | +| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | +| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | +| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | +| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | +| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | +| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the user can pick. | +| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | +| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | ## The parts of an `oracleProperty` diff --git a/packages/oracles/docs/instruments/design-grammar.md b/packages/oracles/docs/instruments/design-grammar.md index 099b3be..e34319d 100644 --- a/packages/oracles/docs/instruments/design-grammar.md +++ b/packages/oracles/docs/instruments/design-grammar.md @@ -6,7 +6,7 @@ retries, or ownership. For an ordinary oracle, design step 2 is enough. **Inputs:** -- the approved sketch and the frozen list from design step 2 +- the approved potential laws and the frozen list from design step 2 - the owned production code - the known examples: docs, issues, and existing tests - one nearby invalid history that the oracle must reject @@ -25,7 +25,7 @@ of range. allow the same observations at the same checkpoints. Name the checkpoints, the assumptions, and the freedoms the contract leaves - open. The user's review of the sketch fixes the intended contract. It is + open. The user's approval of the laws fixes the intended contract. It is not evidence that a model meets it. 2. **Separate what you found from what you added.** Each rule cites its diff --git a/packages/oracles/docs/intro.md b/packages/oracles/docs/intro.md index f2931ab..fce1a8c 100644 --- a/packages/oracles/docs/intro.md +++ b/packages/oracles/docs/intro.md @@ -1,5 +1,9 @@ # What an oracle is +This page is for an engineer who asks what an oracle is. An agent running a +first session does not recite it: it shows oracles working on the +engineer's own code, and uses this page to answer questions. + Types check that a value has the right shape. An oracle checks that the code does what it promises, for every input and sequence of calls the promise covers. It is engineering assurance for software: the way a bridge is @@ -42,13 +46,13 @@ checks that the promise still holds. The gap check then shows what is left: production behavior that no law judges yet. -## How a first session goes +## If you know fuzzing or formal models -1. Look around the code, or one subsystem you name. -2. Probe one area: small scripts that call the real API, to find what holds, - what changes, where the edges are, and where the sources disagree. -3. Sketch candidate oracles from what the probes found. -4. Build the one you choose. +Like a fuzzer, an oracle generates many inputs. Unlike a fuzzer, it does +not only wait for a crash: it compares every result with what the code +promises. Like a formal model, it states the rules the code must follow. +Unlike a proof, it checks the real code by running it, across the histories +it generates. For the story of how TanStack DB uses oracles, read [TanStack DB 1.0 RC1: Going All In on Oracles](https://tanstack.com/blog/tanstack-db-1.0-rc1-going-all-in-on-oracles). diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index bff0cf7..1b89059 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -1,6 +1,6 @@ --- name: oracles-design -description: Design, write, or change an oracle with @tanstack/oracles. Start from an approved sketch, write the oracle file with its law block and code, choose a model, and build histories that reach the states the law needs. Show the evidence the change needs, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. +description: Design, write, or change an oracle with @tanstack/oracles. Start from approved potential laws, write the oracle file with its law block and code, choose a model, and build histories that reach the states the law needs. Show the evidence the change needs, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. --- # Design an oracle @@ -11,7 +11,7 @@ that someone could predict an observation without reading production. The code comes after that. Read this skill in full, then read the [glossary](../../docs/glossary.md) -in full. It defines the terms that the sketch and the evidence use. +in full. It defines the terms that the laws and the evidence use. The [checklist](../../docs/oracle-checklist.md) is the complete list of requirements. This skill cites it by ID, such as ORC-004. @@ -21,43 +21,45 @@ witness, and another round. The common shortcuts are changing production first, editing the model to match production, narrowing histories until a failure goes away, and pinning one example. `oracles-review` §2 lists them all. -## 1. Start from an approved sketch, then write the oracle file once +## 1. Start from approved potential laws, then write the oracle file -**A new oracle starts from an approved sketch** in `oracles/SKETCHES.md` -(`oracles-start` step 4): the user chose it, and no open question is left. -If there is none, do `oracles-start` steps 3 and 4 for this boundary first: -explore with probes, write the sketch, and stop for review. Exploration and -review never go in the oracle file. The file is written once, when the -direction is settled, with its law block and its code together. +**A new oracle starts from potential laws that the user approved**, usually +in a first session (`oracles-start`): the laws that the authority decides. +Open design questions may still be open: they are limits until the user +settles them. If no laws are approved yet, probe the behavior and propose +them first, as `oracles-start` does, and stop for approval. Exploration and +review stay in the conversation, never in the oracle file. The file is +written with its law block and its code together. **Decide whether the user must approve.** - **Strengthening an adopted law** (a missing history, observation, driver - path, or check) reuses the law as it is. Do not write a new sketch, and do + path, or check) reuses the law as it is. Do not propose new laws, and do not stop. Name the law in the evidence. Widening the histories inside the law's stated domain is strengthening. Changing the stated domain, a checkpoint, or `limits` is a law change. - **Changing a law** (its promise, domain, checkpoint, assumptions, or the - freedoms it allows) needs a law delta. Add it to the sketch: the old and - the new block for each changed law, with the probe that shows why. Stop - for the user's review, then update the oracle file. -- **A new law, or missing or contradictory authority,** needs an approved - sketch. An analogy to another documented rule is not authority. + freedoms it allows) needs a law delta. Show the user the old and the new + law, with the probe or result that shows why. Stop for their review, then + rewrite the law block in the same change. +- **A new law, or missing or contradictory authority,** needs the user's + approval of that potential law. An analogy to another documented rule is + not authority. - **The task already approved these laws?** Do not ask again. **Production output is evidence, never authority.** A rule that exists only because production does it is model hacking. When the law and the code -disagree, the sketch's answer decides, not the code. +disagree, the approved law decides, not the code. When the promise is unclear, or two readings predict different behavior, use the [law discovery](../../docs/instruments/law-discovery.md) card. **Use the [design grammar](../../docs/instruments/design-grammar.md) card when any of these is true.** Do not judge whether the law is hard. Check the -list at the sketch, and again after each gap check: +list when you propose the laws, and again after each gap check: -- the sketch has more than five laws for one boundary (adopted, observed, - and open ones all count) +- there are more than five laws for one boundary (adopted, observed, and + open ones all count) - a law has no new case that it predicts - the model has more than one branch per law - the work changes or simplifies an existing model @@ -140,11 +142,11 @@ its documentation promises, and a law that restates the implementation. A model of that law would be the same code, so it could not catch a defect. Keep the law block to about 40 lines of prose, plus the contract code block. -It never mentions the sketch, a review, or an open question: a point the -sketch left undecided appears only as a limit, with its reason. +It never mentions a review or an open question: a point the user has not +decided appears only as a limit, with its reason. -**Then write the code under it.** Each boundary condition in the sketch -becomes a constructed history. Each law's sentence, authority, and limits go +**Then write the code under it.** Each boundary condition from the probes +and the law worksheet becomes a constructed history. Each law's sentence, authority, and limits go in that property's `law` option. Put prose beside the model, the histories, the driver, and the check that says why each one predicts or tests the promised result. @@ -157,18 +159,19 @@ promised result. contract code block: public API calls and their results, with no assertion or helper. Each line states the same law as its sentence. - No law restates the code. -- Each law sentence matches its block in the sketch. If they disagree, the - sketch decides: fix the prose, and tell the user. A new heading for a law - is not a law change. A changed sentence, domain, or checkpoint is. +- Each law sentence matches the law the user approved. If they disagree, the + approved law decides: fix the prose, and tell the user. A new heading for + a law is not a law change. A changed sentence, domain, or checkpoint is. Keep the law block, the `law` option, and the model in step: a change to one -updates the others in the same change (ORC-003). +updates the others in the same change (ORC-003). When a decision changes a +law during the work, rewrite the law block at once, not at the end. ## 2. Freeze what the oracle must preserve Write these down. They stay fixed while you build the oracle. -- The laws and their checkpoints, from the approved sketch. +- The laws and their checkpoints, as the user approved them. - The owned production code: the files or functions the oracle answers for. - The freedoms the contract leaves open, such as the order of independent events, scheduling, or representation. The oracle must accept every @@ -364,6 +367,13 @@ Techniques: [guide § Histories](../../docs/guide.md#histories). - It awaits the work it starts. - It bounds its own run time and work, and throws a structured error when a bound is passed. + - It waits on events, never on the wall clock. Control time with fake + timers, a virtual clock, or deferred promises, so that load on the + machine cannot change a result. + - When something else needs the model (a probe, a second oracle, a + reconstruction), keep the model in a test-free companion module, such + as `parse-model-oracle.ts`. An oracle file registers tests when it is + imported. - It releases what it created. - **The check** compares observations with the model at each checkpoint. It is the only place that reports a violation. For exact values, use diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index bed6e80..a566b4d 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -41,9 +41,9 @@ contains one is not ready. | Shortcut | What to look for | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Law after code** | An oracle written without an approved sketch, a law with no authority, or a law block that disagrees with the `law` option or the model. Also a law block that is a list of fields or law IDs, not prose (ORC-003), or one that carries review material or open questions. | -| **Law as examples** | A law written as a list of cases, not one sentence over the domain. A sketch with no new case that the law predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | -| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired at the sketch or after a gap check, and the design grammar card was not used. | +| **Law after code** | An oracle written without approved potential laws, a law with no authority, or a law block that disagrees with the `law` option or the model. Also a law block that is a list of fields or law IDs, not prose (ORC-003), or one that carries review material or open questions. | +| **Law as examples** | A law written as a list of cases, not one sentence over the domain. A proposed law with no new case that it predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | +| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired when the laws were proposed or after a gap check, and the design grammar card was not used. | | **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | | **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | | **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index b9e3477..e1aef0a 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -1,18 +1,21 @@ --- name: oracles-setup -description: Configure a repository for @tanstack/oracles. Install the pinned packages, add the lint preset and the Vitest config, write the AGENTS.md block, and make the oracle skills findable by coding agents. Then build the first oracle from an approved sketch and run it through the gap check. Use once oracles-start has an approved sketch, or when a repository upgrades its pinned commit. +description: Take an oracle to production with @tanstack/oracles. Pin the packages, add the lint preset and the Vitest config, write the AGENTS.md block, make the oracle skills findable by coding agents, and run the oracle through the gap check. Use when the engineer chooses to ship an oracle from a first session, when a repository adopts @tanstack/oracles, or when it upgrades its pinned commit. Its install and file-placement sections also cover the minimal setup that a first session needs. --- -# Set up @tanstack/oracles +# Take an oracle to production You are setting up oracle testing in this repository. Do the steps in order. Ask the user before each step that writes outside the repository. When no user is available, choose the default this skill gives, record each choice in -your report, and continue. That default covers setup questions only. A -sketch has no default: write it, report it, and stop. +your report, and continue. That default covers setup questions only. +Potential laws have no default: propose them, report them, and stop. -Setup starts once the engineer has chosen a sketch in `oracles-start`. If -there is none yet, load `oracles-start` first. +Setup takes an oracle to production. It usually starts after a first +session (`oracles-start`), once the engineer has an oracle that found a bug +and chooses to ship it. If there is no oracle yet, load `oracles-start` +first. A first session needs only this skill's install section and its +"where oracle files go" section. Setup ends when one oracle passes on production and the gap check leaves no gap open in that oracle's own contract, unless the user decides otherwise. A @@ -248,7 +251,7 @@ is your call, and you will keep building on it. Make it the smallest, clearest code that satisfies the oracles: merge repeated code, remove code that no law needs (after the checks in `oracles-design` step 8), and restructure freely while the oracles pass. -- To find where an oracle would help, or to sketch a new one, read +- To find where an oracle would help, or to propose a new one, read `node_modules/@tanstack/oracles/skills/oracles-start/SKILL.md` in full and follow it. - Before you create or change an oracle, or change production code that an @@ -268,9 +271,15 @@ needs (after the checks in `oracles-design` step 8), and restructure freely whil If lint is an environment limit (step 3), replace the lint line with "Lint: not available here ()." Fill in the two placeholders in step 7, -once their values exist. A run that -stops before step 7 leaves them in place, and its report says so. If the repository has a -`CLAUDE.md` and no `AGENTS.md`, ask the user which file gets the block. +once their values exist. A run that stops before step 7 leaves them in +place, and its report says so. + +Check for both `AGENTS.md` and `CLAUDE.md`. Claude Code reads `CLAUDE.md`, +and most other agents read `AGENTS.md`. If both exist, put the block in +`AGENTS.md` and make sure `CLAUDE.md` refers to it (a line `@AGENTS.md` +imports it). If only `CLAUDE.md` exists, ask the user which file gets the +block. If neither exists, create `AGENTS.md` and a `CLAUDE.md` that holds +only `@AGENTS.md`, after asking. In a workspace with a package-local install, put the block in the package's `AGENTS.md`, where its `node_modules` paths resolve. A root copy must use the @@ -340,9 +349,10 @@ Before writing a global pointer: ## 6. Build the first oracle -The first oracle is the sketch that the engineer chose in `oracles-start`, -with its questions answered. Load `oracles-design`, and follow it from its -first step. +Usually the oracle already exists from the first session. Keep it, and +continue with its gap check. If there is none, build it from the potential +laws that the engineer approved: load `oracles-design`, and follow it from +its first step. If the first oracle fails on unchanged production under an adopted law, it found a defect. The RED is the finding: report it with its replay line, and diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index d254a52..088fce9 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -1,47 +1,83 @@ --- name: oracles-start -description: Introduce @tanstack/oracles to an engineer and find a first oracle in their codebase. Give a short introduction, look around a subsystem they name or the whole repository, explore one area with probes that prove each finding, and write sketches of candidate oracles for the engineer to choose from. Use when someone is trying @tanstack/oracles, asks where an oracle would help, or wants a new oracle and has no approved sketch yet. +description: Run a first session with @tanstack/oracles in an engineer's codebase. Say what the session will do, look around a part of the code they name or the whole repository, probe the real API to prove each finding, propose a potential oracle and its potential laws, then build it and run it until it finds a bug and the engineer has settled some design questions. Present the work, and explain the path to production. Use when someone is trying @tanstack/oracles, or asks where an oracle would help. --- -# Start with oracles +# A first session with oracles -The engineer is trying this tool, often in the mood of "impress me". Earn it -with evidence about their own code: a probe that shows a bug, an -inconsistency between the docs and the code, or a behavior that no test -checks. An agent that only says things is often wrong, and one wrong claim -costs the tool its credibility. So never claim what a probe has not shown. +## Your goal -Probes are the small first instrument. They bootstrap the oracle: each probe -finds a constraint (or its absence), a dynamic, or a boundary, and those -become the oracle's laws and histories. The oracle then explores the whole -design space and checks the production code far more thoroughly than any -probe can. +Be an engaging teacher. The engineer is trying this tool, often in the mood +of "impress me". Show, don't tell. Bring up interesting findings about their +own code, and explain what you are doing and why as you go. Tie each +explanation to something you just showed them: a probe, a finding, a law, or +the oracle file. Do not explain what oracles are unless they ask. +[What an oracle is](../../docs/intro.md) is there for when they do. -Work in these four steps. Stop after each one, show the result, and wait for -the engineer. Then `oracles-design` builds the oracle that the engineer -chooses. +The session ends at one moment: an oracle that runs on their code, has found +a bug, and has pushed them to settle some design questions. Then you present +what you built, and explain how to take it to production. -## 1. Ask, then introduce +An agent that only says things is often wrong, and one wrong claim costs the +tool its credibility. So every finding needs a probe that the engineer can +run again. -Ask one question first: is there something specific they want to try this -on? Then introduce the tool in a few sentences and one example, from the -[introduction](../../docs/intro.md). Keep it short: the engineer came to see -it work. +## How to talk -Offer two ways to start: +- Use these terms, and use them every time: a **probe**, a **finding**, a + **potential oracle** that owns a public behavior, its **potential laws**, + and the **oracle file**. +- Talk naturally. Never name a skill, a step, or a section ("step 3", "the + skill's fallback"). Say what you are doing, in your own words. +- Ask in plain words. Explain any term the engineer may not know in one + line (for example, a lone surrogate is half of a character that takes two + UTF-16 units). +- Ask one clear question when you need a decision. For a possible bug, ask + "Is this a bug, or is it intended?" -- **A subsystem they name:** look closely at that part of the code. -- **No preference:** look around the repository for a good first oracle. +## Where you stop -Say what you will do: read the repository, run small scripts against its -code from a scratch directory, and change no file in the repository until -the engineer chooses an oracle to build. +Stop and wait only when the engineer must decide: + +- which part of the code to look at +- whether a finding is a bug, or intended +- a design question +- whether to approve the potential laws +- whether to approve a more complex design (see below) +- whether to take the oracle to production + +Between those points, keep working, and keep telling them what you are doing. + +## What you change + +Until the engineer chooses to take an oracle to production, change only +these: + +- the minimal installs that an oracle needs to run: `@tanstack/oracles`, + `fast-check`, and `vitest` when they are missing +- a minimal Vitest config that picks up the oracle file, only when the + repository's own config does not +- oracle files, with their prose + +Findings, potential oracles, and potential laws stay in the chat. Keep +probes in a scratch directory outside the repository. Change production code +only when the engineer asks you to. + +## 1. Open + +Say in two or three sentences what this session will do. For example: "I'll +look around your code for a behavior that's easy to break, try it with small +scripts that call your real API, and show you what I find. Then I'll write +an oracle for it with you and run it." + +Then ask one question: is there a part of the code they want to try this on, +or should you look around? ## 2. Look around Make it quick. Read the public API and its docs (the README, the API docs, -the JSDoc), the existing tests, and the git history of fixes (for example -`git log --oneline -i --grep fix`, and the files that fixes change most). +the JSDoc), the existing tests, and the git history of fixes. The files that +fixes change most often tell you more than a search for "fix". Look for behavior that is easy to break, and whose right answer is simpler to state than the code that produces it. Good signs: @@ -56,21 +92,18 @@ to state than the code that produces it. Good signs: Avoid an internal helper whose only law would restate its code. Avoid behavior that nothing documents, unless the engineer wants to decide it. -Report two or three spots, a short paragraph each: what the code seems to -promise and where that is stated, and why it makes a good first oracle. -Recommend one. The engineer picks. +Tell them about two or three spots, a short paragraph each: what the code +seems to promise, where that is stated, and why it would make a good oracle. +Recommend one, and let them choose. -## 3. Explore with probes +## 3. Probe -Read the chosen spot's docs, code, tests, and fix history. Then probe it. +Read the spot's docs, code, tests, and fix history. Then probe it. **A probe** is a small script that calls the real public API with chosen -inputs and prints what happens. Run it from a scratch directory, outside the -repository or in an ignored path, against the unchanged code. Keep each -probe and its output, so the engineer can run it again. - -**Every finding needs a probe.** A claim without one is a suspicion: label -it as a suspicion, or leave it out. +inputs and prints what happens. Import the code the way the repository's own +tests do, and run the probe from a scratch directory against the unchanged +code. Keep each probe and its output. **Probe to find the design, not only bugs.** Look for: @@ -82,67 +115,73 @@ it as a suspicion, or leave it out. capacity, time, and odd input (digits, unicode, lone surrogates, keys that shadow prototype names). - **Inconsistencies:** places where the docs, the code, the tests, and the - examples disagree, and behavior that nobody decided, where two readings - predict different results. + examples disagree, and behavior that nobody decided. When a probe is cheap, run it over a grid or a random sample of inputs, count the failures, and show the smallest one. -**Report each finding with its kind:** +**Find the authority for each promise, and show it.** Look in the README, +the API docs, the JSDoc, the tests' names and bodies, the changelog, source +comments, and any standard that the docs name. Quote it, with the file and +line. The authority is what makes a finding credible, and it becomes the +documentation in the oracle file. When nothing decides a behavior, it is a +design question, not a bug. + +**Present each finding** in this shape, a light touch, with each finding +called a possible bug or a question, never a certainty: + +1. A title that names the behavior: "Object Map key: replay throws". +2. The smallest case, expected against observed, as a short snippet the + engineer can run. +3. **Why I think it's wrong:** the authority, quoted, with its file and line. +4. **Why the tests miss it:** what the existing tests check instead. +5. **How often:** how many of the probe's inputs failed, out of how many. + +Give each finding its kind: | Kind | What it means | | --------------- | ----------------------------------------------------------------------- | -| Bug | The code breaks a promise that a document states. | +| Possible bug | The code breaks a promise that a document or a test states. | | Inconsistency | Two sources disagree: say which ones, and what each says. | | Test gap | A probe shows behavior that no existing test checks. | | Design question | Nothing decides the behavior: give two readings and what each predicts. | -For each finding, give the promise and its source, the input, what -happened, the command that reproduces it, and why the existing tests miss -it. For a bug, ask the engineer two questions: is it unintended, and did -they already know? - -Then say why an oracle would help here: it would generate the whole domain -instead of a few probes, state the law that the probes point at, settle the -open questions, and hold the answer for every later change. For example: - -> Column mapping between snake_case and camelCase. The JSDoc promises a -> round trip. The tests cover `user_id` and `created_at`. A probe over -> 21,844 names found 3,518 that do not round-trip, for example -> `address2_line → address2Line → address2line`. The JSDoc also shows -> `userHTTPSURL → user_https_url`, but the code gives `user_httpsurl`. An -> oracle would generate every name shape, state the round trip as a law, -> settle the acronym rule, and hold it. - -If the probes find nothing, say so plainly: "No confirmed defect in the -paths examined." Then report what they did find: the constraints that held, -and the design questions. - -## 4. Sketch candidate oracles - -Write one to three candidate oracles for the chosen spot in -`oracles/SKETCHES.md`. Create the file if it does not exist, and add to it -if it does: it is the repository's list of oracle ideas. Mark a sketch -"built" once its oracle exists. - -Each sketch establishes the boundary: - -| Question | What to establish | -| ---------------------------------- | --------------------------------------------------------------------------------------------- | -| What promise belongs together? | One caller-visible behavioral contract. | -| Where is it observed? | The public entry points and the checkpoints where it is owed. | -| What does it assume? | What providers, the environment, and neighboring components must supply. | -| What else witnesses it? | Other drivers for the same law: component, integration, end-to-end, or replay. | -| Where does another contract begin? | A different authority, lifecycle, observation boundary, or composition rule. | -| Which source implements it? | The `owns` list that the gap check uses. | -| What public state does it expose? | Each readable field or getter needs a law or a stated limit. A limit leaves its mutants open. | - -Then each sketch gives: - -- **The promise,** in one or two sentences, and what the oracle would own. -- **The contract,** as `call // → result` lines (see `oracles-design` step 1). -- **One block for each law,** from this template. It is where examples - become a general law. +For a possible bug, ask: "Is this a bug, or is it intended?" + +If the probes find nothing, say so plainly: "No confirmed defect in what I +tried." Show what held, and the design questions that came up. + +## 4. Propose a potential oracle + +Say why an oracle would help here. It tries the whole space of inputs and +sequences, not a few probes, states the rule that the probes point at, and +holds the answer for every later change. + +Then present it in the chat: "Here's a potential oracle that owns +, and its potential laws as I see them." + +- What it would own, in one or two sentences. +- The contract, as a few lines of public calls and their results: + + ```ts + klona(x) // → a deep copy: same kind, prototype, and values as x, sharing no object with it + ``` + +- Each potential law, as one "For every …" sentence, with its authority + quoted, and what the probes showed about it. +- The laws people forget, or one line on why each does not apply: + - **No throw on valid input.** When the contract says a call returns a + value, it promises not to throw for valid input. + - **The laws of the kind of function.** An equality check is reflexive and + symmetric. A parser and a serializer round-trip. A setter is idempotent + when set to the same value. +- The open design questions, each with its two readings, and what the probe + showed. Say that these can wait: the oracle's results will help decide + them. + +**Work out each law privately before you propose it.** Fill in this +worksheet for each one, for yourself. Do not show it. It is how you get from +a few examples to a general law: ```markdown ### L1 @@ -163,61 +202,59 @@ Then each sketch gives: - Checkpoint: ``` -How to fill in the fields: - -- **Examples** and **What varies, what stays:** for example, `ms('2 hrs')` - is `7200000` (README, Examples). The number, the spelling, and the spacing - can change. The unit's length decides the answer. -- **Law:** one sentence over the domain. For example: "For every finite - number `n` and every unit name `u` in the table, in any case and with or - without a space, `parse(n + u)` is `n` times the length of `u`." -- **New case:** if you cannot write one, the law is still a list of - examples. Write it again. A new case that only renames a value in the - example (`/:id` for `/:foo`) does not count. -- **Domain edges:** wider than the examples. A block whose domain is only its - own examples is not a law. Keep those examples as `fixed` histories, and as - rebuilt examples of the general law that explains them. -- **Rebuilt examples:** an example that the law gets wrong shows a missing - condition, a second law, or a source that disagrees. Say which. -- **Dynamics** are the model's rules, and the histories vary them. For a pure - function, write "none: one call", and put the variation into the domain. -- **Constraints** hold whatever happens: a count never above its limit, an - equality that is symmetric, a read that returns the last write. The check - tests them at every checkpoint. -- **Boundary conditions** are the edges, where real behavior and most - defects live: empty, zero, `-0`, the size limit, at capacity, the first and - the last item, keys that shadow prototype names, cleanup. A boundary that - the law does not decide goes under open questions. Do not leave it out of - the grammar. -- **Status:** `adopted` if it has an authority, `observed` if production does - it but nobody adopted it, `open` if it is a question. Only `adopted` laws - become expected results. An `observed` or `open` law goes to the user as a - question. -- **Production output is evidence, never authority.** A rule that you write - only because production does it is model hacking. Mark that rule - `observed`, and ask the user. An analogy to another documented rule is not - authority either: mark that law `open`. - -The sketch also gives: - -- **What the probes found,** sorted into constraints, dynamics, and - boundaries. Each one becomes a law, a history dimension, or a constructed - history. Link each probe. -- **The laws people forget,** as blocks, or one line that says why each does - not apply: - - **No throw on valid input.** When the contract says a call returns a - value, it promises not to throw for valid input. - - **The laws of the kind of function.** An equality check is reflexive and - symmetric. A parser and a serializer round-trip. A setter is idempotent - when set to the same value. -- **Open design questions,** each with its two readings and what the probe - showed. - -Check each sketch: from its contract and laws alone, a reader can predict -one observation of each law, and no law restates the code. - -**Stop here.** The engineer picks a sketch and answers its questions. Record -the answers in the sketch. A sketch has no default: with no engineer -present, write it and stop. When the chosen sketch has no open question -left, its direction is settled. Then load `oracles-setup` if the repository -is not configured yet, and then `oracles-design` to build it. +The law must say more than the code does. If a model of it would only repeat +the implementation, look for the public promise that the code serves. + +**A more complex design needs sign-off.** If the potential oracle has more +than five laws, or a law has no new case that it predicts, or the model +needs more than one branch per law, work through the +[design grammar](../../docs/instruments/design-grammar.md) card. Then show +the engineer the design it produced: its parts, and any choice between two +designs. Ask them to approve it with the laws. + +Then ask the engineer to approve the potential laws that the authority +decides. + +## 5. Build and run + +1. **Install what is missing**, and nothing more: `fast-check` and `vitest` + when the repository does not have them. Use the repository's own package + manager. Add a minimal Vitest config only if the repository's own config + would not pick up the oracle file. The `oracles-setup` skill's install and + "where oracle files go" sections have the details. +2. **Write the oracle file.** Follow `oracles-design` from its first step: + the law block's prose, then the model, the histories, the driver, and the + check. As you build, tell the engineer briefly what each part is for: the + model computes the expected answer the simple way, the histories are the + generated inputs and sequences, the driver calls their real code, and the + check compares the two at each step. +3. **Run it on the unchanged code.** A failure under an approved law is a + possible bug. Present it like a probe finding, with the oracle's smallest + failing case as the snippet. +4. **As the engineer settles questions, change the law** and rewrite the law + block's prose in the same step. Before you show results, check that the + prose still says what the code does. + +Do not run the gap check yet, and do not change production code. Both belong +to taking the oracle to production. + +If the oracle finds no bug, say so plainly, show what held, and offer to +build the next oracle at another spot. + +## 6. Present + +When the oracle has found a bug and the engineer has settled some design +questions, stop and present: + +- what you built, and what it found +- an offer to walk through how it works: how the reference model was built, + how the histories are generated, and how the driver calls their code. + Engineers often want this, and it is the best part to teach. +- an invitation to ask questions + +Then explain how to take the oracle to production: a lint preset that keeps +oracle files honest, a block in `AGENTS.md` (or `CLAUDE.md`) so every agent +follows the same rules, and the gap check, which makes small changes to a copy of the +production code and shows each change that no law catches. That is how the +oracle grows toward full coverage. If they want to go ahead, load +`oracles-setup`. From cc881128cefea896222ff8fe67aea5fabaf69f08 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:07:50 -0700 Subject: [PATCH 40/76] docs(oracles): restore what the rewrite dropped, and align start, setup, and design A loss audit of the first-session rewrite found guidance that was lost, and places where the skills now disagreed. Restored in oracles-start: the law worksheet's fill-in guidance (a renamed value is not a new case, a domain made only of its own examples is not a law, status labels and the rule that only adopted laws become expected results, dynamics and constraints, boundaries that the law leaves open), the boundary questions (assumptions, other witnesses, public state), the rule that an unprobed claim is a suspicion, a close look when the engineer names a spot, a fix-history command, a worked finding, a second potential oracle when two fit, and "no default" for potential laws. Aligned: a first session takes only the minimal installs and config from setup, with no lint plugin, scripts, or other config, and it stops at the RED instead of running setup's scratch fix and gap check. The design-grammar triggers come from one list, and a triggered design gets the engineer's sign-off. An answer to an open question is its own review. An approved potential law is an adopted law. A later build without a first session fills in the worksheet first. The README prompt again pins one commit throughout. Co-authored-by: Isaac --- packages/oracles/README.md | 2 +- .../oracles/skills/oracles-design/SKILL.md | 13 ++- .../oracles/skills/oracles-setup/SKILL.md | 14 ++- .../oracles/skills/oracles-start/SKILL.md | 98 ++++++++++++++++--- 4 files changed, 106 insertions(+), 21 deletions(-) diff --git a/packages/oracles/README.md b/packages/oracles/README.md index a469e51..7866144 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -24,7 +24,7 @@ Show me what oracle testing with @tanstack/oracles can find in this repository. 2. Install https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@ as a dev dependency, with this repository's package manager. 3. Read node_modules/@tanstack/oracles/skills/oracles-start/SKILL.md in full, - and follow it. + and follow it. Use wherever a skill asks for the commit to pin. ``` What happens next: diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 1b89059..b6ccdae 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -45,7 +45,11 @@ written with its law block and its code together. - **A new law, or missing or contradictory authority,** needs the user's approval of that potential law. An analogy to another documented rule is not authority. -- **The task already approved these laws?** Do not ask again. +- **The user's answer to an open question is its review.** Rewrite the law + block in the same step, and continue. A change the user did not ask for + (you found that a law is wrong) needs the delta and a stop. +- **The task already approved these laws?** Do not ask again. An approved + potential law is an adopted law. **Production output is evidence, never authority.** A rule that exists only because production does it is model hacking. When the law and the code @@ -65,7 +69,9 @@ list when you propose the laws, and again after each gap check: - the work changes or simplifies an existing model - the gap check leaves three or more gaps in one function -Say in the evidence which trigger fired, or that none did. +Say in the evidence which trigger fired, or that none did. When a trigger +fires before the oracle is written, show the user the design the card +produced, and get their approval with the laws. **Write the law block.** The law block is the file's opening comment. The oracle file is a literate program (ORC-003), so write it as prose. That reader is a maintainer or a serious user who knows @@ -146,7 +152,8 @@ It never mentions a review or an open question: a point the user has not decided appears only as a limit, with its reason. **Then write the code under it.** Each boundary condition from the probes -and the law worksheet becomes a constructed history. Each law's sentence, authority, and limits go +and the law worksheet becomes a constructed history. If there was no first +session, fill in the worksheet from `oracles-start` for each law first. Each law's sentence, authority, and limits go in that property's `law` option. Put prose beside the model, the histories, the driver, and the check that says why each one predicts or tests the promised result. diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index e1aef0a..dc6a4d3 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -11,11 +11,12 @@ user is available, choose the default this skill gives, record each choice in your report, and continue. That default covers setup questions only. Potential laws have no default: propose them, report them, and stop. -Setup takes an oracle to production. It usually starts after a first -session (`oracles-start`), once the engineer has an oracle that found a bug -and chooses to ship it. If there is no oracle yet, load `oracles-start` -first. A first session needs only this skill's install section and its -"where oracle files go" section. +This skill has two uses. A first session (`oracles-start`) uses only its +install section and its "where oracle files go" section, and only for the +minimal setup that an oracle needs to run. Taking an oracle to production +uses all of it. That usually starts after a first session, once the +engineer has an oracle that found a bug and chooses to ship it. If there is +no oracle yet, load `oracles-start` first. Setup ends when one oracle passes on production and the gap check leaves no gap open in that oracle's own contract, unless the user decides otherwise. A @@ -354,6 +355,9 @@ continue with its gap check. If there is none, build it from the potential laws that the engineer approved: load `oracles-design`, and follow it from its first step. +In a first session, stop at the RED: present the bug, and leave the fix and +the gap check for production. When taking the oracle to production: + If the first oracle fails on unchanged production under an adopted law, it found a defect. The RED is the finding: report it with its replay line, and tell the user. Never weaken the oracle to make it pass. The gap check cannot diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 088fce9..2cf981c 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -75,9 +75,12 @@ or should you look around? ## 2. Look around -Make it quick. Read the public API and its docs (the README, the API docs, -the JSDoc), the existing tests, and the git history of fixes. The files that -fixes change most often tell you more than a search for "fix". +If the engineer named a part of the code, look closely there. Otherwise, +look around the repository, and make it quick. Read the public API and its +docs (the README, the API docs, the JSDoc), the existing tests, and the git +history of fixes. The files that fixes change most often tell you more than +a search for "fix": +`git log -i --grep fix --name-only --format= | sort | uniq -c | sort -rn | head`. Look for behavior that is easy to break, and whose right answer is simpler to state than the code that produces it. Good signs: @@ -105,6 +108,9 @@ inputs and prints what happens. Import the code the way the repository's own tests do, and run the probe from a scratch directory against the unchanged code. Keep each probe and its output. +**Every finding needs a probe.** A claim without one is a suspicion: say so, +or leave it out. + **Probe to find the design, not only bugs.** Look for: - **Constraints:** what holds whatever the input or the sequence (a round @@ -146,6 +152,20 @@ Give each finding its kind: | Test gap | A probe shows behavior that no existing test checks. | | Design question | Nothing decides the behavior: give two readings and what each predicts. | +For example: + +> **Column names with a digit don't round-trip.** +> +> ```ts +> camelToSnake(snakeToCamel('address2_line')) // expected 'address2_line', got 'address2line' +> ``` +> +> **Why I think it's wrong:** the JSDoc of `snakeToCamel` says the transform +> is "round-trippable via `camelToSnake`" (`src/column-mapper.ts:71`). +> **Why the tests miss it:** they check `user_id` and `created_at`, which +> have no digits and no single-letter parts. **How often:** 3,518 of the +> 21,844 names I tried. + For a possible bug, ask: "Is this a bug, or is it intended?" If the probes find nothing, say so plainly: "No confirmed defect in what I @@ -158,7 +178,10 @@ sequences, not a few probes, states the rule that the probes point at, and holds the answer for every later change. Then present it in the chat: "Here's a potential oracle that owns -, and its potential laws as I see them." +, and its potential laws as I see them." If two +potential oracles fit the spot well (two public behaviors, or two ways to +draw the boundary), name the second one in a sentence, say what it would +own instead, and recommend one. - What it would own, in one or two sentences. - The contract, as a few lines of public calls and their results: @@ -202,26 +225,77 @@ a few examples to a general law: - Checkpoint: ``` +How to fill in the fields: + +- **Examples** and **What varies, what stays:** for example, `ms('2 hrs')` + is `7200000` (README, Examples). The number, the spelling, and the spacing + can change. The unit's length decides the answer. +- **Law:** one sentence over the domain. For example: "For every finite + number `n` and every unit name `u` in the table, in any case and with or + without a space, `parse(n + u)` is `n` times the length of `u`." +- **New case:** if you cannot write one, the law is still a list of + examples. Write it again. A new case that only renames a value in the + example (`/:id` for `/:foo`) does not count. +- **Domain edges:** wider than the examples. A block whose domain is only its + own examples is not a law. Keep those examples as `fixed` histories, and as + rebuilt examples of the general law that explains them. +- **Rebuilt examples:** an example that the law gets wrong shows a missing + condition, a second law, or a source that disagrees. Say which. +- **Dynamics** are the model's rules, and the histories vary them. For a pure + function, write "none: one call", and put the variation into the domain. +- **Constraints** hold whatever happens: a count never above its limit, an + equality that is symmetric, a read that returns the last write. The check + tests them at every checkpoint. +- **Boundary conditions** are the edges, where real behavior and most + defects live: empty, zero, `-0`, the size limit, at capacity, the first and + the last item, keys that shadow prototype names, cleanup. A boundary that + the law does not decide goes under open questions. Do not leave it out of + the grammar. +- **Status:** `adopted` if it has an authority, `observed` if production does + it but nobody adopted it, `open` if it is a question. Only `adopted` laws + become expected results. An `observed` or `open` law goes to the user as a + question. +- **Production output is evidence, never authority.** A rule that you write + only because production does it is model hacking. Mark that rule + `observed`, and ask the user. An analogy to another documented rule is not + authority either: mark that law `open`. + +Settle these privately too: + +- **What it assumes:** what providers, the environment, or neighboring code + must supply. +- **What else witnesses it:** other ways to observe the same law (a + component, an integration, a replay). +- **What public state it exposes:** each readable field or getter needs a + law or a stated limit. +- **What each probe found:** each constraint, dynamic, or boundary becomes a + law, a history dimension, or a constructed history. Keep a pointer to the + probe that showed it. + The law must say more than the code does. If a model of it would only repeat the implementation, look for the public promise that the code serves. -**A more complex design needs sign-off.** If the potential oracle has more -than five laws, or a law has no new case that it predicts, or the model -needs more than one branch per law, work through the -[design grammar](../../docs/instruments/design-grammar.md) card. Then show +**A more complex design needs sign-off.** When a trigger for the +[design grammar](../../docs/instruments/design-grammar.md) card fires (the +list is in `oracles-design` step 1: for example, more than five laws, or a +law with no new case that it predicts), work through the card. Then show the engineer the design it produced: its parts, and any choice between two designs. Ask them to approve it with the laws. Then ask the engineer to approve the potential laws that the authority -decides. +decides. An approved potential law is an adopted law. Potential laws have no +default: with no engineer present, propose them and stop. ## 5. Build and run 1. **Install what is missing**, and nothing more: `fast-check` and `vitest` when the repository does not have them. Use the repository's own package - manager. Add a minimal Vitest config only if the repository's own config - would not pick up the oracle file. The `oracles-setup` skill's install and - "where oracle files go" sections have the details. + manager. If the repository's own Vitest config would not pick up the oracle + file, add a minimal config for it (or add its glob to the existing + `include`). Add no scripts, lint, or other config yet. The `oracles-setup` + skill's install and "where oracle files go" sections have the details: + from them, take only what these packages and the config need. The lint + plugin waits for production. 2. **Write the oracle file.** Follow `oracles-design` from its first step: the law block's prose, then the model, the histories, the driver, and the check. As you build, tell the engineer briefly what each part is for: the From 171bb5418a0583a2b05b02a739d0a0ec32a503d6 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:18:44 -0700 Subject: [PATCH 41/76] docs(oracles-start): old or other test runners, private worksheet labels, decisions as authority - Say what to do when the repository runs another runner or an older Vitest, and when an install fails. - Keep the worksheet's labels and statuses private until approval. - Several pending decisions end as a short numbered list. - The engineer's answer to a design question is an authority the oracle can check, the usual route to a bug when docs are thin. - Open questions are listed apart from potential laws and do not count toward the design grammar trigger. - Stop for the spot answer unless the first message named one. Co-authored-by: Isaac --- .../oracles/skills/oracles-design/SKILL.md | 3 +- .../oracles/skills/oracles-start/SKILL.md | 36 +++++++++++++++++-- 2 files changed, 35 insertions(+), 4 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index b6ccdae..5233376 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -63,7 +63,8 @@ when any of these is true.** Do not judge whether the law is hard. Check the list when you propose the laws, and again after each gap check: - there are more than five laws for one boundary (adopted, observed, and - open ones all count) + open ones all count; an open question that the oracle file states only as + a limit is not a law) - a law has no new case that it predicts - the model has more than one branch per law - the work changes or simplifies an existing model diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 2cf981c..9e11722 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -33,7 +33,12 @@ run again. line (for example, a lone surrogate is half of a character that takes two UTF-16 units). - Ask one clear question when you need a decision. For a possible bug, ask - "Is this a bug, or is it intended?" + "Is this a bug, or is it intended?" When several decisions are waiting, + end with a short numbered list of them, most important first, so the + engineer can answer by number. +- Show the engineer only what they decide on. Your private notes, their + labels, and their statuses ("adopted", "open") stay private. Until the + engineer approves it, a potential law is only a potential law. ## Where you stop @@ -71,7 +76,8 @@ scripts that call your real API, and show you what I find. Then I'll write an oracle for it with you and run it." Then ask one question: is there a part of the code they want to try this on, -or should you look around? +or should you look around? Stop for the answer. If their first message +already named a part of the code, say that you will start there, and go on. ## 2. Look around @@ -133,6 +139,12 @@ line. The authority is what makes a finding credible, and it becomes the documentation in the oracle file. When nothing decides a behavior, it is a design question, not a bug. +The engineer's answer to a design question is an authority too. Once they +decide it, it becomes a potential law, and the oracle can find where the +code breaks it, just as with a law from the docs. In a repository with few +docs, this is the usual way to a bug: ask the question, and let the oracle +check the answer. + **Present each finding** in this shape, a light touch, with each finding called a possible bug or a question, never a certainty: @@ -200,7 +212,9 @@ own instead, and recommend one. when set to the same value. - The open design questions, each with its two readings, and what the probe showed. Say that these can wait: the oracle's results will help decide - them. + them. List them apart from the potential laws, and do not number them as + laws. Until the engineer decides one, the oracle file states it as a + limit. **Work out each law privately before you propose it.** Fill in this worksheet for each one, for yourself. Do not show it. It is how you get from @@ -296,6 +310,22 @@ default: with no engineer present, propose them and stop. skill's install and "where oracle files go" sections have the details: from them, take only what these packages and the config need. The lint plugin waits for production. + + The framework needs Vitest 4.1 or later. Check before you install: + + - **The repository uses another runner** (mocha, uvu, Jest, node:test): + add Vitest for the oracles only, with a config whose `include` is only + the oracle files. Their own suite and runner stay as they are. Tell + the engineer this in one line as you install. + - **The repository runs an older Vitest:** this is a decision for the + engineer. Tell them that the oracle needs Vitest 4.1 or later, and that + upgrading changes the runner for their own suite too. Offer to upgrade + and then run their suite, to show whether it still passes. Do not + upgrade until they agree. + - **An install fails:** show the error in a line or two, say what you + think caused it, and ask before you try anything that changes more than + the packages above. + 2. **Write the oracle file.** Follow `oracles-design` from its first step: the law block's prose, then the model, the histories, the driver, and the check. As you build, tell the engineer briefly what each part is for: the From 7294946063b3ab01ccf3d8077ffdf7932075cf31 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:19:35 -0700 Subject: [PATCH 42/76] docs(oracles-start): pick the spot when asked to, and how to run a probe Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 9e11722..e01348c 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -103,7 +103,9 @@ behavior that nothing documents, unless the engineer wants to decide it. Tell them about two or three spots, a short paragraph each: what the code seems to promise, where that is stated, and why it would make a good oracle. -Recommend one, and let them choose. +Recommend one, and let them choose. If they asked you to choose ("surprise +me"), say in a line which spot you picked and why, and go on to probe it. +They can redirect you. ## 3. Probe @@ -112,7 +114,9 @@ Read the spot's docs, code, tests, and fix history. Then probe it. **A probe** is a small script that calls the real public API with chosen inputs and prints what happens. Import the code the way the repository's own tests do, and run the probe from a scratch directory against the unchanged -code. Keep each probe and its output. +code, with `node` (or `npx tsx` for TypeScript). If Node warns about the +module type, give the scratch directory a `package.json` with +`"type": "module"`. Keep each probe and its output. **Every finding needs a probe.** A claim without one is a suspicion: say so, or leave it out. From 3f1faf1ff19488299473c4315b77ede6dd8ff8b5 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:21:33 -0700 Subject: [PATCH 43/76] docs(oracles-start): show a design for sign-off in plain lines Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index e01348c..8dc9fcc 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -297,8 +297,10 @@ the implementation, look for the public promise that the code serves. [design grammar](../../docs/instruments/design-grammar.md) card fires (the list is in `oracles-design` step 1: for example, more than five laws, or a law with no new case that it predicts), work through the card. Then show -the engineer the design it produced: its parts, and any choice between two -designs. Ask them to approve it with the laws. +the engineer the design it produced, in a few plain lines: what the model +keeps, how the laws share it, and any choice between two designs, with your +recommendation. Leave out the card's own terms (preservation relation, +primitives, ablation). Ask them to approve it with the laws. Then ask the engineer to approve the potential laws that the authority decides. An approved potential law is an adopted law. Potential laws have no From 1a0f29b0ec16b0837521fc06e3a31e0b439d7293 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:23:35 -0700 Subject: [PATCH 44/76] docs(oracles-start): the opening says what the session will do before it asks Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 8dc9fcc..a410d22 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -70,7 +70,8 @@ only when the engineer asks you to. ## 1. Open -Say in two or three sentences what this session will do. For example: "I'll +Your first message starts with two or three sentences on what this session +will do, before any question. For example: "I'll look around your code for a behavior that's easy to break, try it with small scripts that call your real API, and show you what I find. Then I'll write an oracle for it with you and run it." @@ -78,6 +79,8 @@ an oracle for it with you and run it." Then ask one question: is there a part of the code they want to try this on, or should you look around? Stop for the answer. If their first message already named a part of the code, say that you will start there, and go on. +A request about the whole repository ("what can it find in this repo") +names no part: ask. ## 2. Look around From a04abba4665f382f4bee67891fc7c22cf391de8c Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:24:57 -0700 Subject: [PATCH 45/76] docs(oracles-design): the law block does not tell the story of the session Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 5233376..e4f2225 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -150,7 +150,11 @@ model of that law would be the same code, so it could not catch a defect. Keep the law block to about 40 lines of prose, plus the contract code block. It never mentions a review or an open question: a point the user has not -decided appears only as a limit, with its reason. +decided appears only as a limit, with its reason. Nor does it tell the story +of the session: no "the maintainer approved", "the engineer deferred", or +"as the probe showed". A user's decision is stated as the rule it made, and +a limit's reason is about the code ("`klona` calls the constructor with no +arguments"), not about who has not decided yet. **Then write the code under it.** Each boundary condition from the probes and the law worksheet becomes a constructed history. If there was no first From 06706c698fe3f975d30f583adc81205f65cf9ce9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:25:14 -0700 Subject: [PATCH 46/76] docs(oracles-start): approved potential laws are laws; gloss framework terms Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index a410d22..11529e0 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -26,7 +26,11 @@ run again. - Use these terms, and use them every time: a **probe**, a **finding**, a **potential oracle** that owns a public behavior, its **potential laws**, - and the **oracle file**. + and the **oracle file**. Once the engineer approves a potential law, it is + a **law**; once the oracle runs, it is the **oracle**. +- Explain a framework term in a line the first time you use it (a + calibration is a wrong answer planted in a run to show that the check + rejects it), or leave the term out. - Talk naturally. Never name a skill, a step, or a section ("step 3", "the skill's fallback"). Say what you are doing, in your own words. - Ask in plain words. Explain any term the engineer may not know in one From e1085aa4a4766f2bbfb2aa2b1048f689997baeb9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:25:44 -0700 Subject: [PATCH 47/76] docs(oracles-start): the walk-through follows one bug through the file Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 11529e0..529aad3 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -367,6 +367,12 @@ questions, stop and present: - an offer to walk through how it works: how the reference model was built, how the histories are generated, and how the driver calls their code. Engineers often want this, and it is the best part to teach. + +When they take the walk-through, follow one bug the oracle found through +the file: the history that produced it, what the driver did and saw, what +the model expected, and how the check reported the difference and shrank it. +Show a few lines of the file at each part, not a description of it. + - an invitation to ask questions Then explain how to take the oracle to production: a lint preset that keeps From 5d445346013984a0d787483cc59a899b0241f3c6 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:26:59 -0700 Subject: [PATCH 48/76] docs(oracles): probes import source when the build is missing; fix-history is a start; config name without type module Co-authored-by: Isaac --- packages/oracles/skills/oracles-setup/SKILL.md | 4 +++- packages/oracles/skills/oracles-start/SKILL.md | 12 ++++++++---- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index dc6a4d3..b584218 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -120,7 +120,9 @@ The gap check finds oracle files by their `oracleProperty` call, so the name only has to match the Vitest config and the lint glob. - If the repository has no Vitest config, create `vitest.config.ts` for the - oracle files. Jest and mocha ignore that name. + oracle files. Jest and mocha ignore that name. If its `package.json` has + no `"type": "module"`, name it `vitest.config.mts`, so that Vitest loads + it without a module-type warning. - If another tool would pick up `vitest.config.ts`, name it `vitest.oracle.config.ts`, and pass `--config vitest.oracle.config.ts` to the gap check (step 7) and to the Vitest script. diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 529aad3..eacc876 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -94,6 +94,8 @@ docs (the README, the API docs, the JSDoc), the existing tests, and the git history of fixes. The files that fixes change most often tell you more than a search for "fix": `git log -i --grep fix --name-only --format= | sort | uniq -c | sort -rn | head`. +It is a starting point: skip build and config files, and read the commits +that changed the public API. Look for behavior that is easy to break, and whose right answer is simpler to state than the code that produces it. Good signs: @@ -120,7 +122,8 @@ Read the spot's docs, code, tests, and fix history. Then probe it. **A probe** is a small script that calls the real public API with chosen inputs and prints what happens. Import the code the way the repository's own -tests do, and run the probe from a scratch directory against the unchanged +tests do. If they import a build that is not built yet, import the source +it comes from. Run the probe from a scratch directory against the unchanged code, with `node` (or `npx tsx` for TypeScript). If Node warns about the module type, give the scratch directory a `package.json` with `"type": "module"`. Keep each probe and its output. @@ -300,9 +303,10 @@ Settle these privately too: The law must say more than the code does. If a model of it would only repeat the implementation, look for the public promise that the code serves. -**A more complex design needs sign-off.** When a trigger for the -[design grammar](../../docs/instruments/design-grammar.md) card fires (the -list is in `oracles-design` step 1: for example, more than five laws, or a +**A more complex design needs sign-off.** Before you propose, read the +trigger list in `oracles-design` step 1. When a trigger for the +[design grammar](../../docs/instruments/design-grammar.md) card fires (for +example, more than five laws, or a law with no new case that it predicts), work through the card. Then show the engineer the design it produced, in a few plain lines: what the model keeps, how the laws share it, and any choice between two designs, with your From d614a44b0d9eec324415b75da1d0481c9b6dfd90 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:27:45 -0700 Subject: [PATCH 49/76] docs(oracles): the private worksheet says backing, not status; decisions as dated authority A potential law backed by a document is 'stated' until the engineer approves it, so 'adopted' no longer leaks into the chat before approval. A user's decision is written as a dated maintainer decision, not as the session's story. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 4 +++- packages/oracles/skills/oracles-start/SKILL.md | 14 ++++++++------ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index e4f2225..15172ea 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -154,7 +154,9 @@ decided appears only as a limit, with its reason. Nor does it tell the story of the session: no "the maintainer approved", "the engineer deferred", or "as the probe showed". A user's decision is stated as the rule it made, and a limit's reason is about the code ("`klona` calls the constructor with no -arguments"), not about who has not decided yet. +arguments"), not about who has not decided yet. When a user's decision is +the authority for a law, write it as "Maintainer decision, : ", the same way in the `law` option's `authority`. **Then write the code under it.** Each boundary condition from the probes and the law worksheet becomes a constructed history. If there was no first diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index eacc876..2ec4c2b 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -217,7 +217,8 @@ own instead, and recommend one. ``` - Each potential law, as one "For every …" sentence, with its authority - quoted, and what the probes showed about it. + quoted, and what the probes showed about it. Nothing else from your + worksheet: no backing or status line, no rival, no witness. - The laws people forget, or one line on why each does not apply: - **No throw on valid input.** When the contract says a call returns a value, it promises not to throw for valid input. @@ -247,7 +248,7 @@ a few examples to a general law: - Constraints: - Boundary conditions: - Authority: -- Status: adopted | observed | open +- Backing: stated | observed | open - Rival: - Witness: - Checkpoint: @@ -279,10 +280,11 @@ How to fill in the fields: the last item, keys that shadow prototype names, cleanup. A boundary that the law does not decide goes under open questions. Do not leave it out of the grammar. -- **Status:** `adopted` if it has an authority, `observed` if production does - it but nobody adopted it, `open` if it is a question. Only `adopted` laws - become expected results. An `observed` or `open` law goes to the user as a - question. +- **Backing:** `stated` if an authority states it, `observed` if production + does it but nothing states it, `open` if it is a question. A `stated` law + is what you ask the engineer to approve. It becomes adopted only when they + do, and only adopted laws become expected results. An `observed` or `open` + law goes to the engineer as a question. - **Production output is evidence, never authority.** A rule that you write only because production does it is model hacking. Mark that rule `observed`, and ask the user. An analogy to another documented rule is not From 1d325604ba17c4795143e65e30b1b7dcf5725283 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:28:02 -0700 Subject: [PATCH 50/76] docs(oracles-start): list fast-check and vitest even when npm installed them as peers Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 2ec4c2b..d229b4f 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -330,6 +330,10 @@ default: with no engineer present, propose them and stop. from them, take only what these packages and the config need. The lint plugin waits for production. + Check `package.json`, not `node_modules`: npm installs the peers of + `@tanstack/oracles` without listing them, so add `fast-check` and + `vitest` to the dev dependencies even when they are already on disk. + The framework needs Vitest 4.1 or later. Check before you install: - **The repository uses another runner** (mocha, uvu, Jest, node:test): From 88c759fe529406cc1e28b5310268096c1c9e73af Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:30:00 -0700 Subject: [PATCH 51/76] docs(oracles-start): what to do when Node cannot load the source for a probe Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index d229b4f..7ef7fd9 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -126,7 +126,10 @@ tests do. If they import a build that is not built yet, import the source it comes from. Run the probe from a scratch directory against the unchanged code, with `node` (or `npx tsx` for TypeScript). If Node warns about the module type, give the scratch directory a `package.json` with -`"type": "module"`. Keep each probe and its output. +`"type": "module"`. If Node still cannot load the source as it is (for +example, an extensionless ES module), copy it unchanged into the scratch +directory under a name Node can load, such as `index.mjs`, and say so. +Keep each probe and its output. **Every finding needs a probe.** A claim without one is a suspicion: say so, or leave it out. From 3b486bacca47fbd3c21d0281ed5750b216a59dc2 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:34:05 -0700 Subject: [PATCH 52/76] docs(oracles-start): a commit message that states an intent is an authority to quote Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 7ef7fd9..101830b 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -150,7 +150,7 @@ When a probe is cheap, run it over a grid or a random sample of inputs, count the failures, and show the smallest one. **Find the authority for each promise, and show it.** Look in the README, -the API docs, the JSDoc, the tests' names and bodies, the changelog, source +the API docs, the JSDoc, the tests' names and bodies, the changelog, commit messages that state an intent, source comments, and any standard that the docs name. Quote it, with the file and line. The authority is what makes a finding credible, and it becomes the documentation in the oracle file. When nothing decides a behavior, it is a From 84cdc758cadd15a3f15b7498d175bc29d8743aa5 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:36:12 -0700 Subject: [PATCH 53/76] docs(oracles-start): with no bug, show the oracle catching a planted one in a scratch copy Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 101830b..f4ed1e3 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -369,7 +369,10 @@ Do not run the gap check yet, and do not change production code. Both belong to taking the oracle to production. If the oracle finds no bug, say so plainly, show what held, and offer to -build the next oracle at another spot. +build the next oracle at another spot. To show that it would have caught +one, make one or two small, plausible bugs in a scratch copy of the +production code (never in the repository), run the oracle against the copy, +and show which law catches each. ## 6. Present From 3dbf0eb8ef92f300105a881dbd179771c4cdbaa9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:40:33 -0700 Subject: [PATCH 54/76] docs(oracles-start): the lint that waits is the oracles plugin, not the repository's own Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index f4ed1e3..4aaa3d1 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -330,8 +330,10 @@ default: with no engineer present, propose them and stop. file, add a minimal config for it (or add its glob to the existing `include`). Add no scripts, lint, or other config yet. The `oracles-setup` skill's install and "where oracle files go" sections have the details: - from them, take only what these packages and the config need. The lint - plugin waits for production. + from them, take only what these packages and the config need. The + oracles lint plugin waits for production. If the repository's own lint + already covers the oracle file's folder, run it on the file, as + `oracles-design` says. Check `package.json`, not `node_modules`: npm installs the peers of `@tanstack/oracles` without listing them, so add `fast-check` and From 4d421ca164c08e86275d510787885ea0a01ac5fb Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:40:51 -0700 Subject: [PATCH 55/76] docs(oracles-start): present the same way when the oracles found no bug Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 4aaa3d1..803d905 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -379,7 +379,8 @@ and show which law catches each. ## 6. Present When the oracle has found a bug and the engineer has settled some design -questions, stop and present: +questions, stop and present. Present the same way when the oracles found no +bug and the engineer does not want another one: - what you built, and what it found - an offer to walk through how it works: how the reference model was built, From 00509ed34afbc43a5a3ae149153ab5c9bf3e6dfa Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 13:51:01 -0700 Subject: [PATCH 56/76] docs(oracles): one law per property; always show the contract; pin Vitest 4.1 in a first session - oracles-design: each oracleProperty carries one law; properties may share the model, grammar, and driver. - oracles-start: the potential oracle always shows its contract block, keeps the worksheet's backing word private, and installs vitest@^4.1 (Node 20+), leaving CI Node versions to production. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 6 ++++-- packages/oracles/skills/oracles-start/SKILL.md | 12 ++++++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 15172ea..34c3dc7 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -160,8 +160,10 @@ rule>", the same way in the `law` option's `authority`. **Then write the code under it.** Each boundary condition from the probes and the law worksheet becomes a constructed history. If there was no first -session, fill in the worksheet from `oracles-start` for each law first. Each law's sentence, authority, and limits go -in that property's `law` option. Put prose beside the model, the histories, +session, fill in the worksheet from `oracles-start` for each law first. Each `oracleProperty` carries one law: give +each law its own property, so a failure names the law it breaks. The +properties can share one model, grammar, and driver. Each law's sentence, +authority, and limits go in that property's `law` option. Put prose beside the model, the histories, the driver, and the check that says why each one predicts or tests the promised result. diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 803d905..fa5781a 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -213,7 +213,8 @@ draw the boundary), name the second one in a sentence, say what it would own instead, and recommend one. - What it would own, in one or two sentences. -- The contract, as a few lines of public calls and their results: +- The contract, as a few lines of public calls and their results. Always + include it: it is the quickest way for the engineer to see what you mean. ```ts klona(x) // → a deep copy: same kind, prototype, and values as x, sharing no object with it @@ -221,7 +222,8 @@ own instead, and recommend one. - Each potential law, as one "For every …" sentence, with its authority quoted, and what the probes showed about it. Nothing else from your - worksheet: no backing or status line, no rival, no witness. + worksheet: no backing word (`stated`, `observed`, `open`), no rival, no + witness. - The laws people forget, or one line on why each does not apply: - **No throw on valid input.** When the contract says a call returns a value, it promises not to throw for valid input. @@ -324,8 +326,10 @@ default: with no engineer present, propose them and stop. ## 5. Build and run -1. **Install what is missing**, and nothing more: `fast-check` and `vitest` - when the repository does not have them. Use the repository's own package +1. **Install what is missing**, and nothing more: `fast-check@^4` and + `vitest@^4.1` when the repository does not have them. Vitest 4.1 runs on + Node 20 and later, and Vitest 5 needs Node 22.12 or later. Which Node + versions CI runs is a question for production: do not raise it now. Use the repository's own package manager. If the repository's own Vitest config would not pick up the oracle file, add a minimal config for it (or add its glob to the existing `include`). Add no scripts, lint, or other config yet. The `oracles-setup` From 16426a7ee5bc248d01c8d23d430d6ec440aeedb9 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:06:43 -0700 Subject: [PATCH 57/76] docs(oracles): a decision as authority names who made it, with their GitHub login On a team repository a reader needs to know whom to ask. The login comes from gh, or the engineer; it is written without an @ so a copy into a pull request notifies no one. Co-authored-by: Isaac --- packages/oracles/docs/glossary.md | 2 +- packages/oracles/docs/oracle-checklist.md | 3 ++- packages/oracles/skills/oracles-design/SKILL.md | 7 +++++-- packages/oracles/skills/oracles-start/SKILL.md | 7 +++++-- 4 files changed, 13 insertions(+), 6 deletions(-) diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md index 11c8823..d04bd85 100644 --- a/packages/oracles/docs/glossary.md +++ b/packages/oracles/docs/glossary.md @@ -9,7 +9,7 @@ term for one concept. Do not rotate synonyms for style. | Term | Meaning | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | -| authority | Where a law comes from: an API or contract document and section, or a dated maintainer decision. Production output is evidence, never authority. | +| authority | Where a law comes from: an API or contract document and section, or a maintainer decision with its maintainer (name and GitHub login) and date. Production output is evidence, never authority. | | limits | What a property does not establish. Unresolved parts of a law go here. | | potential oracle | A candidate oracle that the agent proposes in the conversation: the public behavior it would own, its contract as calls and results, and its potential laws (each a "For every" sentence with its authority, and what the probes showed), plus its open questions. The engineer approves the laws that the authority decides before the oracle file is written. | | probe | A small script that calls the real public API with chosen inputs and records what happens. A finding needs one. Probes find the constraints, dynamics, and boundaries that become an oracle's laws and histories. | diff --git a/packages/oracles/docs/oracle-checklist.md b/packages/oracles/docs/oracle-checklist.md index 5bbec3f..d5fbc9f 100644 --- a/packages/oracles/docs/oracle-checklist.md +++ b/packages/oracles/docs/oracle-checklist.md @@ -58,7 +58,8 @@ No requirement here can be met by declaring something out of reach. - **Framework:** `law.statement`, `law.authority`, and `law.limits` are required fields. - **Acceptance evidence:** A reviewer can trace the expected result to an - approved API, contract document and section, or dated maintainer decision, + approved API, contract document and section, or a maintainer decision + with its maintainer and date, and can say what the oracle does not establish. Production output is evidence; it is never authority. - **Not required:** A complete list of every behavior outside the subsystem. diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 34c3dc7..e6befbc 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -155,8 +155,11 @@ of the session: no "the maintainer approved", "the engineer deferred", or "as the probe showed". A user's decision is stated as the rule it made, and a limit's reason is about the code ("`klona` calls the constructor with no arguments"), not about who has not decided yet. When a user's decision is -the authority for a law, write it as "Maintainer decision, : ", the same way in the `law` option's `authority`. +the authority for a law, write it as "Decision by (GitHub +), : ", the same way in the `law` option's +`authority`. On a team, a reader needs to know whom to ask. Take the login +from `gh api user --jq .login`, or ask the user. Write the login without an +`@`, so that a copy into a pull request does not notify anyone. **Then write the code under it.** Each boundary condition from the probes and the law worksheet becomes a constructed history. If there was no first diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index fa5781a..7060a53 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -160,7 +160,10 @@ The engineer's answer to a design question is an authority too. Once they decide it, it becomes a potential law, and the oracle can find where the code breaks it, just as with a law from the docs. In a repository with few docs, this is the usual way to a bug: ask the question, and let the oracle -check the answer. +check the answer. The oracle file records the decision with the +engineer's name and GitHub login, so that their team knows whom to ask. +Find the login with `gh api user --jq .login`. If that fails, ask for it +together with the first decision. **Present each finding** in this shape, a light touch, with each finding called a possible bug or a question, never a certainty: @@ -252,7 +255,7 @@ a few examples to a general law: - Dynamics: - Constraints: - Boundary conditions: -- Authority: +- Authority: - Backing: stated | observed | open - Rival: - Witness: From fd3a7994eaecd79ef6eebae8651223ef6fbff38e Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:08:11 -0700 Subject: [PATCH 58/76] docs(oracles): match main: declareOracle is deleted, legacyCampaigns is removed Co-authored-by: Isaac --- packages/eslint-plugin-oracles/README.md | 4 ++-- packages/oracles/skills/oracles-design/SKILL.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/packages/eslint-plugin-oracles/README.md b/packages/eslint-plugin-oracles/README.md index 94ee2af..13dcdd6 100644 --- a/packages/eslint-plugin-oracles/README.md +++ b/packages/eslint-plugin-oracles/README.md @@ -38,5 +38,5 @@ The decided bans on snapshot matchers (B1), `fc.pre` (B3), production imports in a model (B4), and identical comparison operands (B7) are not rules yet. Reviews check them until they ship. -`settings.oracles.legacyCampaigns` exists only for TanStack DB's migration, -and it will be removed. Do not use it in another repository. +`settings.oracles.legacyCampaigns` is removed. A configuration that still +sets it exempts nothing. diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index e6befbc..5a45f3c 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -436,9 +436,9 @@ Keep the law, its authority, and its limits in `law`, and prose beside each part that says why it predicts the promised result (ORC-003). Put `owns`, the law, the model, the histories, the driver, and the check on -`oracleProperty`. Do not call `declareOracle`: it is legacy, kept only for -TanStack DB's migration. If a repository's existing oracles already use it, -leave them as they are. +`oracleProperty`. An oracle file registers its tests only through +`oracleProperty`, or through a project helper that forwards a literal `owns` +list to it. The options of `oracleProperty` are typed in `node_modules/@tanstack/oracles/src/oracle-property.ts` From 4a85561eaf35bbdfad5654ce636e014132278281 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:10:54 -0700 Subject: [PATCH 59/76] docs(oracles): probes learn what the oracle needs; a passing first oracle runs the gap check; the AGENTS.md block governs from the start - oracles-start: probes exist to learn the code's dynamics, boundaries, and constraints that the oracle needs; the agent writes down what it does not know and probes each point. Bugs are a welcome side effect. - When the first oracle finds no bug, run the gap check (output outside the repository) and present what the surviving changes show, before offering the next oracle. A failing oracle still stops at the RED. - Before the first message, read the block that production adds to AGENTS.md: it governs the agent from the start of the session. - Remove every mention of legacyCampaigns from the docs and skills. Co-authored-by: Isaac --- README.md | 7 +-- packages/eslint-plugin-oracles/README.md | 3 - packages/oracles/README.md | 12 ++-- .../oracles/skills/oracles-setup/SKILL.md | 8 +-- .../oracles/skills/oracles-start/SKILL.md | 55 +++++++++++++------ 5 files changed, 52 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index b833625..10e929c 100644 --- a/README.md +++ b/README.md @@ -154,12 +154,7 @@ required. The rule reports: A framework implementation module is exempt. -No other configuration exempts a file. `settings.oracles.legacyCampaigns`, the -old list of files that kept their raw campaigns while they migrated, is removed. -A configuration that still sets it, to any value, exempts nothing. The rule -reports it as `removedSetting` at line 1 of each oracle file that the -configuration reaches, since ESLint runs a rule once per file. Convert the -listed files to `oracleProperty`, then delete the setting. +No configuration exempts any other file. A non-array `settings.oracles.implementationModules` or `replayWitnessModules` is reported at line 1 as `malformedSetting`, and it is read as an empty list. diff --git a/packages/eslint-plugin-oracles/README.md b/packages/eslint-plugin-oracles/README.md index 13dcdd6..a406a72 100644 --- a/packages/eslint-plugin-oracles/README.md +++ b/packages/eslint-plugin-oracles/README.md @@ -37,6 +37,3 @@ elsewhere, add `oracle/probe-form` with your own glob. The decided bans on snapshot matchers (B1), `fc.pre` (B3), production imports in a model (B4), and identical comparison operands (B7) are not rules yet. Reviews check them until they ship. - -`settings.oracles.legacyCampaigns` is removed. A configuration that still -sets it exempts nothing. diff --git a/packages/oracles/README.md b/packages/oracles/README.md index 7866144..17b62d7 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -33,8 +33,9 @@ What happens next: part of the code in mind. 2. It looks around, and suggests two or three places where an oracle would help. You pick one. -3. It tries that code with probes: small scripts that call your real API. - Each finding comes with a snippet you can rerun, and the doc or test that +3. It tries that code with probes: small scripts that call your real API, + to learn how the code behaves at its edges and over a sequence of calls. + That is what the oracle needs. Each finding comes with a snippet you can rerun, and the doc or test that the behavior seems to break: possible bugs, places where the docs and the code disagree, behavior no test checks, and design questions nobody has decided. @@ -43,12 +44,15 @@ What happens next: code promises. 5. It writes the oracle file and runs it on your unchanged code, telling you what each part does as it goes, until the oracle has found a bug and you - have settled some design questions. + have settled some design questions. If the oracle finds no bug, the agent + runs the gap check, which shows production changes that no law catches, + to dig deeper for a bug or a design question. 6. It shows you what it built, offers to walk you through how it works, and explains how to take the oracle to production. Until you choose to ship it, the agent changes nothing in your repository -except the minimal installs and the oracle file. +except the minimal installs, the oracle file, and the gap check's working +directory, if it runs. The package needs Node 20 or later. The setup skill installs Vitest and fast-check if the repository does not have them. diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index b584218..548ae72 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -205,8 +205,7 @@ export default [ it is TypeScript. Otherwise a typed production file fails to parse before the probe rule runs. For JavaScript production source, set the parser for the oracle files only. -- Do not add an exemption. There is none: `legacyCampaigns` is removed, and - setting it is reported. +- Do not add an exemption. There is none. **An older ESLint (7 or 8, `.eslintrc`).** Do not upgrade the repository's lint. Lint the oracle files with a separate ESLint 9: @@ -357,8 +356,9 @@ continue with its gap check. If there is none, build it from the potential laws that the engineer approved: load `oracles-design`, and follow it from its first step. -In a first session, stop at the RED: present the bug, and leave the fix and -the gap check for production. When taking the oracle to production: +In a first session, stop at the RED: present the bug, and leave the fix for +production. A first session runs the gap check only when its oracle passes +(`oracles-start` step 5). When taking the oracle to production: If the first oracle fails on unchanged production under an adopted law, it found a defect. The RED is the finding: report it with its replay line, and diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 7060a53..3b28ead 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -18,9 +18,12 @@ The session ends at one moment: an oracle that runs on their code, has found a bug, and has pushed them to settle some design questions. Then you present what you built, and explain how to take it to production. -An agent that only says things is often wrong, and one wrong claim costs the -tool its credibility. So every finding needs a probe that the engineer can -run again. +Probes are how you learn what reading the code does not tell you: its +dynamics, boundary conditions, and constraints. You need that understanding +to build the oracle. Bugs and inconsistencies that the probes find along the +way are a welcome side effect. They also keep you honest: an agent that only +says things is often wrong, and one wrong claim costs the tool its +credibility. So every finding needs a probe that the engineer can run again. ## How to talk @@ -67,6 +70,9 @@ these: - a minimal Vitest config that picks up the oracle file, only when the repository's own config does not - oracle files, with their prose +- the gap check's working directory (`.oracle-gaps/`), when you run it. + Tell the engineer it is safe to delete, and that production adds it to + `.gitignore`. Findings, potential oracles, and potential laws stay in the chat. Keep probes in a scratch directory outside the repository. Change production code @@ -74,6 +80,12 @@ only when the engineer asks you to. ## 1. Open +Before your first message, read the block in `oracles-setup` step 4 that +production adds to `AGENTS.md`. It governs you from the start of this +session, though nobody has added it to the repository yet: who owns the +laws and who owns the code, which skill to read before each kind of oracle +work, and the lint rule. + Your first message starts with two or three sentences on what this session will do, before any question. For example: "I'll look around your code for a behavior that's easy to break, try it with small @@ -118,7 +130,11 @@ They can redirect you. ## 3. Probe -Read the spot's docs, code, tests, and fix history. Then probe it. +Read the spot's docs, code, tests, and fix history. Then ask yourself: what +do I not know about this code that I need to know to build its oracle? How +does the state change with each call? Where are the boundaries? What holds +whatever happens? Write those questions down for yourself, and build a probe +for each one. **A probe** is a small script that calls the real public API with chosen inputs and prints what happens. Import the code the way the repository's own @@ -374,14 +390,22 @@ default: with no engineer present, propose them and stop. block's prose in the same step. Before you show results, check that the prose still says what the code does. -Do not run the gap check yet, and do not change production code. Both belong -to taking the oracle to production. +Do not change production code. That belongs to taking the oracle to +production. + +**If the oracle finds no bug, run the gap check to go deeper.** Say so +plainly first, and show what held. The gap check makes small changes to a +copy of the production code that the oracle owns, and shows each change +that no law catches. Each one points at a bug the oracle has not reached, a +design question, or a law nobody has stated yet. Take the command and its +flags from `oracles-setup` step 7, and put its `--out` directory in your +scratch directory, outside the repository. Read each surviving change, probe +it when it is unclear, and present what it shows as a finding: a possible +bug, or a design question for the engineer. Do not try to close every gap: +that is production work. -If the oracle finds no bug, say so plainly, show what held, and offer to -build the next oracle at another spot. To show that it would have caught -one, make one or two small, plausible bugs in a scratch copy of the -production code (never in the repository), run the oracle against the copy, -and show which law catches each. +If the gap check finds nothing worth the engineer's time either, offer to +build the next oracle at another spot. ## 6. Present @@ -393,17 +417,16 @@ bug and the engineer does not want another one: - an offer to walk through how it works: how the reference model was built, how the histories are generated, and how the driver calls their code. Engineers often want this, and it is the best part to teach. +- an invitation to ask questions When they take the walk-through, follow one bug the oracle found through the file: the history that produced it, what the driver did and saw, what the model expected, and how the check reported the difference and shrank it. Show a few lines of the file at each part, not a description of it. -- an invitation to ask questions - Then explain how to take the oracle to production: a lint preset that keeps oracle files honest, a block in `AGENTS.md` (or `CLAUDE.md`) so every agent -follows the same rules, and the gap check, which makes small changes to a copy of the -production code and shows each change that no law catches. That is how the -oracle grows toward full coverage. If they want to go ahead, load +follows the same rules, and the gap check in CI, which makes small changes +to a copy of the production code and shows each change that no law catches. +Closing those gaps is how the oracle grows toward full coverage. If they want to go ahead, load `oracles-setup`. From 4749f61cc0c4727bb0f0d96877fa2c0470166ec3 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:13:35 -0700 Subject: [PATCH 60/76] docs(oracles-start): show the questions the probes will answer in the chat, before probing Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 3b28ead..ed9bfe5 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -133,8 +133,9 @@ They can redirect you. Read the spot's docs, code, tests, and fix history. Then ask yourself: what do I not know about this code that I need to know to build its oracle? How does the state change with each call? Where are the boundaries? What holds -whatever happens? Write those questions down for yourself, and build a probe -for each one. +whatever happens? Show the engineer those questions in the chat, a short +list, before you probe: it shows them how an oracle gets built. Then build a +probe for each one, and answer each question with what its probe showed. **A probe** is a small script that calls the real public API with chosen inputs and prints what happens. Import the code the way the repository's own From 5bc7e2fe7d254428c552976e30faf618dece45f8 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:19:02 -0700 Subject: [PATCH 61/76] docs(oracles): resolve cross-file drift from the day's edits From a semantic drift assay of cc88112..4749f61: - the gap check's --out follows oracles-setup step 7 (in .oracle-gaps/), and oracles-setup names step 7 as part of a first session - the session's ending and the walk-through cover the no-bug path - a decision records name, GitHub login, and date in start and checklist - one Vitest 4.1 / fast-check 4 pin across start, setup, and the README - a helper that forwards literal owns registers tests, in setup as in design - adopted means approved, in the glossary - 'gap check in CI' had no source; it is the gap check over all owned code Co-authored-by: Isaac --- packages/oracles/README.md | 4 ++-- packages/oracles/docs/glossary.md | 2 +- packages/oracles/docs/oracle-checklist.md | 2 +- .../oracles/skills/oracles-setup/SKILL.md | 12 +++++++----- .../oracles/skills/oracles-start/SKILL.md | 19 ++++++++++++------- 5 files changed, 23 insertions(+), 16 deletions(-) diff --git a/packages/oracles/README.md b/packages/oracles/README.md index 17b62d7..cf6cf7f 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -54,8 +54,8 @@ Until you choose to ship it, the agent changes nothing in your repository except the minimal installs, the oracle file, and the gap check's working directory, if it runs. -The package needs Node 20 or later. The setup skill installs Vitest and -fast-check if the repository does not have them. +The package needs Node 20 or later. The first session installs Vitest 4.1 +and fast-check 4 if the repository does not have them. ## What ships in this package diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md index d04bd85..a35a408 100644 --- a/packages/oracles/docs/glossary.md +++ b/packages/oracles/docs/glossary.md @@ -13,7 +13,7 @@ term for one concept. Do not rotate synonyms for style. | limits | What a property does not establish. Unresolved parts of a law go here. | | potential oracle | A candidate oracle that the agent proposes in the conversation: the public behavior it would own, its contract as calls and results, and its potential laws (each a "For every" sentence with its authority, and what the probes showed), plus its open questions. The engineer approves the laws that the authority decides before the oracle file is written. | | probe | A small script that calls the real public API with chosen inputs and records what happens. A finding needs one. Probes find the constraints, dynamics, and boundaries that become an oracle's laws and histories. | -| law status | `adopted` (has an authority), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | +| law status | `adopted` (has an authority, and the maintainer approved it), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | | rival | A plausible different law that predicts a different observation for some history. | | witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | | distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | diff --git a/packages/oracles/docs/oracle-checklist.md b/packages/oracles/docs/oracle-checklist.md index d5fbc9f..356d0c7 100644 --- a/packages/oracles/docs/oracle-checklist.md +++ b/packages/oracles/docs/oracle-checklist.md @@ -59,7 +59,7 @@ No requirement here can be met by declaring something out of reach. required fields. - **Acceptance evidence:** A reviewer can trace the expected result to an approved API, contract document and section, or a maintainer decision - with its maintainer and date, + with its maintainer (name and GitHub login) and date, and can say what the oracle does not establish. Production output is evidence; it is never authority. - **Not required:** A complete list of every behavior outside the subsystem. diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index 548ae72..ac5a77b 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -12,8 +12,9 @@ your report, and continue. That default covers setup questions only. Potential laws have no default: propose them, report them, and stop. This skill has two uses. A first session (`oracles-start`) uses only its -install section and its "where oracle files go" section, and only for the -minimal setup that an oracle needs to run. Taking an oracle to production +install section and its "where oracle files go" section, for the minimal +setup that an oracle needs to run, and step 7's command, when its oracle +passes and it runs the gap check. Taking an oracle to production uses all of it. That usually starts after a first session, once the engineer has an oracle that found a bug and chooses to ship it. If there is no oracle yet, load `oracles-start` first. @@ -28,7 +29,7 @@ Find out, and tell the user: - The package manager, and whether the repository is one package or a workspace. For a workspace, ask which package to start in. -- The test runner. The framework runs on Vitest 4 or later, with fast-check 4. +- The test runner. The framework runs on Vitest 4.1 or later, with fast-check 4. If the package uses another runner, oracles run in a separate Vitest config beside it. Do not change the existing suite. - Where production source lives (for example `src/`). @@ -57,7 +58,7 @@ Tell the user which commit you pinned. pnpm add -D \ https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@ \ https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@ \ - fast-check vitest + fast-check@^4 vitest@^4.1 ``` Add `fast-check` and `vitest` beside them. The framework needs Vitest 4.1 or @@ -377,7 +378,8 @@ call tree, including the module-level data it reads. A default export is If the gap check refuses with "no oracle file", the oracle is not declared: that is a setup failure. -In an oracle file, only `oracleProperty` registers tests. A plain Vitest test +In an oracle file, only `oracleProperty` registers tests, directly or +through a project helper that forwards a literal `owns` list to it. A plain Vitest test there is not a kill, and the gap check reports it. Lint also reports an empty test there: it checks nothing and looks like coverage. Remove it, or use `it.todo`. The driver records production errors as observations: a mutant that crashes production is not a diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index ed9bfe5..7512362 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -16,7 +16,9 @@ the oracle file. Do not explain what oracles are unless they ask. The session ends at one moment: an oracle that runs on their code, has found a bug, and has pushed them to settle some design questions. Then you present -what you built, and explain how to take it to production. +what you built, and explain how to take it to production. If no oracle and no +gap check finds a bug, the session ends when the engineer wants no further +oracle, and you present the same way. Probes are how you learn what reading the code does not tell you: its dynamics, boundary conditions, and constraints. You need that understanding @@ -66,7 +68,7 @@ Until the engineer chooses to take an oracle to production, change only these: - the minimal installs that an oracle needs to run: `@tanstack/oracles`, - `fast-check`, and `vitest` when they are missing + `fast-check`, and `vitest` when `package.json` does not list them - a minimal Vitest config that picks up the oracle file, only when the repository's own config does not - oracle files, with their prose @@ -178,7 +180,8 @@ decide it, it becomes a potential law, and the oracle can find where the code breaks it, just as with a law from the docs. In a repository with few docs, this is the usual way to a bug: ask the question, and let the oracle check the answer. The oracle file records the decision with the -engineer's name and GitHub login, so that their team knows whom to ask. +engineer's name, their GitHub login, and the date, so that their team knows +whom to ask. Find the login with `gh api user --jq .login`. If that fails, ask for it together with the first decision. @@ -399,8 +402,8 @@ plainly first, and show what held. The gap check makes small changes to a copy of the production code that the oracle owns, and shows each change that no law catches. Each one points at a bug the oracle has not reached, a design question, or a law nobody has stated yet. Take the command and its -flags from `oracles-setup` step 7, and put its `--out` directory in your -scratch directory, outside the repository. Read each surviving change, probe +flags from `oracles-setup` step 7, with its `--out` in the package's +`.oracle-gaps/` directory, as that step says. Read each surviving change, probe it when it is unclear, and present what it shows as a finding: a possible bug, or a design question for the engineer. Do not try to close every gap: that is production work. @@ -421,13 +424,15 @@ bug and the engineer does not want another one: - an invitation to ask questions When they take the walk-through, follow one bug the oracle found through -the file: the history that produced it, what the driver did and saw, what +the file (with no bug, one change the gap check showed, or one history that +held): the history that produced it, what the driver did and saw, what the model expected, and how the check reported the difference and shrank it. Show a few lines of the file at each part, not a description of it. Then explain how to take the oracle to production: a lint preset that keeps oracle files honest, a block in `AGENTS.md` (or `CLAUDE.md`) so every agent -follows the same rules, and the gap check in CI, which makes small changes +follows the same rules, and the gap check run over all the owned code, which +makes small changes to a copy of the production code and shows each change that no law catches. Closing those gaps is how the oracle grows toward full coverage. If they want to go ahead, load `oracles-setup`. From da4d6da9f2bc62b7bfbf74ffd59b1174a3554c93 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:22:33 -0700 Subject: [PATCH 62/76] docs(oracles-start): milestones to work toward; the engineer decides when the session ends Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 7512362..09faa2b 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -14,11 +14,11 @@ explanation to something you just showed them: a probe, a finding, a law, or the oracle file. Do not explain what oracles are unless they ask. [What an oracle is](../../docs/intro.md) is there for when they do. -The session ends at one moment: an oracle that runs on their code, has found -a bug, and has pushed them to settle some design questions. Then you present -what you built, and explain how to take it to production. If no oracle and no -gap check finds a bug, the session ends when the engineer wants no further -oracle, and you present the same way. +Work toward these milestones, in order: findings from probes of their real +code; a potential oracle whose laws they approve; an oracle that runs on +their code; and that oracle finding a bug, or the gap check finding what no +law catches, while they settle some design questions. The engineer decides +when the session ends. Probes are how you learn what reading the code does not tell you: its dynamics, boundary conditions, and constraints. You need that understanding @@ -413,9 +413,9 @@ build the next oracle at another spot. ## 6. Present -When the oracle has found a bug and the engineer has settled some design -questions, stop and present. Present the same way when the oracles found no -bug and the engineer does not want another one: +When the oracle or the gap check has found something and the engineer has +settled some design questions, stop and present. Present the same way +whenever the engineer wants to stop: - what you built, and what it found - an offer to walk through how it works: how the reference model was built, From 8f7f1344aaa248c0f3000c6ea0c44ac0168788e2 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:26:49 -0700 Subject: [PATCH 63/76] docs(oracles-start): move install and probe special cases to references/problems.md The skill keeps the main path and a pointer; the agent opens the reference when it hits another test runner, an older Vitest, a failed install, peers already on disk, or a probe that Node cannot load. Co-authored-by: Isaac --- .../oracles/skills/oracles-start/SKILL.md | 51 +++++-------------- .../oracles-start/references/problems.md | 47 +++++++++++++++++ 2 files changed, 60 insertions(+), 38 deletions(-) create mode 100644 packages/oracles/skills/oracles-start/references/problems.md diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 09faa2b..cad8acd 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -141,14 +141,10 @@ probe for each one, and answer each question with what its probe showed. **A probe** is a small script that calls the real public API with chosen inputs and prints what happens. Import the code the way the repository's own -tests do. If they import a build that is not built yet, import the source -it comes from. Run the probe from a scratch directory against the unchanged -code, with `node` (or `npx tsx` for TypeScript). If Node warns about the -module type, give the scratch directory a `package.json` with -`"type": "module"`. If Node still cannot load the source as it is (for -example, an extensionless ES module), copy it unchanged into the scratch -directory under a name Node can load, such as `index.mjs`, and say so. -Keep each probe and its output. +tests do, and run the probe with `node` from a scratch directory against +the unchanged code. If the import or Node gets in the way, open +[Problems you may hit](references/problems.md). Keep each probe and its +output. **Every finding needs a probe.** A claim without one is a suspicion: say so, or leave it out. @@ -350,36 +346,15 @@ default: with no engineer present, propose them and stop. ## 5. Build and run 1. **Install what is missing**, and nothing more: `fast-check@^4` and - `vitest@^4.1` when the repository does not have them. Vitest 4.1 runs on - Node 20 and later, and Vitest 5 needs Node 22.12 or later. Which Node - versions CI runs is a question for production: do not raise it now. Use the repository's own package - manager. If the repository's own Vitest config would not pick up the oracle - file, add a minimal config for it (or add its glob to the existing - `include`). Add no scripts, lint, or other config yet. The `oracles-setup` - skill's install and "where oracle files go" sections have the details: - from them, take only what these packages and the config need. The - oracles lint plugin waits for production. If the repository's own lint - already covers the oracle file's folder, run it on the file, as - `oracles-design` says. - - Check `package.json`, not `node_modules`: npm installs the peers of - `@tanstack/oracles` without listing them, so add `fast-check` and - `vitest` to the dev dependencies even when they are already on disk. - - The framework needs Vitest 4.1 or later. Check before you install: - - - **The repository uses another runner** (mocha, uvu, Jest, node:test): - add Vitest for the oracles only, with a config whose `include` is only - the oracle files. Their own suite and runner stay as they are. Tell - the engineer this in one line as you install. - - **The repository runs an older Vitest:** this is a decision for the - engineer. Tell them that the oracle needs Vitest 4.1 or later, and that - upgrading changes the runner for their own suite too. Offer to upgrade - and then run their suite, to show whether it still passes. Do not - upgrade until they agree. - - **An install fails:** show the error in a line or two, say what you - think caused it, and ask before you try anything that changes more than - the packages above. + `vitest@^4.1` when `package.json` does not list them. Use the + repository's own package manager. If the repository's own Vitest config + would not pick up the oracle file, add a minimal config for it (or add + its glob to the existing `include`). Add no scripts, lint, or other + config yet. The `oracles-setup` skill's install and "where oracle files + go" sections have the details: take only what these packages and the + config need. If the repository uses another test runner or an older + Vitest, or an install fails, open + [Problems you may hit](references/problems.md). 2. **Write the oracle file.** Follow `oracles-design` from its first step: the law block's prose, then the model, the histories, the driver, and the diff --git a/packages/oracles/skills/oracles-start/references/problems.md b/packages/oracles/skills/oracles-start/references/problems.md new file mode 100644 index 0000000..0447b63 --- /dev/null +++ b/packages/oracles/skills/oracles-start/references/problems.md @@ -0,0 +1,47 @@ +# Problems you may hit in a first session + +Open this file when one of these happens. Each entry says what to do, and +what to tell the engineer. + +## Installs + +**The repository uses another test runner** (mocha, uvu, Jest, ava, +`node:test`). Add Vitest for the oracles only, with a config whose `include` +is only the oracle files. Their own suite and runner stay as they are. Tell +the engineer this in one line as you install. + +**The repository runs a Vitest older than 4.1.** This is the engineer's +decision. Tell them that the oracle needs Vitest 4.1 or later, and that +upgrading changes the runner for their own suite too. Offer to upgrade and +then run their suite, to show whether it still passes. Do not upgrade until +they agree. + +**npm installed `fast-check` and `vitest` already.** npm installs the peers +of `@tanstack/oracles` without listing them. Check `package.json`, not +`node_modules`, and add both to the dev dependencies. + +**Which Vitest.** Install `vitest@^4.1`. It runs on Node 20 and later, and +Vitest 5 needs Node 22.12 or later. Which Node versions CI runs is a +question for production: do not raise it now. + +**An install fails.** Show the error in a line or two, say what you think +caused it, and ask before you try anything that changes more than the +packages the oracle needs. + +**Lint.** The oracles lint plugin waits for production. If the +repository's own lint already covers the oracle file's folder, run it on +the file, as `oracles-design` says. + +## Probes + +**The tests import a build that is not built yet.** Import the source that +the build comes from. + +**Node warns about the module type.** Give the scratch directory a +`package.json` with `"type": "module"`. + +**Node cannot load the source as it is** (for example, an extensionless ES +module). Copy it unchanged into the scratch directory, under a name that +Node can load, such as `index.mjs`, and tell the engineer that you did. + +**The code is TypeScript.** Run the probe with `npx tsx`. From 9d16dbbe44fd49dfa464bd042a67bdddc7c32342 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:33:20 -0700 Subject: [PATCH 64/76] docs(oracles): developer and maintainer as the two roles; terms reconciled; a preview feedback log - 'developer' replaces 'engineer' and 'user' for the person the agent works with; 'maintainer' is only the role with authority over the laws. A developer's approval adopts a law and is recorded; on a shared repository the maintainers confirm it in review. A library's end user is 'the caller'. - 'approved' is the act and 'adopted' the state; the glossary glosses owns as a public behavior in conversation; 'sketch' becomes 'draft'; the finding kind 'Test gap' becomes 'Untested behavior'. - While in preview, a first session keeps a feedback log outside the repository and asks the developer to send it to Kyle Mathews. Co-authored-by: Isaac --- packages/oracles/README.md | 4 + packages/oracles/docs/glossary.md | 68 ++++++------ packages/oracles/docs/guide.md | 4 +- .../docs/instruments/design-grammar.md | 12 +-- packages/oracles/docs/instruments/index.md | 2 +- .../docs/instruments/mutant-gap-hunt.md | 4 +- packages/oracles/docs/intro.md | 4 +- packages/oracles/docs/oracle-checklist.md | 4 +- .../oracles/skills/oracles-design/SKILL.md | 54 +++++----- .../skills/oracles-evaluate-review/SKILL.md | 10 +- .../oracles/skills/oracles-review/SKILL.md | 4 +- .../oracles/skills/oracles-setup/SKILL.md | 60 +++++------ .../oracles/skills/oracles-start/SKILL.md | 102 ++++++++++-------- .../oracles-start/references/problems.md | 8 +- 14 files changed, 181 insertions(+), 159 deletions(-) diff --git a/packages/oracles/README.md b/packages/oracles/README.md index cf6cf7f..b2b699d 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -54,6 +54,10 @@ Until you choose to ship it, the agent changes nothing in your repository except the minimal installs, the oracle file, and the gap check's working directory, if it runs. +While the package is in preview, the agent also keeps a feedback log at +`~/oracles-feedback/-.md`, outside your repository. When +you're done, please send it to Kyle Mathews (GitHub KyleAMathews). + The package needs Node 20 or later. The first session installs Vitest 4.1 and fast-check 4 if the repository does not have them. diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md index a35a408..085b167 100644 --- a/packages/oracles/docs/glossary.md +++ b/packages/oracles/docs/glossary.md @@ -6,24 +6,24 @@ term for one concept. Do not rotate synonyms for style. ## Laws and evidence -| Term | Meaning | -| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | -| authority | Where a law comes from: an API or contract document and section, or a maintainer decision with its maintainer (name and GitHub login) and date. Production output is evidence, never authority. | -| limits | What a property does not establish. Unresolved parts of a law go here. | -| potential oracle | A candidate oracle that the agent proposes in the conversation: the public behavior it would own, its contract as calls and results, and its potential laws (each a "For every" sentence with its authority, and what the probes showed), plus its open questions. The engineer approves the laws that the authority decides before the oracle file is written. | -| probe | A small script that calls the real public API with chosen inputs and records what happens. A finding needs one. Probes find the constraints, dynamics, and boundaries that become an oracle's laws and histories. | -| law status | `adopted` (has an authority, and the maintainer approved it), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | -| rival | A plausible different law that predicts a different observation for some history. | -| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | -| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | -| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | -| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | -| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | -| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | -| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the user can pick. | -| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | -| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | +| Term | Meaning | +| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | +| authority | Where a law comes from: an API or contract document and section, or a decision, with who made it (name and GitHub login) and its date. Production output is evidence, never authority. | +| limits | What a property does not establish. Unresolved parts of a law go here. | +| potential oracle | A candidate oracle that the agent proposes in the conversation: the public behavior it would own, its contract as calls and results, and its potential laws (each a "For every" sentence with its authority, and what the probes showed), plus its open questions. The developer approves the laws that the authority decides before the oracle file is written. | +| probe | A small script that calls the real public API with chosen inputs and records what happens. A finding needs one. Probes find the constraints, dynamics, and boundaries that become an oracle's laws and histories. | +| law status | `adopted` (has an authority, and a developer approved it; on a shared repository, its maintainers confirm it in review), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | +| rival | A plausible different law that predicts a different observation for some history. | +| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | +| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | +| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | +| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | +| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | +| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | +| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the developer can pick. | +| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | +| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | ## The parts of an `oracleProperty` @@ -59,22 +59,22 @@ term for one concept. Do not rotate synonyms for style. ## Gaps -| Term | Meaning | -| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| owns | The production files or functions an oracle answers for, listed in the `owns` option of `oracleProperty`. The gap check judges owned code only. | -| gap check | The loop that runs the oracle files against production and against mutants of the owned code. It reports every gap with one instruction. | -| mutant | A small deliberate change to owned production code. Every mutant is deliberately wrong, so do not qualify the word. The original defect is a common mutant to check. | -| kill | A mutant that a law rejects through a registered `oracleProperty` route. A plain test in an oracle file is an unregistered kill, and it does not count. | -| survivor | A mutant that no law rejects. Every survivor is a gap. | -| gap | Owned behavior that the oracles do not judge. It closes only when a law kills it or the code is removed. | -| exit | Any reasoning that puts a gap out of reach: calling a mutant equivalent or unobservable, or accepting a gap. None is allowed. | -| framework bug | A gap that no kill or removal can close. The agent stops work on that gap and reports its key and evidence. | -| exclusion control | ORC-004's control: a nearby invalid history that the grammar rejects. It is required, and distinct from a quiet exclusion, which is a finding. | -| dead code | Code whose removal no legal history can observe. Remove it. | -| invalid mutant | A mutant that no oracle could possibly detect. The framework must not generate one, so meeting one is a framework bug to report. The author never decides that a mutant is invalid, and the label does not close a gap. | -| unowned | Production code that no oracle owns. The gap check reports it as a warning, not as judged. | -| RED | The extended oracle failing on unchanged production. It shows that the oracle can see the defect. | -| GREEN | The same oracle passing after the change. | +| Term | Meaning | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| owns | The production files or functions an oracle answers for, listed in the `owns` option of `oracleProperty`. The gap check judges owned code only. In conversation, an oracle owns a public behavior; its `owns` list names the functions that produce it. | +| gap check | The loop that runs the oracle files against production and against mutants of the owned code. It reports every gap with one instruction. | +| mutant | A small deliberate change to owned production code. Every mutant is deliberately wrong, so do not qualify the word. The original defect is a common mutant to check. | +| kill | A mutant that a law rejects through a registered `oracleProperty` route. A plain test in an oracle file is an unregistered kill, and it does not count. | +| survivor | A mutant that no law rejects. Every survivor is a gap. | +| gap | Owned behavior that the oracles do not judge. It closes only when a law kills it or the code is removed. | +| exit | Any reasoning that puts a gap out of reach: calling a mutant equivalent or unobservable, or accepting a gap. None is allowed. | +| framework bug | A gap that no kill or removal can close. The agent stops work on that gap and reports its key and evidence. | +| exclusion control | ORC-004's control: a nearby invalid history that the grammar rejects. It is required, and distinct from a quiet exclusion, which is a finding. | +| dead code | Code whose removal no legal history can observe. Remove it. | +| invalid mutant | A mutant that no oracle could possibly detect. The framework must not generate one, so meeting one is a framework bug to report. The author never decides that a mutant is invalid, and the label does not close a gap. | +| unowned | Production code that no oracle owns. The gap check reports it as a warning, not as judged. | +| RED | The extended oracle failing on unchanged production. It shows that the oracle can see the defect. | +| GREEN | The same oracle passing after the change. | There is no "equivalent mutant" and no "unobservable effect". A survivor shows a missing history, observation, or law, or dead code. An effect that diff --git a/packages/oracles/docs/guide.md b/packages/oracles/docs/guide.md index 52a5845..1ac1573 100644 --- a/packages/oracles/docs/guide.md +++ b/packages/oracles/docs/guide.md @@ -98,7 +98,7 @@ Which state determines the observation? Before you change production to satisfy a red oracle, check the contract, the model's representation, and the comparison. An unresolved product choice goes -to the user as an `open` law. Judge a reviewer's accusation and their proposed +to the developer as an `open` law. Judge a reviewer's accusation and their proposed fix separately: a correct concern does not make its repair correct. Projection needs the same care. Dropping internal metadata from a comparison @@ -444,7 +444,7 @@ add an empty case. - **Minimal ambiguity:** include the smallest valid input for every branch of a classifier. Rich values that carry several redundant signals do not cover a one-field collision. -- **Name invariance:** renaming a user-controlled name must not change a result +- **Name invariance:** renaming a developer-controlled name must not change a result unless the contract gives the name meaning. - **Representation symmetry:** two forms of one input that the contract treats as the same must give the same diff --git a/packages/oracles/docs/instruments/design-grammar.md b/packages/oracles/docs/instruments/design-grammar.md index e34319d..ed8db26 100644 --- a/packages/oracles/docs/instruments/design-grammar.md +++ b/packages/oracles/docs/instruments/design-grammar.md @@ -25,7 +25,7 @@ of range. allow the same observations at the same checkpoints. Name the checkpoints, the assumptions, and the freedoms the contract leaves - open. The user's approval of the laws fixes the intended contract. It is + open. The developer's approval of the laws fixes the intended contract. It is not evidence that a model meets it. 2. **Separate what you found from what you added.** Each rule cites its @@ -40,7 +40,7 @@ of range. 4. **Record rules and conflicts.** Write each rule as one sentence of the law. When two rules conflict, keep both and mark the conflict as an open question - for the user. Record a priority only when the authority gives one. + for the developer. Record a priority only when the authority gives one. 5. **Check overlaps.** Where two parts share state or timing, ask whether the interaction itself carries a law, such as a cleanup that races a late @@ -79,7 +79,7 @@ of range. - **Refactor:** the allowed behavior stays the same. - **Refine:** the model adds a guarantee or narrows what it allows. - **Revise:** the contract itself changes. That is a law question for the - user. + developer. Do not change the law, a history filter, and an observation in one step. Each needs its own comparison. @@ -90,8 +90,8 @@ of range. laws across oracles, or another production structure that the same oracle accepts. Give each form's route, what it keeps, what it changes, and its cost. Merge forms that differ only in naming or detail. Do not rank them - unless asked. When the choice is the user's, present the forms and let the - user pick. + unless asked. When the choice is the developer's, present the forms and let the + developer pick. 9. **Optionally check a formal model.** For a consequential claim about state, concurrency, retries, or ownership, translate the reduced model into TLA+ @@ -134,6 +134,6 @@ simplify. A simplification that rejects an allowed outcome narrows the contract. - a mapping that hides an observation **Stop when:** every part has a job and a witness, every simplification has -its comparison, and each open conflict is with the user. +its comparison, and each open conflict is with the developer. Adapted from Field Lab's design grammar instrument. diff --git a/packages/oracles/docs/instruments/index.md b/packages/oracles/docs/instruments/index.md index 7f6a141..833e774 100644 --- a/packages/oracles/docs/instruments/index.md +++ b/packages/oracles/docs/instruments/index.md @@ -2,7 +2,7 @@ Use an instrument when a specific uncertainty makes ordinary implementation or review insufficient. Read only the relevant card. Choose and use it within the -user's authorized task; these cards do not require a separate approval for each +developer's authorized task; these cards do not require a separate approval for each reasoning step or a fixed workflow. | Uncertainty | Instrument | Result | diff --git a/packages/oracles/docs/instruments/mutant-gap-hunt.md b/packages/oracles/docs/instruments/mutant-gap-hunt.md index cbc26ff..27566d0 100644 --- a/packages/oracles/docs/instruments/mutant-gap-hunt.md +++ b/packages/oracles/docs/instruments/mutant-gap-hunt.md @@ -58,7 +58,7 @@ Keep test bodies and assertions out of the design brief. separately from the kills. Until the gap check recognizes allowed sets, it stays open in the report. Only the framework closes it. - **A law question:** the observations differ, and the contract only seems - to allow both. Do not record that as acceptable. Give the user the two + to allow both. Do not record that as acceptable. Give the developer the two observations and the history that shows them, and ask whether the law should decide the choice. - **A framework bug:** none of the above fits, such as a mutant that no @@ -107,5 +107,5 @@ branch was never reached. Recheck types after a tool rewrites code. - a decision checked in one build mode while another mode can observe it **Stop when:** each selected mutant has an executed distinction, a removed -piece of dead code, a law question with the user, or a framework bug +piece of dead code, a law question with the developer, or a framework bug reported. Do not infer broad coverage from a kill count. diff --git a/packages/oracles/docs/intro.md b/packages/oracles/docs/intro.md index fce1a8c..fffd152 100644 --- a/packages/oracles/docs/intro.md +++ b/packages/oracles/docs/intro.md @@ -1,8 +1,8 @@ # What an oracle is -This page is for an engineer who asks what an oracle is. An agent running a +This page is for a developer who asks what an oracle is. An agent running a first session does not recite it: it shows oracles working on the -engineer's own code, and uses this page to answer questions. +developer's own code, and uses this page to answer questions. Types check that a value has the right shape. An oracle checks that the code does what it promises, for every input and sequence of calls the promise diff --git a/packages/oracles/docs/oracle-checklist.md b/packages/oracles/docs/oracle-checklist.md index 356d0c7..31c11fd 100644 --- a/packages/oracles/docs/oracle-checklist.md +++ b/packages/oracles/docs/oracle-checklist.md @@ -58,8 +58,8 @@ No requirement here can be met by declaring something out of reach. - **Framework:** `law.statement`, `law.authority`, and `law.limits` are required fields. - **Acceptance evidence:** A reviewer can trace the expected result to an - approved API, contract document and section, or a maintainer decision - with its maintainer (name and GitHub login) and date, + approved API, contract document and section, or a decision with who + made it (name and GitHub login) and its date, and can say what the oracle does not establish. Production output is evidence; it is never authority. - **Not required:** A complete list of every behavior outside the subsystem. diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 5a45f3c..f3bd2b6 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -23,15 +23,15 @@ failure goes away, and pinning one example. `oracles-review` §2 lists them all. ## 1. Start from approved potential laws, then write the oracle file -**A new oracle starts from potential laws that the user approved**, usually +**A new oracle starts from potential laws that the developer approved**, usually in a first session (`oracles-start`): the laws that the authority decides. -Open design questions may still be open: they are limits until the user +Open design questions may still be open: they are limits until the developer settles them. If no laws are approved yet, probe the behavior and propose them first, as `oracles-start` does, and stop for approval. Exploration and review stay in the conversation, never in the oracle file. The file is written with its law block and its code together. -**Decide whether the user must approve.** +**Decide whether the developer must approve.** - **Strengthening an adopted law** (a missing history, observation, driver path, or check) reuses the law as it is. Do not propose new laws, and do @@ -39,21 +39,21 @@ written with its law block and its code together. law's stated domain is strengthening. Changing the stated domain, a checkpoint, or `limits` is a law change. - **Changing a law** (its promise, domain, checkpoint, assumptions, or the - freedoms it allows) needs a law delta. Show the user the old and the new + freedoms it allows) needs a law delta. Show the developer the old and the new law, with the probe or result that shows why. Stop for their review, then rewrite the law block in the same change. -- **A new law, or missing or contradictory authority,** needs the user's +- **A new law, or missing or contradictory authority,** needs the developer's approval of that potential law. An analogy to another documented rule is not authority. -- **The user's answer to an open question is its review.** Rewrite the law - block in the same step, and continue. A change the user did not ask for +- **The developer's answer to an open question is its review.** Rewrite the law + block in the same step, and continue. A change the developer did not ask for (you found that a law is wrong) needs the delta and a stop. - **The task already approved these laws?** Do not ask again. An approved potential law is an adopted law. **Production output is evidence, never authority.** A rule that exists only because production does it is model hacking. When the law and the code -disagree, the approved law decides, not the code. +disagree, the adopted law decides, not the code. When the promise is unclear, or two readings predict different behavior, use the [law discovery](../../docs/instruments/law-discovery.md) card. @@ -71,11 +71,11 @@ list when you propose the laws, and again after each gap check: - the gap check leaves three or more gaps in one function Say in the evidence which trigger fired, or that none did. When a trigger -fires before the oracle is written, show the user the design the card +fires before the oracle is written, show the developer the design the card produced, and get their approval with the laws. **Write the law block.** The law block is the file's opening comment. The oracle file is a literate -program (ORC-003), so write it as prose. That reader is a maintainer or a serious user who knows +program (ORC-003), so write it as prose. That reader is a maintainer, or a developer who uses the library seriously and knows the codebase. Name internal parts without introducing them, and do not explain the language, the library's basic ideas, or what a reader of the code can see at once. Spend the words on what such a reader could get @@ -97,9 +97,9 @@ reader. sentences, that shows the public API promises this oracle maintains. A reader takes it in at one look. Give it no title, and no sentence that introduces it. Write each line as `call // → result`: - - the call is the public API, written the way a user writes it, with + - the call is the public API, written the way a caller writes it, with concrete values where they make the promise clear. - - the result is what the user gets back or sees: a value, or a short + - the result is what the caller gets back or sees: a value, or a short phrase. - a condition goes at the end of the line, as "when ...". - an open law ends with `(open)`. @@ -149,16 +149,16 @@ its documentation promises, and a law that restates the implementation. A model of that law would be the same code, so it could not catch a defect. Keep the law block to about 40 lines of prose, plus the contract code block. -It never mentions a review or an open question: a point the user has not +It never mentions a review or an open question: a point the developer has not decided appears only as a limit, with its reason. Nor does it tell the story -of the session: no "the maintainer approved", "the engineer deferred", or -"as the probe showed". A user's decision is stated as the rule it made, and +of the session: no "the maintainer approved", "the developer deferred", or +"as the probe showed". A developer's decision is stated as the rule it made, and a limit's reason is about the code ("`klona` calls the constructor with no -arguments"), not about who has not decided yet. When a user's decision is +arguments"), not about who has not decided yet. When a developer's decision is the authority for a law, write it as "Decision by (GitHub ), : ", the same way in the `law` option's `authority`. On a team, a reader needs to know whom to ask. Take the login -from `gh api user --jq .login`, or ask the user. Write the login without an +from `gh api user --jq .login`, or ask the developer. Write the login without an `@`, so that a copy into a pull request does not notify anyone. **Then write the code under it.** Each boundary condition from the probes @@ -178,8 +178,8 @@ promised result. contract code block: public API calls and their results, with no assertion or helper. Each line states the same law as its sentence. - No law restates the code. -- Each law sentence matches the law the user approved. If they disagree, the - approved law decides: fix the prose, and tell the user. A new heading for +- Each law sentence matches the law the developer approved. If they disagree, the + adopted law decides: fix the prose, and tell the developer. A new heading for a law is not a law change. A changed sentence, domain, or checkpoint is. Keep the law block, the `law` option, and the model in step: a change to one @@ -190,7 +190,7 @@ law during the work, rewrite the law block at once, not at the end. Write these down. They stay fixed while you build the oracle. -- The laws and their checkpoints, as the user approved them. +- The laws and their checkpoints, as the developer approved them. - The owned production code: the files or functions the oracle answers for. - The freedoms the contract leaves open, such as the order of independent events, scheduling, or representation. The oracle must accept every @@ -209,7 +209,7 @@ failure. Make each weakening its own decision, with its own reason. histories and the model. A known example that the model cannot reproduce shows a missing part or a wrong rule. - When two rules conflict, record the conflict as an open question for the - user. Do not pick a winner. + developer. Do not pick a winner. For the design grammar card, see the triggers in step 1. @@ -239,7 +239,7 @@ can state it: Mixing styles is fine. [Model styles](../../docs/model-styles.md) shows a small example of each style. -When two styles fit the law, sketch both briefly, keep the one that states +When two styles fit the law, draft both briefly, keep the one that states the law most directly, and name the other in the law block, so a reader can choose it later. @@ -491,9 +491,9 @@ it to simplify: reason. - Prefer a change that shrinks the code, or leaves its size the same. If the repository sets a size rule, such as a net-negative diff, follow it. -- For a larger cleanup, sketch two or three designs that pass the same oracle +- For a larger cleanup, draft two or three designs that pass the same oracle (a table instead of branches, one state field instead of three, a - recompute instead of a cache). Give each one's trade-off, and let the user + recompute instead of a cache). Give each one's trade-off, and let the developer choose when the choice is theirs. Do not keep only the first design you find. - Rerun the oracle and the gap check after each cleanup. @@ -534,20 +534,20 @@ comes with one instruction. Do what it says, within the rules below. mutant only moves production to another member of an allowed set, report it with its key, the clause, and the history, apart from the kills. Do not strengthen the law or change production to kill it. It stays open until the - framework recognizes allowed sets. The user decides whether that blocks the + framework recognizes allowed sets. The developer decides whether that blocks the work. - **A stated limit does not close a gap.** If a survivor differs from the original only on inputs that the law's limits leave out (a null-prototype object, say), the gap is open. Report it with that reason, and recommend widening the grammar and the law to say what the code promises there. If that behavior is truly not part of the contract, the code that the mutant - changes is a production question for the owner. Either way, the user + changes is a production question for the owner. Either way, the developer decides. - **A gap that existed before your change still counts** when it is in a function you changed. Report it with its law question. Do not take an exit. - **Never propose an exit.** If a gap can be neither killed nor removed, it is a framework bug. Stop work on that gap, report its key and evidence to the - user, and keep working on the other gaps. + developer, and keep working on the other gaps. - **A mutant that no oracle could possibly detect is an invalid mutant.** The framework must not generate it, so it is a framework bug: report it the same way. You do not decide that a mutant is invalid, and the label does not close diff --git a/packages/oracles/skills/oracles-evaluate-review/SKILL.md b/packages/oracles/skills/oracles-evaluate-review/SKILL.md index a258721..f6b9664 100644 --- a/packages/oracles/skills/oracles-evaluate-review/SKILL.md +++ b/packages/oracles/skills/oracles-evaluate-review/SKILL.md @@ -15,7 +15,7 @@ Use the [glossary](../../docs/glossary.md) terms in the ledger and the report. ## 1. Write the ledger first Before you test or edit anything, write an append-only ledger of the raw -review in a task-local scratch file. Do not commit it unless the user asks. +review in a task-local scratch file. Do not commit it unless the developer asks. - Give every item a stable ID in source order: each finding, footnote, capped item, non-blocking comment, and testing idea. @@ -38,7 +38,7 @@ for behavior an adopted law or decision covers, cited), `design-decision` kept). There is no `skip`. Deferral is rare. A confirmed, in-scope defect stays `confirmed-open` until it -is fixed, unless the user agrees to postpone it. A deferral records the open +is fixed, unless the developer agrees to postpone it. A deferral records the open work, why it cannot be done now, the remaining risk, an owner, and the condition for resuming. @@ -54,8 +54,8 @@ For each confirmed behavioral finding, decide which kind it is. - **A law change: present it and wait.** The finding questions what is promised: a new law, a narrower or revised law, a new checkpoint, an obligation on a collaborator, two laws in conflict, or a rule with no - authority. Give the user the candidate law and its rival, the evidence, - what each choice changes for users, and your recommendation. + authority. Give the developer the candidate law and its rival, the evidence, + what each choice changes for callers, and your recommendation. A finding can contain both. Fix the enforcement part, and present the law question. @@ -177,7 +177,7 @@ Lead with the law changes that need a decision. Then: - **Losses:** the totals by disposition, and every open evidence or coverage gap. -If the user asks whether the pull request is ready, give a separate verdict: +If the developer asks whether the pull request is ready, give a separate verdict: `ready`, `not ready` (a confirmed in-scope defect or an open decision blocks it), `readiness unproven` (a decisive witness is missing), or `not assessed`. A complete ledger and a green run do not make a pull request ready. diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md index a566b4d..afe0428 100644 --- a/packages/oracles/skills/oracles-review/SKILL.md +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -54,7 +54,7 @@ contains one is not ready. | **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | | **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | | **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | -| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the user. | +| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the developer. | | **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | | **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable, or that counts a gap as closed because a stated limit covers it. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | @@ -89,7 +89,7 @@ hunt is your main job. Check each definition against the state that the law needs. - **A late or coarse checkpoint.** The check runs only at the end of a history, or observes only a summary, so a wrong intermediate observation - disappears. Ask whether a user could see the wrong value before the + disappears. Ask whether a caller could see the wrong value before the checkpoint. - **A limit that hides in-scope work.** A stated limit excludes a legal case that the law promises. Check that each limit excludes exactly what its diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index ac5a77b..e306ed2 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -1,13 +1,13 @@ --- name: oracles-setup -description: Take an oracle to production with @tanstack/oracles. Pin the packages, add the lint preset and the Vitest config, write the AGENTS.md block, make the oracle skills findable by coding agents, and run the oracle through the gap check. Use when the engineer chooses to ship an oracle from a first session, when a repository adopts @tanstack/oracles, or when it upgrades its pinned commit. Its install and file-placement sections also cover the minimal setup that a first session needs. +description: Take an oracle to production with @tanstack/oracles. Pin the packages, add the lint preset and the Vitest config, write the AGENTS.md block, make the oracle skills findable by coding agents, and run the oracle through the gap check. Use when the developer chooses to ship an oracle from a first session, when a repository adopts @tanstack/oracles, or when it upgrades its pinned commit. Its install and file-placement sections also cover the minimal setup that a first session needs. --- # Take an oracle to production You are setting up oracle testing in this repository. Do the steps in order. -Ask the user before each step that writes outside the repository. When no -user is available, choose the default this skill gives, record each choice in +Ask the developer before each step that writes outside the repository. When no +developer is available, choose the default this skill gives, record each choice in your report, and continue. That default covers setup questions only. Potential laws have no default: propose them, report them, and stop. @@ -16,16 +16,16 @@ install section and its "where oracle files go" section, for the minimal setup that an oracle needs to run, and step 7's command, when its oracle passes and it runs the gap check. Taking an oracle to production uses all of it. That usually starts after a first session, once the -engineer has an oracle that found a bug and chooses to ship it. If there is +developer has an oracle that found a bug and chooses to ship it. If there is no oracle yet, load `oracles-start` first. Setup ends when one oracle passes on production and the gap check leaves no -gap open in that oracle's own contract, unless the user decides otherwise. A +gap open in that oracle's own contract, unless the developer decides otherwise. A stated limit does not close a gap. It does not end when the packages are installed. ## 1. Read the repository -Find out, and tell the user: +Find out, and tell the developer: - The package manager, and whether the repository is one package or a workspace. For a workspace, ask which package to start in. @@ -36,14 +36,14 @@ Find out, and tell the user: - The lint setup: ESLint 9 or later with a flat config, an older ESLint with `.eslintrc`, or no ESLint (for example Biome only). - The Node version that CI and the `engines` field use. The package needs - Node 20 or later. If CI runs an older Node, tell the user, and offer a + Node 20 or later. If CI runs an older Node, tell the developer, and offer a separate Node 20 job for the oracles. Do not change the existing jobs. With - no user present, do not change CI: report the mismatch. + no developer present, do not change CI: report the mismatch. - The repository's TypeScript version. Do not change it (step 2). - The package manager and its runner for package binaries: `pnpm exec`, `npx`, or `yarn`. Use them wherever this skill shows `pnpm add` or `pnpm exec`. -- Which coding agents the user works with. Look for `.claude/`, `.agents/`, +- Which coding agents the developer works with. Look for `.claude/`, `.agents/`, `.codex/`, `AGENTS.md`, and `CLAUDE.md`, and ask. ## 2. Install two pinned packages @@ -52,7 +52,7 @@ The packages are not on npm yet. Install both from pkg.pr.new, pinned to the same commit of TanStack/oracles. If you were not given one, use the current commit on main: the first 7 characters of the hash that `git ls-remote https://github.com/TanStack/oracles refs/heads/main` prints. -Tell the user which commit you pinned. +Tell the developer which commit you pinned. ```sh pnpm add -D \ @@ -63,7 +63,7 @@ pnpm add -D \ Add `fast-check` and `vitest` beside them. The framework needs Vitest 4.1 or later. If the repository already runs an older Vitest, upgrading it is the -user's decision: ask, and with no user present, do not upgrade. Report it as +developer's decision: ask, and with no developer present, do not upgrade. Report it as a blocker. For lint, also add `eslint@^9` if the repository has no ESLint, and `@typescript-eslint/parser@^8` if nothing parses TypeScript yet. Name those majors: a package manager can pick @@ -88,7 +88,7 @@ repository's TypeScript, and do not add a typecheck that uses it. Use the repository's own package manager. With pnpm 11, an install that skips a dependency's build script stops every later pnpm command (`ERR_PNPM_IGNORED_BUILDS`), and pnpm writes an `allowBuilds` placeholder into -`pnpm-workspace.yaml`. Answer it, usually `false`, and tell the user. +`pnpm-workspace.yaml`. Answer it, usually `false`, and tell the developer. A run that installs from a local tarball has no commit to pin or check. Say which tarball you installed, and note that `package.json` now points at a @@ -280,7 +280,7 @@ place, and its report says so. Check for both `AGENTS.md` and `CLAUDE.md`. Claude Code reads `CLAUDE.md`, and most other agents read `AGENTS.md`. If both exist, put the block in `AGENTS.md` and make sure `CLAUDE.md` refers to it (a line `@AGENTS.md` -imports it). If only `CLAUDE.md` exists, ask the user which file gets the +imports it). If only `CLAUDE.md` exists, ask the developer which file gets the block. If neither exists, create `AGENTS.md` and a `CLAUDE.md` that holds only `@AGENTS.md`, after asking. @@ -299,14 +299,14 @@ pointer. A pointer must read the copy that the chosen package installed: `/node_modules/@tanstack/oracles/skills//SKILL.md`, for the package chosen in step 1. Check that the file exists there. -Use that path in the `AGENTS.md` block (step 4) too. Offer the user these +Use that path in the `AGENTS.md` block (step 4) too. Offer the developer these choices. They can pick more than one. -| Choice | Writes | Use when | -| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -| **AGENTS.md only** | Step 4's block | The user's agents read `AGENTS.md`. This is always done. | -| **Repository pointers** | A pointer skill per skill in `.agents/skills/` and `.claude/skills/` | Everyone who clones the repository gets the skills. Commit them. | -| **Global pointers** | A pointer skill per skill in `~/.agents/skills/` and `~/.claude/skills/` (and any other skills directory the user's agents read) | The user wants the skills in every repository that uses `@tanstack/oracles`. | +| Choice | Writes | Use when | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | +| **AGENTS.md only** | Step 4's block | The developer's agents read `AGENTS.md`. This is always done. | +| **Repository pointers** | A pointer skill per skill in `.agents/skills/` and `.claude/skills/` | Everyone who clones the repository gets the skills. Commit them. | +| **Global pointers** | A pointer skill per skill in `~/.agents/skills/` and `~/.claude/skills/` (and any other skills directory the developer's agents read) | The developer wants the skills in every repository that uses `@tanstack/oracles`. | Write a pointer skill as `//SKILL.md` for each of `oracles-start`, `oracles-design`, `oracles-review`, @@ -314,7 +314,7 @@ Write a pointer skill as `//SKILL.md` for each of Before you write repository pointers, check that git does not ignore their directories (`git check-ignore .claude/skills .agents/skills`). If one is -ignored, tell the user: they can commit the pointers in the other +ignored, tell the developer: they can commit the pointers in the other directory, or add an exception to `.gitignore`. - Copy `name` and `description` exactly from the package's `SKILL.md`, so the @@ -346,7 +346,7 @@ installed version, and never breaks before install. Before writing a global pointer: -- Show the user the exact paths. +- Show the developer the exact paths. - If a skill with the same name exists, show it and ask. Never overwrite it silently. @@ -354,7 +354,7 @@ Before writing a global pointer: Usually the oracle already exists from the first session. Keep it, and continue with its gap check. If there is none, build it from the potential -laws that the engineer approved: load `oracles-design`, and follow it from +laws that the developer approved: load `oracles-design`, and follow it from its first step. In a first session, stop at the RED: present the bug, and leave the fix for @@ -363,11 +363,11 @@ production. A first session runs the gap check only when its oracle passes If the first oracle fails on unchanged production under an adopted law, it found a defect. The RED is the finding: report it with its replay line, and -tell the user. Never weaken the oracle to make it pass. The gap check cannot +tell the developer. Never weaken the oracle to make it pass. The gap check cannot judge a failing baseline yet, so make the smallest local fix on a scratch -copy or a local branch (never pushed or committed to the user's branch), run +copy or a local branch (never pushed or committed to the developer's branch), run the gap check there, and say in the report that it ran on the -patched code. With the user's approval, follow `oracles-design` step 7's +patched code. With the developer's approval, follow `oracles-design` step 7's defect path for the real fix. Put the oracle's `owns` list on its `oracleProperty` call. An entry is a file @@ -415,17 +415,17 @@ that instruction, and follow `oracles-design` for what a gap means. There are no equivalent mutants and no exits. A surviving mutant shows only that the current histories did not expose that code. If you believe the code is dead, first follow `oracles-design` step 8 to rule out a missing history, -observation, or law. Removing it is a production change, so tell the user, +observation, or law. Removing it is a production change, so tell the developer, and rerun the oracle and the gap check. If a gap can be neither killed nor removed, stop work on that gap and report its key and evidence to the -user. +developer. Now fill in the two placeholders in the `AGENTS.md` block: the oracle file glob and the gap check command you ran. ## 8. Report -Tell the user: +Tell the developer: - what was installed (both pinned URLs) and where configuration changed - which pointers were written, and where @@ -433,7 +433,7 @@ Tell the user: - the gap check's totals, and every gap left, grouped as: - open because the inputs that expose it lie outside the law's stated limits: name those inputs, and recommend widening the grammar and the - law. The user decides. Do not count these as closed. + law. The developer decides. Do not count these as closed. - allowed alternatives (`oracles-design` step 8), apart from the kills - framework or CLI limitations (for example a timeout from a synchronous infinite loop, or a guard that TypeScript narrowing makes constant) @@ -443,4 +443,4 @@ Tell the user: - any defect the oracle found, with its witness Do not call setup done while a gap is open in the oracle's own contract, -unless the user decides otherwise. +unless the developer decides otherwise. diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index cad8acd..547fb0d 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -1,13 +1,13 @@ --- name: oracles-start -description: Run a first session with @tanstack/oracles in an engineer's codebase. Say what the session will do, look around a part of the code they name or the whole repository, probe the real API to prove each finding, propose a potential oracle and its potential laws, then build it and run it until it finds a bug and the engineer has settled some design questions. Present the work, and explain the path to production. Use when someone is trying @tanstack/oracles, or asks where an oracle would help. +description: Run a first session with @tanstack/oracles in a developer's codebase. Say what the session will do, look around a part of the code they name or the whole repository, probe the real API to prove each finding, propose a potential oracle and its potential laws, then build it and run it until it finds a bug and the developer has settled some design questions. Present the work, and explain the path to production. Use when someone is trying @tanstack/oracles, or asks where an oracle would help. --- # A first session with oracles ## Your goal -Be an engaging teacher. The engineer is trying this tool, often in the mood +Be an engaging teacher. The developer is trying this tool, often in the mood of "impress me". Show, don't tell. Bring up interesting findings about their own code, and explain what you are doing and why as you go. Tie each explanation to something you just showed them: a probe, a finding, a law, or @@ -17,7 +17,7 @@ the oracle file. Do not explain what oracles are unless they ask. Work toward these milestones, in order: findings from probes of their real code; a potential oracle whose laws they approve; an oracle that runs on their code; and that oracle finding a bug, or the gap check finding what no -law catches, while they settle some design questions. The engineer decides +law catches, while they settle some design questions. The developer decides when the session ends. Probes are how you learn what reading the code does not tell you: its @@ -25,33 +25,33 @@ dynamics, boundary conditions, and constraints. You need that understanding to build the oracle. Bugs and inconsistencies that the probes find along the way are a welcome side effect. They also keep you honest: an agent that only says things is often wrong, and one wrong claim costs the tool its -credibility. So every finding needs a probe that the engineer can run again. +credibility. So every finding needs a probe that the developer can run again. ## How to talk - Use these terms, and use them every time: a **probe**, a **finding**, a **potential oracle** that owns a public behavior, its **potential laws**, - and the **oracle file**. Once the engineer approves a potential law, it is + and the **oracle file**. Once the developer approves a potential law, it is a **law**; once the oracle runs, it is the **oracle**. - Explain a framework term in a line the first time you use it (a calibration is a wrong answer planted in a run to show that the check rejects it), or leave the term out. - Talk naturally. Never name a skill, a step, or a section ("step 3", "the skill's fallback"). Say what you are doing, in your own words. -- Ask in plain words. Explain any term the engineer may not know in one +- Ask in plain words. Explain any term the developer may not know in one line (for example, a lone surrogate is half of a character that takes two UTF-16 units). - Ask one clear question when you need a decision. For a possible bug, ask "Is this a bug, or is it intended?" When several decisions are waiting, end with a short numbered list of them, most important first, so the - engineer can answer by number. -- Show the engineer only what they decide on. Your private notes, their + developer can answer by number. +- Show the developer only what they decide on. Your private notes, their labels, and their statuses ("adopted", "open") stay private. Until the - engineer approves it, a potential law is only a potential law. + developer approves it, a potential law is only a potential law. ## Where you stop -Stop and wait only when the engineer must decide: +Stop and wait only when the developer must decide: - which part of the code to look at - whether a finding is a bug, or intended @@ -64,7 +64,7 @@ Between those points, keep working, and keep telling them what you are doing. ## What you change -Until the engineer chooses to take an oracle to production, change only +Until the developer chooses to take an oracle to production, change only these: - the minimal installs that an oracle needs to run: `@tanstack/oracles`, @@ -73,12 +73,25 @@ these: repository's own config does not - oracle files, with their prose - the gap check's working directory (`.oracle-gaps/`), when you run it. - Tell the engineer it is safe to delete, and that production adds it to + Tell the developer it is safe to delete, and that production adds it to `.gitignore`. Findings, potential oracles, and potential laws stay in the chat. Keep probes in a scratch directory outside the repository. Change production code -only when the engineer asks you to. +only when the developer asks you to. + +## A feedback log, while this is a preview + +`@tanstack/oracles` is in preview, and its maintainers want to know where +this session was rough. Keep a log at +`~/oracles-feedback/-.md`, outside the repository. +Append a short entry whenever a skill or doc was unclear, wrong, or +contradicted another; whenever you had to guess; whenever a command, +install, or config step failed or needed a workaround; and whenever +something took much longer than it should have. Each entry gives the time, +what you were doing, what happened, the skill file and section, and what +would have made it smooth. Keep entries short, and keep working: the log +is for later, not a reason to stop. ## 1. Open @@ -102,7 +115,7 @@ names no part: ask. ## 2. Look around -If the engineer named a part of the code, look closely there. Otherwise, +If the developer named a part of the code, look closely there. Otherwise, look around the repository, and make it quick. Read the public API and its docs (the README, the API docs, the JSDoc), the existing tests, and the git history of fixes. The files that fixes change most often tell you more than @@ -122,7 +135,7 @@ to state than the code that produces it. Good signs: - code that git history shows was fixed more than once Avoid an internal helper whose only law would restate its code. Avoid -behavior that nothing documents, unless the engineer wants to decide it. +behavior that nothing documents, unless the developer wants to decide it. Tell them about two or three spots, a short paragraph each: what the code seems to promise, where that is stated, and why it would make a good oracle. @@ -135,7 +148,7 @@ They can redirect you. Read the spot's docs, code, tests, and fix history. Then ask yourself: what do I not know about this code that I need to know to build its oracle? How does the state change with each call? Where are the boundaries? What holds -whatever happens? Show the engineer those questions in the chat, a short +whatever happens? Show the developer those questions in the chat, a short list, before you probe: it shows them how an oracle gets built. Then build a probe for each one, and answer each question with what its probe showed. @@ -171,12 +184,12 @@ line. The authority is what makes a finding credible, and it becomes the documentation in the oracle file. When nothing decides a behavior, it is a design question, not a bug. -The engineer's answer to a design question is an authority too. Once they +The developer's answer to a design question is an authority too. Once they decide it, it becomes a potential law, and the oracle can find where the code breaks it, just as with a law from the docs. In a repository with few docs, this is the usual way to a bug: ask the question, and let the oracle check the answer. The oracle file records the decision with the -engineer's name, their GitHub login, and the date, so that their team knows +developer's name, their GitHub login, and the date, so that their team knows whom to ask. Find the login with `gh api user --jq .login`. If that fails, ask for it together with the first decision. @@ -186,19 +199,19 @@ called a possible bug or a question, never a certainty: 1. A title that names the behavior: "Object Map key: replay throws". 2. The smallest case, expected against observed, as a short snippet the - engineer can run. + developer can run. 3. **Why I think it's wrong:** the authority, quoted, with its file and line. 4. **Why the tests miss it:** what the existing tests check instead. 5. **How often:** how many of the probe's inputs failed, out of how many. Give each finding its kind: -| Kind | What it means | -| --------------- | ----------------------------------------------------------------------- | -| Possible bug | The code breaks a promise that a document or a test states. | -| Inconsistency | Two sources disagree: say which ones, and what each says. | -| Test gap | A probe shows behavior that no existing test checks. | -| Design question | Nothing decides the behavior: give two readings and what each predicts. | +| Kind | What it means | +| ----------------- | ----------------------------------------------------------------------- | +| Possible bug | The code breaks a promise that a document or a test states. | +| Inconsistency | Two sources disagree: say which ones, and what each says. | +| Untested behavior | A probe shows behavior that no existing test checks. | +| Design question | Nothing decides the behavior: give two readings and what each predicts. | For example: @@ -233,7 +246,7 @@ own instead, and recommend one. - What it would own, in one or two sentences. - The contract, as a few lines of public calls and their results. Always - include it: it is the quickest way for the engineer to see what you mean. + include it: it is the quickest way for the developer to see what you mean. ```ts klona(x) // → a deep copy: same kind, prototype, and values as x, sharing no object with it @@ -252,7 +265,7 @@ own instead, and recommend one. - The open design questions, each with its two readings, and what the probe showed. Say that these can wait: the oracle's results will help decide them. List them apart from the potential laws, and do not number them as - laws. Until the engineer decides one, the oracle file states it as a + laws. Until the developer decides one, the oracle file states it as a limit. **Work out each law privately before you propose it.** Fill in this @@ -306,12 +319,12 @@ How to fill in the fields: the grammar. - **Backing:** `stated` if an authority states it, `observed` if production does it but nothing states it, `open` if it is a question. A `stated` law - is what you ask the engineer to approve. It becomes adopted only when they + is what you ask the developer to approve. It becomes adopted only when they do, and only adopted laws become expected results. An `observed` or `open` - law goes to the engineer as a question. + law goes to the developer as a question. - **Production output is evidence, never authority.** A rule that you write only because production does it is model hacking. Mark that rule - `observed`, and ask the user. An analogy to another documented rule is not + `observed`, and ask the developer. An analogy to another documented rule is not authority either: mark that law `open`. Settle these privately too: @@ -334,14 +347,17 @@ trigger list in `oracles-design` step 1. When a trigger for the [design grammar](../../docs/instruments/design-grammar.md) card fires (for example, more than five laws, or a law with no new case that it predicts), work through the card. Then show -the engineer the design it produced, in a few plain lines: what the model +the developer the design it produced, in a few plain lines: what the model keeps, how the laws share it, and any choice between two designs, with your recommendation. Leave out the card's own terms (preservation relation, primitives, ablation). Ask them to approve it with the laws. -Then ask the engineer to approve the potential laws that the authority -decides. An approved potential law is an adopted law. Potential laws have no -default: with no engineer present, propose them and stop. +Then ask the developer to approve the potential laws that the authority +decides. When the developer approves a potential law, it becomes an adopted +law, and the oracle file records their decision. If they do not maintain the +repository, its maintainers confirm the laws when they review the pull +request. Potential laws have no default: with no developer present, propose +them and stop. ## 5. Build and run @@ -358,14 +374,14 @@ default: with no engineer present, propose them and stop. 2. **Write the oracle file.** Follow `oracles-design` from its first step: the law block's prose, then the model, the histories, the driver, and the - check. As you build, tell the engineer briefly what each part is for: the + check. As you build, tell the developer briefly what each part is for: the model computes the expected answer the simple way, the histories are the generated inputs and sequences, the driver calls their real code, and the check compares the two at each step. -3. **Run it on the unchanged code.** A failure under an approved law is a +3. **Run it on the unchanged code.** A failure under an adopted law is a possible bug. Present it like a probe finding, with the oracle's smallest failing case as the snippet. -4. **As the engineer settles questions, change the law** and rewrite the law +4. **As the developer settles questions, change the law** and rewrite the law block's prose in the same step. Before you show results, check that the prose still says what the code does. @@ -380,23 +396,25 @@ design question, or a law nobody has stated yet. Take the command and its flags from `oracles-setup` step 7, with its `--out` in the package's `.oracle-gaps/` directory, as that step says. Read each surviving change, probe it when it is unclear, and present what it shows as a finding: a possible -bug, or a design question for the engineer. Do not try to close every gap: +bug, or a design question for the developer. Do not try to close every gap: that is production work. -If the gap check finds nothing worth the engineer's time either, offer to +If the gap check finds nothing worth the developer's time either, offer to build the next oracle at another spot. ## 6. Present -When the oracle or the gap check has found something and the engineer has +When the oracle or the gap check has found something and the developer has settled some design questions, stop and present. Present the same way -whenever the engineer wants to stop: +whenever the developer wants to stop: - what you built, and what it found - an offer to walk through how it works: how the reference model was built, how the histories are generated, and how the driver calls their code. - Engineers often want this, and it is the best part to teach. + Developers often want this, and it is the best part to teach. - an invitation to ask questions +- the path to the feedback log, and a request to send it to Kyle Mathews + (GitHub KyleAMathews) when they are done When they take the walk-through, follow one bug the oracle found through the file (with no bug, one change the gap check showed, or one history that diff --git a/packages/oracles/skills/oracles-start/references/problems.md b/packages/oracles/skills/oracles-start/references/problems.md index 0447b63..41be290 100644 --- a/packages/oracles/skills/oracles-start/references/problems.md +++ b/packages/oracles/skills/oracles-start/references/problems.md @@ -1,16 +1,16 @@ # Problems you may hit in a first session Open this file when one of these happens. Each entry says what to do, and -what to tell the engineer. +what to tell the developer. ## Installs **The repository uses another test runner** (mocha, uvu, Jest, ava, `node:test`). Add Vitest for the oracles only, with a config whose `include` is only the oracle files. Their own suite and runner stay as they are. Tell -the engineer this in one line as you install. +the developer this in one line as you install. -**The repository runs a Vitest older than 4.1.** This is the engineer's +**The repository runs a Vitest older than 4.1.** This is the developer's decision. Tell them that the oracle needs Vitest 4.1 or later, and that upgrading changes the runner for their own suite too. Offer to upgrade and then run their suite, to show whether it still passes. Do not upgrade until @@ -42,6 +42,6 @@ the build comes from. **Node cannot load the source as it is** (for example, an extensionless ES module). Copy it unchanged into the scratch directory, under a name that -Node can load, such as `index.mjs`, and tell the engineer that you did. +Node can load, such as `index.mjs`, and tell the developer that you did. **The code is TypeScript.** Run the probe with `npx tsx`. From 1a9e80ef545947a90da977900ae97f445a686427 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:34:32 -0700 Subject: [PATCH 65/76] docs(oracles-start): keep the preview feedback log in the repository, not the home directory Co-authored-by: Isaac --- packages/oracles/README.md | 6 +++--- packages/oracles/skills/oracles-start/SKILL.md | 5 +++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/packages/oracles/README.md b/packages/oracles/README.md index b2b699d..0c17871 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -54,9 +54,9 @@ Until you choose to ship it, the agent changes nothing in your repository except the minimal installs, the oracle file, and the gap check's working directory, if it runs. -While the package is in preview, the agent also keeps a feedback log at -`~/oracles-feedback/-.md`, outside your repository. When -you're done, please send it to Kyle Mathews (GitHub KyleAMathews). +While the package is in preview, the agent also keeps a feedback log, +`oracles-feedback.md`, at the root of your repository. Don't commit it. +When you're done, please send it to Kyle Mathews (GitHub KyleAMathews). The package needs Node 20 or later. The first session installs Vitest 4.1 and fast-check 4 if the repository does not have them. diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 547fb0d..5cb23ae 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -72,6 +72,7 @@ these: - a minimal Vitest config that picks up the oracle file, only when the repository's own config does not - oracle files, with their prose +- the feedback log, `oracles-feedback.md`, while this is a preview - the gap check's working directory (`.oracle-gaps/`), when you run it. Tell the developer it is safe to delete, and that production adds it to `.gitignore`. @@ -83,8 +84,8 @@ only when the developer asks you to. ## A feedback log, while this is a preview `@tanstack/oracles` is in preview, and its maintainers want to know where -this session was rough. Keep a log at -`~/oracles-feedback/-.md`, outside the repository. +this session was rough. Keep a log in `oracles-feedback.md` at the root of +the repository. Do not commit it. Append a short entry whenever a skill or doc was unclear, wrong, or contradicted another; whenever you had to guess; whenever a command, install, or config step failed or needed a workaround; and whenever From 835ec4bc9067422d430f7293c7cf9251a936e1c5 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:36:42 -0700 Subject: [PATCH 66/76] docs(oracles-start): link to the AGENTS.md block in oracles-setup step 4 Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 5cb23ae..2e7d41c 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -96,8 +96,9 @@ is for later, not a reason to stop. ## 1. Open -Before your first message, read the block in `oracles-setup` step 4 that -production adds to `AGENTS.md`. It governs you from the start of this +Before your first message, read the block that production adds to +`AGENTS.md`: it is in [`oracles-setup` step 4](../oracles-setup/SKILL.md), +under "Write the AGENTS.md block". It governs you from the start of this session, though nobody has added it to the repository yet: who owns the laws and who owns the code, which skill to read before each kind of oracle work, and the lint rule. From 7d086f37c087389e499bf412f9088c43e075ed56 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:37:32 -0700 Subject: [PATCH 67/76] docs(oracles-start): when to ask for the GitHub login; create the feedback log at the start Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 2e7d41c..50df4fa 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -84,8 +84,8 @@ only when the developer asks you to. ## A feedback log, while this is a preview `@tanstack/oracles` is in preview, and its maintainers want to know where -this session was rough. Keep a log in `oracles-feedback.md` at the root of -the repository. Do not commit it. +this session was rough. Create `oracles-feedback.md` at the root of the +repository when the session starts, and do not commit it. Append a short entry whenever a skill or doc was unclear, wrong, or contradicted another; whenever you had to guess; whenever a command, install, or config step failed or needed a workaround; and whenever @@ -194,7 +194,8 @@ check the answer. The oracle file records the decision with the developer's name, their GitHub login, and the date, so that their team knows whom to ask. Find the login with `gh api user --jq .login`. If that fails, ask for it -together with the first decision. +with the first finding or design question you bring them, not with the +question of where to look. **Present each finding** in this shape, a light touch, with each finding called a possible bug or a question, never a certainty: From 48ca60cdfee04552f4fa15e64f8e85bba57afa1b Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:40:38 -0700 Subject: [PATCH 68/76] docs(oracles): examples that do not prime a validation repository; the repo's module-type warning is harmless The contract and law-block examples used klona, a library the skills are validated on; an agent noted that the example could prime its findings. They now use a fictional slugify. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 11 +++++------ packages/oracles/skills/oracles-start/SKILL.md | 3 ++- .../skills/oracles-start/references/problems.md | 6 ++++-- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index f3bd2b6..2e835d8 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -89,9 +89,8 @@ reader. checkpoints where the promise is owed, and the owned code. A reader who stops here knows what the file is for. Name the promise itself. "The stylesheet matches the options the README documents" names no promise. For - example: "This oracle owns `klona`'s deep-copy promise for the default - mode, checked on the value that `klona(x)` returns. It owns - `src/index.js#klona`." + example: "This oracle owns `slugify`'s promise for titles, checked on the + string that `slugify(title)` returns. It owns `src/slugify.ts#slugify`." 2. **The contract at a glance:** a short code block, right after those sentences, that shows the public API promises this oracle maintains. A @@ -112,7 +111,7 @@ reader. answer. For example: ```ts - klona(x) // → a deep copy: same kind, prototype, and values as x, sharing no object with it + slugify('Hello, World!') // → 'hello-world': lowercase words joined by '-' typography({ scaleRatio: 2, baseFontSize: '16px' }) // → h1 font size 32px (ratio × base) typography({ scaleRatio: 2, baseFontSize: '15.5px' }) // → h1 font size 31px (open) ``` @@ -153,8 +152,8 @@ It never mentions a review or an open question: a point the developer has not decided appears only as a limit, with its reason. Nor does it tell the story of the session: no "the maintainer approved", "the developer deferred", or "as the probe showed". A developer's decision is stated as the rule it made, and -a limit's reason is about the code ("`klona` calls the constructor with no -arguments"), not about who has not decided yet. When a developer's decision is +a limit's reason is about the code ("`slugify` drops every character +outside ASCII"), not about who has not decided yet. When a developer's decision is the authority for a law, write it as "Decision by (GitHub ), : ", the same way in the `law` option's `authority`. On a team, a reader needs to know whom to ask. Take the login diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 50df4fa..af371c3 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -252,7 +252,8 @@ own instead, and recommend one. include it: it is the quickest way for the developer to see what you mean. ```ts - klona(x) // → a deep copy: same kind, prototype, and values as x, sharing no object with it + slugify('Hello, World!') // → 'hello-world': lowercase words joined by '-' + slugify(slugify(title)) // → slugify(title) ``` - Each potential law, as one "For every …" sentence, with its authority diff --git a/packages/oracles/skills/oracles-start/references/problems.md b/packages/oracles/skills/oracles-start/references/problems.md index 41be290..5d1affd 100644 --- a/packages/oracles/skills/oracles-start/references/problems.md +++ b/packages/oracles/skills/oracles-start/references/problems.md @@ -37,8 +37,10 @@ the file, as `oracles-design` says. **The tests import a build that is not built yet.** Import the source that the build comes from. -**Node warns about the module type.** Give the scratch directory a -`package.json` with `"type": "module"`. +**Node warns about the module type.** If the warning names the scratch +directory, give it a `package.json` with `"type": "module"`. If it names the +repository's `package.json`, the warning is harmless: run the probe with +`node --no-warnings`, and do not change their `package.json`. **Node cannot load the source as it is** (for example, an extensionless ES module). Copy it unchanged into the scratch directory, under a name that From 3512d0cdbb0209290af981635d9be10437665668 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:44:34 -0700 Subject: [PATCH 69/76] docs(oracles-start): before production the lint rule means the repo's lint; no gap check on a failing oracle Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index af371c3..8636060 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -101,7 +101,9 @@ Before your first message, read the block that production adds to under "Write the AGENTS.md block". It governs you from the start of this session, though nobody has added it to the repository yet: who owns the laws and who owns the code, which skill to read before each kind of oracle -work, and the lint rule. +work, and the lint rule. Until production, the lint rule means the +repository's own lint: the oracles lint plugin comes with production, and +its absence is not a problem to report. Your first message starts with two or three sentences on what this session will do, before any question. For example: "I'll @@ -390,7 +392,8 @@ them and stop. prose still says what the code does. Do not change production code. That belongs to taking the oracle to -production. +production. Do not offer the gap check while the oracle fails: it cannot +judge a failing run. **If the oracle finds no bug, run the gap check to go deeper.** Say so plainly first, and show what held. The gap check makes small changes to a From 6001ec1656378624a1d512cd713844cea48ffee1 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:46:06 -0700 Subject: [PATCH 70/76] docs(oracles-design): lint in a first session is the repository's own Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 2e835d8..50117b1 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -446,6 +446,8 @@ The options of `oracleProperty` are typed in **Lint as you go.** Run the repository's lint on each oracle file and each changed production file after you edit it, and again before you finish. Fix every report. Do not disable a rule. +In a first session, the oracles lint plugin is not installed yet: run the +repository's own lint, if it covers the oracle file, and go on. Techniques: [guide § The driver and the check](../../docs/guide.md#the-driver-and-the-check). From 0c2b7d78818dd15aa52f569177f508e41b78288d Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:51:06 -0700 Subject: [PATCH 71/76] docs(oracles-design): B6 does not forbid the check of a no-throw law Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index 50117b1..a12903e 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -271,6 +271,10 @@ choose it later. B1, B3, B4, and B7 are review checks until lint ships them. +B6 is about a check that stands in for a richer law. When the adopted law is +itself "returns without throwing on valid input", its check may assert only +that, and it needs its own calibration. + Techniques: [guide § The model](../../docs/guide.md#the-model). ## 4. Account for every distinction From 68856a3e8f3a42e01e87b042fa97220969584872 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 14:56:36 -0700 Subject: [PATCH 72/76] docs(oracles): always build the strongest model the laws need; never offer a weaker one as a choice A first session asked the developer 'Full model, or checks only?'. The term meant nothing to them, and the answer should always be the strongest model. How the oracle is built is the agent's decision. Co-authored-by: Isaac --- packages/oracles/skills/oracles-design/SKILL.md | 4 ++++ packages/oracles/skills/oracles-start/SKILL.md | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md index a12903e..5019a24 100644 --- a/packages/oracles/skills/oracles-design/SKILL.md +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -214,6 +214,10 @@ For the design grammar card, see the triggers in step 1. ## 3. Choose the model +Build the strongest model the approved laws need: one that predicts every +observation the laws make a claim about. The choice of model is yours, so do +not ask the developer to pick between a complete model and partial checks. + Pick by the shape of the law, in this order, and use the first style that can state it: diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 8636060..4ae1f4c 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -345,6 +345,13 @@ Settle these privately too: law, a history dimension, or a constructed history. Keep a pointer to the probe that showed it. +**Always build the strongest oracle the laws need.** Its model predicts every +observation that the approved laws make a claim about. How the oracle is +built is your decision, not the developer's: never offer them a weaker +design, such as checks without a model, a partial model, or fewer +observations, as a choice. If a law truly cannot be modeled yet, say which +part, and state it as a limit. + The law must say more than the code does. If a model of it would only repeat the implementation, look for the public promise that the code serves. From 0aff4ddf89651d47af8c94eadc29f8d38efd668d Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 15:03:17 -0700 Subject: [PATCH 73/76] docs(oracles): the package README is one sentence and the prompt While only invited people try it, the README holds a one-sentence overview and the prompt. The session itself explains the rest. Co-authored-by: Isaac --- README.md | 8 +- packages/oracles/README.md | 191 +------------------------------------ 2 files changed, 5 insertions(+), 194 deletions(-) diff --git a/README.md b/README.md index 10e929c..3f83381 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,9 @@ every commit with [pkg.pr.new](https://pkg.pr.new). ## Packages -To try it, paste the prompt in [Try it](packages/oracles/README.md#try-it) -into your coding agent. To adopt it, start with -[`packages/oracles/README.md`](packages/oracles/README.md): -the package ships setup, design, and review skills for coding agents, the -oracle checklist, and a glossary. +To try it, paste the prompt in +[`packages/oracles/README.md`](packages/oracles/README.md) into your coding +agent. - `packages/oracles` (`@tanstack/oracles`): the oracle framework, its skills and docs, and the `oracle-gaps` CLI. The CLI lists the production behavior that a diff --git a/packages/oracles/README.md b/packages/oracles/README.md index 0c17871..f2c0282 100644 --- a/packages/oracles/README.md +++ b/packages/oracles/README.md @@ -1,19 +1,7 @@ # @tanstack/oracles -Oracle testing for TypeScript on Vitest and fast-check: - -1. State a law. -2. Generate the histories where the law matters. -3. Check production against an independent model. - -The gap check then lists the owned production behavior that no law judges. - -Not on npm yet. Install a commit from pkg.pr.new (see [Try it](#try-it)). - -## Try it - -Paste this prompt into your coding agent at the root of the repository you -want to test. Any agent that can run shell commands works. +Oracle testing for TypeScript: an agent states what your code promises, then +checks your real code against an independent model of those promises. ```text Show me what oracle testing with @tanstack/oracles can find in this repository. @@ -26,178 +14,3 @@ Show me what oracle testing with @tanstack/oracles can find in this repository. 3. Read node_modules/@tanstack/oracles/skills/oracles-start/SKILL.md in full, and follow it. Use wherever a skill asks for the commit to pin. ``` - -What happens next: - -1. The agent says what the session will do, and asks whether you have a - part of the code in mind. -2. It looks around, and suggests two or three places where an oracle would - help. You pick one. -3. It tries that code with probes: small scripts that call your real API, - to learn how the code behaves at its edges and over a sequence of calls. - That is what the oracle needs. Each finding comes with a snippet you can rerun, and the doc or test that - the behavior seems to break: possible bugs, places where the docs and the - code disagree, behavior no test checks, and design questions nobody has - decided. -4. It proposes a potential oracle for that behavior, with its potential laws. - You approve the laws. This is the step that needs you: you decide what the - code promises. -5. It writes the oracle file and runs it on your unchanged code, telling you - what each part does as it goes, until the oracle has found a bug and you - have settled some design questions. If the oracle finds no bug, the agent - runs the gap check, which shows production changes that no law catches, - to dig deeper for a bug or a design question. -6. It shows you what it built, offers to walk you through how it works, and - explains how to take the oracle to production. - -Until you choose to ship it, the agent changes nothing in your repository -except the minimal installs, the oracle file, and the gap check's working -directory, if it runs. - -While the package is in preview, the agent also keeps a feedback log, -`oracles-feedback.md`, at the root of your repository. Don't commit it. -When you're done, please send it to Kyle Mathews (GitHub KyleAMathews). - -The package needs Node 20 or later. The first session installs Vitest 4.1 -and fast-check 4 if the repository does not have them. - -## What ships in this package - -| Path | What it is | -| --------------------------------- | -------------------------------------------------------------------------------------------------- | -| `skills/oracles-start/` | A first session: a look around, probes, a potential oracle, and the oracle built and run. | -| `skills/oracles-setup/` | Taking an oracle to production: setup, the gap check, upgrades, making the skills findable. | -| `skills/oracles-design/` | How to design, write, or change an oracle: from approved laws to model, histories, and gaps. | -| `skills/oracles-evaluate-review/` | How to act on a review: a lossless ledger, RED on unchanged production, and closing the bug class. | -| `skills/oracles-review/` | How to review oracle work. | -| `docs/intro.md` | What an oracle is, for when someone asks. | -| `docs/oracle-checklist.md` | The complete requirements, by ID (ORC-001 to ORC-016). | -| `docs/guide.md` | Techniques for each design and review step. | -| `docs/model-styles.md` | One small example of each model style. | -| `docs/glossary.md` | The oracle terms. | -| `docs/instruments/` | Short reasoning cards that the skills route to. | -| `oracle-gaps/` | The gap check (`oracle-gaps` binary). | - -## An oracle - -One `oracleProperty` call holds the oracle's parts. - -```ts -import fc from 'fast-check' -import { differs, oracleProperty, pass } from '@tanstack/oracles' -import { clamp } from '../src/clamp' - -type History = { n: number; max: number } - -// The type is number, so the domain includes -0. -const histories = fc.record({ - n: fc.oneof(fc.integer({ min: -100, max: 100 }), fc.constant(-0)), - max: fc.integer({ min: 0, max: 50 }), -}) - -oracleProperty({ - id: 'clamp-bound', - owns: ['src/clamp.ts'], - law: { - statement: - 'clamp(n, max) returns max when n is above max, and n otherwise.', - authority: 'README, "clamp"', - limits: - 'Integers in [-100, 100] and -0 for n, and [0, 50] for max. No NaN or infinities.', - }, - histories: { - arbitrary: histories, - runs: 100, - fixedSeeds: [1], - states: ['above max', 'at max', 'negative zero at max'], - measured: ['below max'], - constructed: [ - { - name: 'n above max', - enters: ['above max'], - build: ({ max }) => ({ n: max + 1, max }), - }, - { - name: 'n at max', - enters: ['at max'], - build: ({ max }) => ({ n: max, max }), - }, - { - // -0 is not above 0, so the result is n (-0), not max (0). - name: '-0 at max 0', - enters: ['negative zero at max'], - build: () => ({ n: -0, max: 0 }), - }, - ], - }, - // The law restated without production. - model: (plan: History) => ({ result: Math.min(plan.n, plan.max) }), - // The real entry point; the driver records and decides nothing. - driver: (plan, observe) => observe('result', clamp(plan.n, plan.max)), - check: (_plan, observations, expected) => - differs( - 'result', - 'clamp bound', - observations.find((o) => o.checkpoint === 'result')?.value, - expected.result, - ) ?? pass, - reached: (plan) => [ - Object.is(plan.n, -0) && plan.max === 0 - ? 'negative zero at max' - : plan.n > plan.max - ? 'above max' - : plan.n === plan.max - ? 'at max' - : 'below max', - ], - calibrations: [ - { - name: 'returns n unclamped', - law: 'clamp bound', - corrupt: (trace, plan) => - plan.n > plan.max - ? trace.map((o) => - o.checkpoint === 'result' ? { ...o, value: plan.n } : o, - ) - : undefined, - }, - ], -}) -``` - -A generated property runs one test per fixed seed and one random test. It -also runs every constructed history, a determinism test, and each -calibration. Each run reports how many histories entered each state. - -## API - -| Export | Purpose | -| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `oracleProperty(options)` | Registers an oracle. Options: `id`, `owns`, `law` (`statement`, `authority`, `limits`), `histories` (an enumerated list, or `{ arbitrary, runs, fixedSeeds, states, measured, constructed, fixed }`), `resolve`, `model`, `driver`, `check`, `reached`, `predict`, `calibrations`, `label`, `timeout`. | -| `pass`, `violation(...)`, `differs(checkpoint, law, actual, expected)` | The check's verdicts. `differs` returns a violation with a diff, or `undefined`, so comparisons chain with `??`. | -| `forwardCut(...)` | Checks an observation against an ordered list of allowed states that may be skipped but never revisited. | -| `useOracleClock(...)` | Runs the rest of a driver on virtual timers and `Date`, with a bound on fired timers (law `bounded clock`). | -| `scheduleFrom(...)`, `withinTurns(...)` | Controlled scheduling helpers for drivers. | -| `configureOracles({ envPrefix, registry, retiredVariables })` | Optional. Changes the environment-variable prefix or limits the property ids. Call it before any oracle module is imported. | -| `OracleViolation`, `OracleModelError`, `OracleDataError` | The framework's failure types. | - -Observations and enumerated histories must be data: primitives, plain -arrays and objects, Maps, Sets, and Errors. A driver converts anything else -before it records it. - -`@tanstack/oracles/declaration` is an entry point for browser or native -modules. It loads none of the main entry's Node built-ins. - -## The gap check - -```sh -pnpm exec oracle-gaps --package --source -``` - -The gap check mutates the owned source and runs the oracles against each -mutant. Its report lists every gap with one -instruction. The other options and the report format are still changing. -The skills describe how to work with the report. - -A gap closes only when a law rejects the mutant or the code is removed. There -are no equivalent mutants. From bc4544ca77773533d5f3610da995d6daf1eecf87 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 15:41:51 -0700 Subject: [PATCH 74/76] docs(oracles-start): talk about the developer's code in its own terms, not the skill's From manual review: an opening that copied the skill's example ('a behavior that's easy to break and has a simple rule behind it') sounded nonsensical, and a probe plan listed the skill's categories ('dynamics, constraints, boundaries, timing, and lifecycle') instead of questions about the code. Co-authored-by: Isaac --- .../oracles/skills/oracles-start/SKILL.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 4ae1f4c..2b6e3c8 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -38,6 +38,10 @@ credibility. So every finding needs a probe that the developer can run again. rejects it), or leave the term out. - Talk naturally. Never name a skill, a step, or a section ("step 3", "the skill's fallback"). Say what you are doing, in your own words. +- Talk about their code in its own terms, not in this skill's words. Words + such as "dynamics", "constraints", "boundaries", "a behavior that's easy to + break", or "a simple rule behind it" are for you. The developer should hear + about scrolling, retries, cache entries, or whatever their code does. - Ask in plain words. Explain any term the developer may not know in one line (for example, a lone surrogate is half of a character that takes two UTF-16 units). @@ -106,10 +110,13 @@ repository's own lint: the oracles lint plugin comes with production, and its absence is not a problem to report. Your first message starts with two or three sentences on what this session -will do, before any question. For example: "I'll -look around your code for a behavior that's easy to break, try it with small -scripts that call your real API, and show you what I find. Then I'll write -an oracle for it with you and run it." +will do, before any question. Write them for this repository, in plain words: +name what the library does, and what you will try. For a scroll-aware header +component, for example: "I'll read how Headroom decides when to hide and +show the header, then mount it in a fake browser and scroll it around to see +whether it does what the docs say. Anything odd, I'll show you with a script +you can rerun. Then we'll write an oracle for the rules we agree on, and run +it." Do not reuse that wording. Then ask one question: is there a part of the code they want to try this on, or should you look around? Stop for the answer. If their first message @@ -153,7 +160,9 @@ Read the spot's docs, code, tests, and fix history. Then ask yourself: what do I not know about this code that I need to know to build its oracle? How does the state change with each call? Where are the boundaries? What holds whatever happens? Show the developer those questions in the chat, a short -list, before you probe: it shows them how an oracle gets built. Then build a +list, before you probe: it shows them how an oracle gets built. Write each +question about their code ("If you scroll up by one pixel, does the header +come back?"), not as a category ("timing", "lifecycle"). Then build a probe for each one, and answer each question with what its probe showed. **A probe** is a small script that calls the real public API with chosen From 2179c2f925d96a63b22fa320b2442317838f4882 Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 15:42:27 -0700 Subject: [PATCH 75/76] docs(oracles-start): examples from a fictional retry helper, not a repository under test Co-authored-by: Isaac --- packages/oracles/skills/oracles-start/SKILL.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 2b6e3c8..2bc139f 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -41,7 +41,7 @@ credibility. So every finding needs a probe that the developer can run again. - Talk about their code in its own terms, not in this skill's words. Words such as "dynamics", "constraints", "boundaries", "a behavior that's easy to break", or "a simple rule behind it" are for you. The developer should hear - about scrolling, retries, cache entries, or whatever their code does. + about timeouts, retries, cache entries, or whatever their code does. - Ask in plain words. Explain any term the developer may not know in one line (for example, a lone surrogate is half of a character that takes two UTF-16 units). @@ -111,12 +111,11 @@ its absence is not a problem to report. Your first message starts with two or three sentences on what this session will do, before any question. Write them for this repository, in plain words: -name what the library does, and what you will try. For a scroll-aware header -component, for example: "I'll read how Headroom decides when to hide and -show the header, then mount it in a fake browser and scroll it around to see -whether it does what the docs say. Anything odd, I'll show you with a script -you can rerun. Then we'll write an oracle for the rules we agree on, and run -it." Do not reuse that wording. +name what the library does, and what you will try. For a retry helper, for +example: "I'll read how `retry()` decides when to try again and how long it +waits, then call it with fake failures to see whether it does what the docs +say. Anything odd, I'll show you with a script you can rerun. Then we'll +write an oracle for the rules we agree on, and run it." Do not reuse that wording. Then ask one question: is there a part of the code they want to try this on, or should you look around? Stop for the answer. If their first message @@ -161,8 +160,8 @@ do I not know about this code that I need to know to build its oracle? How does the state change with each call? Where are the boundaries? What holds whatever happens? Show the developer those questions in the chat, a short list, before you probe: it shows them how an oracle gets built. Write each -question about their code ("If you scroll up by one pixel, does the header -come back?"), not as a category ("timing", "lifecycle"). Then build a +question about their code ("If the second try fails too, does it wait +longer than after the first?"), not as a category ("timing", "lifecycle"). Then build a probe for each one, and answer each question with what its probe showed. **A probe** is a small script that calls the real public API with chosen From 23af0bf8a1b12b817679cc4ca6616a5b70164e4e Mon Sep 17 00:00:00 2001 From: Kyle Mathews Date: Wed, 11 Nov 2026 16:48:16 -0700 Subject: [PATCH 76/76] docs(oracles-start): present spots, explain every new word, oracle first then fix, close gaps until something turns up From the 2026-11-11 UX test (Kyle Mathews and Tanner Linsley) and its evaluate-review ledger, validated with first sessions on klona, quick-lru, and p-queue: - Open with plain sentences about this repository (no quotable example), look around with impact signals (the most important API, README examples, user-facing code, strongly stated promises in old code, rare edge cases that take a system down, repeated fixes), present two or three spots, and stop for the choice. - A table of uncommon words, each explained in one line the first time it comes up, including gap check and gap. - Findings carry 'why it matters' and are ranked by impact; the first small one says why small bugs appear; trivial ones are left out of the findings and fixed later with the others. - Oracle first: a bug a probe finds goes into the oracle before any fix. Confirmed bugs are fixed right away, with the oracle failing before and passing after. - When the oracle passes, close gaps at once: ask why the oracle missed each one, widen the histories or the check under the approved laws, and keep going until a bug or a design question turns up. - The gap check's report goes to a temporary directory; no cleanup talk. - 'Take the oracle to production' becomes 'finish setting up the repository for oracles', offered when the developer wants to keep the work. oracles-setup is retitled to match. - problems.md: the repository's lint picking up the oracle file. Co-authored-by: Isaac --- .../oracles/skills/oracles-setup/SKILL.md | 16 +- .../oracles/skills/oracles-start/SKILL.md | 227 ++++++++++++------ .../oracles-start/references/problems.md | 11 +- 3 files changed, 164 insertions(+), 90 deletions(-) diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md index e306ed2..492100c 100644 --- a/packages/oracles/skills/oracles-setup/SKILL.md +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -1,9 +1,9 @@ --- name: oracles-setup -description: Take an oracle to production with @tanstack/oracles. Pin the packages, add the lint preset and the Vitest config, write the AGENTS.md block, make the oracle skills findable by coding agents, and run the oracle through the gap check. Use when the developer chooses to ship an oracle from a first session, when a repository adopts @tanstack/oracles, or when it upgrades its pinned commit. Its install and file-placement sections also cover the minimal setup that a first session needs. +description: Finish setting up a repository for oracles with @tanstack/oracles. Pin the packages, add the lint preset and the Vitest config, write the AGENTS.md block, make the oracle skills findable by coding agents, and run the oracle through the gap check. Use when the developer wants to keep the work of a first session (a commit, or a pull request), when a repository adopts @tanstack/oracles, or when it upgrades its pinned commit. Its install and file-placement sections also cover the minimal setup that a first session needs. --- -# Take an oracle to production +# Finish setting up a repository for oracles You are setting up oracle testing in this repository. Do the steps in order. Ask the developer before each step that writes outside the repository. When no @@ -14,9 +14,9 @@ Potential laws have no default: propose them, report them, and stop. This skill has two uses. A first session (`oracles-start`) uses only its install section and its "where oracle files go" section, for the minimal setup that an oracle needs to run, and step 7's command, when its oracle -passes and it runs the gap check. Taking an oracle to production -uses all of it. That usually starts after a first session, once the -developer has an oracle that found a bug and chooses to ship it. If there is +passes and it runs the gap check. Finishing the setup uses all of it. That +usually starts after a first session, when the developer wants to keep the +work: to commit it, or to open a pull request. If there is no oracle yet, load `oracles-start` first. Setup ends when one oracle passes on production and the gap check leaves no @@ -357,9 +357,9 @@ continue with its gap check. If there is none, build it from the potential laws that the developer approved: load `oracles-design`, and follow it from its first step. -In a first session, stop at the RED: present the bug, and leave the fix for -production. A first session runs the gap check only when its oracle passes -(`oracles-start` step 5). When taking the oracle to production: +A first session fixes each bug the developer confirms right away, oracle +first, and runs the gap check only when its oracle passes (`oracles-start` +step 5). When finishing the setup: If the first oracle fails on unchanged production under an adopted law, it found a defect. The RED is the finding: report it with its replay line, and diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md index 2bc139f..e3ab235 100644 --- a/packages/oracles/skills/oracles-start/SKILL.md +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -1,6 +1,6 @@ --- name: oracles-start -description: Run a first session with @tanstack/oracles in a developer's codebase. Say what the session will do, look around a part of the code they name or the whole repository, probe the real API to prove each finding, propose a potential oracle and its potential laws, then build it and run it until it finds a bug and the developer has settled some design questions. Present the work, and explain the path to production. Use when someone is trying @tanstack/oracles, or asks where an oracle would help. +description: Run a first session with @tanstack/oracles in a developer's codebase. Say what the session will do, look around and present a few spots, probe the real API to prove each finding, propose a potential oracle and its potential laws, then build it and run it, fix confirmed bugs oracle first, and close gaps until it finds bugs or design questions. Present the work, and say what finishing the repository's setup would add. Use when someone is trying @tanstack/oracles, or asks where an oracle would help. --- # A first session with oracles @@ -16,9 +16,14 @@ the oracle file. Do not explain what oracles are unless they ask. Work toward these milestones, in order: findings from probes of their real code; a potential oracle whose laws they approve; an oracle that runs on -their code; and that oracle finding a bug, or the gap check finding what no -law catches, while they settle some design questions. The developer decides -when the session ends. +their code; and that oracle finding a bug, or closing gaps until one turns +up, while they settle some design questions. Do not wrap up before the +oracle has found a bug or brought up a design question. The developer can +stop you at any time. + +Part of what the session shows, without saying so, is how an oracle works +through a codebase: each bug goes into the oracle first, the fix follows, +and the gap check points at the next thing to cover. Probes are how you learn what reading the code does not tell you: its dynamics, boundary conditions, and constraints. You need that understanding @@ -33,9 +38,34 @@ credibility. So every finding needs a probe that the developer can run again. **potential oracle** that owns a public behavior, its **potential laws**, and the **oracle file**. Once the developer approves a potential law, it is a **law**; once the oracle runs, it is the **oracle**. -- Explain a framework term in a line the first time you use it (a - calibration is a wrong answer planted in a run to show that the check - rejects it), or leave the term out. +- Explain each uncommon word in one line the first time you use it, tied to + what is on the screen. Do not explain them all up front. The words, and a + line for each: + + | Word | One line | + | ------------------- | ----------------------------------------------------------------------------------------------------- | + | probe | a small script that calls your real code with chosen inputs and prints what happens | + | finding | something a probe showed: a possible bug, two sources that disagree, untested behavior, or a question | + | law | one promise the code makes, as a "for every …" sentence, with where the promise is written | + | oracle | a test that checks the laws against your real code over many generated inputs | + | oracle file | the file that holds the oracle: the laws in prose first, then the code | + | model | the expected answer, computed the simple way, without your code | + | history | one generated input or sequence of calls that the oracle tries | + | constructed history | a history written by hand to reach one edge case every run | + | driver | the part that calls your real code and records what it does | + | check | the part that compares what your code did with the model | + | shrinking | cutting a failing history down to the smallest one that still fails | + | seed | the number that makes a run's random histories the same every time | + | calibration | a wrong answer planted in a run, to show that the check rejects it | + | owns | the functions an oracle answers for | + | gap check | makes small changes to a copy of the owned code, and shows each change that no law notices | + | gap | one such change: behavior the oracle does not check yet | + | authority | where a promise is written: the docs, a test, a comment, or your decision | + | limit | what the oracle does not check, and why | + + Use the same word for the same thing every time. Leave out a word the + session does not need. + - Talk naturally. Never name a skill, a step, or a section ("step 3", "the skill's fallback"). Say what you are doing, in your own words. - Talk about their code in its own terms, not in this skill's words. Words @@ -57,19 +87,20 @@ credibility. So every finding needs a probe that the developer can run again. Stop and wait only when the developer must decide: -- which part of the code to look at +- which of the spots you present to dig into - whether a finding is a bug, or intended - a design question - whether to approve the potential laws - whether to approve a more complex design (see below) -- whether to take the oracle to production +- whether to finish setting up the repository for oracles, when they want to + keep the work (a commit, or a pull request) Between those points, keep working, and keep telling them what you are doing. ## What you change -Until the developer chooses to take an oracle to production, change only -these: +Until the developer wants to keep the work and finish setting up the +repository, change only these: - the minimal installs that an oracle needs to run: `@tanstack/oracles`, `fast-check`, and `vitest` when `package.json` does not list them @@ -77,13 +108,12 @@ these: repository's own config does not - oracle files, with their prose - the feedback log, `oracles-feedback.md`, while this is a preview -- the gap check's working directory (`.oracle-gaps/`), when you run it. - Tell the developer it is safe to delete, and that production adds it to - `.gitignore`. +- the smallest production fix for a bug the developer confirmed, after the + oracle fails on it (see step 5) Findings, potential oracles, and potential laws stay in the chat. Keep -probes in a scratch directory outside the repository. Change production code -only when the developer asks you to. +probes, and the gap check's report, in a temporary directory outside the +repository (`mktemp -d`). Say nothing about cleaning them up. ## A feedback log, while this is a preview @@ -100,28 +130,25 @@ is for later, not a reason to stop. ## 1. Open -Before your first message, read the block that production adds to +Before your first message, read the block that finishing the setup adds to `AGENTS.md`: it is in [`oracles-setup` step 4](../oracles-setup/SKILL.md), under "Write the AGENTS.md block". It governs you from the start of this session, though nobody has added it to the repository yet: who owns the laws and who owns the code, which skill to read before each kind of oracle -work, and the lint rule. Until production, the lint rule means the -repository's own lint: the oracles lint plugin comes with production, and -its absence is not a problem to report. +work, and the lint rule. Until the setup is finished, the lint rule means +the repository's own lint: the oracles lint plugin comes with the setup, +and its absence is not a problem to report. Your first message starts with two or three sentences on what this session -will do, before any question. Write them for this repository, in plain words: -name what the library does, and what you will try. For a retry helper, for -example: "I'll read how `retry()` decides when to try again and how long it -waits, then call it with fake failures to see whether it does what the docs -say. Anything odd, I'll show you with a script you can rerun. Then we'll -write an oracle for the rules we agree on, and run it." Do not reuse that wording. - -Then ask one question: is there a part of the code they want to try this on, -or should you look around? Stop for the answer. If their first message -already named a part of the code, say that you will start there, and go on. -A request about the whole repository ("what can it find in this repo") -names no part: ask. +will do, before any question. Write them for this repository, in plain words, +in your own sentences: name what the library does, which of its behaviors +you will try, how you will try them (calling the real code with chosen +inputs), and that you will then write an oracle with them for the rules you +agree on. Mention nothing about this skill or its steps. + +Then go straight on to look around. If their first message named a part of +the code, say that you will start there, and go on. They can name a part at +any time, and you start there. ## 2. Look around @@ -135,23 +162,33 @@ It is a starting point: skip build and config files, and read the commits that changed the public API. Look for behavior that is easy to break, and whose right answer is simpler -to state than the code that produces it. Good signs: - +to state than the code that produces it. Rank the candidates by how much a +bug there would hurt the people who use the code. Good signs: + +- the most important API: the call most users make first, and most often +- code the README's examples and getting-started section show +- behavior users see directly, and depend on +- a promise stated strongly (a guarantee in the docs, an invariant in a + comment), above all in code that has not changed in years: nobody has + checked it since it was written +- a rare edge case that would bring a user's system down when it hits +- code that git history shows was fixed more than once: people keep + reporting bugs there, so the tests are not capturing the whole picture - a round trip (parse and serialize, encode and decode, copy and compare) -- documented examples and defaults - a cache, a queue, or other state that changes over a sequence of calls - an ordering, timing, or normalization rule -- an invariant stated in a comment -- code that git history shows was fixed more than once Avoid an internal helper whose only law would restate its code. Avoid behavior that nothing documents, unless the developer wants to decide it. -Tell them about two or three spots, a short paragraph each: what the code -seems to promise, where that is stated, and why it would make a good oracle. -Recommend one, and let them choose. If they asked you to choose ("surprise -me"), say in a line which spot you picked and why, and go on to probe it. -They can redirect you. +Quick probes to choose the spots are fine. Show a result only later, with +the probe that proves it. + +Present two or three spots, a short paragraph each: what the code seems to +promise, where that is stated, why a bug there would matter, and why it +would make a good oracle. Recommend one, and stop for their choice. If they +already asked you to choose ("surprise me"), say in a line which spot you +picked and why, and go on to probe it. ## 3. Probe @@ -210,12 +247,24 @@ question of where to look. **Present each finding** in this shape, a light touch, with each finding called a possible bug or a question, never a certainty: -1. A title that names the behavior: "Object Map key: replay throws". +1. A title that names the behavior: "Retry delay: the second wait is + shorter than the first". 2. The smallest case, expected against observed, as a short snippet the developer can run. 3. **Why I think it's wrong:** the authority, quoted, with its file and line. -4. **Why the tests miss it:** what the existing tests check instead. -5. **How often:** how many of the probe's inputs failed, out of how many. +4. **Why it matters:** who would hit it, and what would happen to them. +5. **Why the tests miss it:** what the existing tests check instead. +6. **How often:** how many of the probe's inputs failed, out of how many. + +Put the findings in order of their potential impact on users, the worst +first. The first time you show a small finding, say in a line why it is +there: an oracle captures the code's design and then enforces all of it, +trying every input it can generate, so it turns up small bugs as well as +serious ones. Expect a mix. + +Leave out a bug so trivial that no real user would ever hit it. Keep a note +of it: the oracle will catch it too, and you fix it with the others when it +comes time to fix, and list it in the wrap-up. Give each finding its kind: @@ -228,20 +277,26 @@ Give each finding its kind: For example: -> **Column names with a digit don't round-trip.** +> **Retry delay: the second wait is shorter than the first.** > > ```ts -> camelToSnake(snakeToCamel('address2_line')) // expected 'address2_line', got 'address2line' +> await retry(failTwice, { retries: 3, delay: 100 }) // waits 100 ms, then 50 ms; expected 100, then 200 > ``` > -> **Why I think it's wrong:** the JSDoc of `snakeToCamel` says the transform -> is "round-trippable via `camelToSnake`" (`src/column-mapper.ts:71`). -> **Why the tests miss it:** they check `user_id` and `created_at`, which -> have no digits and no single-letter parts. **How often:** 3,518 of the -> 21,844 names I tried. +> **Why I think it's wrong:** the README says the delay "doubles after each +> failed try" (`README.md:42`). **Why it matters:** a service that is already +> overloaded gets its retries faster, not slower. **Why the tests miss it:** +> they check only the first delay. **How often:** 412 of the 2,000 schedules +> I tried. For a possible bug, ask: "Is this a bug, or is it intended?" +**Oracle first.** When a probe finds a bug, do not fix production yet, even +when the fix is obvious. The bug goes into the oracle: a law that it breaks, +and a history that reaches it. The fix comes after the oracle fails on it +(step 5). That way the oracle, not the probe, proves the fix, and holds it +for every later change. + If the probes find nothing, say so plainly: "No confirmed defect in what I tried." Show what held, and the design questions that came up. @@ -402,35 +457,49 @@ them and stop. 3. **Run it on the unchanged code.** A failure under an adopted law is a possible bug. Present it like a probe finding, with the oracle's smallest failing case as the snippet. -4. **As the developer settles questions, change the law** and rewrite the law +4. **Fix each bug the developer confirms, right away, oracle first.** The + oracle already fails on it. Make the smallest production fix, rerun the + oracle until it passes, and show both runs: the failure before, the pass + after. Follow the defect path in `oracles-design` step 7. Most developers + want the fix at once; with the oracle failing first, the fix is proved. +5. **As the developer settles questions, change the law** and rewrite the law block's prose in the same step. Before you show results, check that the prose still says what the code does. -Do not change production code. That belongs to taking the oracle to -production. Do not offer the gap check while the oracle fails: it cannot +**When the oracle passes, close gaps at once, without stopping.** Say in a +line that the oracle passed and what held, then run the gap check, and +explain what it does the first time you mention it. Take the command and +its flags from `oracles-setup` step 7, with `--out` in a temporary directory +outside the repository. Do not run it while the oracle fails: it cannot judge a failing run. -**If the oracle finds no bug, run the gap check to go deeper.** Say so -plainly first, and show what held. The gap check makes small changes to a -copy of the production code that the oracle owns, and shows each change -that no law catches. Each one points at a bug the oracle has not reached, a -design question, or a law nobody has stated yet. Take the command and its -flags from `oracles-setup` step 7, with its `--out` in the package's -`.oracle-gaps/` directory, as that step says. Read each surviving change, probe -it when it is unclear, and present what it shows as a finding: a possible -bug, or a design question for the developer. Do not try to close every gap: -that is production work. +For each gap, oracle first: ask why the oracle missed it. Usually the +histories never reach the case, or the check does not look at that result. +Widen the histories or the check, under the laws already approved, and run +the oracle again on the unchanged code: + +- If the oracle now fails, it found a bug. Present it, and fix it as above + once the developer confirms it. +- If closing the gap needs a rule that no approved law states, that is a + design question. Bring it to the developer. +- If the oracle passes and the gap is closed, go on to the next one. -If the gap check finds nothing worth the developer's time either, offer to -build the next oracle at another spot. +Group the gaps by function, and tell the developer what you are doing as +you go. Keep going until a bug or a design question turns up. When the +gaps under the approved laws are closed and nothing has turned up, say so, +pick the next spot from your look-around, and keep going. ## 6. Present -When the oracle or the gap check has found something and the developer has -settled some design questions, stop and present. Present the same way -whenever the developer wants to stop: +When the oracle has found bugs, or brought up design questions that the +developer has settled, stop and present. Present the same way whenever the +developer wants to stop: -- what you built, and what it found +- what it found, first: each bug, with what it was, its fix, and the oracle + run before and after +- the trivial bugs you left out of the findings and fixed along the way, in + one line each +- what you built - an offer to walk through how it works: how the reference model was built, how the histories are generated, and how the driver calls their code. Developers often want this, and it is the best part to teach. @@ -439,15 +508,13 @@ whenever the developer wants to stop: (GitHub KyleAMathews) when they are done When they take the walk-through, follow one bug the oracle found through -the file (with no bug, one change the gap check showed, or one history that -held): the history that produced it, what the driver did and saw, what +the file: the history that produced it, what the driver did and saw, what the model expected, and how the check reported the difference and shrank it. Show a few lines of the file at each part, not a description of it. -Then explain how to take the oracle to production: a lint preset that keeps -oracle files honest, a block in `AGENTS.md` (or `CLAUDE.md`) so every agent -follows the same rules, and the gap check run over all the owned code, which -makes small changes -to a copy of the production code and shows each change that no law catches. -Closing those gaps is how the oracle grows toward full coverage. If they want to go ahead, load -`oracles-setup`. +When the developer wants to keep the work (commit it, or open a pull +request), say what finishing the repository's setup adds: a lint preset +that keeps oracle files honest, a block in `AGENTS.md` (or `CLAUDE.md`) so +every agent follows the same rules, and the gap check over all the owned +code. Then load `oracles-setup`, and do it. Do not call this "production" +or "shipping": it is setting up their repository for oracles. diff --git a/packages/oracles/skills/oracles-start/references/problems.md b/packages/oracles/skills/oracles-start/references/problems.md index 5d1affd..83089a7 100644 --- a/packages/oracles/skills/oracles-start/references/problems.md +++ b/packages/oracles/skills/oracles-start/references/problems.md @@ -22,16 +22,23 @@ of `@tanstack/oracles` without listing them. Check `package.json`, not **Which Vitest.** Install `vitest@^4.1`. It runs on Node 20 and later, and Vitest 5 needs Node 22.12 or later. Which Node versions CI runs is a -question for production: do not raise it now. +question for when the repository's setup is finished: do not raise it +now. **An install fails.** Show the error in a line or two, say what you think caused it, and ask before you try anything that changes more than the packages the oracle needs. -**Lint.** The oracles lint plugin waits for production. If the +**Lint.** The oracles lint plugin comes when the repository's setup is +finished. If the repository's own lint already covers the oracle file's folder, run it on the file, as `oracles-design` says. +**The repository's lint picks up the oracle file.** If its lint (xo, ESLint +over every file) now fails on the oracle file or the Vitest config, tell the +developer in one line. Leaving them out of that lint is part of finishing +the setup. Do not change their lint config in a first session. + ## Probes **The tests import a build that is not built yet.** Import the source that