diff --git a/.changeset/ship-oracle-skills.md b/.changeset/ship-oracle-skills.md new file mode 100644 index 0000000..8a7904b --- /dev/null +++ b/.changeset/ship-oracle-skills.md @@ -0,0 +1,6 @@ +--- +'@tanstack/oracles': patch +'@tanstack/eslint-plugin-oracles': patch +--- + +Ship agent skills and docs inside `@tanstack/oracles`. The package now includes `skills/` with `oracles-setup`, `oracles-design`, `oracles-review`, and `oracles-evaluate-review`. It also includes `docs/`: the oracle checklist (version 1, ORC-001 to ORC-016), a glossary, a techniques guide, model-style examples, and six instrument cards. A coding agent reads them from `node_modules`. `@tanstack/eslint-plugin-oracles` gains a README that lists its configs and rules. diff --git a/README.md b/README.md index 6e44a81..3f83381 100644 --- a/README.md +++ b/README.md @@ -9,8 +9,12 @@ every commit with [pkg.pr.new](https://pkg.pr.new). ## Packages -- `packages/oracles` (`@tanstack/oracles`): the oracle framework, and the - `oracle-gaps` CLI. The CLI lists the production behaviour that a +To try it, paste the prompt in +[`packages/oracles/README.md`](packages/oracles/README.md) into your coding +agent. + +- `packages/oracles` (`@tanstack/oracles`): the oracle framework, its skills + and docs, and the `oracle-gaps` CLI. The CLI lists the production behavior that a project's owning oracles leave unexercised or unjudged. A project runs it as `oracle-gaps --package --source `, with `--env-prefix` set to the prefix that the project passes to @@ -148,12 +152,7 @@ required. The rule reports: A framework implementation module is exempt. -No other configuration exempts a file. `settings.oracles.legacyCampaigns`, the -old list of files that kept their raw campaigns while they migrated, is removed. -A configuration that still sets it, to any value, exempts nothing. The rule -reports it as `removedSetting` at line 1 of each oracle file that the -configuration reaches, since ESLint runs a rule once per file. Convert the -listed files to `oracleProperty`, then delete the setting. +No configuration exempts any other file. A non-array `settings.oracles.implementationModules` or `replayWitnessModules` is reported at line 1 as `malformedSetting`, and it is read as an empty list. diff --git a/packages/eslint-plugin-oracles/README.md b/packages/eslint-plugin-oracles/README.md new file mode 100644 index 0000000..a406a72 --- /dev/null +++ b/packages/eslint-plugin-oracles/README.md @@ -0,0 +1,39 @@ +# @tanstack/eslint-plugin-oracles + +Lint rules for oracle files written with `@tanstack/oracles`. They report the +shortcuts that make an oracle pass without enforcing its law, wherever a +syntactic check can see them. They need no type information, so an agent's +post-edit hook can run them on one file quickly. + +Install it at the same pkg.pr.new commit as `@tanstack/oracles`. The +`oracles-setup` skill in that package configures it. + +## Configs + +| Config | Files | Rules | +| --------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | +| `configs.recommended` | Files whose names contain `oracle` (`.ts`, `.tsx`, `.js`, `.jsx`, `.mjs`) | `no-disabled`, `no-oracle-disable`, `no-unit-test`, `oracle-property-only`, `pure-model`, `replayable-seed` | +| `configs.production` | `src/**` and `packages/*/src/**` | `probe-form` | + +The `recommended` config also sets `linterOptions.noInlineConfig` for oracle +files, so a directive comment cannot switch a rule off there. Fix a report; +do not disable the rule. + +The `production` config covers `src/**` and `packages/*/src/**`. For source +elsewhere, add `oracle/probe-form` with your own glob. + +## Rules + +| Rule | Reports | +| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `oracle/no-disabled` | An oracle test that stops running: `.only`, a skipped body, a switch-off condition, or one run with no fixed history. | +| `oracle/no-unit-test` | A test in an oracle file that reaches no declared check or model: an ordinary unit test. | +| `oracle/no-oracle-disable` | A directive comment that would silence an oracle rule. Oracle rules cannot be disabled inline. | +| `oracle/oracle-property-only` | A raw campaign (`fc.assert`, `fc.check`, or a `.prop` registration) or a legacy campaign API in an oracle file. Only `oracleProperty` declares a generated-history law. | +| `oracle/pure-model` | A model, check, or reach classifier that reads a clock, ambient randomness, or a timer. | +| `oracle/probe-form` | A state probe in production code that is not in the one form the production build strips. | +| `oracle/replayable-seed` | A property seed that the shared replay variables cannot select. | + +The decided bans on snapshot matchers (B1), `fc.pre` (B3), production imports +in a model (B4), and identical comparison operands (B7) are not rules yet. +Reviews check them until they ship. diff --git a/packages/oracles/README.md b/packages/oracles/README.md new file mode 100644 index 0000000..f2c0282 --- /dev/null +++ b/packages/oracles/README.md @@ -0,0 +1,16 @@ +# @tanstack/oracles + +Oracle testing for TypeScript: an agent states what your code promises, then +checks your real code against an independent model of those promises. + +```text +Show me what oracle testing with @tanstack/oracles can find in this repository. + +1. Find the current commit of TanStack/oracles. Run + `git ls-remote https://github.com/TanStack/oracles refs/heads/main`, and use + the first 7 characters of the hash as . +2. Install https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@ as a dev + dependency, with this repository's package manager. +3. Read node_modules/@tanstack/oracles/skills/oracles-start/SKILL.md in full, + and follow it. Use wherever a skill asks for the commit to pin. +``` diff --git a/packages/oracles/docs/glossary.md b/packages/oracles/docs/glossary.md new file mode 100644 index 0000000..085b167 --- /dev/null +++ b/packages/oracles/docs/glossary.md @@ -0,0 +1,94 @@ +# Oracle glossary + +This glossary owns the terms for oracle work with `@tanstack/oracles`. The +project's own glossary or architecture documents own its domain terms. Use one +term for one concept. Do not rotate synonyms for style. + +## Laws and evidence + +| Term | Meaning | +| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| law | A promised behavior, stated so that a reader could predict an observation without reading production. An `oracleProperty` states it in `law.statement`. | +| authority | Where a law comes from: an API or contract document and section, or a decision, with who made it (name and GitHub login) and its date. Production output is evidence, never authority. | +| limits | What a property does not establish. Unresolved parts of a law go here. | +| potential oracle | A candidate oracle that the agent proposes in the conversation: the public behavior it would own, its contract as calls and results, and its potential laws (each a "For every" sentence with its authority, and what the probes showed), plus its open questions. The developer approves the laws that the authority decides before the oracle file is written. | +| probe | A small script that calls the real public API with chosen inputs and records what happens. A finding needs one. Probes find the constraints, dynamics, and boundaries that become an oracle's laws and histories. | +| law status | `adopted` (has an authority, and a developer approved it; on a shared repository, its maintainers confirm it in review), `observed` (production does it, nobody has adopted it), or `open` (a question for the maintainer). Only an adopted law becomes an expected result. | +| rival | A plausible different law that predicts a different observation for some history. | +| witness | A history that shows something: that a state is reached, that two laws differ, or that a distinction matters. | +| distinguishing witness | A legal history where a law and its rival, or two model states, predict different observations. | +| allowed set | The answers a model accepts where the contract leaves a choice open, such as an order of independent events. The check accepts any member. This is how an oracle keeps an implementation's freedom. | +| dynamics | How the result or the state changes with each action or input. The model's rules state them, and the histories vary them. | +| constraint | What holds in every history, whatever happens, such as a count that never exceeds its limit. The check tests it at every checkpoint. | +| boundary condition | An edge of the domain or the lifecycle (empty, zero, `-0`, a size limit, at capacity, cleanup), where real behavior and most defects live. Each one needs a constructed history or an open question. | +| adjacent form | A different design that keeps every adopted law: another representation, allowed-set choice, or production structure. Offered so the developer can pick. | +| oracle | The parts that judge production behavior: law, histories, model, driver, and check. | +| oracle file | A test file whose tests are oracles. Its name contains `oracle`. | + +## The parts of an `oracleProperty` + +| Term | Meaning | +| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| history | Starting conditions plus a sequence of actions or events, including timing. | +| history grammar | The values, actions, constraints, and schedules that can form legal histories: the fast-check arbitrary and its resolver. | +| plan | The concrete actions that `resolve(history)` gives. The driver runs it, and the model and check read it. It never holds expected observations. | +| model | The part that gives the expected result of a plan without production. It is pure. It may read the inputs that the driver recorded in the trace, never production's outputs. | +| driver | The part that runs a plan through the real production entry point and records observations. It returns nothing, so it cannot decide a verdict. | +| observation | One recorded public fact, recorded with `observe(checkpoint, value)`. It must be data. | +| trace | The ordered observations of one history. | +| checkpoint | The exact point where the contract owes an observation. | +| check | The part that compares the trace with the model and reports a violation. Only the check can fail a law. | +| violation | A check's report that a history broke a law at a checkpoint. | +| calibration | A named wrong answer (a corrupted trace or a mutant driver) and the law that must reject it. | +| campaign | One run of a generated `oracleProperty`: a fixed-seed test or the random test. | +| replay | Rerunning a recorded failure from the command that the failure prints. | +| shrinking | Reducing a failing history while it still fails the same way. | +| bounded exhaustiveness | Running every case inside stated finite bounds, with an enumerated history source. | + +## Reach + +| Term | Meaning | +| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| reach | The declared states that a history entered. | +| declared state | A state in `states`. A constructed history must enter it. | +| measured state | A state in `measured`. The run reports how often histories enter it, but nothing requires it. | +| constructed history | A history built from a generated one so that it enters named declared states. Only constructed histories make reach required. | +| fixed history | A named history, such as a regression. It is not reach. | +| state probe | A production call, `probing && globalThis.__oracleProbe?.(name, condition)`, that reports where production decides to enter a state. Production builds strip it. | +| probe-only code | The probe statements and their guard. The gap check does not mutate them. | + +## Gaps + +| Term | Meaning | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| owns | The production files or functions an oracle answers for, listed in the `owns` option of `oracleProperty`. The gap check judges owned code only. In conversation, an oracle owns a public behavior; its `owns` list names the functions that produce it. | +| gap check | The loop that runs the oracle files against production and against mutants of the owned code. It reports every gap with one instruction. | +| mutant | A small deliberate change to owned production code. Every mutant is deliberately wrong, so do not qualify the word. The original defect is a common mutant to check. | +| kill | A mutant that a law rejects through a registered `oracleProperty` route. A plain test in an oracle file is an unregistered kill, and it does not count. | +| survivor | A mutant that no law rejects. Every survivor is a gap. | +| gap | Owned behavior that the oracles do not judge. It closes only when a law kills it or the code is removed. | +| exit | Any reasoning that puts a gap out of reach: calling a mutant equivalent or unobservable, or accepting a gap. None is allowed. | +| framework bug | A gap that no kill or removal can close. The agent stops work on that gap and reports its key and evidence. | +| exclusion control | ORC-004's control: a nearby invalid history that the grammar rejects. It is required, and distinct from a quiet exclusion, which is a finding. | +| dead code | Code whose removal no legal history can observe. Remove it. | +| invalid mutant | A mutant that no oracle could possibly detect. The framework must not generate one, so meeting one is a framework bug to report. The author never decides that a mutant is invalid, and the label does not close a gap. | +| unowned | Production code that no oracle owns. The gap check reports it as a warning, not as judged. | +| RED | The extended oracle failing on unchanged production. It shows that the oracle can see the defect. | +| GREEN | The same oracle passing after the change. | + +There is no "equivalent mutant" and no "unobservable effect". A survivor +shows a missing history, observation, or law, or dead code. An effect that +no check sees is an effect that the driver does not record. + +## Model alignment rule + +For every model state, action, and observation, apply one of these rules: + +1. If it represents a production concept, use the production term. +2. If it combines production concepts, name the abstraction and list what it + combines. +3. If it splits one production concept, qualify each part and state the split. +4. If it exists only in the model, say so. Do not give it a production name. + +Aligned names do not mean copied logic. A model should not use production +queues, caches, or helpers merely to share their names. diff --git a/packages/oracles/docs/guide.md b/packages/oracles/docs/guide.md new file mode 100644 index 0000000..1ac1573 --- /dev/null +++ b/packages/oracles/docs/guide.md @@ -0,0 +1,464 @@ +# Oracle guide: techniques + +These techniques support `oracles-design` and `oracles-review`. They add no +requirements; the [checklist](oracle-checklist.md) holds them all, and this +guide cites it by ID. Terms follow the [glossary](glossary.md). + +## The model + +### Make the model easy to distrust + +Prefer a pure function or a small state transition. Keep production queues, +caches, classifiers, and helpers out of the expected result (ORC-002). A +reader should be able to challenge the model without learning the +implementation first. + +Name a partial model honestly: if it predicts values but not callback counts, +say so in `limits` (ORC-001). + +Keep the model's records separate from production's records. If production +mutates a shared object, the expected answer can change with it, and both sides +agree for the wrong reason. Copy the fields the model needs. + +Independent code also needs independent reasoning. A model that calls +production's comparator cannot see a comparator defect. Reuse ordinary test +mechanics freely, but name any semantic helper that both sides trust. A small +model is easier to inspect. It is not correct because it is small. + +### Decompose the model by law + +A model can become as hard to trust as production. Do not keep adding state +until it is a second implementation. + +When the contract has independent laws, treat the oracle as a small graph: + +- A node owns one rule and the least state needed to predict it. +- An edge records a real dependency between two rules. +- The driver can feed the same action to several nodes. +- The check composes their observations at the checkpoint. + +A failure then names the law that diverged, and a reviewer can inspect each +rule alone. Plain functions and Maps are usually enough. + +Do not split state that a legal continuation can observe only as a whole. If +two states look equal to the proposed nodes, try a legal continuation of one +or more actions. If it can +produce different promised results, add the missing edge, or keep that state in +one model. Decomposition does not mean one model per assertion. Group facts +that form one state machine, and keep independent policies separate. + +### Choose the judgment that fits + +A full reference works when the answer is cheap to compute independently. +Other promises need other judgments: + +| Method | Useful judgment | Limit | +| ----------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------- | +| Simple model | Recompute the allowed result, or step a small abstract state. | The model can omit a distinction or encode the wrong rule. | +| Differential comparison | Run the same work through another implementation or formulation. | Both paths can share a bug. Their semantics must actually agree. | +| Metamorphic relation | Transform a history, and check a relation between the two results. | The transformation's premises must hold. | +| Partial invariant | Check one promise, such as "no request completes twice". | A pass says nothing about other properties. | + +Choose the smallest judgment that answers the question. Combining +complementary checks can help. Making every oracle compute every observation +usually does not. + +### Keep only state that can matter + +Removing fields until a model looks simple is not a design method. Test each +merge of states instead (ORC-008). + +Example: a handle supports a row. Replacing handle `a` with `b` moves that +support. Releasing a retired handle does nothing. + +| History | Supported rows | Live support count | +| --------------------------------------------- | -------------- | ------------------ | +| Acquire `a` for `x` | `{x}` | 1 | +| Acquire `a`, then replace it with `b` for `x` | `{x}` | 1 | + +The two states look equal. Now release `a`. The first history loses support +for `x`. The second keeps it through `b`. A model with only rows and a count +cannot answer both. + +So: find two states that the model treats as equal, and try a legal next +action. If their promised observations differ, or an action is legal in only +one, the model erased something that matters. The witness justifies this one +distinction. It does not prove the model handles every race. + +### Know where the expected answer comes from + +Two questions sit behind "the model says so". Who established the behavior? +Which state determines the observation? + +- A finite model can lose permitted values. For example, a model that holds + scores only up to three makes `score > 3` an empty predicate. Fix the model, + not production. +- A model can read the wrong state. For example, it can use locally edited + values where the contract uses source values. + +Before you change production to satisfy a red oracle, check the contract, the +model's representation, and the comparison. An unresolved product choice goes +to the developer as an `open` law. Judge a reviewer's accusation and their proposed +fix separately: a correct concern does not make its repair correct. + +Projection needs the same care. Dropping internal metadata from a comparison +can be valid. Dropping a field the contract promises is not, whatever it is +called. Write down one difference the comparison may ignore and one it must +keep. + +## Histories + +### Describe the input domain as a grammar + +Before choosing arbitraries, describe the language of legal histories. Separate +three kinds of fact: + +- **Dynamics:** the actions that move the system. +- **Constraints:** invariants, interfaces, and forbidden combinations. +- **Boundary conditions:** the starting state, scale, provider behavior, and + value domains where the rules apply. + +Mark which rules come from the contract and which you inferred. Map +containment, overlap, and dependency. Do not force overlapping concerns into a +tree to make the generator neat. + +ORC-004's four controls apply to any claim of generated coverage: + +1. **Reconstruction:** can the grammar rebuild every known valid witness? +2. **Ablation:** does removing each axis, rule, or overlap lose a promised case + or admit a forbidden one? +3. **Range:** does an independent marginal case need only new values, or does + it expose a missing rule? +4. **Exclusion:** does the grammar reject a nearby invalid history? + +Do not take a flat Cartesian product of unrelated axes because the tool makes +it easy. Two false greens are common. A grammar that cannot rebuild a known bug +omits a path. A grammar that generates valid and invalid histories without +telling them apart hides policy in filters. + +### Build legal histories + +Generate only actions that are legal in the current state. Choose the action +kind first, then a key that it applies to. Put legality in the grammar or in +`resolve`, so every shrunk history stays legal; `fc.pre` is banned (B3). + +Design action dependencies so that smaller histories stay meaningful under +shrinking. + +The structural cases that matter become constructed histories, for example: + +- no rows +- a width of zero +- a tie at a boundary +- repeated changes to one key + +Each constructed history names the declared state it enters (ORC-015). Then +let random generation vary the values and lengths around them. A larger run +count is not a substitute for a constructed history. + +### Generate histories that reach the problem + +Once the model can tell the relevant states apart, the histories must put +production into them. Random values alone do not create reuse, failed +startup, or work after recovery. + +Look for relationships and transitions: + +- the same key versus a fresh key +- one owner versus a surviving sibling +- rejection before versus after acquisition +- release before versus after completion +- recovery followed by another operation + +Keep healthy peers alive where isolation is part of the promise. A history +that ends at "recovery succeeded" may never show broken state on the next use. + +Fresh disjoint keys can simplify a driver but exclude the reused-key +interaction that the law needs. Repeated `true` reports do not cover +`true → false → true` while earlier work is pending. These are different +histories, not different random values. + +Widening a generator can lose reach in two ways: + +- A rewrite can make a previously possible history impossible, even while it + adds a dimension. +- A larger domain can make a useful history rarer at the same run budget. + +Check possibility with a constructed history, and frequency with the reach +counts. + +### Control the event, not its returned Promise + +An async operation can have several boundaries: + +```text +request invoked → response delivered → writes applied → result visible + ↘ returned Promise settles +``` + +The contract says which boundaries come before which. The driver must control +and observe the boundaries that the law depends on. + +If an operation applies its work eagerly, delaying its returned Promise does +not delay that work. An unconditional `await` can let a queued microtask +repair a mismatch before the check sees it. A driver that reads only settled +state cannot expose an earlier wrong observation. + +Put an explicit hold and release at the event you need. For a law about +visibility before application: + +1. Production requests data. +2. The driver receives the response, and holds delivery of its writes. +3. The driver records what is visible while delivery is held. +4. The driver releases delivery, and production applies the writes. +5. The driver records what is visible at the promised checkpoint. + +The hold must be real. A driver that delivers the writes and only delays +returning from its handler controls a different event. A scheduler helps only +within the boundaries it controls. A seed cannot schedule external I/O that the +driver does not intercept. + +### Check that timing helpers establish their premises + +A driver often needs helpers such as "wait until the request is blocked" or +"let the message reach the worker". Each helper establishes a premise. If it +does less than its name says, the oracle can pass without ever reaching the +history. + +Example: a helper released a lock in the same turn that it sent a close +message. The message was still crossing a task boundary, so the earlier +request finished first. Close never ran during the wait, and the mutants it +targeted passed. + +For each timing helper, do one of these: + +- Assert the premise at its checkpoint, for example from a lock inventory or a + delivery counter. +- Add a calibration that only a history meeting the premise can reject + (ORC-006). + +A helper that waits a fixed time proves no delivery. It still needs one of +these checks. + +### One law can need two schedules + +Enumerating more positions is not always enough. Example: a request settles +after four completion obligations, `r0` to `r3`. Success must wait for every +obligation, and a rejection must not wait for unrelated later work. For a +selected obligation `r1`, two schedules ask different questions: + +| Question | Schedule | Observation | +| ---------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------ | +| Can settlement ignore `r1`? | Hold `r1`; complete `r0`, `r2`, and `r3`. | The request is still pending. | +| Does a rejection of `r1` wait for peers? | Complete `r0`; hold `r2` and `r3`; reject `r1`. | The request rejects at once, with the partial result of `r0` only. | + +Completing every peer first isolates the first question but removes the +pending sibling that the second question needs. At the last obligation there is +no later peer, so that case cannot distinguish the delay. + +### Keep the driver on its side of the boundary + +A test provider can fix the consumer by accident. Example: a paging provider +removed already-delivered rows before it applied the requested limit. That +supplied progress the consumer never requested, and hid a repeated-page bug. + +State what the request asks, what the provider promises, and which progress +production must arrange. The same applies to reentry. If a callback triggers a +second operation only under a condition, assert that the callback and the +trigger happened. "If reached, check it" is not evidence that reentry was +tested. + +When one law must hold in two environments, give each environment its own +driver, and share the law, model, histories, and check (ORC-014). + +## The driver and the check + +### Observe what the contract promises + +Choose observations from the promise. Membership, values, order, +multiplicity, callback boundaries, error identity, settlement timing, durable +storage, and resource ownership are different facts. Record the ones the law +needs, not all of them. + +- If an unprojected sort field changes, compare the new order, not only whether + the result changed. +- If a callback count matters, check the count before looping over the + callbacks. A loop over no callbacks checks nothing. +- If a failed write must not reach storage later, read the stored data after a + later successful operation. + +Each comparison should separate the specific wrong behavior from the allowed +one. + +### Let observations represent violations + +Make generated inputs legal. Do not make recorded outputs unable to be +illegal. + +Suppose production completes one request twice. A map keyed by request that +keeps the last completion hides the violation. Record both events, then compare +the promised count. Do not fold the raw list back into a map before the check +sees it. + +Likewise: + +- Do not reset a change tracker before checking the parts that must stay + unchanged. +- Do not discard values that "should" never appear. +- Do not normalize away an order that the contract promises. + +Normalize only what the comparison permits. + +### Check incremental results and publication separately + +An incremental system emits changes, while a model often computes a whole +result. Compare compatible objects: accumulate the changes, then compare that +state with a full recomputation at the promised checkpoint. Check individual +batches too when the contract makes their boundaries visible. + +A correct final result cannot show that every earlier publication was +coherent. Choose checkpoints that can expose the promised failure, such as a +torn publication of old and new halves. Do not add delays until the result is +stable. + +When scheduling allows several results, check for **one allowed explanation of +the whole observed history**. Do not approve each field against a different +allowed execution: pieces that are each allowed may never occur together. A +bounded oracle checks sampled histories. It is not a proof of every behavior. + +### Make the comparison prove its usefulness + +Suppose an update check asserts only that the result changed. An empty result +passes it. Compare against the exact expected value instead. + +Then calibrate the check against known wrong answers (ORC-006): + +```ts +// The allowed answer passes; a wrong order and an arbitrary change fail. +check(plan, [{ checkpoint: 'ids', value: [1, 2] }], { ids: [1, 2] }) // pass +check(plan, [{ checkpoint: 'ids', value: [2, 1] }], { ids: [1, 2] }) // violation +check(plan, [{ checkpoint: 'ids', value: [] }], { ids: [1, 2] }) // violation +``` + +In an `oracleProperty`, write these as `calibrations`, so the framework runs +them. Calibration proves the check is sensitive. Reach proves the path ran. +These are different claims: a perfect comparison on a path that never runs +protects nothing. + +### Compare formulations when a model might share the bug + +A plain model can copy a wrong semantic assumption. A second formulation gives +another way to disagree. This applies when a plausible shared fault and a +meaningfully different formulation have been named (ORC-011). + +- First state how the two formulations match on empty inputs, duplicates, + order, and projection. A transformation that changes the answer cannot + show that the two formulations agree. +- A metamorphic relation transforms a history and checks a known relation + between the results. Examples: incremental equals recompute, partitions that + recombine to the whole, a round trip. +- Agreement between two paths does not prove that they are independent. Paths + that share an operator or a copied code path share its faults. A matching + count can hide wrong rows. Use each formulation for the faults it can + separate. + +## Evidence + +### Test the test + +"The oracle passed" combines several claims: + +1. Was the property selected? +2. Did it run? +3. Did it reach the intended path? +4. Did its comparison execute? +5. Would that comparison reject the fault under discussion? + +Treat them as separate evidence. Reach counts, calibrations, and the gap +check answer them for an `oracleProperty`. + +For a mutant, record what happened, not only "killed": + +- A comparison rejected the intended wrong answer. +- A timeout exposed a missing progress obligation. +- Setup failed before the comparison. +- Execution failed outside the comparison after setup, such as a crash. +- The changed code never ran. +- The mutant survived. This is a gap (see the + [mutant gap hunt](instruments/mutant-gap-hunt.md), step 3). + +These outcomes call for different conclusions. If deleting one clause leaves +the same obligation enforced elsewhere, survival does not show that the +obligation is unnecessary. If an oracle kills a compound change, it may not +distinguish each part. + +### Preserve the violation, not merely a red exit + +Shrinking keeps the first failure kind (ORC-010). Keep the violated law, the +checkpoint, and the original history beside the reduced one. A replay that +ran but did not reproduce the mismatch is not a reproduction. + +Capture evidence before cleanup can change it. Copy what the law needs before +a rollback runs. A serializer can lose object identity or reorder arrays. Keep +the original form when normalization would erase the disputed distinction. + +A challenge that shaped the design is a regression case afterwards, not a +held-out test of that design. + +### Vary machinery and premises separately + +When comparing implementation candidates, hold the law, the legal histories, +and the checkpoints fixed. If removing an index or cache changes no public +result, it is dead code, unless a law states the work it saves, such as a +count of reads. State that law and measure it, or remove the code. + +If a premise changes instead, derive which guarantees still follow. Examples: +delivery may duplicate, ordering may tie, or the client may see only part of +the source. A failure under a changed premise may call for more provider data +or a law decision, not another algorithm. + +### Simplify machinery without changing the question + +Before removing or combining oracles, name what will remain: the valid +history, the expected outcome, the production path, the observation, and the +oracle that receives them. "Retry the parent" turning into "delete the parent" +changes the behavior, even when the replacement is shorter and green. + +Other reductions are legitimate. An expectation can contradict the contract. +Two branches can enforce the same law. Removing a redundant branch need not +remove its law. + +Share stable mechanics, such as cleanup, where they are truly interchangeable. +Keep domain truth small and reviewable. Several independent models should not +grow into one miniature implementation to reduce duplication. + +### Reusable boundary laws + +Consider these laws when the contract has the matching boundary. They are +prompts, not requirements. If one does not apply, say why in `limits`; do not +add an empty case. + +- **Real-provider conformance:** freeze representative values from each + supported provider version. Show that the driver's stand-in accepts them + before it stands in for that provider (ORC-014). +- **Minimal ambiguity:** include the smallest valid input for every branch of a + classifier. Rich values that carry several redundant signals do not cover a + one-field collision. +- **Name invariance:** renaming a developer-controlled name must not change a result + unless the contract gives the name meaning. +- **Representation symmetry:** two forms of one input that the contract treats + as the same must give the same + public result or the same documented error. +- **Await-boundary transitions:** hold each relevant `await`. Change ownership, + leadership, generation, abort, cleanup, or restart state, then release it. + Compare the result with the contract for that transition. +- **Local and remote agreement:** data sent across a transport must mean the + same on both sides. Live local references stay local. +- **Partial-construction cleanup:** fail each construction step after it + acquires a resource. Keep the primary error, and show that every acquired + resource is released exactly once. +- **Value and work:** when bounded work is promised, check the exact result and + a deterministic work count. A correct result alone does not establish the + work law. + +An unchecked prompt is not evidence that the law it names holds. diff --git a/packages/oracles/docs/instruments/challenge-review.md b/packages/oracles/docs/instruments/challenge-review.md new file mode 100644 index 0000000..e5757e1 --- /dev/null +++ b/packages/oracles/docs/instruments/challenge-review.md @@ -0,0 +1,76 @@ +# Challenge review + +**Use when:** a model, oracle, or proposed law needs a concrete challenge. + +**Inputs:** one artifact or candidate, its sources, claimed domain, and success +criterion. Prefer a fresh reviewer that has not seen the author's desired +verdict. Without that separation, report a self-critique rather than independent +validation. + +## Operation + +Start with a plausible way this account could be wrong. Select the relevant +levels; they are different questions, not a mandatory sequence: + +| Level | Challenge | +| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Enforcement | Find a legal history or wrong output that the model, generator, driver, or recorder cannot distinguish. Can the intended production path remain unexercised while the test passes? | +| Formulation | Find an omitted precondition, excessive guarantee, wrong checkpoint, or ambiguous observation. Try a correct alternative implementation as well as a wrong one. | +| Suitability | Question whether the promised behavior is desirable, safe, or conventional. Name the external contract, standard, user consequence, or engineering reason that makes the objection applicable. | +| Composition | Find obligations that conflict, duplicate each other, or require a shared ordering or authority rule. Use the tension or restructuring card if that is the material uncertainty. | +| Concepts | Question the chosen identity, state, ownership, or boundary distinctions. Describe the same case without relying on its current names and check what is lost or becomes visible. | + +For a suspected information gap, use [law discovery](law-discovery.md#operation), +step 5. + +### Locate the repair + +The level names what to challenge. Once evidence shows a problem, use +this table to locate the repair: + +| Finding or question | Possible repair location | +| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | +| Production disagrees with an accepted law. | Production code. | +| The reference predicts behavior the accepted law does not allow. | Reference model. | +| An accepted law over-promises, under-promises, or conflicts with another law. | Authorized contract decision, then model and test updates. | +| The law needs information the production path lacks at the promised checkpoint. | Provider, protocol, architecture, or an authorized contract decision. | +| The check rejects an allowed intermediate observation or accepts a wrong one. | Evaluation: checkpoint, history grammar, driver, recorder, or comparison. | +| Correct behavior still fails the user's need. | Requirement and its suitability for the intended use. | + +These locations can overlap. Name the evidence for each before choosing a +repair; an assertion change cannot resolve a code defect or supply missing +information. + +An accepted contract remains an obligation even if the current path lacks the +information needed to meet it. A repair might require additional provider +data. Moving the promised checkpoint or narrowing the guarantee needs an +authorized contract decision; changing an assertion supplies no data. + +For each material challenge, name the broken claim and construct a small legal +history or concrete failure scene. State the competing expected observations, +the evidence that could decide between them, and a possible repair condition. +Check that the challenge is within scope and that its standard actually applies. +An outside-domain case may motivate a scope change; label that proposal. + +When execution is authorized and practical, run the discriminating experiment. +A wrong-answer control must reach the intended comparison to demonstrate rejection. +Use the oracle guide's calibration and mutant-outcome rules. A source-level +argument can remain useful when execution is unavailable; label it accurately. +For a systematic search across plausible wrong implementations, use the +[mutant gap hunt](mutant-gap-hunt.md) to design semantic edits and classify +survivors. + +**Return:** supported defects, design objections, and untested hypotheses as +separate kinds of finding, with sources, witnesses, outcomes, and repair +conditions. For bugs, identify the test gap that permitted the class of failure. + +**Control:** explain what would refute the criticism. Check whether the challenge +would also reject a valid implementation or depend on a condition the contract +excludes. Do not presume a defect merely because the stance is critical. + +**Distortion:** generic pessimism, irrelevant external standards, author capture, +or treating a preferred design as evidence of a current correctness bug. + +**Stop when:** the material challenges have evidence or an explicit unresolved +experiment. No supported finding is a valid result. Challenging a law does not +silently replace it. diff --git a/packages/oracles/docs/instruments/design-grammar.md b/packages/oracles/docs/instruments/design-grammar.md new file mode 100644 index 0000000..ed8db26 --- /dev/null +++ b/packages/oracles/docs/instruments/design-grammar.md @@ -0,0 +1,139 @@ +# Design grammar + +**Use when:** the law has many interacting rules, the model keeps growing, a +simplification is proposed, or the law makes claims about state, concurrency, +retries, or ownership. For an ordinary oracle, design step 2 is enough. + +**Inputs:** + +- the approved potential laws and the frozen list from design step 2 +- the owned production code +- the known examples: docs, issues, and existing tests +- one nearby invalid history that the oracle must reject + +Keep the invalid history and the examples fixed before you simplify anything. +A case that shaped the design is a regression case afterwards, not evidence +of range. + +## Operation + +1. **State the preservation relation.** Say which one the work keeps: + - **Reconstruction:** the model accounts for the documented behavior. + - **Named laws:** the model keeps each adopted law. Behavior that no law + names may change. + - **Same observations:** under fixed assumptions, the old and new models + allow the same observations at the same checkpoints. + + Name the checkpoints, the assumptions, and the freedoms the contract leaves + open. The developer's approval of the laws fixes the intended contract. It is + not evidence that a model meets it. + +2. **Separate what you found from what you added.** Each rule cites its + authority, or you mark it as added. Production output is evidence, never + authority. A rule that only reproduces production's current behavior can + freeze a defect into the law. + +3. **Admit a part only for a job.** A part is an action, a state field, or an + observation. Admit it only when it rebuilds a sourced behavior, decides + legality, or keeps a named law. A renamed production component is not a + part. + +4. **Record rules and conflicts.** Write each rule as one sentence of the law. + When two rules conflict, keep both and mark the conflict as an open question + for the developer. Record a priority only when the authority gives one. + +5. **Check overlaps.** Where two parts share state or timing, ask whether the + interaction itself carries a law, such as a cleanup that races a late + completion. Keep the function of that interaction. Its original shape can + change. + +6. **Reconstruct, then simplify.** + - Rebuild every known example with the histories and the model. Revise the + model if it cannot. + - Remove each part and each rule in turn. If nothing breaks (no example, no + law, no rejected invalid history), remove it or demote it to a comment. + - Then try a different representation. Removing one part at a time does not + find a joint replacement. Examples: + - Three disjoint sets of ids become one map from id to status. + - A stored value becomes a derived one. + - Two states with the same futures merge. + - A case table becomes one general rule. + - Count the complexity that moves into helpers, adapters, or mappings. + Moving it does not remove it. + + Call the result a reduced model, not a minimal one. Local removal shows only + that no tried simplification succeeded. + +7. **Compare every simplification in both directions.** Under the same + assumptions and checkpoints, find: + - histories the new model allows and the old one rejects (a weakening) + - histories the new model rejects and the old one allows (a strengthening) + + Report one of: weakening, strengthening, both, or no difference found within + the stated scope. Name the scope, such as "sampled, 2,000 histories" or + "every history up to 6 steps". No difference found in a sample is not + equivalence. A mapping between the two models is part of what a reviewer + checks: it must not hide an observation to make the models agree. + + Keep three kinds of change apart: + - **Refactor:** the allowed behavior stays the same. + - **Refine:** the model adds a guarantee or narrows what it allows. + - **Revise:** the contract itself changes. That is a law question for the + developer. + + Do not change the law, a history filter, and an observation in one step. + Each needs its own comparison. + +8. **Offer adjacent forms.** Keep design diversity. From the parts and rules, + sketch two to four forms that keep every adopted law but differ in a real + way: another representation, another allowed-set choice, another split of + laws across oracles, or another production structure that the same oracle + accepts. Give each form's route, what it keeps, what it changes, and its + cost. Merge forms that differ only in naming or detail. Do not rank them + unless asked. When the choice is the developer's, present the forms and let the + developer pick. + +9. **Optionally check a formal model.** For a consequential claim about state, + concurrency, retries, or ownership, translate the reduced model into TLA+ + or a similar model checker. + - Add state only where it changes allowed future behavior. Write derived + values as definitions. + - Classify each obligation: a state invariant, a property of transitions or + histories, or a progress property. Progress needs a stated fairness + assumption. Do not add fairness only to make a check pass. + - Mark a property that the transition relation enforces by construction. It + shows a choice in the model, not a fact about production. + - Run a passing configuration and a deliberately weakened one that must + produce the expected counterexample. Without the second run, the check + has not shown that it can find the failure. + - Report the exact mode: an exhaustive finite model, a depth-limited run, + simulation, or an incomplete run. Record the constraints and overrides. + + A passing model check does not show that the model matches production. The + oracle's histories and checks do that. + +**Return:** + +- the preservation relation, with its checkpoints, assumptions, and freedoms +- the parts, each with its job and a witness +- the rules with their sources, and the open conflicts +- the reduced model, and the comparison for each simplification +- the adjacent forms, each with what it keeps, changes, and costs +- formal-check results with their mode, when run +- the limits: what the work does not establish + +**Control:** keep the known examples and the invalid history fixed while you +simplify. A simplification that rejects an allowed outcome narrows the contract. + +**Distortion:** + +- forcing elegance: an awkward rule may be earned, so keep it with its witness +- calling a reduced model minimal +- calling two models equivalent because a sample found no difference +- writing production's current behavior into the law +- a mapping that hides an observation + +**Stop when:** every part has a job and a witness, every simplification has +its comparison, and each open conflict is with the developer. + +Adapted from Field Lab's design grammar instrument. diff --git a/packages/oracles/docs/instruments/index.md b/packages/oracles/docs/instruments/index.md new file mode 100644 index 0000000..833e774 --- /dev/null +++ b/packages/oracles/docs/instruments/index.md @@ -0,0 +1,36 @@ +# Design and review instruments + +Use an instrument when a specific uncertainty makes ordinary implementation or +review insufficient. Read only the relevant card. Choose and use it within the +developer's authorized task; these cards do not require a separate approval for each +reasoning step or a fixed workflow. + +| Uncertainty | Instrument | Result | +| ----------------------------------------------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| What behavior should this subsystem promise? | [Law discovery](law-discovery.md) | A sourced candidate law, competing predictions, and a distinguishing witness. | +| Could the implementation, model, or law itself be wrong? | [Challenge review](challenge-review.md) | Concrete challenges at named levels, their evidence, and repair conditions. | +| Would a plausible wrong implementation still pass the oracle? | [Mutant gap hunt](mutant-gap-hunt.md) | A semantic mutant and what its survival shows: a missing history, observation, or law, dead code, a law question, or a framework bug. | +| Do sound obligations conflict under the same conditions? | [Tension scan](tension-scan.md) | A traced collision, missing distinction, or no supported tension. | +| Should these laws or concepts be merged, split, replaced, or removed? | [Law restructuring](law-restructuring.md) | An alternative organization and a check of what it preserves or loses. | +| Is this model the smallest one that keeps the law, and what does a simplification change? | [Design grammar](design-grammar.md) | Parts with jobs and witnesses, open rule conflicts, a reduced model, and a two-way comparison for each simplification. | + +These are reasoning aids, not additional oracle conformance requirements. The +[checklist](../oracle-checklist.md) owns testing requirements. Use the +[glossary](../glossary.md) and the relevant architecture and contract sources. +The `oracles-design` and `oracles-review` skills route oracle work here; the cards can also help +with architecture and other design questions. + +Keep these apart: source statements, observed behavior, generated examples, +hypotheses, and design judgments. An existing contract constrains correctness +claims, and a design proposal may challenge it. Only the authority +appropriate to the task can change a contract. An instrument result does not +change product expectations by itself. + +Record the question, the operation, the source pointers, the concrete +witness, and the open part in the task's normal notes. No finding, no +supported tension, and a failed simplification are useful results. A +reasoning witness is not an executed test. Passing sampled histories is not +proof. + +Adapted from Field Lab instruments. The challenge review card also permits +criticism of a candidate's standard and concepts. No Field Lab install is needed. diff --git a/packages/oracles/docs/instruments/law-discovery.md b/packages/oracles/docs/instruments/law-discovery.md new file mode 100644 index 0000000..5ac9cc4 --- /dev/null +++ b/packages/oracles/docs/instruments/law-discovery.md @@ -0,0 +1,76 @@ +# Law discovery + +**Use when:** a bug, example, or implementation exists but the promised behavior +is unclear; several supplied reports may share a law; or an existing law might +conflate distinct obligations. + +**Inputs:** the question, contract sources, known examples, the existing oracle +and its limits, and the public observation that matters. A missing authority is an +input gap, not permission to infer a promise from production. + +With several reports or examples, ask whether one caller-visible obligation +predicts their different symptoms. Name its common preconditions. Check an +apparent outlier and a nearby case that the obligation should permit. If those +cases need different promises, split the proposal. A descriptive cluster is a +lead, not evidence. + +## Operation + +1. State the obligation in domain terms before describing machinery: for which + legal inputs or histories, under which preconditions, what must be observable + at which checkpoint? Include what can still be pending or unspecified. +2. Trace its authority. Separate accepted contract, observed current behavior, + mathematical consequence, and proposed design. Explain why the obligation + matters to a caller. A mathematical rule still needs an argument that it is + the right rule for this API. +3. Form a plausible rival interpretation. Derive both predictions for the + smallest legal case that separates them, without consulting production for + the expected answer. If authority does not select a prediction, retain the + design question instead of turning one guess into an assertion. +4. Compare an ordinary case with a nearby boundary case. Vary a relevant + condition such as cardinality, ordering, identity, authority, or observation + time. Keep other conditions fixed where possible. Check the boundary is + within the claimed domain. +5. Identify the checkpoint where the contract requires the proposed result; a + permitted pending state does not establish a gap. Check whether the result + is determined by the information available to the responsible production + path there. Seek two legal underlying histories that give the path the same + initial state and inputs through that checkpoint but require incompatible + public observations. If they exist, name the missing information or provider + obligation. A candidate law may be scoped explicitly; do not silently narrow + an accepted promise, which requires authority to revise. +6. Describe the smallest independent model that enacts the obligation and limits + from step 1 and makes the rival and boundary predictions from steps 3–4. + Retain the source facts needed to examine any information gap from step 5, + even when the production path has not received them. + For any proposed merge of model states, ask whether a legal continuation can + distinguish them. Map model abstractions to the project vocabulary. + +The information check asks what production can receive. The model-state test +asks what the reference must remember. Example: a path must publish a full top +ten after a deletion, but it receives only the old top ten and the deletion. +Two datasets can then need different results while the path sees the same +inputs. If the path can request the replacement rows, and the provider +supplies them before publication, the path has the information. Judge it at +its own checkpoint. + +**Return:** the candidate law and status, its source, rival predictions, a +concrete distinguishing witness, and the next experiment or unresolved decision. +For an information gap, include the indistinguishable histories and the +unresolved source, protocol, or scope decision. +For a simple law, a short paragraph and a worked example suffice. + +**Control:** try a legal alternative implementation. Does the proposed law +reject it only because its scheduling or representation differs? Also try a +plausible wrong result: could the proposed observation distinguish it? If an +information gap is claimed, try a nearby path that supplies the missing input, +such as a provider backfill. Check whether that input distinguishes the +histories before the checkpoint; if it does, the original impossibility +argument no longer applies to that path. + +**Distortion:** freezing an implementation accident into a law; treating a vivid +edge case as representative; or overstating what is owed at an intermediate cut. + +**Stop when:** the law is precise enough to enact independently, or a named +missing authority or fact prevents choosing between predictions. Continue other +authorized work while keeping that choice unresolved. diff --git a/packages/oracles/docs/instruments/law-restructuring.md b/packages/oracles/docs/instruments/law-restructuring.md new file mode 100644 index 0000000..2dfd76f --- /dev/null +++ b/packages/oracles/docs/instruments/law-restructuring.md @@ -0,0 +1,50 @@ +# Law restructuring + +**Use when:** laws overlap, need a missing distinction, or might be better +expressed by merging, splitting, replacing, or removing them. + +**Inputs:** the original laws with their authority, motivations, examples, and +limits. Include the oracles that enforce them. Do not simplify labels without first +recovering the obligations they represent. + +## Operation + +1. Decompose the laws into traceable obligations, conditions, distinctions, + and promised observations. Check that these parts can reconstruct each + original claim; invented categories can lose the essential constraint. + If public wording and an oracle disagree, retain both claims and + the unresolved authority. A passing test alone cannot retire public wording. +2. Propose a different organization and state the operation: merge under a + common principle, split by authority or checkpoint, replace a concept, or + retire an obligation with an explicit reason. Fewer words or model fields + alone do not establish improvement. +3. Map every original obligation to where it is preserved, narrowed, strengthened, + rejected, or still unresolved in the candidate. State what insight originally + made the obligation necessary. A claim may be overturned, but needs a reason + and a design decision rather than disappearing from the mapping. +4. Derive old and proposed predictions for a witness exercising each distinction + that the reorganization could erase. Compare both allowed and forbidden + outcomes. Check whether the change accidentally forbids a valid implementation + or admits one of the formerly rejected wrong designs. +5. When an independent preservation check is useful and authorized, give a fresh + agent the original obligation, its sources, and one candidate. Hide sibling + candidates and the author's preferred outcome. Ask what insight survives, + what is overturned, and what smallest repair is needed. Without that separation, + describe the result as a trace audit, not independent validation. + +**Return:** the proposed organization, the old-to-new obligation mapping, +witnesses and their evidence status, losses, and unresolved design decisions. +Keeping the existing laws separate is a valid result. + +**Control:** reconstruct each original obligation from the candidate and check +its discriminating witness. A semantic change may be justified, but it is a +contract change, not a refactor. Merging prose does not require merging oracles +whose models or production boundaries differ. + +**Distortion:** neat categories that erase obligations, speculative connections +presented as established, or preserving familiar wording instead of useful +insight. + +**Stop when:** the alternative's preservation and losses are explicit, or the +source obligations cannot be reconstructed. Do not adopt a replacement merely +because this operation produced it. diff --git a/packages/oracles/docs/instruments/mutant-gap-hunt.md b/packages/oracles/docs/instruments/mutant-gap-hunt.md new file mode 100644 index 0000000..27566d0 --- /dev/null +++ b/packages/oracles/docs/instruments/mutant-gap-hunt.md @@ -0,0 +1,111 @@ +# Mutant gap hunt + +**Use when:** a recent refactor, a suspicious green oracle, a survivor whose +missing history is not obvious, or an open review question leaves doubt that +the oracle would reject a plausible wrong behavior. One carefully chosen +mutant can answer a useful question. A campaign is optional. + +**Inputs:** + +- the changed code and the commits that shaped it +- the claimed law and its authority +- the oracle's prose and its `limits` +- the gap check's report, and any open review questions +- the public observation that could distinguish a mistake + +Keep test bodies and assertions out of the design brief. + +## Operation + +1. **Design plausible mistakes before you read the tests.** + - Start with semantic, type-plausible edits near the changed boundaries: a + lost operand, a wrong prior value, a premature clear, a duplicated event, + a changed order, or a skipped callback. + - Turn each open review question into a candidate. + - If code is called dead, try removing it. + - Spread candidates across the affected files and boundaries. Do not + cluster them where edits are easy. + - Hand-write candidates at boundaries a designer may miss, such as wrappers, + lazy getters, and caches. + - If you inspect the assertions before drafting, label the design as + self-generated. +2. **Make each edit replayable.** + - Record each mutant in the gap check's authored form: + `{ id, source, find, replace, intent, author }`. `find` must occur + exactly once in `source` at the pinned revision, and the result must + parse. + - Change one decision per mutant. A compound kill does not say which part + the comparison rejected. + - Run each candidate against an unchanged, pinned baseline, and restore + the code exactly. + - A crash outside the comparison, a timeout, a setup failure, or an + unreached edit is not a kill. Record which one it was, not a score. +3. **Interrogate every survivor.** A survivor is a gap. Find which one of + these it shows: + - **A missing history:** no legal history reaches the mutated decision, + or none reaches it under the condition where it matters. Name the history + and add the generator dimension or constructed history for it. + - **A missing observation:** the mutant changes something the driver does + not record. Record it at the checkpoint the contract owes it. + - **A missing or loose law:** the change is observed, but no law rejects + it. State the law and find its authority (use + [law discovery](law-discovery.md)). + - **Dead code:** no legal history can observe the decision. Remove the code. + - **An allowed alternative:** the observations differ, and the adopted law + already allows both, as an allowed set with its clause. The mutant is a + valid implementation, so do not strengthen the law or change production + to kill it. Report it with the mutant's key, the clause, and the history, + separately from the kills. Until the gap check recognizes allowed sets, + it stays open in the report. Only the framework closes it. + - **A law question:** the observations differ, and the contract only seems + to allow both. Do not record that as acceptable. Give the developer the two + observations and the history that shows them, and ask whether the law + should decide the choice. + - **A framework bug:** none of the above fits, such as a mutant that no + oracle could detect. Stop work on that gap, and report its key and + evidence. Calling a mutant invalid does not close the gap. + + There is no "equivalent" class, and no class where the agent accepts a + survivor. Probe the nearest unusual legal history before you call a path + unreachable. + + To confirm a missing history or observation, run one focused probe on both + revisions. Expect a pass on the baseline and a failure on the mutant at the + intended assertion. If the contract predicts the mutant's result instead, + the baseline is RED: investigate a product bug. Then remove the temporary + probe, and check that `git status` is clean. + +4. **Repair the oracle, when authorized.** + - Extend the oracle with the missing history, observation, or law. + - Show that the extension rejects the motivating mutant and a nearby + plausible wrong design, and still accepts every allowed alternative. + - For a bug on the baseline, show RED on production before its repair. + - Then run the gap check again. + +**Return:** for each chosen mutant: + +- its pinned revision and exact edit +- the observed run outcome, using ORC-006's categories in the + [checklist](../oracle-checklist.md) +- which step 3 class it showed, with the distinguishing + history + +Keep raw logs task-local unless the task asks to publish them. + +**Control:** public observations decide product claims. Check that each probe +reaches the path and the premise it names. A green suite does not prove that +no gap exists. Use a positive control when instrumentation reports that a +branch was never reached. Recheck types after a tool rewrites code. + +**Distortion:** + +- mutants chosen to satisfy the existing tests +- syntax failures counted as semantic kills +- private-state differences treated as product bugs +- an oracle extension that still accepts the motivating mutant +- a negative witness that accepts any failure +- a decision checked in one build mode while another mode can observe it + +**Stop when:** each selected mutant has an executed distinction, a removed +piece of dead code, a law question with the developer, or a framework bug +reported. Do not infer broad coverage from a kill count. diff --git a/packages/oracles/docs/instruments/tension-scan.md b/packages/oracles/docs/instruments/tension-scan.md new file mode 100644 index 0000000..a51b62b --- /dev/null +++ b/packages/oracles/docs/instruments/tension-scan.md @@ -0,0 +1,37 @@ +# Tension scan + +**Use when:** individually sensible demands appear to collide, or new evidence +may have changed an earlier conflict. + +**Inputs:** the obligations and their sources, the condition in which they +interact, and any earlier formulation. Record which statements are accepted +contracts, user goals, current behavior, or your own inferences. + +## Operation + +1. Establish why each demand matters independently. If only one has support, + return that asymmetry rather than manufacturing two equally sound sides. +2. State the collision concretely: we need A and B, but under C, satisfying A + undermines B through mechanism M. Work through a legal history or design + consequence; a pair of opposing slogans is insufficient. +3. Look for a missing distinction in scope, authority, ordering, ownership, or + checkpoint. Recompute the consequences with that distinction explicit. +4. Check whether both original obligations survive. Removing a guarantee can + be a proposed tradeoff, but does not demonstrate that the tension dissolved. +5. On a recheck, identify the evidence that makes the tension remain live, + sharpen, move, weaken, or dissolve. Read it against the original purpose; + recent or vivid material alone does not justify changing the question. + +**Return:** the sourced demands, concrete collision and mechanism, possible +missing distinction, and current status; or no supported tension, with the +missing support identified. Preserve materially different interpretations when +the available authority cannot choose among them. + +**Control:** restate the case without an A-versus-B framing. Does the conflict +still exist? Keep a generated collision separate from an observed one. + +**Distortion:** invented symmetry, apparent resolution by deleting an obligation, +or using an ordinary clarification to claim a major design discovery. + +**Stop when:** the collision or its dissolution is demonstrated at the stated +scope, or a specific missing fact or design decision is identified. diff --git a/packages/oracles/docs/intro.md b/packages/oracles/docs/intro.md new file mode 100644 index 0000000..fffd152 --- /dev/null +++ b/packages/oracles/docs/intro.md @@ -0,0 +1,58 @@ +# What an oracle is + +This page is for a developer who asks what an oracle is. An agent running a +first session does not recite it: it shows oracles working on the +developer's own code, and uses this page to answer questions. + +Types check that a value has the right shape. An oracle checks that the code +does what it promises, for every input and sequence of calls the promise +covers. It is engineering assurance for software: the way a bridge is +checked against its load rating, not against one truck. + +## From a test to an oracle + +A unit test records one answer. Give a sort function `[3, 1, 2]`, and check +that it returns `[1, 2, 3]`. You chose the input, and you supplied the +answer. + +A property test generalizes the input. A generator makes thousands of +lists, including empty ones, repeated values, and negative numbers. The test +checks one rule for each: the result holds exactly the original elements, in +ascending order. You still write the rule by hand. + +An oracle supplies the answer for any input. It is a small, independent +model that computes the expected result the simple way, even when production +uses complicated machinery to compute it fast. For a live query that keeps +the ten highest-priority open tasks up to date, the model keeps the tasks in +a plain `Map`, filters, sorts, and takes ten. The oracle gives the same +generated changes to the model and to the real code, and compares what a +user would see after each one. + +An isolated test preserves an answer. An oracle preserves the understanding +that produces the answers. + +## What oracles find + +Oracles find bugs that example tests miss, because the generator tries the +combinations nobody wrote down. They also find design questions: places +where the docs, the code, and the tests disagree, or behavior that nobody +decided. Writing the oracle forces each decision, and the oracle then holds +it. + +With an oracle in place, an agent can replace the machinery behind a promise +(a faster algorithm, less code, a different data structure) while the oracle +checks that the promise still holds. + +The gap check then shows what is left: production behavior that no law +judges yet. + +## If you know fuzzing or formal models + +Like a fuzzer, an oracle generates many inputs. Unlike a fuzzer, it does +not only wait for a crash: it compares every result with what the code +promises. Like a formal model, it states the rules the code must follow. +Unlike a proof, it checks the real code by running it, across the histories +it generates. + +For the story of how TanStack DB uses oracles, read +[TanStack DB 1.0 RC1: Going All In on Oracles](https://tanstack.com/blog/tanstack-db-1.0-rc1-going-all-in-on-oracles). diff --git a/packages/oracles/docs/model-styles.md b/packages/oracles/docs/model-styles.md new file mode 100644 index 0000000..08d2b31 --- /dev/null +++ b/packages/oracles/docs/model-styles.md @@ -0,0 +1,296 @@ +# Model styles, by example + +`oracles-design` step 3 lists six model styles in order. Use the first style +that can state the law. This page gives one small example of each. The +examples use made-up domains. Copy the shape, not the domain. + +Two rules hold for every style: + +- **Each model rule is one sentence of the law.** A reviewer can point from a + sentence of `law.statement` to the rule that enforces it, and back. +- **Legality is not the model.** If the grammar or `resolve` needs state to + pick legal actions (counts, phases, open handles), compute it in a separate + pure `legal(prefix, action)` guard. Keep the model a pure function from the + plan to the expected result. + +## 1. A pure function of the inputs or the whole history + +Use this by default. The expected observation is a formula over the plan. + +Law: "A page holds the rows that match the filter, sorted by `id`, from +`offset`, at most `limit` rows." + +```ts +type Row = { id: number; tag: string } +type PagePlan = { rows: Array; tag: string; offset: number; limit: number } + +function expectedPage(plan: PagePlan): Array { + return plan.rows + .filter((row) => row.tag === plan.tag) + .map((row) => row.id) + .sort((a, b) => a - b) + .slice(plan.offset, plan.offset + plan.limit) +} +``` + +The model ignores how production finds the rows (an index, a cursor, a cache). +A formula like this also forces you to state edge cases: an `offset` past the +end gives an empty page, and a `limit` of 0 gives an empty page. + +## 2. A naive reference structure and an abstraction + +Use this for a data structure with a simple abstract view. Keep a plain copy +of that view, and compare it with what production shows at each checkpoint. + +Law: "A cache of capacity `n` keeps the `n` most recently used keys. `get` +returns the stored value of a kept key, and `undefined` otherwise." + +```ts +type CacheAction = + { kind: 'set'; key: string; value: number } | { kind: 'get'; key: string } + +function expectedReads( + capacity: number, + actions: Array, +): Array { + // The reference: keys from least to most recently used. + const order: Array = [] + const values = new Map() + const reads: Array = [] + const touch = (key: string) => { + order.splice(order.indexOf(key), 1) + order.push(key) + } + for (const action of actions) { + if (action.kind === 'set') { + if (!values.has(action.key)) order.push(action.key) + values.set(action.key, action.value) + touch(action.key) + if (order.length > capacity) values.delete(order.shift()!) + } else { + const value = values.get(action.key) + if (value !== undefined) touch(action.key) + reads.push(value) + } + } + return reads +} +``` + +The driver records only what the contract exposes: each `get` result. If the +contract also promises iteration order, the driver records the key order +(the abstraction of production's state) at the same checkpoint. + +## 3. A transition table for a lifecycle + +Use this when the law names phases. Each row is one rule, with an id and its +authority. Reach is the set of rows that fired, so transition coverage is +measured, not guessed. + +Law: "A connection opens only from `idle`. A failure while connecting returns +it to `idle`. Closing is final." + +```ts +type Phase = 'idle' | 'connecting' | 'open' | 'closed' +type ConnEvent = 'connect' | 'connected' | 'fail' | 'close' +type TableRow = { + id: string + from: ReadonlyArray + event: ConnEvent + to: Phase + authority: string +} + +const rows: ReadonlyArray = [ + { + id: 'connect', + from: ['idle'], + event: 'connect', + to: 'connecting', + authority: 'README, Lifecycle', + }, + { + id: 'connected', + from: ['connecting'], + event: 'connected', + to: 'open', + authority: 'README, Lifecycle', + }, + { + id: 'fail', + from: ['connecting'], + event: 'fail', + to: 'idle', + authority: 'README, Lifecycle', + }, + { + id: 'close', + from: ['idle', 'connecting', 'open'], + event: 'close', + to: 'closed', + authority: 'README, Lifecycle', + }, +] + +function expectedPhases(events: Array): { + phases: Array + fired: Array +} { + let phase: Phase = 'idle' + const phases: Array = [] + const fired: Array = [] + for (const event of events) { + const row = rows.find((r) => r.event === event && r.from.includes(phase)) + // No row means the event has no effect in this phase. + if (row) { + phase = row.to + fired.push(row.id) + } + phases.push(phase) + } + return { phases, fired } +} +``` + +Return `fired` from `reached`, so each row id is a reach state. A row that +never fires shows a missing history, or a rule that the law does not need. + +A table costs more lines than a formula. Use it when the phases are part of +the contract, or when you need to show which transitions the histories cover. +A table may also call a formula for one rule, such as a timeout guard. + +## 4. A pure step function + +Use this for a stateful API whose answers depend on what came before, without +named phases. The step function handles every legal action. + +Law: "A withdrawal succeeds only when it does not exceed the balance. A +deposit always succeeds. Each call returns the new balance, or `rejected`." + +```ts +type Account = { balance: number } +type AccountAction = + { kind: 'deposit'; amount: number } | { kind: 'withdraw'; amount: number } +type AccountResult = number | 'rejected' + +function step( + state: Account, + action: AccountAction, +): { state: Account; expected: AccountResult } { + switch (action.kind) { + case 'deposit': + return { + state: { balance: state.balance + action.amount }, + expected: state.balance + action.amount, + } + case 'withdraw': + return action.amount <= state.balance + ? { + state: { balance: state.balance - action.amount }, + expected: state.balance - action.amount, + } + : { state, expected: 'rejected' } + } +} + +function expectedResults(actions: Array): Array { + let state: Account = { balance: 0 } + return actions.map((action) => { + const next = step(state, action) + state = next.state + return next.expected + }) +} +``` + +There is one `switch` on the action kind. A rejected withdrawal is an +expected result, not an illegal history, so the grammar generates it. + +## 5. Relations between runs + +Use this when the exact output is hard to compute. State a relation that must +hold between two runs, or between a run and its inverse. + +Law: "`parse(serialize(value))` equals `value` for every value in the domain." + +```ts +type Value = { name: string; tags: Array } + +function roundTripHolds( + serialize: (value: Value) => string, + parse: (text: string) => Value, + value: Value, +): boolean { + const back = parse(serialize(value)) + // Compare element by element. Joining the tags would accept [''] -> [] and + // ['a\u0000b'] -> ['a', 'b'], which lose information. + return ( + back.name === value.name && + back.tags.length === value.tags.length && + back.tags.every((tag, i) => tag === value.tags[i]) + ) +} +``` + +In an oracle, the driver records `value` and `parse(serialize(value))`, and the +check compares them. A relation can hold for a useless implementation: a +`serialize` that returns `""` and a `parse` that returns a fixed value +satisfy many symmetries. So each relation needs a calibration that shows it +fails. Good calibrations here are traces whose parsed tags lose an empty +string (`['']` becomes `[]`) or split one tag in two (`['a\u0000b']` becomes +`['a', 'b']`). + +Other useful relations: an incremental result equals a recompute from scratch, +and the order of independent updates does not change the final state. + +## 6. An allowed set of outcomes + +Use this when the contract leaves a choice open. Predict every allowed +outcome, cite the clause for each, and accept any member. + +Law: "After `save`, the store notifies `written` and `indexed`, in either +order (README, Events: the two listeners are independent)." + +```ts +type Notice = 'written' | 'indexed' + +function allowedNotices(): Array> { + return [ + ['written', 'indexed'], + ['indexed', 'written'], + ] +} + +function noticesAllowed(observed: Array): boolean { + return allowedNotices().some( + (allowed) => + allowed.length === observed.length && + allowed.every((notice, i) => notice === observed[i]), + ) +} +``` + +Never pick the order that production happens to use: that would reject a +valid implementation. A mutant that only switches production to the other +allowed order survives a sound oracle. The law already answers the question, +so report it as an allowed alternative: its key, the clause, and the history. +Do not strengthen the law to kill it. Until the gap check recognizes allowed +sets, it stays open in the report (see the +[mutant gap hunt](instruments/mutant-gap-hunt.md) card). + +## Mixing styles + +Mix styles when one law has parts of different shapes. For example, a +lifecycle table can call a formula for its timeout rule: + +```ts +// A row's guard that is a formula, not more table rows. +const retiresNow = ( + unheldSince: number, + now: number, + gcTime: number, +): boolean => now > unheldSince + gcTime +``` + +The formula states the timing rule in one line. The table keeps the phases +and makes each transition a row. diff --git a/packages/oracles/docs/oracle-checklist.md b/packages/oracles/docs/oracle-checklist.md new file mode 100644 index 0000000..31c11fd --- /dev/null +++ b/packages/oracles/docs/oracle-checklist.md @@ -0,0 +1,324 @@ +# Oracle checklist + +Version 1 (2026-11-10). Cite requirements as `ORC-NNN (checklist v1)`. + +This is the complete list of requirements for an oracle that uses +`@tanstack/oracles`. The review skill reads it in full. The design skill +cites it by ID. Other guides explain techniques; they do not add +requirements unless they cite an ID here. + +It was ported from TanStack DB's oracle rubric. The IDs are the same, so +existing citations keep working. [Changes from the rubric](#changes-from-tanstack-dbs-rubric) +lists every difference. + +## How to read a requirement + +`MUST` is required when its trigger applies. `SHOULD` is expected, unless a +review accepts an explicit, technically grounded reason to do otherwise. +`MAY` is optional. + +Only these uppercase words inside a requirement block create a requirement. +Lowercase words such as "must" in explanations do not. + +Each requirement says what **the framework enforces** and what is left for +the author and the reviewer. Do not work around an enforced part. A +workaround is a finding, even when the tests pass. + +An **important generated property** is an `oracleProperty` that protects a +reusable law, claims coverage beyond named examples, or demonstrates a +bug-class repair. A fixed regression or a bounded enumeration is not one +merely because it uses test data. + +Score the stated obligation. Do not score the presence of a heading, comment +template, or helper. + +## There are no exits + +No requirement here can be met by declaring something out of reach. + +- There are no equivalent mutants. A surviving mutant shows a missing + history, observation, or law, or dead code. +- An effect that no check observes means that the driver does not record it. + Record it, or remove the code that causes it. +- The only exclusions are probe-only code and the `super()` region of a + derived constructor. +- A gap that nobody can kill or remove is a framework bug. Stop and report it + to the maintainer with its key and evidence. A mutant that no oracle could + possibly detect is invalid, and the framework must not generate it. The + author does not decide that a mutant is invalid, and the label does not + close a gap. + +--- + +### ORC-001: Contract authority and limits + +- **Trigger:** A test computes or constrains an expected product result. +- **Obligation:** The oracle MUST identify the promised law, its authority, + and the limits of the claimed result. +- **Framework:** `law.statement`, `law.authority`, and `law.limits` are + required fields. +- **Acceptance evidence:** A reviewer can trace the expected result to an + approved API, contract document and section, or a decision with who + made it (name and GitHub login) and its date, + and can say what the oracle does not establish. Production output is + evidence; it is never authority. +- **Not required:** A complete list of every behavior outside the subsystem. + +### ORC-002: Independent judgment + +- **Trigger:** An oracle compares production with an expected result. +- **Obligation:** The expected result MUST be derived independently of the + production semantic machinery that it judges. +- **Author:** The model receives the plan and the trace. It may read the + inputs that the driver recorded, never production's outputs. The framework + does not enforce this; a review checks it. +- **Acceptance evidence:** The model, alternate implementation, or relation + does not import or reproduce the relevant production classifier, + transition, comparator, or state machine without a separately justified + trusted base. +- **Not required:** Independence of the driver. The driver is expected to call + the real production entry point. + +### ORC-003: Distinguishable oracle responsibilities + +- **Trigger:** Any oracle. +- **Obligation:** The oracle MUST be a literate program. + - Opening prose MUST state the law and its limits before the mechanics. + - Prose beside the model MUST explain why its independent rule predicts the + promised result. + - The oracle MUST explain its legal histories, production path, public + observations, and the checkpoint where the check compares, beside the + relevant code. + - The contract, model, history grammar, driver, and check MUST remain + distinguishable. + - A change to the executable law MUST update its prose in the same change. +- **Framework:** `oracleProperty` takes these as separate parts: `law`, + `histories`, `model`, `driver`, and `check`. Only the check can report a + violation. +- **Acceptance evidence:** A reader learns the contract from the prose and + code together. A compact oracle may use one opening comment and short notes + beside its code. +- **Does not satisfy:** Code alone, headings without explanations, or + explanations that exist only in a pull request. + +### ORC-004: Generated-history grammar controls + +- **Trigger:** A generated property claims coverage of a legal history or + input grammar. +- **Obligation:** The change MUST account for reconstruction, ablation, range, + and exclusion for that grammar. +- **Acceptance evidence:** + - **Reconstruction:** every known valid witness within the claimed scope can + be generated. + - **Ablation:** each declared axis, rule, or overlap has a stated semantic + contribution. + - **Range:** the bounded domains and marginal cases are stated. + - **Exclusion:** a nearby invalid history or state is named, and the grammar + rejects it. +- **Contract changes:** A change that narrows the grammar, a filter, or a + validity predicate removes histories from the claim. It is a contract + change, and it MUST say which previously legal histories it removes and why. +- **Evidence location:** Executable checks are preferred when they are cheap + to keep. Comments or pull-request evidence MAY carry the rest. +- **Not required:** One test per control, or a Cartesian product of unrelated + axes. + +### ORC-005: Production path and observation + +- **Trigger:** Any oracle used as product evidence. +- **Obligation:** The driver MUST exercise the named production entry point + and MUST record the promised public observation at the named checkpoint. +- **Framework:** The driver records observations with + `observe(checkpoint, value)`. Observations must be data; anything else fails + its history with an `OracleDataError`. +- **Acceptance evidence:** The intended path ran, and the check ran at the checkpoint. The observations + can represent the violations relevant to the law, including duplicates, + omissions, order, or partial output when those are contractual. +- **Information sufficiency:** When a concrete hypothesis says a public result + is owed at a checkpoint before the production path has the facts it needs, + the evidence MUST show whether those facts arrive by then. If they do not, + the claim is unresolved. A green comparison after later delivery MUST NOT be + cited as proof of the earlier result. +- **Not required:** Observing internal facts that the contract does not + expose. + +### ORC-006: Checker calibration + +- **Trigger:** An important generated property, or a claimed oracle repair. +- **Obligation:** The evidence MUST name at least one plausible wrong answer + or design and show that the relevant law rejects it. +- **Framework:** A `calibration` names a wrong answer, as a `corrupt` trace or + a mutant `driver`, and the law that must reject it. The calibration test + fails if that law does not report it. +- **Acceptance evidence:** The wrong answer reaches the intended checkpoint + and the law reports the distinguishing failure. +- **Conditional obligation:** When a production mutant is run, the evidence + MUST classify its outcome as one of: assertion failure, timeout, setup + failure, runtime failure outside the intended comparison, unreached path, or + survival. A survivor is a gap (see [There are no exits](#there-are-no-exits)). +- **Not required:** Checking in a mutant. + +### ORC-007: Fixed and random campaigns with direct replay + +- **Trigger:** An important generated property. +- **Obligation:** The same property, generators, driver, check, and budget + MUST run with fixed seeds and without a seed, and a failure MUST be + replayable directly. +- **Framework:** Enforced. A generated `oracleProperty` runs one test per + fixed seed and one random test with the same arbitrary, check, and budget. + A failure prints how to replay it. +- **Author:** Do not work around this with a raw `fc.assert`, `fc.check`, or `.prop` + in an oracle file. The `oracle/oracle-property-only` lint rule reports them. + +### ORC-008: Stateful-model minimality + +- **Trigger:** A change introduces, removes, combines, or splits state in a + stateful model. +- **Obligation:** The change MUST explain whether a legal continuation (one + or more actions) can distinguish states that the model proposes to treat as + equal, by a later observation or by which actions are legal. +- **Acceptance evidence:** A distinguishing history justifies retained state. + Removed state needs a reasoned argument that it cannot change a promised + observation or the legality of an action. +- **Not required:** An executable mutant for every model field. + +### ORC-009: Vocabulary mapping + +- **Trigger:** A model combines or splits production concepts, or introduces + a model-only term that could be mistaken for a production concept. +- **Obligation:** The oracle MUST map the differing concepts explicitly. It + MUST use the project's term for a shared domain concept, and the + [package glossary](glossary.md)'s term for an oracle concept. +- **Acceptance evidence:** A reviewer can translate model actions, states, and + checkpoints to production boundaries without guessing. +- **Not required:** A mapping for local data holders whose meaning is + unambiguous. + +### ORC-010: Failure fidelity and cleanup + +- **Trigger:** Shrinking, capture, or cleanup can replace or erase an oracle + failure. +- **Obligation:** The harness MUST preserve the original violated law and + checkpoint, keep cleanup diagnostics distinguishable, and release its + resources. +- **Framework:** Enforced for shrinking. Once a history fails one way, a + smaller history that fails another way counts as passing, so the reported + counterexample still shows the first failure. +- **Author:** The driver releases what it created before it returns or + throws. An `AggregateError` whose `cause` and `errors` keep the distinctions + satisfies this requirement. + +### ORC-011: Independent second formulation + +- **Trigger:** A reviewer names a plausible semantic fault that production and + the model could share, and a meaningfully different formulation could + distinguish it. +- **Obligation:** The change SHOULD add that formulation. If it does not, the + evidence MUST say why it is impractical and keep the risk in the oracle's + limits. +- **Acceptance evidence:** The second formulation states its equivalence or + metamorphic relation, including differences in ordering, projection, + duplicates, and empty results. +- **Does not satisfy:** Agreement between two structurally identical paths. + +### ORC-012: Review evidence + +- **Trigger:** A change claims that an oracle is new, repaired, or fully + audited against this checklist. +- **Obligation:** The review evidence MUST record the outcome of every other + applicable requirement and MUST name the requirements that do not apply, + each with a reason tied to its trigger. +- **Bug-class closure:** A claim that a change eliminates a bug class MUST name + its contract × history × production path × observation boundary, the + original and nearby distinguishing witnesses, and a wrong design rejected at + the intended checkpoint. Unresolved in-scope combinations of contract, history, production path, + and observation boundary go in the oracle's + `limits`. A known reachable counterexample prevents a closure claim. +- **Evidence location:** Evidence that decides the verdict and lives outside + the oracle MUST be in a versioned record tied to the reviewed commit. An + editable pull-request description alone does not satisfy this requirement. + +### ORC-013: Distinguishing witness for a reusable boundary law + +- **Trigger:** An oracle is cited as protecting a reusable conditional, + threshold, or range law beyond its named examples. +- **Obligation:** The evidence MUST show a legal witness that reaches the + law's premise, and a nearby in-domain witness whose observation + distinguishes the law from a plausible wrong boundary or consequence. If + those witnesses are unavailable, the claim MUST be limited to the tested + cases, and the gap MUST be stated in `limits`. +- **Acceptance evidence:** The evidence names the wrong rule and the case that + rejects it. For a numeric threshold, cases on opposite sides are not enough + if a misplaced threshold still produces every expected result. + +### ORC-014: Controlled-premise handoff + +- **Trigger:** A controlled provider or host supplies a premise that matters to + an oracle result, and that result is cited as evidence for behavior with the + real provider or host. +- **Obligation:** The evidence MUST identify that premise. It MUST either + point to a receiving witness in which the real provider supplies the same + premise, or state the cross-boundary claim as unresolved in `limits`. +- **Shared law:** When both environments check the same law, the receiving + witness SHOULD be another driver for the same oracle. It reuses the law, + model, histories, and check, and supplies only its entry point, premise + control, and observations. +- **Not required:** An end-to-end run inside every component oracle. An oracle + whose claim is limited to the controlled boundary needs no real-provider + witness. + +### ORC-015: Declared reach + +- **Trigger:** A change adds or changes an important generated property, its + histories, or its driver, or cites one as evidence for a bug-class repair. +- **Obligation:** The property MUST declare the states that its laws depend + on, and a history MUST be constructed to enter each one. +- **Framework:** Enforced. Every state in `states` must be named by a + constructed history. Every campaign runs each constructed history, and one + that does not enter its named state fails. Half of the generated histories + are also constructed, so the histories after each state are explored. A + state that random histories reach only by chance goes in `measured`, which + is reported but not required. +- **Author:** Reach is a predicate over the plan or trace, or a production + probe. A literal example is not reach. A `fixed` history is a regression, + not reach. A law that needs a state entered at least N times in each + campaign needs N constructed histories that enter it. +- **Acceptance evidence:** The run reports each declared state and how many + histories entered it. + +### ORC-016: Law-only oracle files + +- **Trigger:** A file defines an oracle, its histories, or a campaign, + including a compile-time type oracle. +- **Obligation:** The file MUST NOT contain ordinary example-based unit tests + or standalone concrete type assertions. Fixed regressions MUST be inputs to + the same model, driver, and check as the campaign, and MUST NOT state their + own expected answers. A type oracle MUST apply a reusable independent type + law to generated legal inputs. +- **Framework:** The `oracle/no-unit-test` lint rule reports unit tests in + oracle files. The gap check counts only kills through a registered + `oracleProperty` route; a plain test in an oracle file is reported as an + unregistered kill. +- **Does not satisfy:** Renaming a unit test, adding contract prose, or + wrapping fixed expectations in a property runner. + +--- + +## Changes from TanStack DB's rubric + +Each change below keeps the requirement's purpose. A citation of the +rubric's ID still names the same requirement. + +| ID | Change | Reason | +| ------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | +| All | "Framework" and "Author" notes added | Say what `@tanstack/oracles` enforces, so nobody re-implements or works around it. | +| All | Coverage-map owner entries replaced by the oracle's `limits` | Adopters have no coverage map; code ownership belongs to tools such as CODEOWNERS (maintainer, 2026-11-10). | +| ORC-004 | Narrowing a grammar, filter, or validity predicate is a contract change | Maintainer, 2026-11-10. | +| ORC-006 | "Equivalence within the tested domain" removed from mutant outcomes | There are no equivalent mutants (maintainer, 2026-11-10). | +| ORC-007 | Rewritten as an enforced requirement | `oracleProperty` runs the campaigns and prints the replay command. | +| ORC-008 | "A legal next action" becomes "a legal continuation (one or more actions)" | Two states can agree after one action and differ after two, such as a FIFO queue's tail. | +| ORC-009 | Oracle terms come from the package glossary | The project glossary keeps domain terms. | +| ORC-015 | Reach comes from constructed histories, not `createOracleReach` | `oracleProperty` is the only campaign API (maintainer, 2026-11-09). | + +TanStack DB adds its own requirements, such as its coverage map, on top of +this checklist in its repository. diff --git a/packages/oracles/package.json b/packages/oracles/package.json index b6e094e..ca0fdb7 100644 --- a/packages/oracles/package.json +++ b/packages/oracles/package.json @@ -48,6 +48,8 @@ "files": [ "dist", "src", + "skills", + "docs", "oracle-gaps", "!oracle-gaps/**/*.property.test.mjs", "!oracle-gaps/gap-report-model-oracle.mjs", diff --git a/packages/oracles/skills/oracles-design/SKILL.md b/packages/oracles/skills/oracles-design/SKILL.md new file mode 100644 index 0000000..5019a24 --- /dev/null +++ b/packages/oracles/skills/oracles-design/SKILL.md @@ -0,0 +1,635 @@ +--- +name: oracles-design +description: Design, write, or change an oracle with @tanstack/oracles. Start from approved potential laws, write the oracle file with its law block and code, choose a model, and build histories that reach the states the law needs. Show the evidence the change needs, and close every gap. Use for a new oracle, an oracle repair, or a production change in owned code. Also use for any change to a model, history grammar, driver, or check. +--- + +# Design an oracle + +An oracle states a law, generates the histories where the law matters, and +checks production against an independent model. Make the law clear enough +that someone could predict an observation without reading production. The +code comes after that. + +Read this skill in full, then read the [glossary](../../docs/glossary.md) +in full. It defines the terms that the laws and the evidence use. +The [checklist](../../docs/oracle-checklist.md) is the complete list of +requirements. This skill cites it by ID, such as ORC-004. + +**Work in this order.** Reviewers check the order as well as the result. A +shortcut moves the work into review, where each finding costs a fix, a new +witness, and another round. The common shortcuts are changing production +first, editing the model to match production, narrowing histories until a +failure goes away, and pinning one example. `oracles-review` §2 lists them all. + +## 1. Start from approved potential laws, then write the oracle file + +**A new oracle starts from potential laws that the developer approved**, usually +in a first session (`oracles-start`): the laws that the authority decides. +Open design questions may still be open: they are limits until the developer +settles them. If no laws are approved yet, probe the behavior and propose +them first, as `oracles-start` does, and stop for approval. Exploration and +review stay in the conversation, never in the oracle file. The file is +written with its law block and its code together. + +**Decide whether the developer must approve.** + +- **Strengthening an adopted law** (a missing history, observation, driver + path, or check) reuses the law as it is. Do not propose new laws, and do + not stop. Name the law in the evidence. Widening the histories inside the + law's stated domain is strengthening. Changing the stated domain, a + checkpoint, or `limits` is a law change. +- **Changing a law** (its promise, domain, checkpoint, assumptions, or the + freedoms it allows) needs a law delta. Show the developer the old and the new + law, with the probe or result that shows why. Stop for their review, then + rewrite the law block in the same change. +- **A new law, or missing or contradictory authority,** needs the developer's + approval of that potential law. An analogy to another documented rule is + not authority. +- **The developer's answer to an open question is its review.** Rewrite the law + block in the same step, and continue. A change the developer did not ask for + (you found that a law is wrong) needs the delta and a stop. +- **The task already approved these laws?** Do not ask again. An approved + potential law is an adopted law. + +**Production output is evidence, never authority.** A rule that exists only +because production does it is model hacking. When the law and the code +disagree, the adopted law decides, not the code. + +When the promise is unclear, or two readings predict different behavior, use +the [law discovery](../../docs/instruments/law-discovery.md) card. + +**Use the [design grammar](../../docs/instruments/design-grammar.md) card +when any of these is true.** Do not judge whether the law is hard. Check the +list when you propose the laws, and again after each gap check: + +- there are more than five laws for one boundary (adopted, observed, and + open ones all count; an open question that the oracle file states only as + a limit is not a law) +- a law has no new case that it predicts +- the model has more than one branch per law +- the work changes or simplifies an existing model +- the gap check leaves three or more gaps in one function + +Say in the evidence which trigger fired, or that none did. When a trigger +fires before the oracle is written, show the developer the design the card +produced, and get their approval with the laws. + +**Write the law block.** The law block is the file's opening comment. The oracle file is a literate +program (ORC-003), so write it as prose. That reader is a maintainer, or a developer who uses the library seriously and knows +the codebase. Name internal parts without introducing them, and do not +explain the language, the library's basic ideas, or what a reader of the +code can see at once. Spend the words on what such a reader could get +wrong. Do not write about "a careless reader": state the case. Do not paste the template's fields or a list of law IDs. +Write it as continuous prose in this order. Do not label the parts with +the names below ("The hook", "The mechanism"): they are for you, not the +reader. + +1. **What this oracle owns**, in one or two plain sentences, about 40 words + in all: the concrete promise it maintains, the public entry point and + checkpoints where the promise is owed, and the owned code. A reader who + stops here knows what the file is for. Name the promise itself. "The + stylesheet matches the options the README documents" names no promise. For + example: "This oracle owns `slugify`'s promise for titles, checked on the + string that `slugify(title)` returns. It owns `src/slugify.ts#slugify`." + +2. **The contract at a glance:** a short code block, right after those + sentences, that shows the public API promises this oracle maintains. A + reader takes it in at one look. Give it no title, and no sentence that + introduces it. Write each line as `call // → result`: + - the call is the public API, written the way a caller writes it, with + concrete values where they make the promise clear. + - the result is what the caller gets back or sees: a value, or a short + phrase. + - a condition goes at the end of the line, as "when ...". + - an open law ends with `(open)`. + - when several laws share one call, write the call once, and put one + `// → result` line under it for each law. + + Keep it to about five results. A call may span several lines. Never write + an assertion, a test helper, or a checkpoint expression there: those + belong in the code. The block never shows how production computes the + answer. For example: + + ```ts + slugify('Hello, World!') // → 'hello-world': lowercase words joined by '-' + typography({ scaleRatio: 2, baseFontSize: '16px' }) // → h1 font size 32px (ratio × base) + typography({ scaleRatio: 2, baseFontSize: '15.5px' }) // → h1 font size 31px (open) + ``` + +3. **The hook:** the one concrete case that shows where the promise is + easy to get wrong. Write it as code where you can. Choose its form by + the oracle's shape: + + | Shape | Hook | + | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | + | State, timing, or order with a surprising rule | A question that names the surprise: "Why does a ready transaction wait?" | + | A pure function with documented examples | The documented example, then the nearby case that a careless reading gets wrong: ratio 2 and base 16px give 32px, so what does base 15.5px give? | + | The obvious implementation is wrong | The plausible implementation and the one input where it fails: why a deep copy is not a JSON round trip | + + When two forms fit, choose the one whose case a reader is most likely + to get wrong. + +4. **The mechanism:** the two or three moving parts that make the + behavior necessary, in plain sentences. Give each a short name (the + window, the head, the applied receipt), and use the same name in the + law names. When you write the code, use it in the `states` and the + calibrations too. +5. **The laws:** each one sentence over its domain ("For every ..."), with + the source of its promise in the documentation's own words where you + can. Then the edges where it is easy to get wrong, and what a caller + sees when the law breaks. +6. **How the oracle knows:** what the model reads to predict each law, + which histories reach the edges, what the driver observes and does not + observe and why, and where the check compares. +7. **Limits and open questions**, last, each with its reason. + +Avoid the two generic openings: a summary that says the code does what +its documentation promises, and a law that restates the implementation. A +model of that law would be the same code, so it could not catch a defect. + +Keep the law block to about 40 lines of prose, plus the contract code block. +It never mentions a review or an open question: a point the developer has not +decided appears only as a limit, with its reason. Nor does it tell the story +of the session: no "the maintainer approved", "the developer deferred", or +"as the probe showed". A developer's decision is stated as the rule it made, and +a limit's reason is about the code ("`slugify` drops every character +outside ASCII"), not about who has not decided yet. When a developer's decision is +the authority for a law, write it as "Decision by (GitHub +), : ", the same way in the `law` option's +`authority`. On a team, a reader needs to know whom to ask. Take the login +from `gh api user --jq .login`, or ask the developer. Write the login without an +`@`, so that a copy into a pull request does not notify anyone. + +**Then write the code under it.** Each boundary condition from the probes +and the law worksheet becomes a constructed history. If there was no first +session, fill in the worksheet from `oracles-start` for each law first. Each `oracleProperty` carries one law: give +each law its own property, so a failure names the law it breaks. The +properties can share one model, grammar, and driver. Each law's sentence, +authority, and limits go in that property's `law` option. Put prose beside the model, the histories, +the driver, and the check that says why each one predicts or tests the +promised result. + +**Check the law block:** + +- From the prose alone, a reader can predict one observation of each law. + For a no-throw law, the observation is that the call returns. +- It starts with one or two sentences on what the oracle owns, then the + contract code block: public API calls and their results, with no assertion + or helper. Each line states the same law as its sentence. +- No law restates the code. +- Each law sentence matches the law the developer approved. If they disagree, the + adopted law decides: fix the prose, and tell the developer. A new heading for + a law is not a law change. A changed sentence, domain, or checkpoint is. + +Keep the law block, the `law` option, and the model in step: a change to one +updates the others in the same change (ORC-003). When a decision changes a +law during the work, rewrite the law block at once, not at the end. + +## 2. Freeze what the oracle must preserve + +Write these down. They stay fixed while you build the oracle. + +- The laws and their checkpoints, as the developer approved them. +- The owned production code: the files or functions the oracle answers for. +- The freedoms the contract leaves open, such as the order of independent + events, scheduling, or representation. The oracle must accept every + allowed choice. + +Do not weaken a check, history, or law in the same step that clears a +failure. Make each weakening its own decision, with its own reason. + +**Name the parts, then reconstruct.** Before you trust a model: + +- List its actions, its state, and its observations. Give each one its job: + the law sentence it serves, or the legality it decides. +- Write each model rule as one sentence of the law. Mark any rule that you + added rather than found in the authority. +- Rebuild the known examples (docs, issues, existing tests) with the + histories and the model. A known example that the model cannot reproduce + shows a missing part or a wrong rule. +- When two rules conflict, record the conflict as an open question for the + developer. Do not pick a winner. + +For the design grammar card, see the triggers in step 1. + +## 3. Choose the model + +Build the strongest model the approved laws need: one that predicts every +observation the laws make a claim about. The choice of model is yours, so do +not ask the developer to pick between a complete model and partial checks. + +Pick by the shape of the law, in this order, and use the first style that +can state it: + +1. **The observation is a function of the inputs or the whole history.** + Write a pure function from the plan to the expected observation. This is + the default. +2. **A data structure with a simple abstract view.** Write a naive reference + structure, such as an array, a Map, or a sorted copy, and compare + production's observable state with it at each checkpoint. +3. **A lifecycle or protocol with named phases.** Write a transition table + whose rows are `{ from, event, guard, to, effect }`. +4. **A stateful API without natural phases.** Write a pure step function, + `(state, action) → { state, expected }`, that is defined for every legal + action. +5. **The exact output is hard to compute.** State relations between runs, + such as a round trip, incremental equals recompute, or order independence. + Each relation needs a calibration that shows it can fail. +6. **The contract allows several outcomes.** Predict the **allowed set**, and + cite the contract clause for each choice. Never pick one outcome that the + contract leaves open. + +Mixing styles is fine. [Model styles](../../docs/model-styles.md) shows a +small example of each style. + +When two styles fit the law, draft both briefly, keep the one that states +the law most directly, and name the other in the law block, so a reader can +choose it later. + +**Structure rules:** + +- One dispatch per role. The resolver, model, and driver may each branch on + the action kind once. +- The model is pure. It takes the plan and returns expectations. It never + reads production or driver values, never mutates its input, and keeps no + hidden state. +- Reach is a predicate over the plan or trace, or a production probe. It is + not a side effect inside the model. +- Every model rule cites its authority. +- Use the project's terms for domain concepts. For anything the model + combines, splits, or invents, apply the glossary's model alignment rule + (ORC-009). + +**Banned patterns** (each is a finding): + +| Pattern | Enforced by | +| ------------------------------------------------------------------------------- | ---------------------------------- | +| B1: expected values captured from production (`toMatchSnapshot` and similar) | review (lint pending) | +| B2: a raw campaign (`fc.assert`, `fc.check`, `.prop`) in an oracle file | lint `oracle/oracle-property-only` | +| B3: `fc.pre` in an oracle file. Legality belongs in the grammar or `resolve`. | review (lint pending) | +| B4: a model that imports runtime code from production | review (lint pending) | +| B6: a check whose only assertions are "does not throw", "defined", or "settles" | review (warning) | +| B7: the same expression on both sides of a comparison | review (lint pending) | +| B8: a model that branches on a case name or a literal id from the grammar | review (warning) | +| Unit tests or fixed expected answers in an oracle file | lint `oracle/no-unit-test` | + +B1, B3, B4, and B7 are review checks until lint ships them. + +B6 is about a check that stands in for a richer law. When the adopted law is +itself "returns without throwing on valid input", its check may assert only +that, and it needs its own calibration. + +Techniques: [guide § The model](../../docs/guide.md#the-model). + +## 4. Account for every distinction + +Show what each law, model field, and model branch contributes. + +- **A law:** give a legal history that the other laws allow and this law + rejects. +- **A model state or field:** give a legal continuation (one or more + actions) after which two states that differ only in it must give different + observations, or allow different actions (ORC-008). A FIFO queue holding + `[1, 2]` and one holding `[1, 3]` agree after one dequeue and differ after + two. +- **A branch or exception:** give the history that needs it. If none exists, + remove it. + +If you cannot find a witness, look for a simpler representation before +adding more machinery. For example, three disjoint sets of ids that move +between each other may be one map from id to status. Count complexity that +you move into helpers, adapters, or mappings. Moving it does not remove it. +The [design grammar](../../docs/instruments/design-grammar.md) card shows how +to compare a simplification with the model it replaces. + +**Laws stay simple. History generators may be elaborate.** Reaching a hard +state can need a sophisticated generator. Put that work in the histories, not +in the model. + +## 5. Build histories that reach the law + +- **Grammar.** The fast-check arbitrary and `resolve` produce only legal + histories. Legality belongs here, never in a filter after generation. +- **Changing a filter or validity predicate is a contract change.** Narrowing + the grammar removes histories from the claim. Say which previously legal + histories it removes and why (ORC-004). +- **Reach.** Declare in `states` each state that a law needs, and give each + one a constructed history that enters it (ORC-015). Put a state that you + only count in `measured`. A literal example or a `fixed` history is not + reach. A constructed history must be one the arbitrary could generate, + within its bounds. Use a constant when the arbitrary can generate it and it + cannot collide with a generated value (a fixed boundary shape). Otherwise + derive it from the generated history (for example "the initial value plus + one"). +- **Every arbitrary must take its histories back.** Each `fixed` and + constructed history goes back through the arbitrary, and one that cannot + fails when the file loads, naming the history. Give each `.map` its inverse + (`.map(f, unmap)`, where `unmap` throws on a value it cannot map back), and + avoid `.chain`. Build objects with `fc.record` or `fc.tuple` over primitive + leaves. `fc.constant` and `fc.constantFrom` take back only primitives, or + the same object reference. +- **Choose the string unit that covers the law's domain.** fast-check's + default strings include lone surrogates, and slicing by code unit splits + astral characters. `fc.string({ unit: 'binary' })` gives every code point + except lone surrogates, control characters included. `unit: 'grapheme'` + gives printable graphemes only, so it leaves out control characters, and + a `fixed` history that uses one is refused. When the law is about any + JavaScript string, lone surrogates are a boundary condition that the law + must decide. +- **Cover the law's whole domain.** A mutant can only show what the histories + vary. If the grammar covers part of the domain the law states (for example, + fractions in quarter steps only), say so in `limits`. Better, widen it. +- **Probes.** Where production decides to enter a state, bind the state to a + production probe, and predict entry from the plan alone. +- **ORC-004 controls.** For a claim about generated coverage, account for: + - reconstruction: every known valid example can be generated + - ablation: what each axis or rule contributes + - range: the bounded domains and marginal cases + - exclusion: a nearby invalid history that the grammar rejects + +**Choose the run budget.** Random runs explore around the critical states. +They do not reach them. + +- **Cover every critical state by construction.** Each state that a law + needs is a declared state with a constructed history, and a probe where + production decides to enter it. Never raise `runs` to reach a state. With + an enumerated source (`histories: { enumerated, states, measured }`), every + history runs, so a required state needs no constructed history: the `reach` + test fails if no history entered it. No enumerated history's label may be + `reach` or `determinism`: those name the framework's own tests. +- **Need a state at least N times?** Give it N constructed histories. +- **Give at least one fixed seed** in `fixedSeeds`. The random campaign always + runs beside it. A kill that only the random campaign finds is a gap until you + keep its history as a fixed or constructed history. +- **Start `runs` at 100** for a fast, synchronous driver, and at 25 to 50 + for a driver that awaits timers or I/O. +- **Keep it fast.** The gap check runs every owning oracle once for each + mutant, so oracle time multiplies. Aim for each oracle file to finish in a + few seconds in a normal run. Lower `runs` before you remove a dimension. +- **Read the reach counts.** If a measured state is rare, add a constructed + history or change the generator's weights. Do not only add runs. +- **Run longer sweeps with the run multiplier** (`_RUNS_MULTIPLIER`), + locally or on a schedule, not by raising `runs` in the file. +- **Lowering `runs`, removing a fixed seed, or removing a constructed history + is a weakening.** Give its reason in the change. + +Techniques: [guide § Histories](../../docs/guide.md#histories). + +## 6. Write the driver and the check + +- **The driver** calls the real production entry point and records public + observations with `observe(checkpoint, value)`. + - It records production errors, unhandled rejections, and warnings as + observations, so the check can reject them. A mutant that crashes + production is not a kill. + - It observes the contract's return values. + - It records only data: primitives other than symbols, plain arrays and + objects, Maps, Sets, and Errors. A symbol, a Date, a function, or a class + instance throws `OracleDataError`. Convert it first, for example to its + description or its time value. + - It identifies kinds with brand checks (`node:util` `types`, or + `Object.prototype.toString`), not `instanceof`, when a law is about + prototypes. `instanceof` follows a corrupted prototype chain. + - It awaits the work it starts. + - It bounds its own run time and work, and throws a structured error when + a bound is passed. + - It waits on events, never on the wall clock. Control time with fake + timers, a virtual clock, or deferred promises, so that load on the + machine cannot change a result. + - When something else needs the model (a probe, a second oracle, a + reconstruction), keep the model in a test-free companion module, such + as `parse-model-oracle.ts`. An oracle file registers tests when it is + imported. + - It releases what it created. +- **The check** compares observations with the model at each checkpoint. It + is the only place that reports a violation. For exact values, use + `differs(...) ?? pass`. For an allowed set, accept any member. +- **Calibration.** Give at least one `calibration`: a wrong answer, as a + corrupted trace or a mutant driver, and the law that must reject it + (ORC-006). +- **Ownership.** Put the owned code on the property's `owns` list. An entry is + a file (`src/x.ts`) or one function and its static call tree + (`src/x.ts#run`, `src/x.ts#Class.method`). Own only what the laws cover. The + gap check judges only owned code, and the gap check's `--source` selects + files. + - A method of an object that a function returns (`mitt().emit`) has no name + of its own. Own it through the enclosing function (`src/index.ts#mitt`). + That entry also brings in the sibling methods: cover them, or report their + gaps as outside this oracle's scope. + - The tree follows what the TypeScript checker resolves. A call whose + target is typed only by an interface or a type parameter (`this.#queue.filter` + where `#queue: QueueType`) reaches no function body, so it is not followed. + If the law depends on that method, own the implementation too + (`source/priority-queue.ts#PriorityQueue.filter`). + - A default export is named `default`, in any form (`export default +function`, `export default () => ...`, `module.exports = fn`): + `index.js#default`. `exports.name = fn` is named `name`. + - Module-level data (`const UNIT_MS = {...}`) is its own unit, named by its + variable. An owned function that reads it owns it, and what its + initializer refers to. An entry can also name it: `src/index.ts#UNIT_MS`. + A destructured declaration (`const { a, b } = ...`) is not a unit: if a law + depends on one, own the whole file. + - When an entry pulls in another contract (a batching API in the same + file, or a helper in a default export's call tree that serves another + promise), write that contract's own oracle, or report its gaps as + outside this oracle's scope. + - Owned code that serves another module's contract belongs to that module's + oracle. Owned code with no behavior anyone observes is possible dead code: + investigate it as step 8 says. + - Several oracles may own the same function. A mutant counts as killed when + any of its owners kills it. + +Keep the law, its authority, and its limits in `law`, and prose beside each +part that says why it predicts the promised result (ORC-003). + +Put `owns`, the law, the model, the histories, the driver, and the check on +`oracleProperty`. An oracle file registers its tests only through +`oracleProperty`, or through a project helper that forwards a literal `owns` +list to it. + +The options of `oracleProperty` are typed in +`node_modules/@tanstack/oracles/src/oracle-property.ts` +(`OraclePropertyOptions`). Read them there. + +**Lint as you go.** Run the repository's lint on each oracle file and each +changed production file after you edit it, and again before you finish. Fix +every report. Do not disable a rule. +In a first session, the oracles lint plugin is not installed yet: run the +repository's own lint, if it covers the oracle file, and go on. + +Techniques: [guide § The driver and the check](../../docs/guide.md#the-driver-and-the-check). + +## 7. Show the evidence that fits the change + +The evidence depends on the kind of work: + +| Work | Evidence | +| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Fix a defect** | The extended oracle fails on unchanged production (RED). The repaired production passes (GREEN). Neighboring laws still pass. The original defect, as a mutant, fails the oracle. | +| **Refactor without changing behavior** | The oracle files are unchanged, or only strengthened, and pass on old and new production with the same seed. Every mutant key present in both gap runs that was killed on the old code is still killed. Every new key in a changed function is killed, or its survivor directive is acted on. Keys that disappear need nothing. No RED is expected. | +| **Add or strengthen an oracle over correct production** | Production passes. A calibration (a wrong answer or a corrupted trace) and a mutant fail the intended law. No RED is expected. | + +A refactor that needs a changed law, check, or history is not a pure +refactor: treat the law change first (step 1). + +**To fix a defect:** + +1. Run the extended oracle file on **unchanged** production. It must fail on + the defect. This is RED. A setup failure is not RED: fix the oracle first. + (Until the gap check reports a failing baseline, run the oracle file with + Vitest for this step.) +2. Change production, and rerun the oracle after each attempt. It also shows + when an attempt breaks a neighboring law. +3. Run the oracle again. It passes. This is GREEN. +4. Show that a mutant, such as the original defect, fails the + oracle. + +For a defect fix, do not change production before step 1. A later run does +not make up for a skipped RED. + +**Clean up production once GREEN.** The oracle now protects the law, so use +it to simplify: + +- Refactor the code around the change. Do not insert a new branch beside an + old one that does almost the same thing. +- Remove states, flags, branches, and recovery paths that no law needs. A + surviving mutant shows only that the current histories did not expose the + code. Before you remove it, use the mutant gap hunt to rule out a missing + history, observation, or law, and give the contract's reason that no legal + history can observe it. A green rerun of the same oracle is not that + reason. +- Prefer a change that shrinks the code, or leaves its size the same. If the + repository sets a size rule, such as a net-negative diff, follow it. +- For a larger cleanup, draft two or three designs that pass the same oracle + (a table instead of branches, one state field instead of three, a + recompute instead of a cache). Give each one's trade-off, and let the developer + choose when the choice is theirs. Do not keep only the first design you + find. +- Rerun the oracle and the gap check after each cleanup. + +Techniques: [guide § Evidence](../../docs/guide.md#evidence). + +## 8. Close every gap + +Run the gap check on the owned code, then work from its report. Exit 2 is a +refusal: do what its `fix:` line says. Exit 3 is a CLI bug: report it with +`/error.log`, and do not change the oracle. Every gap +comes with one instruction. Do what it says, within the rules below. + +- **There are no equivalent mutants.** A surviving mutant shows a missing + history, observation, or law, or it shows dead code. +- **An effect that no check sees** is one that the driver does not record, + or one that no history makes different. A changed condition can still lead + to the same result when both branches agree on every generated input. Then + add a history where they diverge. Otherwise, record the effect. The + `not-observed` directive gives both steps. +- **Code whose removal no legal history can observe is dead.** The gap check + alone cannot show this, because its histories are a sample. First rule out + a missing history, observation, or law. Then remove the code in the same + change, give the contract's reason in the evidence (it is a production + change), and rerun the oracle and the gap check. +- **A law that kills nothing** owns no mutated code, or its witness never + varies in the histories. Check its witness and its constructed history. If + the law still kills nothing, it may restate another law: compare the two. + If every change that breaks it also breaks the other law, it restates + that law: merge them, as a law change. The report's "First law to reject, + by law" table credits a kill only to the first law that rejects a + history. A later law on the same history is not checked, so a law with no + count is not shown to be useless. Before you restate or delete it, run hand + mutants aimed at that law alone. +- **The only exclusions are probe-only code and the `super()` region of a + derived constructor.** +- **A survivor that the adopted law allows is an allowed alternative.** If a + mutant only moves production to another member of an allowed set, report it + with its key, the clause, and the history, apart from the kills. Do not + strengthen the law or change production to kill it. It stays open until the + framework recognizes allowed sets. The developer decides whether that blocks the + work. +- **A stated limit does not close a gap.** If a survivor differs from the + original only on inputs that the law's limits leave out (a null-prototype + object, say), the gap is open. Report it with that reason, and recommend + widening the grammar and the law to say what the code promises there. If + that behavior is truly not part of the contract, the code that the mutant + changes is a production question for the owner. Either way, the developer + decides. +- **A gap that existed before your change still counts** when it is in a + function you changed. Report it with its law question. Do not take an exit. +- **Never propose an exit.** If a gap can be neither killed nor removed, it is + a framework bug. Stop work on that gap, report its key and evidence to the + developer, and keep working on the other gaps. +- **A mutant that no oracle could possibly detect is an invalid mutant.** The + framework must not generate it, so it is a framework bug: report it the same + way. You do not decide that a mutant is invalid, and the label does not close + its gap. +- **Validity follows the types.** The framework judges a mutant over the + domain that the TypeScript types admit, so `number` includes `-0` and `NaN`. + A narrower domain stated only in the law's prose, such as "positive + integers", does not make a mutant invalid. Cover the type's whole domain, + or narrow the type. +- **Never write "equivalent" or "needs a maintainer's decision" in your + report.** For a survivor you cannot kill, give the histories you tried and + what they showed, and follow its directive. +- **Temporary, a CLI limitation:** a mutant that makes production loop forever + synchronously shows as a timeout. The framework's breakage laws will judge + it, and they are not built yet. Report it as a known CLI limitation, not an + oracle gap. Do not add driver timeouts for it, and keep working. +- **Temporary, a known CLI bug:** a gap on a guard that is constant under + TypeScript's narrowing is an invalid mutant that the CLI should not have + generated. Two forms: a branch after `x satisfies never`, and the test for + the last member of an exhaustive union (after the other members' branches + return or continue). Record it in your report under CLI limitations, with + its line, and keep working. Do not remove the guard and do not change + production: the guard protects untyped callers. Every other mutant on that + line must still be killed. + +- **Few mutants are weak evidence.** If the gap check generates few mutants + for an owned function, or none on a law's decisions (such as a return + expression), write authored mutants for the laws it misses. Put them in a + directory and pass it with `--mutants `. The form is in the + [mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) card. +- **Three or more gaps in one function** fire the design grammar trigger + (step 1). Stop, read the card, and write its parts list for that function + before you add a history. Adding histories first is the shortcut "Skipped + design grammar". + +Use the [mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) card for +a survivor whose missing history is not obvious. + +## 9. Report the evidence + +The evidence counts only as a set. One item alone proves little. Give the +items that fit the kind of change (step 7). + +- **Law:** the law and its authority, and whether it was reused, changed, or + new. +- **Reach:** each declared state and how many histories entered it. +- **Before and after:** for a defect fix, RED on unchanged production and + GREEN after the change. For a refactor, the same oracle passing on old and + new production with the same seed. For a new oracle over correct code, + production passing. +- **Mutant:** a mutant, such as the original defect, and the law that + rejected it, plus the calibrations that fail. For a refactor, the shared + keys still killed, and each new key killed or acted on. +- **Gaps:** the gap check's totals, and any gap reported as a framework bug. +- **Limits:** the states, paths, and observations that the evidence does not + cover. + +Put the evidence block in the pull request description. When a change claims +that an oracle is new, repaired, or fully audited (ORC-012), also keep the +evidence in a versioned record tied to the reviewed commit. + +When review findings arrive, load `oracles-evaluate-review`. + +## Instruments + +Use a card when one of these uncertainties is real. There is no required +sequence. + +| Uncertainty | Card | +| -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | +| The promise is unclear, or two readings predict different behavior | [Law discovery](../../docs/instruments/law-discovery.md) | +| Production and the model may share a mistake | [Challenge review](../../docs/instruments/challenge-review.md) | +| Two obligations collide under some condition | [Tension scan](../../docs/instruments/tension-scan.md) | +| Laws overlap, or their concepts seem wrong | [Law restructuring](../../docs/instruments/law-restructuring.md) | +| The model keeps growing, a simplification is proposed, or the law has many interacting rules | [Design grammar](../../docs/instruments/design-grammar.md) | +| The oracle is green but may accept a wrong implementation | [Mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md) | diff --git a/packages/oracles/skills/oracles-evaluate-review/SKILL.md b/packages/oracles/skills/oracles-evaluate-review/SKILL.md new file mode 100644 index 0000000..f6b9664 --- /dev/null +++ b/packages/oracles/skills/oracles-evaluate-review/SKILL.md @@ -0,0 +1,183 @@ +--- +name: oracles-evaluate-review +description: Evaluate a review of oracle work done with @tanstack/oracles. Keep a lossless ledger of every finding, and check each claim against the code. Extend the oracle so it shows each confirmed bug on unchanged production, then fix production and close the bug class. Use when review findings arrive for a branch that touches an oracle or owned code. +--- + +# Evaluate a review + +Use a review to close the class of bug it reveals, not only the reported +case. A passing reproduction or a complete ledger is a diagnosis. Closure +needs an oracle that rejects the original behavior and its neighbors. + +Read this skill in full. Before you edit any oracle, load `oracles-design`. +Use the [glossary](../../docs/glossary.md) terms in the ledger and the report. + +## 1. Write the ledger first + +Before you test or edit anything, write an append-only ledger of the raw +review in a task-local scratch file. Do not commit it unless the developer asks. + +- Give every item a stable ID in source order: each finding, footnote, capped + item, non-blocking comment, and testing idea. +- Keep the original claim and the proposed fix. Do not merge overlapping + items. Cross-reference them. +- Record the file and line, the severity, and the commit you check against. + +For every item, keep three judgments apart: + +1. **Is it true** on the reviewed or the current commit? +2. **What does this change do about it:** fix now, already fixed, ask for a + decision, or no action? +3. **What lasting value does it have:** a test idea, a law, a cleanup, or a + later investigation? + +Use only these dispositions: `fixed-now`, `confirmed-open`, `stale` (with the +surviving idea recorded), `refuted` (with evidence), `accepted-design` (only +for behavior an adopted law or decision covers, cited), `design-decision` +(with the exact question), and `duplicate` (linked, with its unique details +kept). There is no `skip`. + +Deferral is rare. A confirmed, in-scope defect stays `confirmed-open` until it +is fixed, unless the developer agrees to postpone it. A deferral records the open +work, why it cannot be done now, the remaining risk, an owner, and the +condition for resuming. + +## 2. Sort the findings + +For each confirmed behavioral finding, decide which kind it is. + +- **An enforcement gap: fix it, do not ask.** The adopted law is right, but + the oracle does not enforce it or production breaks it. The cause is a + missing generator dimension, a missing history or timing, a missing + observation, an unreached path, a driver seam, a check that passes when it + should fail, or a production defect against an adopted law. +- **A law change: present it and wait.** The finding questions what is + promised: a new law, a narrower or revised law, a new checkpoint, an + obligation on a collaborator, two laws in conflict, or a rule with no + authority. Give the developer the candidate law and its rival, the evidence, + what each choice changes for callers, and your recommendation. + +A finding can contain both. Fix the enforcement part, and present the law +question. + +An enforcement gap reuses the adopted law, so `oracles-design` step 1 does not +stop for it. Name the law in the ledger and the evidence. + +## 3. Plan the oracle changes for the whole review + +Before you work on single findings, look at all of them together. Findings +that look separate often share one missing history dimension, schedule, +driver path, or observation. + +1. Group the findings by the state, boundary, history dimension, or + observation they share. +2. For each finding, name the oracle change that would have generated it, + and why the current oracle missed it. +3. Choose the smallest set of oracle changes that generates every finding. + Prefer one dimension that reaches several findings and their neighbors + over one pinned example per finding. +4. Make those changes first. Run the extended oracle on **unchanged** + production. Record which findings it reproduces (RED), which it misses, and + any new counterexamples. A miss is a gap in the plan: revise the plan. +5. Then fix production, with the extended oracle as feedback on all findings + at once. + +## 4. Turn each bug into a family + +A reviewer's bug shows one history. The oracle must cover the family it +belongs to. + +- **State the causal condition, in writing:** the relationship that makes + the history fail, as one sentence over histories: "For every history where + a completion outlives the operation that started it, ...". Then write one + history, different from the reported one, that the sentence says also + fails, and run it. +- **Add it to the grammar:** a generator dimension or a constructed history + that varies the incidental details (counts, order, values, timing) and + keeps the condition. A pinned example alone is not the fix. Keep the + original history as a `fixed` regression. +- **Search the neighbors:** run the extended oracle and look for related + failures that the reviewer did not report. Record each one as a separate + ledger item. +- **Keep a nearby passing case:** a history close to the bug that must still + pass, such as the same operations in a legal order. It stops the repair from + rejecting valid behavior. + +## 5. Use the right evidence + +Do not accept or reject a behavioral claim from reading code alone. + +| Claim | Evidence | +| --------------------------------------- | --------------------------------------------------------------------- | +| Wrong product behavior | A failing oracle history or focused probe on the same path. | +| A check that passes when it should fail | A wrong implementation or corrupted trace that the check accepts. | +| Missing generator coverage | Reach counts or a sample showing that the regime never occurs. | +| Async or lifecycle | Controlled promises, events, or a controlled clock, not timing alone. | +| Work or performance | Deterministic counts, not elapsed time. | +| Docs or maintainability | Direct source evidence and a focused check. | + +If the current code passes, check whether the review targeted an older +commit, missed a precondition, or found a bug that a later change turned into +another one. Mark a claim `refuted` only when a probe reaches the same path +and every stated precondition and disproves it. Otherwise ask the reviewer for +the smallest failing history. + +For each confirmed bug: + +1. Show RED: the extended oracle fails on unchanged production. +2. Make the simplest production change that enforces the law across its + histories. +3. Show GREEN: the same oracle and its neighbors pass. +4. Show that a mutant, such as the original defect, fails the oracle. +5. Run the gap check on the owned code, and close every gap + (`oracles-design` step 8). + +Mark the item `fixed-now` only when the repair and the oracle coverage are +both complete. If only the reported history is fixed, keep the class open. + +## 6. Account for the whole law + +For each law the review touched, answer: + +- Is it fully in the model and in reachable histories? +- Is it checked at every checkpoint the contract owes? +- Which mutant shows that the check is sensitive? +- Which neighboring laws does the repair keep? + +A missing link is open work. A neighboring green test does not supply it. + +If the branch moves during the evaluation, mark earlier evidence stale and +rerun the affected items on the new commit. + +## 7. Audit for losses + +Before the final report: + +1. Return to the raw review, not a summary. +2. Account for every item, including footnotes and items you dismissed. +3. Compare each item with the final diff. +4. Recheck every law and sibling failure against the final model, histories, + driver, checkpoints, and calibration. +5. Check the totals: raw items = fixed-now + confirmed-open + stale + + refuted + accepted-design + design-decision + duplicate (+ any agreed + deferral). + +## 8. Report + +Lead with the law changes that need a decision. Then: + +- **Laws:** for each law, what the oracle proves, the decisive experiment + (RED, the neighbor, the mutant), why the suite missed it, and what remains + open. +- **The ledger:** every item, with its verdict, action, and lasting value. +- **The reviewer:** accuracy, depth, signal-to-noise, and fix quality, with a + hire recommendation and evidence. +- **Changes:** what you changed and the exact verification. Keep RED, a + mutant's result, and the repaired branch's GREEN separate. +- **Losses:** the totals by disposition, and every open evidence or coverage + gap. + +If the developer asks whether the pull request is ready, give a separate verdict: +`ready`, `not ready` (a confirmed in-scope defect or an open decision blocks +it), `readiness unproven` (a decisive witness is missing), or `not assessed`. +A complete ledger and a green run do not make a pull request ready. diff --git a/packages/oracles/skills/oracles-review/SKILL.md b/packages/oracles/skills/oracles-review/SKILL.md new file mode 100644 index 0000000..afe0428 --- /dev/null +++ b/packages/oracles/skills/oracles-review/SKILL.md @@ -0,0 +1,174 @@ +--- +name: oracles-review +description: Review oracle work done with @tanstack/oracles. Challenge whether an oracle enforces its law, whether it is the right law, and whether the work took a shortcut. Use for a new or changed oracle, or a production change in owned code. Also use for a proposed change to what an oracle promises. +--- + +# Review oracle work + +Ask two questions: does the oracle enforce its stated law, and is that the +right law to promise? A passing oracle can coexist with a wrong law. A +failing one can coexist with correct production. + +Read the [checklist](../../docs/oracle-checklist.md) and the +[glossary](../../docs/glossary.md) in full before you judge anything. Judge +conformance by the checklist's requirements, not by headings or templates. +After a handoff or a context reset, read them again: another agent's claim +that it read them does not give you their definitions. + +The repository may add its own review rules on top of this skill, in its +`AGENTS.md` or its own review skill. Apply both. + +## 1. Rebuild one observation yourself + +- Follow the law's authority to its source. Read the neighboring promises + there. The oracle's summary can omit an obligation that it should check. +- From the contract alone, without production, predict one promised + observation for one history. +- Then trace what the oracle actually does with that history: + - the model's rule + - the legal histories that reach it + - the production path the driver calls + - what the driver records + - the check at the checkpoint + +Every claimed model output must reach the check at its promised +checkpoint. A final state can hide a wrong intermediate observation. + +## 2. Check the order of the work + +These are the shortcuts. Report each one as a finding. A change that +contains one is not ready. + +| Shortcut | What to look for | +| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Law after code** | An oracle written without approved potential laws, a law with no authority, or a law block that disagrees with the `law` option or the model. Also a law block that is a list of fields or law IDs, not prose (ORC-003), or one that carries review material or open questions. | +| **Law as examples** | A law written as a list of cases, not one sentence over the domain. A proposed law with no new case that it predicts, an empty template field, a table in place of the blocks, or no block for a law people forget. | +| **Skipped design grammar** | A trigger from `oracles-design` step 1 fired when the laws were proposed or after a gap check, and the design grammar card was not used. | +| **Production first** | A defect fix in owned code without a RED result from the extended oracle on the code before the change. A later run does not count. | +| **Unproven refactor** | A refactor of owned code without the refactor evidence in `oracles-design` step 7. A refactor needs no RED. | +| **Re-asking an adopted law** | Stopping for approval to strengthen enforcement of a law that is already adopted, or that the task already approved. | +| **Model hacking** | A model rule changed to match production output without an authority. Production output is evidence, not authority. | +| **Unreachable claim** | A claim about a state that no constructed history enters, or a reach count nobody measured. Also a state that a law depends on but that sits in `measured`, where nothing requires it. | +| **Quiet exclusion** | Any skip, narrowed grammar, added filter or validity predicate, relaxed comparison, or exemption, other than probe-only code and the `super()` region of a derived constructor. Adopters have no exclusion list. | +| **Pinned only** | A fixed example where the law needs a generator dimension. | +| **Lowered budget** | Fewer `runs`, a removed fixed seed, or a removed constructed history, without a stated reason. | +| **Inserted, not refactored** | A production change that adds a branch, flag, or path beside a similar old one. Or a change that leaves code that no law needs. | +| **Unit test in an oracle file** | An ordinary test or a fixed expected answer in an oracle file. | +| **"Pre-existing" as a dismissal** | A reachable counterexample labeled as older than the change, and then left without a fix or a decision from the developer. | +| **No mutant** | A claim that a defect is closed, without a mutant, such as the original defect, that fails the oracle. | +| **An exit** | Any reasoning that calls a mutant "equivalent", an effect "unobservable", or a gap acceptable, or that counts a gap as closed because a stated limit covers it. A gap is closed only by a kill or by removing dead code. A gap that can be neither is a framework bug to report. | + +The evidence counts only as a set, chosen by the kind of change +(`oracles-design` step 7): law, reach, before-and-after runs, mutant, gaps, +limits. A missing item is a finding, even when the rest is strong. + +## 3. Hunt what the tools cannot see + +Lint, reach counts, the gap check, and calibrations miss the weaknesses +below. Hunt for each one across the whole oracle, not only the diff. This +hunt is your main job. + +- **Run but not checked.** A history executes the code, but the check + never observes its effect. Reach passes anyway. Find a line that the oracle + claims, and name the mutant on it that would survive. +- **A model that copies production.** The model mirrors production's + branches, classifications, or helpers, so both agree on the same wrong + answer. Compare the model's structure with production, not only its + results. +- **A grammar narrower than the law.** The law claims a value domain, action, + or case that the grammar never generates. Examples: + - a type that the generator never produces + - an exclusion stated in prose but missing from `limits` + - a size setting that makes part of the claimed range unreachable + + Check each claim in the law against a generator dimension. + +- **Easy declared states and probes.** A declared state or probe sits where + it is always true, or uses a weaker condition than the state it names. Its + count rises without showing that histories enter the interesting state. + Check each definition against the state that the law needs. +- **A late or coarse checkpoint.** The check runs only at the end of a + history, or observes only a summary, so a wrong intermediate observation + disappears. Ask whether a caller could see the wrong value before the + checkpoint. +- **A limit that hides in-scope work.** A stated limit excludes a legal case + that the law promises. Check that each limit excludes exactly what its + reason covers. +- **A dismissed survivor.** A surviving mutant is called equivalent, + unobservable, or out of scope, or is called invalid and treated as closed. + That is an exit, and a finding. The survivor shows a missing history, + observation, or law, dead code, or a framework bug. Reporting a suspected + invalid mutant as a framework bug, with its key and evidence, is correct and + is not an exit. +- **An invariant guard without a witness.** A throw for a contradictory + collaborator signal or an impossible state has no history that reaches it. + Require a history step that simulates the contradictory signal. + Expect production to fail at the boundary, before it releases established + state or publishes success. + +## 4. Choose challenges for the real uncertainty + +Use the [instrument cards](../../docs/instruments/) when they fit. Do not run +every card, and do not require a finding at every level. + +- **[Challenge review](../../docs/instruments/challenge-review.md):** + challenges enforcement, the law's statement, its suitability, or its + concepts. Name the level each challenge addresses. +- **[Law discovery](../../docs/instruments/law-discovery.md):** for a law with + no clear authority, or two readings that predict different behavior. +- **[Tension scan](../../docs/instruments/tension-scan.md):** for demands that + collide under a concrete condition. +- **[Law restructuring](../../docs/instruments/law-restructuring.md):** for + whether merging, splitting, or replacing laws gives a better account. +- **[Design grammar](../../docs/instruments/design-grammar.md):** for a + simplification or a growing model: what each part does, and what the change + newly allows or rejects. +- **[Mutant gap hunt](../../docs/instruments/mutant-gap-hunt.md):** for whether + a plausible wrong implementation can pass. + +Ask these questions of every model and grammar: + +- **What does each distinction contribute?** For each law, model field, and + branch, is there a legal history that needs it? If not, it is a candidate + for removal or for a simpler representation (ORC-008). +- **Is the complexity in the right place?** Laws stay simple, and generators may + be elaborate (`oracles-design` §4). +- **Did the domain shrink?** A changed filter, validity predicate, or grammar + bound removes histories from the claim. That is a contract change, and it + needs its own reason (ORC-004). +- **Does the oracle keep the contract's freedoms?** Where the contract allows + several outcomes, the model must predict the allowed set. An oracle that + accepts only the current implementation's choice is a finding. + +When you can, use a fresh reviewer for an independent challenge. Give it the +work, the sources, and the task, but not the author's preferred verdict. +Label a self-review as one. + +Techniques: [guide](../../docs/guide.md), especially [§ Evidence](../../docs/guide.md#evidence) and [§ Reusable boundary laws](../../docs/guide.md#reusable-boundary-laws). + +## 5. Report findings + +For each finding, give: + +1. The claim you challenge. +2. Its source or authority. +3. The smallest legal witness, or a concrete design consequence. +4. The evidence status: run, read from source, or a proposed experiment. +5. The repair, or the next experiment. + +- **A confirmed bug:** say why the existing oracle missed it, and name the + smallest general improvement: a generator dimension, an observation, or a + law. A pinned example alone is not the repair. +- **Defects and proposals stay separate.** A defect breaks an accepted + contract. A proposal changes the contract. A proposal must name the + external standard or engineering reason behind it, and its cost. Existing + behavior does not prove itself correct, and a preference does not prove a + violation. +- **Setup failures and timeouts are not kills.** Nor are they evidence that a + law rejected a mutant. + +Report the supported findings and the remaining uncertainty. Do not invent findings, retire laws, or rewrite expectations to look +decisive. + +If the review leads to oracle edits, load `oracles-design` before you edit. +To act on the findings, load `oracles-evaluate-review`. diff --git a/packages/oracles/skills/oracles-setup/SKILL.md b/packages/oracles/skills/oracles-setup/SKILL.md new file mode 100644 index 0000000..492100c --- /dev/null +++ b/packages/oracles/skills/oracles-setup/SKILL.md @@ -0,0 +1,446 @@ +--- +name: oracles-setup +description: Finish setting up a repository for oracles with @tanstack/oracles. Pin the packages, add the lint preset and the Vitest config, write the AGENTS.md block, make the oracle skills findable by coding agents, and run the oracle through the gap check. Use when the developer wants to keep the work of a first session (a commit, or a pull request), when a repository adopts @tanstack/oracles, or when it upgrades its pinned commit. Its install and file-placement sections also cover the minimal setup that a first session needs. +--- + +# Finish setting up a repository for oracles + +You are setting up oracle testing in this repository. Do the steps in order. +Ask the developer before each step that writes outside the repository. When no +developer is available, choose the default this skill gives, record each choice in +your report, and continue. That default covers setup questions only. +Potential laws have no default: propose them, report them, and stop. + +This skill has two uses. A first session (`oracles-start`) uses only its +install section and its "where oracle files go" section, for the minimal +setup that an oracle needs to run, and step 7's command, when its oracle +passes and it runs the gap check. Finishing the setup uses all of it. That +usually starts after a first session, when the developer wants to keep the +work: to commit it, or to open a pull request. If there is +no oracle yet, load `oracles-start` first. + +Setup ends when one oracle passes on production and the gap check leaves no +gap open in that oracle's own contract, unless the developer decides otherwise. A +stated limit does not close a gap. It does not end when the packages are installed. + +## 1. Read the repository + +Find out, and tell the developer: + +- The package manager, and whether the repository is one package or a + workspace. For a workspace, ask which package to start in. +- The test runner. The framework runs on Vitest 4.1 or later, with fast-check 4. + If the package uses another runner, oracles run in a separate Vitest config + beside it. Do not change the existing suite. +- Where production source lives (for example `src/`). +- The lint setup: ESLint 9 or later with a flat config, an older ESLint with + `.eslintrc`, or no ESLint (for example Biome only). +- The Node version that CI and the `engines` field use. The package needs + Node 20 or later. If CI runs an older Node, tell the developer, and offer a + separate Node 20 job for the oracles. Do not change the existing jobs. With + no developer present, do not change CI: report the mismatch. +- The repository's TypeScript version. Do not change it (step 2). +- The package manager and its runner for package binaries: `pnpm exec`, + `npx`, or `yarn`. Use them wherever this skill shows `pnpm add` or + `pnpm exec`. +- Which coding agents the developer works with. Look for `.claude/`, `.agents/`, + `.codex/`, `AGENTS.md`, and `CLAUDE.md`, and ask. + +## 2. Install two pinned packages + +The packages are not on npm yet. Install both from pkg.pr.new, pinned to the +same commit of TanStack/oracles. If you were not given one, use the current +commit on main: the first 7 characters of the hash that +`git ls-remote https://github.com/TanStack/oracles refs/heads/main` prints. +Tell the developer which commit you pinned. + +```sh +pnpm add -D \ + https://pkg.pr.new/TanStack/oracles/@tanstack/oracles@ \ + https://pkg.pr.new/TanStack/oracles/@tanstack/eslint-plugin-oracles@ \ + fast-check@^4 vitest@^4.1 +``` + +Add `fast-check` and `vitest` beside them. The framework needs Vitest 4.1 or +later. If the repository already runs an older Vitest, upgrading it is the +developer's decision: ask, and with no developer present, do not upgrade. Report it as +a blocker. For lint, also add `eslint@^9` +if the repository has no ESLint, and `@typescript-eslint/parser@^8` if +nothing parses TypeScript yet. Name those majors: a package manager can pick +a prerelease otherwise. If the repository's ESLint is older than 9 (xo +included), install every package here with `--legacy-peer-deps` (npm), and see +the fallback in step 3. With that flag, npm skips peers, so check that +Vitest's own peer `vite` is installed, and add it if it is not. + +Never add, upgrade, or downgrade the repository's `typescript` for oracles. +The gap check uses its own TypeScript and never reads the project's, from +commit `d8adfee` on. Pin a commit at or after `d8adfee`. Oracle files need +TypeScript 5 or newer. If the repository's `tsc` is older, keep the oracle +files and the Vitest config out of its run (its `include` or `exclude`). +Report any typecheck errors that existed before setup as pre-existing. + +The gap check's own TypeScript installs as `typescript-for-tanstack-oracles`, +so it does not replace the repository's. In a repository with no TypeScript, +or after an incremental npm install, npm can still link its `tsc` into +`node_modules/.bin` until the next `npm ci`. Do not treat that `tsc` as the +repository's TypeScript, and do not add a typecheck that uses it. + +Use the repository's own package manager. With pnpm 11, an install that skips +a dependency's build script stops every later pnpm command +(`ERR_PNPM_IGNORED_BUILDS`), and pnpm writes an `allowBuilds` placeholder into +`pnpm-workspace.yaml`. Answer it, usually `false`, and tell the developer. + +A run that installs from a local tarball has no commit to pin or check. Say +which tarball you installed, and note that `package.json` now points at a +local path that must be replaced before anything is committed. + +To upgrade, change both URLs to the new commit at once. Then do step 4 and +step 5 again (copy each pointer's `name` and `description` again, since they +can change), and rerun the gap check (step 7). + +## 3. Configure + +**Framework.** The defaults use the `ORACLE` environment-variable prefix and +accept every property id. Call `configureOracles` only to change that, in a +Vitest setup file that runs before any oracle module is imported. + +**Where oracle files go.** Put oracle files in a test directory, named +`*oracle*.test.ts`, for example `tests/parse.oracle.property.test.ts`. If the +repository's main runner is not Vitest, check that it does not also pick up +the oracle files: + +- If the runner loads every file in its test directory (uvu, mocha), put the + oracle files in their own directory, such as `oracles/`. +- If its default glob matches `*.test.ts` (better-node-test, Jest), name the + oracle files `*.oracle.ts`, and include that name in the Vitest config. +- If the repository's own Vitest run would also pick them up (a config with + no `include`, or one whose setup needs a server), name the oracle files + `*.oracle.ts`, and give them their own `vitest.oracle.config.ts`. + +The gap check finds oracle files by their `oracleProperty` call, so the name +only has to match the Vitest config and the lint glob. + +- If the repository has no Vitest config, create `vitest.config.ts` for the + oracle files. Jest and mocha ignore that name. If its `package.json` has + no `"type": "module"`, name it `vitest.config.mts`, so that Vitest loads + it without a module-type warning. +- If another tool would pick up `vitest.config.ts`, name it + `vitest.oracle.config.ts`, and pass `--config vitest.oracle.config.ts` to + the gap check (step 7) and to the Vitest script. +- If the repository has a Vitest config, add the oracle glob to its existing + `include`. Keep the globs that are there: replacing them drops the + existing suite, since an `include` replaces Vitest's default. + +```ts +// vitest.config.ts +import { defineConfig } from 'vitest/config' + +export default defineConfig({ + test: { include: ['tests/**/*oracle*.test.ts'] }, +}) +``` + +If the production source uses syntax that Vite cannot parse (Flow types, +for example), add a Vite plugin to the oracle config that strips it, such as +one built on `flow-remove-types`. + +Add a script for it, such as `test:oracles`. Leave the existing suite +unchanged. In a published package, keep the new files out of the published +package (its `files` field or `.npmignore`): the configs, `AGENTS.md`, +`.agents/`, and `.claude/`. + +If the repository's own lint or formatter covers the oracle files (for +example, an older ESLint with Prettier over `test/`), keep the oracle files +out of its globs, or match its style. Say which in your report. + +**Lint.** Add both presets to the ESLint flat config: + +```js +import oracles from '@tanstack/eslint-plugin-oracles' + +export default [ + // ...the repository's existing config + oracles.configs.recommended, + oracles.configs.production, +] +``` + +- If you spread a preset and set your own `ignores` on it, merge + `...preset.ignores` into them. The presets ignore tool configs + (`eslint.config.*`, `vite.config.*`, `vitest.config.*`, `*.oracle.config.*`). +- `recommended` applies to oracle files and turns off inline directives there, + so a disable comment has no effect. +- `production` checks the probe form in `src/**` and `packages/*/src/**`. If + production source lives elsewhere (for example `source/`, or a root-level + `index.js`), add + the rule with that glob. The config object must register the plugin itself: + + ```js + { files: ['index.js'], plugins: { oracle: oracles }, rules: { 'oracle/probe-form': 'error' } } + ``` + +- If the repository lints with xo, keep the oracle files and the new config + files out of xo (its `ignores`), and lint them with the oracle ESLint + config. Do not install `eslint@^9` as a dev dependency: xo loads ESLint + from the hoisted copy, and a second major breaks it. Use the + `npx -p eslint@9` fallback below instead, and run it with + `--no-inline-config`: + xo's own disable comments in production files name rules that this config + does not load, and ESLint reports each one. +- **TypeScript 7:** typescript-eslint does not support TypeScript 7 yet. If the + repository's ESLint config cannot parse TypeScript, the oracle lint cannot + run. Installing `@typescript-eslint/parser` fails with `ERESOLVE`, since its + TypeScript peer range ends before 7. Do not force the install. Report it as + an environment limit, with that reason. Do not change the repository's + TypeScript. +- **TypeScript older than 4.8:** `@typescript-eslint/parser` 8 fails to + load, and the failure takes down the whole flat config. Treat it the same + way: an environment limit, reported, with no parser in the config. +- The plugin parses the modules an oracle file imports with that file's + parser. A `.js` oracle file that imports a TypeScript module gets an + `unparsedModule` report: write the oracle file in TypeScript, with a + TypeScript parser. +- If ESLint does not already parse TypeScript, add `@typescript-eslint/parser` + as a dev dependency. Set it as `languageOptions.parser` for every TypeScript + file that the presets cover: the oracle files, and the production source if + it is TypeScript. Otherwise a typed production file fails to parse before + the probe rule runs. For JavaScript production source, set the parser for + the oracle files only. +- Do not add an exemption. There is none. + +**An older ESLint (7 or 8, `.eslintrc`).** Do not upgrade the repository's +lint. Lint the oracle files with a separate ESLint 9: + +1. Install the plugin with `--legacy-peer-deps` (npm), or the package + manager's equivalent. +2. Write a flat `eslint.oracle.config.mjs` with both presets and a TypeScript + parser for the oracle files and the production source. +3. Run it with an ESLint 9 that the plugin can find: the plugin loads + `eslint` at run time, from where the plugin is installed. `npx -p eslint@9` + is not enough, because npx runs the repository's local ESLint when one + exists. With pnpm, use `pnpm dlx eslint@9 -c eslint.oracle.config.mjs + `. Otherwise, install `eslint@^9` in the + workspace package that holds the plugin, if that does not replace the + ESLint that the repository's own lint uses. If neither works, lint is an + environment limit: report it. Add a `lint:oracles` script. + +The config's parser import resolves from the repository, so it uses the +repository's own `@typescript-eslint/parser`. + +Run the linter once and confirm that it loads the rules. Then check that the +rules are live: a deliberately wrong line in an oracle file must produce an +error. Use `import fc from 'fast-check'` and then `fc.assert(...)`. The rule +follows the fast-check import, so an `fc` that is not imported passes. +Remove the probe again. Before the first oracle file exists, do this check through the linter's `--stdin` with +an oracle file name, since a glob with no files fails. + +## 4. Write the AGENTS.md block + +Add this block to `AGENTS.md` (create the file if needed). On an upgrade, +replace everything between the markers and keep the rest of the file. + +```md + + +## Oracle tests + +This repository tests its laws with `@tanstack/oracles`. + +**Who owns what.** The laws belong to the people who maintain this +repository: what the code promises, the authority for each promise, and its +limits. Propose a law, but do not adopt or change one without their +approval. The production code that the oracles own is yours. How it is built +is your call, and you will keep building on it. Make it the smallest, clearest +code that satisfies the oracles: merge repeated code, remove code that no law +needs (after the checks in `oracles-design` step 8), and restructure freely while the oracles pass. + +- To find where an oracle would help, or to propose a new one, read + `node_modules/@tanstack/oracles/skills/oracles-start/SKILL.md` in full and + follow it. +- Before you create or change an oracle, or change production code that an + oracle owns, read `node_modules/@tanstack/oracles/skills/oracles-design/SKILL.md` + in full and follow it. +- Before you review oracle work, read + `node_modules/@tanstack/oracles/skills/oracles-review/SKILL.md` in full. +- Before you act on review findings, read + `node_modules/@tanstack/oracles/skills/oracles-evaluate-review/SKILL.md` in + full. +- Lint each oracle file and each changed production file after you edit it. + Fix every report. Do not disable a rule. +- Oracle files: ``. Gap check: ``. + + +``` + +If lint is an environment limit (step 3), replace the lint line with "Lint: +not available here ()." Fill in the two placeholders in step 7, +once their values exist. A run that stops before step 7 leaves them in +place, and its report says so. + +Check for both `AGENTS.md` and `CLAUDE.md`. Claude Code reads `CLAUDE.md`, +and most other agents read `AGENTS.md`. If both exist, put the block in +`AGENTS.md` and make sure `CLAUDE.md` refers to it (a line `@AGENTS.md` +imports it). If only `CLAUDE.md` exists, ask the developer which file gets the +block. If neither exists, create `AGENTS.md` and a `CLAUDE.md` that holds +only `@AGENTS.md`, after asking. + +In a workspace with a package-local install, put the block in the package's +`AGENTS.md`, where its `node_modules` paths resolve. A root copy must use the +full paths, such as `packages//node_modules/@tanstack/oracles/...`. + +## 5. Make the skills findable + +The skills ship inside the package, so agents find them only through a +pointer. A pointer must read the copy that the chosen package installed: + +- **One package, or a workspace with a root install:** + `node_modules/@tanstack/oracles/skills//SKILL.md`. +- **A package-local install in a workspace:** + `/node_modules/@tanstack/oracles/skills//SKILL.md`, + for the package chosen in step 1. Check that the file exists there. + +Use that path in the `AGENTS.md` block (step 4) too. Offer the developer these +choices. They can pick more than one. + +| Choice | Writes | Use when | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | +| **AGENTS.md only** | Step 4's block | The developer's agents read `AGENTS.md`. This is always done. | +| **Repository pointers** | A pointer skill per skill in `.agents/skills/` and `.claude/skills/` | Everyone who clones the repository gets the skills. Commit them. | +| **Global pointers** | A pointer skill per skill in `~/.agents/skills/` and `~/.claude/skills/` (and any other skills directory the developer's agents read) | The developer wants the skills in every repository that uses `@tanstack/oracles`. | + +Write a pointer skill as `//SKILL.md` for each of +`oracles-start`, `oracles-design`, `oracles-review`, +`oracles-evaluate-review`, and `oracles-setup`. + +Before you write repository pointers, check that git does not ignore their +directories (`git check-ignore .claude/skills .agents/skills`). If one is +ignored, tell the developer: they can commit the pointers in the other +directory, or add an exception to `.gitignore`. + +- Copy `name` and `description` exactly from the package's `SKILL.md`, so the + agent selects it for the same requests. +- The body says only where the real skill is: + +```md +--- +name: oracles-design +description: +--- + +Read `/skills/oracles-design/SKILL.md` in full, and follow it. +It matches the installed version of `@tanstack/oracles`. If that file does not +exist, run the repository's install, then look again. If it is still missing, +`@tanstack/oracles` is not installed there: say so and stop. +``` + +- In a **repository pointer**, `` is the path above, relative + to the repository root, such as `packages/app/node_modules/@tanstack/oracles`. +- In a **global pointer**, write instead: "Find the nearest `package.json` at + or above the files you are working on whose dependencies include + `@tanstack/oracles`. Use the `node_modules/@tanstack/oracles` in that + package's directory." Then the pointer loads the right version in every + workspace package. + +Use pointers, not copies or symbolic links: a pointer always reads the +installed version, and never breaks before install. + +Before writing a global pointer: + +- Show the developer the exact paths. +- If a skill with the same name exists, show it and ask. Never overwrite it + silently. + +## 6. Build the first oracle + +Usually the oracle already exists from the first session. Keep it, and +continue with its gap check. If there is none, build it from the potential +laws that the developer approved: load `oracles-design`, and follow it from +its first step. + +A first session fixes each bug the developer confirms right away, oracle +first, and runs the gap check only when its oracle passes (`oracles-start` +step 5). When finishing the setup: + +If the first oracle fails on unchanged production under an adopted law, it +found a defect. The RED is the finding: report it with its replay line, and +tell the developer. Never weaken the oracle to make it pass. The gap check cannot +judge a failing baseline yet, so make the smallest local fix on a scratch +copy or a local branch (never pushed or committed to the developer's branch), run +the gap check there, and say in the report that it ran on the +patched code. With the developer's approval, follow `oracles-design` step 7's +defect path for the real fix. + +Put the oracle's `owns` list on its `oracleProperty` call. An entry is a file +(`src/index.ts`) or one function in it (`src/index.ts#parse`, or +`src/x.ts#Class.method`). A function entry owns that function and its static +call tree, including the module-level data it reads. A default export is +`src/index.ts#default`. For a first oracle, own only the function it covers. +If the gap check refuses with "no oracle file", the oracle is not +declared: that is a setup failure. + +In an oracle file, only `oracleProperty` registers tests, directly or +through a project helper that forwards a literal `owns` list to it. A plain Vitest test +there is not a kill, and the gap check reports it. Lint also reports an empty test +there: it checks nothing and looks like coverage. Remove it, or use `it.todo`. The driver records +production errors as observations: a mutant that crashes production is not a +kill until the check rejects the recorded error. + +## 7. Run the gap check + +Run the gap check on the owned source: + +```sh +pnpm exec oracle-gaps --package --source \ + --seed --out /.oracle-gaps/out +``` + +Add `--config ` (relative to the package) when the oracle Vitest config +is not Vite's default name. The baseline prints which config it used. + +- `--source` selects files. `owns` can name a single function in a file, so + the report judges only the owned functions. Code in the file that no oracle + owns appears as unowned warnings. +- Pass `--seed`, so the baseline and every mutant run the same histories. + Pass `--out` inside the repository. (Both become defaults later.) +- Add `.oracle-gaps/` to `.gitignore`. +- Read the exit code. **0:** the run completed; work from its report. + **2:** refused. It prints `refused: ` and one `fix: `. + Do what the `fix:` line says, then run again. **3:** an internal error, "a + bug in oracle-gaps", with the stack in `/error.log`. Report it as a CLI + bug with that file, and do not change the oracle to get around it. + +Run it after the oracle passes on production. Each gap in the report comes +with one instruction. Follow +that instruction, and follow `oracles-design` for what a gap means. There +are no equivalent mutants and no exits. A surviving mutant shows only that +the current histories did not expose that code. If you believe the code is +dead, first follow `oracles-design` step 8 to rule out a missing history, +observation, or law. Removing it is a production change, so tell the developer, +and rerun the oracle and the gap check. If a gap can be neither killed +nor removed, stop work on that gap and report its key and evidence to the +developer. + +Now fill in the two placeholders in the `AGENTS.md` block: the oracle file +glob and the gap check command you ran. + +## 8. Report + +Tell the developer: + +- what was installed (both pinned URLs) and where configuration changed +- which pointers were written, and where +- the first oracle's law and its result +- the gap check's totals, and every gap left, grouped as: + - open because the inputs that expose it lie outside the law's stated + limits: name those inputs, and recommend widening the grammar and the + law. The developer decides. Do not count these as closed. + - allowed alternatives (`oracles-design` step 8), apart from the kills + - framework or CLI limitations (for example a timeout from a synchronous + infinite loop, or a guard that TypeScript narrowing makes constant) + - other contracts that a whole-file entry pulled in, listed as outside + this oracle's scope + - every other gap, with its instruction +- any defect the oracle found, with its witness + +Do not call setup done while a gap is open in the oracle's own contract, +unless the developer decides otherwise. diff --git a/packages/oracles/skills/oracles-start/SKILL.md b/packages/oracles/skills/oracles-start/SKILL.md new file mode 100644 index 0000000..e3ab235 --- /dev/null +++ b/packages/oracles/skills/oracles-start/SKILL.md @@ -0,0 +1,520 @@ +--- +name: oracles-start +description: Run a first session with @tanstack/oracles in a developer's codebase. Say what the session will do, look around and present a few spots, probe the real API to prove each finding, propose a potential oracle and its potential laws, then build it and run it, fix confirmed bugs oracle first, and close gaps until it finds bugs or design questions. Present the work, and say what finishing the repository's setup would add. Use when someone is trying @tanstack/oracles, or asks where an oracle would help. +--- + +# A first session with oracles + +## Your goal + +Be an engaging teacher. The developer is trying this tool, often in the mood +of "impress me". Show, don't tell. Bring up interesting findings about their +own code, and explain what you are doing and why as you go. Tie each +explanation to something you just showed them: a probe, a finding, a law, or +the oracle file. Do not explain what oracles are unless they ask. +[What an oracle is](../../docs/intro.md) is there for when they do. + +Work toward these milestones, in order: findings from probes of their real +code; a potential oracle whose laws they approve; an oracle that runs on +their code; and that oracle finding a bug, or closing gaps until one turns +up, while they settle some design questions. Do not wrap up before the +oracle has found a bug or brought up a design question. The developer can +stop you at any time. + +Part of what the session shows, without saying so, is how an oracle works +through a codebase: each bug goes into the oracle first, the fix follows, +and the gap check points at the next thing to cover. + +Probes are how you learn what reading the code does not tell you: its +dynamics, boundary conditions, and constraints. You need that understanding +to build the oracle. Bugs and inconsistencies that the probes find along the +way are a welcome side effect. They also keep you honest: an agent that only +says things is often wrong, and one wrong claim costs the tool its +credibility. So every finding needs a probe that the developer can run again. + +## How to talk + +- Use these terms, and use them every time: a **probe**, a **finding**, a + **potential oracle** that owns a public behavior, its **potential laws**, + and the **oracle file**. Once the developer approves a potential law, it is + a **law**; once the oracle runs, it is the **oracle**. +- Explain each uncommon word in one line the first time you use it, tied to + what is on the screen. Do not explain them all up front. The words, and a + line for each: + + | Word | One line | + | ------------------- | ----------------------------------------------------------------------------------------------------- | + | probe | a small script that calls your real code with chosen inputs and prints what happens | + | finding | something a probe showed: a possible bug, two sources that disagree, untested behavior, or a question | + | law | one promise the code makes, as a "for every …" sentence, with where the promise is written | + | oracle | a test that checks the laws against your real code over many generated inputs | + | oracle file | the file that holds the oracle: the laws in prose first, then the code | + | model | the expected answer, computed the simple way, without your code | + | history | one generated input or sequence of calls that the oracle tries | + | constructed history | a history written by hand to reach one edge case every run | + | driver | the part that calls your real code and records what it does | + | check | the part that compares what your code did with the model | + | shrinking | cutting a failing history down to the smallest one that still fails | + | seed | the number that makes a run's random histories the same every time | + | calibration | a wrong answer planted in a run, to show that the check rejects it | + | owns | the functions an oracle answers for | + | gap check | makes small changes to a copy of the owned code, and shows each change that no law notices | + | gap | one such change: behavior the oracle does not check yet | + | authority | where a promise is written: the docs, a test, a comment, or your decision | + | limit | what the oracle does not check, and why | + + Use the same word for the same thing every time. Leave out a word the + session does not need. + +- Talk naturally. Never name a skill, a step, or a section ("step 3", "the + skill's fallback"). Say what you are doing, in your own words. +- Talk about their code in its own terms, not in this skill's words. Words + such as "dynamics", "constraints", "boundaries", "a behavior that's easy to + break", or "a simple rule behind it" are for you. The developer should hear + about timeouts, retries, cache entries, or whatever their code does. +- Ask in plain words. Explain any term the developer may not know in one + line (for example, a lone surrogate is half of a character that takes two + UTF-16 units). +- Ask one clear question when you need a decision. For a possible bug, ask + "Is this a bug, or is it intended?" When several decisions are waiting, + end with a short numbered list of them, most important first, so the + developer can answer by number. +- Show the developer only what they decide on. Your private notes, their + labels, and their statuses ("adopted", "open") stay private. Until the + developer approves it, a potential law is only a potential law. + +## Where you stop + +Stop and wait only when the developer must decide: + +- which of the spots you present to dig into +- whether a finding is a bug, or intended +- a design question +- whether to approve the potential laws +- whether to approve a more complex design (see below) +- whether to finish setting up the repository for oracles, when they want to + keep the work (a commit, or a pull request) + +Between those points, keep working, and keep telling them what you are doing. + +## What you change + +Until the developer wants to keep the work and finish setting up the +repository, change only these: + +- the minimal installs that an oracle needs to run: `@tanstack/oracles`, + `fast-check`, and `vitest` when `package.json` does not list them +- a minimal Vitest config that picks up the oracle file, only when the + repository's own config does not +- oracle files, with their prose +- the feedback log, `oracles-feedback.md`, while this is a preview +- the smallest production fix for a bug the developer confirmed, after the + oracle fails on it (see step 5) + +Findings, potential oracles, and potential laws stay in the chat. Keep +probes, and the gap check's report, in a temporary directory outside the +repository (`mktemp -d`). Say nothing about cleaning them up. + +## A feedback log, while this is a preview + +`@tanstack/oracles` is in preview, and its maintainers want to know where +this session was rough. Create `oracles-feedback.md` at the root of the +repository when the session starts, and do not commit it. +Append a short entry whenever a skill or doc was unclear, wrong, or +contradicted another; whenever you had to guess; whenever a command, +install, or config step failed or needed a workaround; and whenever +something took much longer than it should have. Each entry gives the time, +what you were doing, what happened, the skill file and section, and what +would have made it smooth. Keep entries short, and keep working: the log +is for later, not a reason to stop. + +## 1. Open + +Before your first message, read the block that finishing the setup adds to +`AGENTS.md`: it is in [`oracles-setup` step 4](../oracles-setup/SKILL.md), +under "Write the AGENTS.md block". It governs you from the start of this +session, though nobody has added it to the repository yet: who owns the +laws and who owns the code, which skill to read before each kind of oracle +work, and the lint rule. Until the setup is finished, the lint rule means +the repository's own lint: the oracles lint plugin comes with the setup, +and its absence is not a problem to report. + +Your first message starts with two or three sentences on what this session +will do, before any question. Write them for this repository, in plain words, +in your own sentences: name what the library does, which of its behaviors +you will try, how you will try them (calling the real code with chosen +inputs), and that you will then write an oracle with them for the rules you +agree on. Mention nothing about this skill or its steps. + +Then go straight on to look around. If their first message named a part of +the code, say that you will start there, and go on. They can name a part at +any time, and you start there. + +## 2. Look around + +If the developer named a part of the code, look closely there. Otherwise, +look around the repository, and make it quick. Read the public API and its +docs (the README, the API docs, the JSDoc), the existing tests, and the git +history of fixes. The files that fixes change most often tell you more than +a search for "fix": +`git log -i --grep fix --name-only --format= | sort | uniq -c | sort -rn | head`. +It is a starting point: skip build and config files, and read the commits +that changed the public API. + +Look for behavior that is easy to break, and whose right answer is simpler +to state than the code that produces it. Rank the candidates by how much a +bug there would hurt the people who use the code. Good signs: + +- the most important API: the call most users make first, and most often +- code the README's examples and getting-started section show +- behavior users see directly, and depend on +- a promise stated strongly (a guarantee in the docs, an invariant in a + comment), above all in code that has not changed in years: nobody has + checked it since it was written +- a rare edge case that would bring a user's system down when it hits +- code that git history shows was fixed more than once: people keep + reporting bugs there, so the tests are not capturing the whole picture +- a round trip (parse and serialize, encode and decode, copy and compare) +- a cache, a queue, or other state that changes over a sequence of calls +- an ordering, timing, or normalization rule + +Avoid an internal helper whose only law would restate its code. Avoid +behavior that nothing documents, unless the developer wants to decide it. + +Quick probes to choose the spots are fine. Show a result only later, with +the probe that proves it. + +Present two or three spots, a short paragraph each: what the code seems to +promise, where that is stated, why a bug there would matter, and why it +would make a good oracle. Recommend one, and stop for their choice. If they +already asked you to choose ("surprise me"), say in a line which spot you +picked and why, and go on to probe it. + +## 3. Probe + +Read the spot's docs, code, tests, and fix history. Then ask yourself: what +do I not know about this code that I need to know to build its oracle? How +does the state change with each call? Where are the boundaries? What holds +whatever happens? Show the developer those questions in the chat, a short +list, before you probe: it shows them how an oracle gets built. Write each +question about their code ("If the second try fails too, does it wait +longer than after the first?"), not as a category ("timing", "lifecycle"). Then build a +probe for each one, and answer each question with what its probe showed. + +**A probe** is a small script that calls the real public API with chosen +inputs and prints what happens. Import the code the way the repository's own +tests do, and run the probe with `node` from a scratch directory against +the unchanged code. If the import or Node gets in the way, open +[Problems you may hit](references/problems.md). Keep each probe and its +output. + +**Every finding needs a probe.** A claim without one is a suspicion: say so, +or leave it out. + +**Probe to find the design, not only bugs.** Look for: + +- **Constraints:** what holds whatever the input or the sequence (a round + trip, an order, uniqueness, a bound), and where it stops holding. +- **Dynamics:** how the result or the state changes with each call or + input. +- **Boundaries:** empty, zero, `-0`, limits, the first and the last item, + capacity, time, and odd input (digits, unicode, lone surrogates, keys that + shadow prototype names). +- **Inconsistencies:** places where the docs, the code, the tests, and the + examples disagree, and behavior that nobody decided. + +When a probe is cheap, run it over a grid or a random sample of inputs, +count the failures, and show the smallest one. + +**Find the authority for each promise, and show it.** Look in the README, +the API docs, the JSDoc, the tests' names and bodies, the changelog, commit messages that state an intent, source +comments, and any standard that the docs name. Quote it, with the file and +line. The authority is what makes a finding credible, and it becomes the +documentation in the oracle file. When nothing decides a behavior, it is a +design question, not a bug. + +The developer's answer to a design question is an authority too. Once they +decide it, it becomes a potential law, and the oracle can find where the +code breaks it, just as with a law from the docs. In a repository with few +docs, this is the usual way to a bug: ask the question, and let the oracle +check the answer. The oracle file records the decision with the +developer's name, their GitHub login, and the date, so that their team knows +whom to ask. +Find the login with `gh api user --jq .login`. If that fails, ask for it +with the first finding or design question you bring them, not with the +question of where to look. + +**Present each finding** in this shape, a light touch, with each finding +called a possible bug or a question, never a certainty: + +1. A title that names the behavior: "Retry delay: the second wait is + shorter than the first". +2. The smallest case, expected against observed, as a short snippet the + developer can run. +3. **Why I think it's wrong:** the authority, quoted, with its file and line. +4. **Why it matters:** who would hit it, and what would happen to them. +5. **Why the tests miss it:** what the existing tests check instead. +6. **How often:** how many of the probe's inputs failed, out of how many. + +Put the findings in order of their potential impact on users, the worst +first. The first time you show a small finding, say in a line why it is +there: an oracle captures the code's design and then enforces all of it, +trying every input it can generate, so it turns up small bugs as well as +serious ones. Expect a mix. + +Leave out a bug so trivial that no real user would ever hit it. Keep a note +of it: the oracle will catch it too, and you fix it with the others when it +comes time to fix, and list it in the wrap-up. + +Give each finding its kind: + +| Kind | What it means | +| ----------------- | ----------------------------------------------------------------------- | +| Possible bug | The code breaks a promise that a document or a test states. | +| Inconsistency | Two sources disagree: say which ones, and what each says. | +| Untested behavior | A probe shows behavior that no existing test checks. | +| Design question | Nothing decides the behavior: give two readings and what each predicts. | + +For example: + +> **Retry delay: the second wait is shorter than the first.** +> +> ```ts +> await retry(failTwice, { retries: 3, delay: 100 }) // waits 100 ms, then 50 ms; expected 100, then 200 +> ``` +> +> **Why I think it's wrong:** the README says the delay "doubles after each +> failed try" (`README.md:42`). **Why it matters:** a service that is already +> overloaded gets its retries faster, not slower. **Why the tests miss it:** +> they check only the first delay. **How often:** 412 of the 2,000 schedules +> I tried. + +For a possible bug, ask: "Is this a bug, or is it intended?" + +**Oracle first.** When a probe finds a bug, do not fix production yet, even +when the fix is obvious. The bug goes into the oracle: a law that it breaks, +and a history that reaches it. The fix comes after the oracle fails on it +(step 5). That way the oracle, not the probe, proves the fix, and holds it +for every later change. + +If the probes find nothing, say so plainly: "No confirmed defect in what I +tried." Show what held, and the design questions that came up. + +## 4. Propose a potential oracle + +Say why an oracle would help here. It tries the whole space of inputs and +sequences, not a few probes, states the rule that the probes point at, and +holds the answer for every later change. + +Then present it in the chat: "Here's a potential oracle that owns +, and its potential laws as I see them." If two +potential oracles fit the spot well (two public behaviors, or two ways to +draw the boundary), name the second one in a sentence, say what it would +own instead, and recommend one. + +- What it would own, in one or two sentences. +- The contract, as a few lines of public calls and their results. Always + include it: it is the quickest way for the developer to see what you mean. + + ```ts + slugify('Hello, World!') // → 'hello-world': lowercase words joined by '-' + slugify(slugify(title)) // → slugify(title) + ``` + +- Each potential law, as one "For every …" sentence, with its authority + quoted, and what the probes showed about it. Nothing else from your + worksheet: no backing word (`stated`, `observed`, `open`), no rival, no + witness. +- The laws people forget, or one line on why each does not apply: + - **No throw on valid input.** When the contract says a call returns a + value, it promises not to throw for valid input. + - **The laws of the kind of function.** An equality check is reflexive and + symmetric. A parser and a serializer round-trip. A setter is idempotent + when set to the same value. +- The open design questions, each with its two readings, and what the probe + showed. Say that these can wait: the oracle's results will help decide + them. List them apart from the potential laws, and do not number them as + laws. Until the developer decides one, the oracle file states it as a + limit. + +**Work out each law privately before you propose it.** Fill in this +worksheet for each one, for yourself. Do not show it. It is how you get from +a few examples to a general law: + +```markdown +### L1 + +- Examples: +- What varies, what stays: +- Law: For every , . +- Domain edges: +- New case: +- Rebuilt examples: +- Dynamics: +- Constraints: +- Boundary conditions: +- Authority: +- Backing: stated | observed | open +- Rival: +- Witness: +- Checkpoint: +``` + +How to fill in the fields: + +- **Examples** and **What varies, what stays:** for example, `ms('2 hrs')` + is `7200000` (README, Examples). The number, the spelling, and the spacing + can change. The unit's length decides the answer. +- **Law:** one sentence over the domain. For example: "For every finite + number `n` and every unit name `u` in the table, in any case and with or + without a space, `parse(n + u)` is `n` times the length of `u`." +- **New case:** if you cannot write one, the law is still a list of + examples. Write it again. A new case that only renames a value in the + example (`/:id` for `/:foo`) does not count. +- **Domain edges:** wider than the examples. A block whose domain is only its + own examples is not a law. Keep those examples as `fixed` histories, and as + rebuilt examples of the general law that explains them. +- **Rebuilt examples:** an example that the law gets wrong shows a missing + condition, a second law, or a source that disagrees. Say which. +- **Dynamics** are the model's rules, and the histories vary them. For a pure + function, write "none: one call", and put the variation into the domain. +- **Constraints** hold whatever happens: a count never above its limit, an + equality that is symmetric, a read that returns the last write. The check + tests them at every checkpoint. +- **Boundary conditions** are the edges, where real behavior and most + defects live: empty, zero, `-0`, the size limit, at capacity, the first and + the last item, keys that shadow prototype names, cleanup. A boundary that + the law does not decide goes under open questions. Do not leave it out of + the grammar. +- **Backing:** `stated` if an authority states it, `observed` if production + does it but nothing states it, `open` if it is a question. A `stated` law + is what you ask the developer to approve. It becomes adopted only when they + do, and only adopted laws become expected results. An `observed` or `open` + law goes to the developer as a question. +- **Production output is evidence, never authority.** A rule that you write + only because production does it is model hacking. Mark that rule + `observed`, and ask the developer. An analogy to another documented rule is not + authority either: mark that law `open`. + +Settle these privately too: + +- **What it assumes:** what providers, the environment, or neighboring code + must supply. +- **What else witnesses it:** other ways to observe the same law (a + component, an integration, a replay). +- **What public state it exposes:** each readable field or getter needs a + law or a stated limit. +- **What each probe found:** each constraint, dynamic, or boundary becomes a + law, a history dimension, or a constructed history. Keep a pointer to the + probe that showed it. + +**Always build the strongest oracle the laws need.** Its model predicts every +observation that the approved laws make a claim about. How the oracle is +built is your decision, not the developer's: never offer them a weaker +design, such as checks without a model, a partial model, or fewer +observations, as a choice. If a law truly cannot be modeled yet, say which +part, and state it as a limit. + +The law must say more than the code does. If a model of it would only repeat +the implementation, look for the public promise that the code serves. + +**A more complex design needs sign-off.** Before you propose, read the +trigger list in `oracles-design` step 1. When a trigger for the +[design grammar](../../docs/instruments/design-grammar.md) card fires (for +example, more than five laws, or a +law with no new case that it predicts), work through the card. Then show +the developer the design it produced, in a few plain lines: what the model +keeps, how the laws share it, and any choice between two designs, with your +recommendation. Leave out the card's own terms (preservation relation, +primitives, ablation). Ask them to approve it with the laws. + +Then ask the developer to approve the potential laws that the authority +decides. When the developer approves a potential law, it becomes an adopted +law, and the oracle file records their decision. If they do not maintain the +repository, its maintainers confirm the laws when they review the pull +request. Potential laws have no default: with no developer present, propose +them and stop. + +## 5. Build and run + +1. **Install what is missing**, and nothing more: `fast-check@^4` and + `vitest@^4.1` when `package.json` does not list them. Use the + repository's own package manager. If the repository's own Vitest config + would not pick up the oracle file, add a minimal config for it (or add + its glob to the existing `include`). Add no scripts, lint, or other + config yet. The `oracles-setup` skill's install and "where oracle files + go" sections have the details: take only what these packages and the + config need. If the repository uses another test runner or an older + Vitest, or an install fails, open + [Problems you may hit](references/problems.md). + +2. **Write the oracle file.** Follow `oracles-design` from its first step: + the law block's prose, then the model, the histories, the driver, and the + check. As you build, tell the developer briefly what each part is for: the + model computes the expected answer the simple way, the histories are the + generated inputs and sequences, the driver calls their real code, and the + check compares the two at each step. +3. **Run it on the unchanged code.** A failure under an adopted law is a + possible bug. Present it like a probe finding, with the oracle's smallest + failing case as the snippet. +4. **Fix each bug the developer confirms, right away, oracle first.** The + oracle already fails on it. Make the smallest production fix, rerun the + oracle until it passes, and show both runs: the failure before, the pass + after. Follow the defect path in `oracles-design` step 7. Most developers + want the fix at once; with the oracle failing first, the fix is proved. +5. **As the developer settles questions, change the law** and rewrite the law + block's prose in the same step. Before you show results, check that the + prose still says what the code does. + +**When the oracle passes, close gaps at once, without stopping.** Say in a +line that the oracle passed and what held, then run the gap check, and +explain what it does the first time you mention it. Take the command and +its flags from `oracles-setup` step 7, with `--out` in a temporary directory +outside the repository. Do not run it while the oracle fails: it cannot +judge a failing run. + +For each gap, oracle first: ask why the oracle missed it. Usually the +histories never reach the case, or the check does not look at that result. +Widen the histories or the check, under the laws already approved, and run +the oracle again on the unchanged code: + +- If the oracle now fails, it found a bug. Present it, and fix it as above + once the developer confirms it. +- If closing the gap needs a rule that no approved law states, that is a + design question. Bring it to the developer. +- If the oracle passes and the gap is closed, go on to the next one. + +Group the gaps by function, and tell the developer what you are doing as +you go. Keep going until a bug or a design question turns up. When the +gaps under the approved laws are closed and nothing has turned up, say so, +pick the next spot from your look-around, and keep going. + +## 6. Present + +When the oracle has found bugs, or brought up design questions that the +developer has settled, stop and present. Present the same way whenever the +developer wants to stop: + +- what it found, first: each bug, with what it was, its fix, and the oracle + run before and after +- the trivial bugs you left out of the findings and fixed along the way, in + one line each +- what you built +- an offer to walk through how it works: how the reference model was built, + how the histories are generated, and how the driver calls their code. + Developers often want this, and it is the best part to teach. +- an invitation to ask questions +- the path to the feedback log, and a request to send it to Kyle Mathews + (GitHub KyleAMathews) when they are done + +When they take the walk-through, follow one bug the oracle found through +the file: the history that produced it, what the driver did and saw, what +the model expected, and how the check reported the difference and shrank it. +Show a few lines of the file at each part, not a description of it. + +When the developer wants to keep the work (commit it, or open a pull +request), say what finishing the repository's setup adds: a lint preset +that keeps oracle files honest, a block in `AGENTS.md` (or `CLAUDE.md`) so +every agent follows the same rules, and the gap check over all the owned +code. Then load `oracles-setup`, and do it. Do not call this "production" +or "shipping": it is setting up their repository for oracles. diff --git a/packages/oracles/skills/oracles-start/references/problems.md b/packages/oracles/skills/oracles-start/references/problems.md new file mode 100644 index 0000000..83089a7 --- /dev/null +++ b/packages/oracles/skills/oracles-start/references/problems.md @@ -0,0 +1,56 @@ +# Problems you may hit in a first session + +Open this file when one of these happens. Each entry says what to do, and +what to tell the developer. + +## Installs + +**The repository uses another test runner** (mocha, uvu, Jest, ava, +`node:test`). Add Vitest for the oracles only, with a config whose `include` +is only the oracle files. Their own suite and runner stay as they are. Tell +the developer this in one line as you install. + +**The repository runs a Vitest older than 4.1.** This is the developer's +decision. Tell them that the oracle needs Vitest 4.1 or later, and that +upgrading changes the runner for their own suite too. Offer to upgrade and +then run their suite, to show whether it still passes. Do not upgrade until +they agree. + +**npm installed `fast-check` and `vitest` already.** npm installs the peers +of `@tanstack/oracles` without listing them. Check `package.json`, not +`node_modules`, and add both to the dev dependencies. + +**Which Vitest.** Install `vitest@^4.1`. It runs on Node 20 and later, and +Vitest 5 needs Node 22.12 or later. Which Node versions CI runs is a +question for when the repository's setup is finished: do not raise it +now. + +**An install fails.** Show the error in a line or two, say what you think +caused it, and ask before you try anything that changes more than the +packages the oracle needs. + +**Lint.** The oracles lint plugin comes when the repository's setup is +finished. If the +repository's own lint already covers the oracle file's folder, run it on +the file, as `oracles-design` says. + +**The repository's lint picks up the oracle file.** If its lint (xo, ESLint +over every file) now fails on the oracle file or the Vitest config, tell the +developer in one line. Leaving them out of that lint is part of finishing +the setup. Do not change their lint config in a first session. + +## Probes + +**The tests import a build that is not built yet.** Import the source that +the build comes from. + +**Node warns about the module type.** If the warning names the scratch +directory, give it a `package.json` with `"type": "module"`. If it names the +repository's `package.json`, the warning is harmless: run the probe with +`node --no-warnings`, and do not change their `package.json`. + +**Node cannot load the source as it is** (for example, an extensionless ES +module). Copy it unchanged into the scratch directory, under a name that +Node can load, such as `index.mjs`, and tell the developer that you did. + +**The code is TypeScript.** Run the probe with `npx tsx`.