Repository navigation
docs(oracles): ship setup, design, and review skills, with a prompt to try them - #7
Open
KyleAMathews wants to merge 86 commits into
Open
KyleAMathews wants to merge 86 commits into
KyleAMathews wants to merge 86 commits into
Conversation
…checklist
The package now carries what an adopter's coding agent needs, inside
node_modules:
- skills/oracles-setup: install both pinned packages, configure lint,
write the AGENTS.md block, make the skills findable through repository
or global pointer skills, and run a first oracle and gap check.
- skills/oracles-design: law sheet first, model choice, a witness for
every distinction, histories and reach, driver and check, RED, GREEN,
hostile mutant, and closing every gap with no exits.
- skills/oracles-review: rebuild one observation, the shortcut table,
the weaknesses tools cannot see, and the finding format.
- docs/oracle-checklist.md: ORC-001 to ORC-016 (version 1), ported from
TanStack DB's rubric with the same IDs and a table of every change.
- docs/glossary.md, docs/guide.md, and five instrument cards.
- READMEs for both packages; `files` now includes skills and docs.
The skills teach `oracleProperty({ owns })` and no `law.owner`, so this
waits for those framework changes before it merges.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
🦋 Changeset detectedLatest commit: 23af0bf The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Title check |  |
This assessment has not completed. The check will be retried on the next pre-merge check run. |
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled.
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
commit: |
`oracles-design` step 3 named six model styles but showed no example of any but a pure function. docs/model-styles.md gives a small, generic, type-checked example of each: a pure function, a naive reference structure, a transition table with derived reach, a pure step function, a relation between runs with its calibration, and an allowed set. It also states two rules from the model prototypes: each model rule is one sentence of the law, and legality lives in a separate pure guard. Co-authored-by: Isaac <no-reply@databricks.com>
Oracle reviews were sometimes read as security work. The skills and docs now avoid words that suggest it: the adversarial-review card is the challenge-review card, attacks are challenges, a hostile mutant is a mutant (as TanStack DB's glossary already asks), a fault-injected step is a step that simulates a signal, and "bypass", "defeat", and "payload" are reworded. Links follow the renamed card. Co-authored-by: Isaac <no-reply@databricks.com>
Design step 2 now asks the agent to name the model's parts and rules, map each rule to a sentence of the law, rebuild the known examples before trusting the model, and record rule conflicts as open questions. docs/instruments/design-grammar.md carries the heavier method for a hard law or a growing model: a stated preservation relation, parts admitted only for a job, overlaps, reconstruction and simplification by a different representation, a two-way comparison for every simplification with refactor, refine, and revise kept apart, and an optional TLA+ check with a weakened-configuration control. The design and review skills and the card index route to it. Co-authored-by: Isaac <no-reply@databricks.com>
- New skill oracles-evaluate-review: a lossless ledger of review findings, enforcement gaps fixed and law changes brought to the user, one oracle plan for the whole review run against unchanged production, each bug turned into a family of histories with a nearby passing case, and a loss audit. - oracles-design: the approved law sheet becomes the law block at the top of the oracle file; a run-budget heuristic (critical states by construction and probes, starting runs, fast files, longer sweeps by the run multiplier, a lowered budget is a weakening); lint after each edit; a production cleanup step after GREEN; the evidence block goes in the pull request description. - oracles-review: findings for a lowered budget and for inserted rather than refactored code; a law block that disagrees with the law option. - oracles-setup: the AGENTS.md block names the new skill and asks for lint after each edit; pointer skills include the new skill. - Skill descriptions and a few sentences follow the STE structural rules. Co-authored-by: Isaac <no-reply@databricks.com>
#6 made `recommended` turn off inline directives and widened the `production` glob to `src/**`, so setup now adds both presets as they are, without the workaround blocks. The plugin README lists no-oracle-disable and the new glob. The design skill now asks for at least one fixed seed (an empty list will be refused) and treats a random-only kill as a gap until its history is kept. Co-authored-by: Isaac <no-reply@databricks.com>
Two fresh agents set up the package in vercel/ms (Jest and Biome, no ESLint) and developit/mitt (mocha, ESLint 7), using only the packed packages and the skills. mitt reached zero gaps. ms stopped correctly on a law that nobody can decide for a library the adopter does not own. Setup now covers what they hit: - the Node version against CI, the lint setup, and the command runner - `eslint@^9` when the repository has none, a stable parser major, and TypeScript 5 or newer for the oracle files, kept out of an older `tsc` - where oracle files go, and a Vitest config beside another runner - a separate ESLint 9 run for an ESLint 7 or 8 repository, with a check that the rules are live - `owns` entries that name one function - `--source` selecting files, an explicit `--seed`, `--out` inside the repository, and `.oracle-gaps/` in `.gitignore` - dead code that the gap check finds counts as a production change The design skill teaches function-level `owns`, dead-code removal as a production change, and what a law that kills nothing means. Co-authored-by: Isaac <no-reply@databricks.com>
- Evidence by kind of change: a defect fix shows RED, GREEN, and the defect as a mutant; a behavior-preserving refactor shows the same oracle passing on old and new production with kill sets compared; a new oracle over correct code shows production passing and calibrations failing. Review no longer treats a refactor without RED as a shortcut. - Approval follows the semantics: strengthening an adopted law reuses it without a new sheet; a law change needs a law delta; a new law or missing authority needs the full sheet; an approval the task already gave stands. - A boundary-selection step before the law sheet, with an example. - Model state is justified by a legal continuation (one or more actions), not only the next action, in the design skill, ORC-008, the guide, and the law-discovery card. - Dead code needs a contract reason before removal; a surviving mutant only shows what the current histories did not expose. - A survivor that the adopted law already allows is reported as an allowed alternative, not sent back as a law question. - Setup: pointers and the AGENTS.md block read the chosen package's installed copy in a workspace; the TypeScript parser covers production source as well as oracle files. - The round-trip example compares tags element by element and names the two corruptions the joined comparison accepted as calibrations. Co-authored-by: Isaac <no-reply@databricks.com>
The framework judges a mutant over the domain the operand types admit, so a narrower domain stated only in prose does not make a mutant invalid. Co-authored-by: Isaac <no-reply@databricks.com>
Four fresh agents ran the branch's skills on four workflow branches: a behavior-preserving refactor, a review with two findings, a planted defect, and a package-local workspace. Each took the right branch. Their findings, plus the framework session's answers: - Refactor evidence follows the gate's rule: oracle files unchanged or only strengthened and passing on old and new code with the same seed; shared killed keys still killed; new keys in changed functions killed or acted on. - Design: an allowed alternative is its own outcome; a gap that existed before a change still counts in a changed function; widening histories inside the stated domain is strengthening; a grammar that covers part of the law's domain says so; constructed histories must fit the arbitrary; the driver records production errors; own the whole file when an owned function reads module data; several oracles may own one function; public state needs a law or a limit; an exhaustiveness-guard survivor is a known CLI bug, recorded without stopping or removing the guard. - Review: a law-dependent state in `measured` is an unreachable claim. - Setup: name the oracle config `vitest.config.ts` (the gap check loads only that name), pnpm 11's allowBuilds placeholder, refresh pointers on upgrade, the AGENTS.md block in a package-local workspace, keep oracle files out of the repository's own lint or match its style, install before a pointer gives up, and a path for runs with no user. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
With #11 merged, the README sample type-checks and the gap check counts it as an oracle file. It still left one gap: `>` to `>=` differs at clamp(-0, 0), and the histories never generated -0. The example now covers its type's domain, as the design skill teaches: -0 is in the grammar, a constructed history enters "negative zero at max", and the gap check kills all three mutants. Co-authored-by: Isaac <no-reply@databricks.com>
KyleAMathews
marked this pull request as ready for review
November 11, 2026 01:13
Three fresh agents adopted the package in p-limit (async JavaScript, xo, AVA), dequal (uvu with esm), and nanostores (TypeScript 7, oxlint). All three followed the method, and dequal's agent found real library defects. Their findings, plus the framework session's answers: - Setup: offer a separate Node 20 job when CI is older; use the repository's package manager for install commands; place oracle files where the existing runner will not load them (their own directory, or a `*.oracle.ts` name); probe-form for root-level source; xo repositories; the TypeScript 7 plugin bug as a temporary note; the live-rule check through stdin before any oracle file exists; choose a function with a documented promise; a first oracle that is RED found a defect (report it, never weaken the oracle, run the gap check on a scratch fix until the CLI judges a failing baseline); group the open gaps in the report. - Design: an analogy is not authority; laws people forget (no throw on valid input, the algebraic laws of the kind of function); constructed values derived from the generated history; whole-file entries that pull in another contract; owned code that serves another module; where `oracleProperty`'s options are typed; do not copy exit wording into the agent's own report; a synchronous-spin timeout is a CLI limitation. Co-authored-by: Isaac <no-reply@databricks.com>
…mmar
Agents go from examples to a general law only when made to write it
down, and no agent in three validation rounds opened the design-grammar
card, because it triggered on judgment ("a hard law").
- Design step 1: before the law sheet, each law gets a written block:
the examples with sources, what varies and what stays, the law as one
"For every ..." sentence with its edges, a new case it predicts, and
every example rebuilt from the sentence alone.
- Five concrete triggers send the agent to the design-grammar card; the
evidence says which fired.
- Review: "Law as examples" and "Skipped design grammar" findings.
- Evaluate-review: the causal condition is written as a sentence over
histories, with a second history it predicts.
- Setup: TypeScript 7 is an environment limit (typescript-eslint), and
the plugin's `unparsedModule` report for a `.js` oracle file that
imports TypeScript (#13).
- Design: the interim exit-wording bullet is gone with #12; the rule not
to write those words in a report stays.
Co-authored-by: Isaac <no-reply@databricks.com>
- Design step 1, item 6: sort each law into dynamics (how things change, stated by the model's rules), constraints (what holds in every history), and boundary conditions (the edges, where real behavior and most defects live). Every boundary gets a constructed history or an open question; it is never left out of the grammar. Every defect the validation agents found was at a boundary. - Adjacent forms keep design diversity: the design-grammar card's new step 8 sketches two to four forms that keep every adopted law, for the user to pick; design step 3 names a second fitting model style; design step 7 sketches alternative cleanup designs instead of keeping the first. - Glossary: dynamics, constraint, boundary condition, adjacent form. Co-authored-by: Isaac <no-reply@databricks.com>
The law sheet is now a template that the agent fills in for each law. Its examples, law sentence, new case, and dynamics, constraints, and boundary conditions are fields, and the summary table is only an index. In round 4, one of four agents skipped the written block because it was prose before a table. Other fixes from round 4 (validation findings V64 to V96): - design: arbitraries must take histories back (`.map` needs its inverse); string units; observations are data; brand checks, not instanceof; methods of a returned object are owned through the enclosing function; the whole-file entry only when module data affects a law; authored mutants when operators are sparse; triggers checked again after each gap check, counting every row. - setup: the no-user default covers setup questions only; the end condition is the first oracle's own contract; keep an existing Vitest include; register the plugin in a custom probe-form block; xo and TS 7 installs; the live-rule probe needs the fast-check import; placeholders on a stopped run; scratch copy wording. - review: the shortcut rows match the template and the later trigger check. Co-authored-by: Isaac <no-reply@databricks.com>
…Script The framework credits a kill to the first law that a history violates, so a later law can look idle. The design skill now says to aim hand mutants at that law before restating or deleting it. The setup skill reports a TypeScript copy that npm hoisted from @tanstack/oracles as a known packaging bug, so it is not taken as the repository's own TypeScript. Both are temporary: the framework fixes are queued. Co-authored-by: Isaac <no-reply@databricks.com>
The law block in an oracle file is now prose for a reader who has not seen the sheet: a question for a title, how the code works, the documented promises, how the model states them, and what the driver observes. The sheet's fields stay in the review record. The AGENTS.md block (template 2) says who owns what. The maintainers own the laws. The agent owns the production code that the oracles cover, and makes it the smallest, clearest code that passes them. Round-5 validation fixes: - A stated limit no longer closes a gap. Setup's end condition, its report groups, design step 8, and the review's exit row now say so. - A block whose domain is its own example is a fixed history, not a law. - The string-unit advice names `binary` (every code point except lone surrogates), since `grapheme` has no control characters. - A guard that TypeScript narrowing makes constant (the last member of an exhaustive union) is a known CLI bug. Other mutants on its line still count. - A `not-observed` survivor can need a history where the branches diverge. - The design-grammar trigger after a gap check is a stop, not a note. - An idle law that another law covers is merged. No empty tests in oracle files. `vitest.config.mts` in a CommonJS package. The AGENTS.md lint line when lint cannot run. Co-authored-by: Isaac <no-reply@databricks.com>
The law sheet is now the new oracle file before it has any code: an opening comment that states the laws as prose (ORC-003), and a review-notes comment with the per-law template that turns examples into a general law. After approval, the review notes are deleted, and the code is added under the prose. A law delta edits the prose in place. This removes the step that copied an approved sheet into the file. Co-authored-by: Isaac <no-reply@databricks.com>
…story is a start; config name without type module Co-authored-by: Isaac <no-reply@databricks.com>
…ons as dated authority A potential law backed by a document is 'stated' until the engineer approves it, so 'adopted' no longer leaks into the chat before approval. A user's decision is written as a dated maintainer decision, not as the session's story. Co-authored-by: Isaac <no-reply@databricks.com>
…ed them as peers Co-authored-by: Isaac <no-reply@databricks.com>
…a probe Co-authored-by: Isaac <no-reply@databricks.com>
…hority to quote Co-authored-by: Isaac <no-reply@databricks.com>
…one in a scratch copy Co-authored-by: Isaac <no-reply@databricks.com>
…he repository's own Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…test 4.1 in a first session - oracles-design: each oracleProperty carries one law; properties may share the model, grammar, and driver. - oracles-start: the potential oracle always shows its contract block, keeps the worksheet's backing word private, and installs vitest@^4.1 (Node 20+), leaving CI Node versions to production. Co-authored-by: Isaac <no-reply@databricks.com>
…GitHub login On a team repository a reader needs to know whom to ask. The login comes from gh, or the engineer; it is written without an @ so a copy into a pull request notifies no one. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…is removed Co-authored-by: Isaac <no-reply@databricks.com>
…acle runs the gap check; the AGENTS.md block governs from the start - oracles-start: probes exist to learn the code's dynamics, boundaries, and constraints that the oracle needs; the agent writes down what it does not know and probes each point. Bugs are a welcome side effect. - When the first oracle finds no bug, run the gap check (output outside the repository) and present what the surviving changes show, before offering the next oracle. A failing oracle still stops at the RED. - Before the first message, read the block that production adds to AGENTS.md: it governs the agent from the start of the session. - Remove every mention of legacyCampaigns from the docs and skills. Co-authored-by: Isaac <no-reply@databricks.com>
… chat, before probing Co-authored-by: Isaac <no-reply@databricks.com>
From a semantic drift assay of cc88112..4749f61: - the gap check's --out follows oracles-setup step 7 (in .oracle-gaps/), and oracles-setup names step 7 as part of a first session - the session's ending and the walk-through cover the no-bug path - a decision records name, GitHub login, and date in start and checklist - one Vitest 4.1 / fast-check 4 pin across start, setup, and the README - a helper that forwards literal owns registers tests, in setup as in design - adopted means approved, in the glossary - 'gap check in CI' had no source; it is the gap check over all owned code Co-authored-by: Isaac <no-reply@databricks.com>
…when the session ends Co-authored-by: Isaac <no-reply@databricks.com>
…es/problems.md The skill keeps the main path and a pointer; the agent opens the reference when it hits another test runner, an older Vitest, a failed install, peers already on disk, or a probe that Node cannot load. Co-authored-by: Isaac <no-reply@databricks.com>
…ciled; a preview feedback log - 'developer' replaces 'engineer' and 'user' for the person the agent works with; 'maintainer' is only the role with authority over the laws. A developer's approval adopts a law and is recorded; on a shared repository the maintainers confirm it in review. A library's end user is 'the caller'. - 'approved' is the act and 'adopted' the state; the glossary glosses owns as a public behavior in conversation; 'sketch' becomes 'draft'; the finding kind 'Test gap' becomes 'Untested behavior'. - While in preview, a first session keeps a feedback log outside the repository and asks the developer to send it to Kyle Mathews. Co-authored-by: Isaac <no-reply@databricks.com>
… not the home directory Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…dback log at the start Co-authored-by: Isaac <no-reply@databricks.com>
…e repo's module-type warning is harmless The contract and law-block examples used klona, a library the skills are validated on; an agent noted that the example could prime its findings. They now use a fictional slugify. Co-authored-by: Isaac <no-reply@databricks.com>
… lint; no gap check on a failing oracle Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…offer a weaker one as a choice A first session asked the developer 'Full model, or checks only?'. The term meant nothing to them, and the answer should always be the strongest model. How the oracle is built is the agent's decision. Co-authored-by: Isaac <no-reply@databricks.com>
While only invited people try it, the README holds a one-sentence overview and the prompt. The session itself explains the rest. Co-authored-by: Isaac <no-reply@databricks.com>
…, not the skill's
From manual review: an opening that copied the skill's example ('a behavior
that's easy to break and has a simple rule behind it') sounded nonsensical,
and a probe plan listed the skill's categories ('dynamics, constraints,
boundaries, timing, and lifecycle') instead of questions about the code.
Co-authored-by: Isaac <no-reply@databricks.com>
…pository under test Co-authored-by: Isaac <no-reply@databricks.com>
…rst then fix, close gaps until something turns up From the 2026-11-11 UX test (Kyle Mathews and Tanner Linsley) and its evaluate-review ledger, validated with first sessions on klona, quick-lru, and p-queue: - Open with plain sentences about this repository (no quotable example), look around with impact signals (the most important API, README examples, user-facing code, strongly stated promises in old code, rare edge cases that take a system down, repeated fixes), present two or three spots, and stop for the choice. - A table of uncommon words, each explained in one line the first time it comes up, including gap check and gap. - Findings carry 'why it matters' and are ranked by impact; the first small one says why small bugs appear; trivial ones are left out of the findings and fixed later with the others. - Oracle first: a bug a probe finds goes into the oracle before any fix. Confirmed bugs are fixed right away, with the oracle failing before and passing after. - When the oracle passes, close gaps at once: ask why the oracle missed each one, widen the histories or the check under the approved laws, and keep going until a bug or a design question turns up. - The gap check's report goes to a temporary directory; no cleanup talk. - 'Take the oracle to production' becomes 'finish setting up the repository for oracles', offered when the developer wants to keep the work. oracles-setup is retitled to match. - problems.md: the repository's lint picking up the oracle file. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An adopter can now paste one prompt into a coding agent and get oracle testing set up the way TanStack DB does it. The package ships four skills and the docs they cite inside
node_modules. The agent reads them from there, so the instructions always match the installed version.Before this change, the method lived only in TanStack DB, in a 1,175-line rubric that agents did not read in full. Outside agents skipped the required reading, wrote code before they stated a law, and called surviving mutants "equivalent". These skills replace that rubric for adopters. They close the exits that agents took, and they keep the decisions that need a person with that person.
Try it
The package README now starts with this prompt. An adopter pastes it into an agent at the root of their repository:
The agent installs both packages at that commit, adds the lint presets, and writes an
AGENTS.mdblock. Then it writes the first oracle file as prose laws only, and stops. The adopter reads the laws and decides what the code promises. After approval, the agent adds the code, runs the oracle on unchanged production, and runs the gap check. The skills work only after this PR merges, becausemaindoes not have them yet.Who owns what
The
AGENTS.mdblock states the split. The maintainers own the laws: what the code promises, the authority for each promise, and its limits. The agent owns the production code that the oracles cover. It makes that code the smallest, clearest code that satisfies the oracles, and it restructures freely while they pass.oracles-setupAGENTS.mdblock, makes the skills findable, and takes a first oracle through the gap check.oracles-designoracles-evaluate-revieworacles-reviewHow the skills keep the agent on the path
The skills make the right move the obvious one. They do not rely on the agent to judge when a rule applies.
observedand asks the user.What the validation runs found
Fresh agents ran the skills on 12 open-source libraries over five rounds. Each agent got only the packed packages and one library it had not seen. Each round's findings changed the skills, or went to the framework as an issue.
DataViewcopy loses its window, and a null-prototype object comes back withObject.prototype. In dequal, an ownconstructorkey breaks equality. Nobody filed these upstream yet.A sixth round is in progress on three libraries that the maintainer owns.
The checklist and the docs
docs/oracle-checklist.mdholds ORC-001 to ORC-016, ported from TanStack DB's rubric as version 1. The IDs did not change, so existing citations still work. ORC-003 requires each oracle file to be a literate program. A table at the end lists every difference from the rubric and its reason.docs/glossary.mddefines the oracle terms, including law sheet, exit, framework bug, invalid mutant, allowed set, and boundary condition.docs/model-styles.mdgives one small, type-checked example of each of six model styles.docs/guide.mdholds the rubric's technique sections, made generic. It adds no requirements.docs/instruments/holds six reasoning cards. The design grammar card names the parts of a model, records rule conflicts, and sketches adjacent designs, so the user can choose between them.Interim rules that remain
Two framework changes wait for the maintainer's decision on the type domain. The first skips guards that TypeScript narrowing makes constant (F11). The second adds mutation operators (F17). Until they land, the design skill reports a constant guard as a CLI limitation, and it leaves production unchanged. That guard is either a branch after
satisfies neveror the test for the last member of an exhaustive union.Merge order
mainalready has everything that the skills teach, through 84cd34a (feat(oracles): owns on oracleProperty; a run that judges nothing fails; namespaced output #11 to feat(oracles): --config, wrapper configs that match the config's module format, and no listener warning with many contexts #26).docs/oracles-package-skills) bumps DB's pin to this commit and merges. It turns DB'soracle-reviewandoracle-authoringinto addenda that load these skills.Verification
clamporacle, built from the packed packages at this branch's head, passestscand 4 of 4 Vitest tests. The gap check on it reports 1 mutant, 1 killed, 0 gaps, and exits 0.git ls-remotereturns the current commit, and both pkg.pr.new URLs resolve at its 7-character hash.tests/package.test.tspasses, sonpm packincludesskills/anddocs/. All 53 relative links and anchors in the new docs resolve, and Prettier passes.Where to start reading
Start with
packages/oracles/skills/oracles-design/SKILL.md, thenpackages/oracles/docs/oracle-checklist.md. The setup skill and the package README show the API as an adopter uses it.This pull request and its description were written by Isaac.