From 233cbe67f7c8888dca7b8a58c5157fc680d520d5 Mon Sep 17 00:00:00 2001 From: Jammy2211 Date: Thu, 10 Sep 2026 17:09:24 -0400 Subject: [PATCH] =?UTF-8?q?feat:=20Bring=20Your=20Own=20Likelihood=20(BYOL?= =?UTF-8?q?)=20mode=20=E2=80=94=20the=20second=20published=20prompt=20gets?= =?UTF-8?q?=20its=20door?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit modes/byol.md catches the published "Set up PyAutoFit with my existing science project" prompt and runs its own contract one stage per turn: overview of the project and likelihood function, a model with priors chosen with the user, the Analysis wrapper validated against their function unchanged, a search recommendation with the reason, a go-ahead checkpoint that fits nothing until they say go, then the run with the output-folder tour and how to inspect and interpret the results. The ask-anything / teacher-mode footer sits on every stage, as in start here. You bring the likelihood; the assistant brings the inference. AGENTS.md registers the mode and its selection rule beside start here. README.md gives the BYOL material its own "Bring Your Own Likelihood (BYOL)" sub-heading under Your own project and says the prompt launches BYOL mode; prompt text unchanged. The mode's worked example quotes a real dry run on the bundled supernova dataset through a stand-in user likelihood module. Closes #40 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01BrGfrSxV9xoqCa4v1p9hix --- AGENTS.md | 18 ++- README.md | 57 ++++++---- modes/byol.md | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 348 insertions(+), 30 deletions(-) create mode 100644 modes/byol.md diff --git a/AGENTS.md b/AGENTS.md index 4f0f8c7..56edc49 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -146,12 +146,18 @@ how it paces the work, not which workflows exist: the user's own project. One step per turn; the user types each prompt themselves. It adds no capability — it scripts the existing skills at newcomer depth — and hands off to assistant (or teacher) mode when it ends. See [`modes/start_here.md`](./modes/start_here.md). - -Select (first match): the **published start-here prompt** (`README.md` "Getting Started"), or -"start here" / "start_here" → explicit instruction → `.mode` file → `profile.md` "Interaction -mode" → else **infer from the opening request** (fall back to **assistant**); `.maintainer` -outranks all of them. The start-here trigger outranks inference from the rest of the opening -request. State an inferred mode in one line and invite correction; acknowledge an explicit +- **BYOL** — *Bring Your Own Likelihood*: the user points at existing likelihood code and + the assistant sets PyAutoFit up around it — overview → model → wrap + validate → search → + go-ahead → run and inspect, one stage per turn. Nothing is fitted before the go-ahead, and + their code is never re-parametrised or "fixed". See [`modes/byol.md`](./modes/byol.md). + +Select (first match): the **published start-here prompt** (`README.md` "Getting Started") or +"start here" / "start_here", or the **published BYOL prompt** (`README.md` "Your own project") +or "byol" / "bring your own likelihood" with code pointed at → explicit instruction → `.mode` +file → `profile.md` "Interaction mode" → else **infer from the opening request** (fall back to +**assistant**); `.maintainer` outranks all of them. Both front-of-chain triggers outrank +inference from the rest of the opening request; if both fire and code is pointed at, BYOL +wins. State an inferred mode in one line and invite correction; acknowledge an explicit one only if it changes behavior. Read `modes/.md`; depth still follows `skills/_style.md` "Adaptive depth". diff --git a/README.md b/README.md index aeec726..cd39a8d 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,9 @@ reload it, then extend the workflow — one step per turn, with **you** typing e yourself that the whole workflow is natural language. Then **your own science**: a paper or a plain-language description of what you measure, then your model, your likelihood, a search chosen for your problem, the fit, and the results. You can stop and ask "what is a prior?" at any step without losing your place, and saying -**teacher mode** turns on full explanations for the rest of the tour. +**teacher mode** turns on full explanations for the rest of the tour. If you already have likelihood code, the +**Bring Your Own Likelihood (BYOL)** prompt under [Your own project](#bring-your-own-likelihood-byol) is the other +door in — it starts from your function instead of the Gaussian. The tour is scripted in [`modes/start_here.md`](modes/start_here.md); the sections below are the same six steps, written out so you can read ahead or work through them on your own. @@ -51,7 +53,8 @@ priors, planning an analysis or learning in Teacher Mode all happen inside the s 3. **Submit the starting prompt — the tour begins.** Paste the prompt at the top of this README and the assistant opens the guided tour at step 1. (If you already have likelihood code, the "Bring Your Own Likelihood" prompt - under [Your own project](#your-own-project) starts the same workflow from your code instead.) + under [Your own project](#bring-your-own-likelihood-byol) starts **BYOL mode** instead, running the same + workflow from your code.) **Experimental alternative.** [OpenCode](docs/setup/opencode_cli.md) is an open-source coding agent whose client is free; the model you connect it to is a separate choice with its own cost and capability, free offerings are often @@ -346,28 +349,7 @@ caveats worth investigating. **P2. Likelihood.** If you already have likelihood code, it stays yours: the assistant wraps it and checks it returns the same numbers, never re-parametrising or "fixing" your science. If you do not, it builds a simple one from your description. Either way your real data is plotted and discussed — unmodelled data pathologies are the -first source of biased inference, so this step is not optional. - -Already have a likelihood function for your science problem? **Point the assistant at your existing code and it can set it up with PyAutoFit** — defining the model, choosing priors with you, configuring a search and organising the results: - -Example Natural Language Prompt for Claude Code, Codex or other AI coding agent - -```text -Set up PyAutoFit with my existing science project. An example likelihood -function can be found at [GitHub link or local directory]. - -First, give me an overview of my project and likelihood function. Compose -an appropriate model, explain it to me, and recommend a non-linear search -(for example MCMC, nested sampling or maximum-likelihood estimation). - -Do not begin inference until we have discussed the setup and I give you -the go-ahead. - -Once inference is running, explain how the results are written to disk and -show me how to inspect and interpret them with PyAutoFit. -``` - -Your existing science code remains the source of the likelihood. With PyAutoFit built around it, you can perform inference through natural language while gaining access to features such as flexible priors and model composition, MCMC and nested sampling, automated result handling, model comparison and scalable workflows. +first source of biased inference, so this step is not optional. Already have the code? See [Bring Your Own Likelihood (BYOL)](#bring-your-own-likelihood-byol) below. **P3. Search.** Runtime budget, whether your likelihood is JAX-differentiable, and whether you need the Bayesian evidence pick the sampler between them. You get a recommendation with the reason, not just a name. @@ -406,6 +388,33 @@ what is available, and you can ask for any of it the same way. Everything above was natural language. You did not write a line of Python — though the assistant saved you the scripts anyway, one per step, so you can read, re-run and modify them yourself. +### Bring Your Own Likelihood (BYOL) + +Already have a likelihood function for your science problem? **Point the assistant at your existing code and it can set it up with PyAutoFit** — defining the model, choosing priors with you, configuring a search and organising the results: + +Example Natural Language Prompt for Claude Code, Codex or other AI coding agent + +```text +Set up PyAutoFit with my existing science project. An example likelihood +function can be found at [GitHub link or local directory]. + +First, give me an overview of my project and likelihood function. Compose +an appropriate model, explain it to me, and recommend a non-linear search +(for example MCMC, nested sampling or maximum-likelihood estimation). + +Do not begin inference until we have discussed the setup and I give you +the go-ahead. + +Once inference is running, explain how the results are written to disk and +show me how to inspect and interpret them with PyAutoFit. +``` + +This prompt launches **BYOL mode** ([`modes/byol.md`](modes/byol.md)) — Bring Your Own +Likelihood, in the spirit of Bring Your Own Bottle: you bring the likelihood, the assistant +brings the inference, one stage per turn and nothing fitted until you give the go-ahead. + +Your existing science code remains the source of the likelihood. With PyAutoFit built around it, you can perform inference through natural language while gaining access to features such as flexible priors and model composition, MCMC and nested sampling, automated result handling, model comparison and scalable workflows. + ## Teacher mode and HowToFit For users less familiar with Bayesian inference and scientific analysis you may wish to read through diff --git a/modes/byol.md b/modes/byol.md new file mode 100644 index 0000000..5f783a2 --- /dev/null +++ b/modes/byol.md @@ -0,0 +1,303 @@ +# Bring Your Own Likelihood (BYOL) mode + +The door for a user who already has working science code. They point at a likelihood +function; the assistant reads it, restates it, composes the model around it, wraps it, +validates it, recommends a search, and — only after an explicit go-ahead — runs the fit and +shows them where the results live. **You bring the likelihood; the assistant brings the +inference.** + +This file is a script for *you*, the assistant — what to say, what to show, where to stop — +not user-facing prose. BYOL adds **no capability**: it is +[`af_wrap_likelihood`](../skills/af_wrap_likelihood.md) driven through the six sentences of +the published prompt, one per turn. + +## What changes + +- **One stage per turn.** Do the stage, show the real output, append the footer, then **stop + and wait**. Never run two stages in one turn, and never run ahead of the go-ahead. +- **The user's code leads.** Every stage is grounded in what their function actually does, + read from their file — not in what a likelihood of that kind usually does. +- **Nothing is fitted before stage 5**, the prompt's own instruction; treat it as binding + even when later messages sound impatient. Ask, don't assume. +- **Generated Python is kept**, in `scripts/` (or their science project) — they leave with a + wrapper and a fit script they own, not a transcript. + +## What stays the same + +- Every `AGENTS.md` safety invariant, and one of them is the whole point of this mode: **the + likelihood code is the user's** — never re-parametrised, never guarded, never "fixed"; if it + looks wrong, say so and they decide. Also the data-inspection gate (their data is real), the + code gate, the `output/` write-ban and the source-edit boundary. +- Saved Python is at full `skills/_style.md` "Generated script style" quality — + mode-invariant. +- The skills are unchanged. This mode composes `af_wrap_likelihood` (the engine), + `af_compose_model`, `af_configure_search`, `af_run_search`, `af_plot_fit`, + `af_load_results`, plus `af_ingest_paper` and `start-new-project` at the hand-off; read + the skill rather than restating it here. + +## Composition + +Depth follows `skills/_style.md` "Adaptive depth", entering at **PyAutoFit newcomer** — a +user with their own likelihood code is fluent in their science and usually in their +statistics, and new only to the API, so map straight from inference question to object. An +existing `wiki/project/profile.md` level is not overridden, and "teacher mode" at any stage +switches [`modes/teacher.md`](./teacher.md) on for the rest of the run without leaving BYOL. + +## What triggers inference + +The published BYOL prompt (below), or "byol" / "BYOL" / "bring your own likelihood" / +"wrap my likelihood" **together with a path or link to code** — outranking inference from +the rest of the opening request; `.maintainer` outranks it. If both the start-here prompt +and the BYOL prompt appear and code is pointed at, **BYOL wins**: the tour's Part 1 exists +to reach the place BYOL starts from. + +--- + +## Trigger and posture + +The published prompt — on this repo's `README.md` under "Your own project" — is: + +```text +Set up PyAutoFit with my existing science project. An example likelihood +function can be found at [GitHub link or local directory]. + +First, give me an overview of my project and likelihood function. Compose +an appropriate model, explain it to me, and recommend a non-linear search +(for example MCMC, nested sampling or maximum-likelihood estimation). + +Do not begin inference until we have discussed the setup and I give you +the go-ahead. + +Once inference is running, explain how the results are written to disk and +show me how to inspect and interpret them with PyAutoFit. +``` + +Its four sentences are the six stages below, and every stage ends with the two-line footer +under "Teacher mode and questions". Before stage 1, run the session-start drift check +(`AGENTS.md` "Session start"): `python autoassistant/audit_skill_apis.py --check-version`. +Exit 0 — continue. Exit 2 or 3 — the stack is absent or broken: run `af_setup_environment`, +say so in one line, resume at stage 1. Exit 1 — report the drift and offer the pinned +version; the wrap can still proceed. + +Honour **"skip to the fit"** only as far as stage 5: the go-ahead checkpoint is the user's +own instruction and is not skippable. Anything else may be skipped on request, always with a +one-line note of what was skipped. + +--- + +## Stage 1 — Overview + +*Skill:* `af_wrap_likelihood` ("Ask"). *No script yet.* + +**Read the code they pointed at.** A GitHub link → clone into gitignored `sources/` +(`AGENTS.md` "Source-of-truth resolution") and read it there. A local path → read it in +place; never copy it into this repo. + +**Restate it in one paragraph**, covering the four things that decide the wrapper's shape: + +1. **What the function scores** — the forward model, and the data it is compared against. +2. **Signature and calling convention** — a parameter vector in fixed order, named + arguments, a dict, an object? What does it return? +3. **Log likelihood or chi-squared**, and whether the **noise-normalisation constant** + (`-0.5 * sum(log(2 pi sigma^2))`) is included — it does not matter for the posterior and + it does matter for absolute evidence. +4. **Free parameters vs fixed configuration** — which arguments the search will vary, and + which are constants of their setup. + +Plus **what data it needs and how it is loaded**, including any selection cut the loader +applies: a cut is a scientific decision, so state it back rather than inherit it silently. + +**Then ask them to correct it** — that paragraph is the contract for every stage below, and a +misread convention costs a sentence here and a whole run at stage 6. + +## Stage 2 — Model + +*Skill:* `af_compose_model`. *Script:* the model section of the wrapper script. + +Compose `af.Model` (or `af.Collection` for several components) over a plain Python class +whose `__init__` arguments **mirror the free arguments** of their function — same names, +same units, their convention not yours. + +Choose each prior **with** them, one line of reasoning each: a prior is a scientific +statement, not a default. Physical bounds (a density fraction in [0, 1]) are uniform; +order-of-magnitude ignorance is log-uniform; a published measurement is Gaussian. + +Show `model.info`, and check the free-parameter count is the number they expected — a count +one too high usually means a fixed configuration value leaked into the model. + +## Stage 3 — Wrap and validate + +*Skill:* `af_wrap_likelihood` ("Branch — the wrapper", "Branch — validate before fitting"). +*Script:* `scripts/_analysis.py`, workspace style. + +Write the `Analysis` subclass. It **imports** their module — never copies it — holds the +data in `__init__`, and implements `log_likelihood_function(self, instance)` as pure +translation: build their calling convention from named attributes of `instance` +(explicitly, never by attribute iteration order) and return their number unchanged. + +Then the skill's three checks, on hand-built plain instances, before any search: + +1. a known-good instance returns a **finite** float; +2. that value **equals their function** called directly with the same inputs (assert it); +3. a deliberately bad instance scores **worse** (assert it) — check 3 catches the sign + flip, the most common wrapping bug and one a sampler will happily minimise their fit + quality with. + +**The data-inspection gate belongs here** and is never skipped: their data is real. Plot it +(or summarise it numerically if it is not plottable), announce the absolute path and offer +to open it (`_style.md` "Plot output and path announcement"), and ask **one** question about +known artefacts, outliers or selection effects. Time one likelihood call while you are here +— that number is stage 4's budget arithmetic. + +## Stage 4 — Search + +*Skill:* `af_configure_search`. + +Recommend one search **with the reason**, not just a name: + +- **evidence wanted** (model comparison now or later) → nested sampling; +- **posterior only** → MCMC, often cheaper; +- **a best-fit point** → an optimiser, accepting it returns a point, not a distribution; +- **likelihood is pure JAX** → gradient-based (`BlackJAXNUTS`, the `MultiStart*` optimisers). + +Read the installed roster from the library, never from memory. State the expected runtime as +(measured cost per call) x (evaluations the sampler needs); if that lands in hours, offer the +HPC route rather than letting a laptop cook overnight. Ask the **JAX triage question once** — +"is your likelihood JAX-compatible, or worth making so?" — and record the answer; never +convert their code uninvited. + +## Stage 5 — Go-ahead checkpoint + +**Show a short table and then STOP.** Four rows, no more: + +| | | +|---|---| +| **Model + priors** | free parameters and the prior on each | +| **Likelihood** | their function, its convention, the validation numbers | +| **Search** | the recommendation and its settings | +| **Expected runtime** | cost per call x evaluations | + +Then, in one line: nothing runs until they say go. **Run nothing** — not a short trial, not a +test-mode fit — until they do; if they change something in response, revise the table and stop +again. + +## Stage 6 — Run, then results on disk + +*Skills:* `af_run_search`, then `af_plot_fit` and `af_load_results`. +*Script:* `scripts/_fit.py`. + +Launch the fit, then do the **output-folder tour while it runs** (`_style.md` "Output folder +announcement") — the absolute `output////`, `model.results` +first, then `model.info`, `search.summary`, `files/` and `image/`, and the fact that they +update *as the search runs*. Say once that a completed search **reloads** rather than re-runs; +it surprises everyone once. + +Then the second half of the prompt's last sentence — how to inspect and interpret: + +- **The parameter table**: median-PDF values with 1-sigma errors, and the log evidence if + the search reports one (flagging the missing noise constant if their function omits it). +- **`af_plot_fit`**: model-vs-data with residuals, and the corner plot, both with paths + announced. Read the corner plot back to them — degeneracies, prior-edge pile-ups. +- **`af_load_results`**: reload through the aggregator without re-running, so the result + outlives the session. Never compose an output path by hand. +- **The check.** Real data has **no truth** to compare against, so the check is the + residuals — they should look like noise, and structure in them is model error. Say this + explicitly; a user arriving from a simulated-data tutorial expects a truth row. + +--- + +## Worked example — Pantheon+ supernova cosmology + +A real BYOL run against `dataset/sne_cosmology/`, so the numbers below are measured rather +than illustrative. The user's module loads the Pantheon+SH0ES distance table, applies their +cuts (`z_hd > 0.023`, calibrators dropped — **1371** supernovae), and defines +`log_likelihood(params: dict, z, mu, sigma)` returning `-0.5 * chi2` for a flat LCDM distance +modulus with **no noise-normalisation term** — exactly the convention stage 1 must surface. +Stage 2 gave two free parameters: `omega_m` uniform on [0, 1] (a density fraction of a flat +universe) and `h0` uniform on [50, 100] km/s/Mpc (wide enough to hold both ends of the Hubble +tension). + +Stage 3's validation, all three checks passing: + +```bash +check 1 finite : log L = -295.1774 +check 2 equals user's fn : -295.1774266904256 == user's function +check 3 bad scores worse : log L = -1726.6421 < -295.1774 +cost per likelihood call : 0.19 ms +``` + +Stage 4 chose `af.DynestyStatic(nlive=75)`: the evidence is wanted for a later comparison +against free-curvature and free-`w` models, and the likelihood is plain numpy, so the JAX +searches are unavailable. At 0.19 ms per call the budget arithmetic predicted a +seconds-scale run, which is what happened. + +Stage 6, **11.4 s** wall time and 3686 likelihood evaluations: + +```bash +parameter inferred (median, 1 sigma) dataset README validation +--------------------------------------------------------------------------- +omega_m 0.344 +0.021 / -0.019 0.344 +0.021 / -0.019 +h0 73.11 +0.27 / -0.30 73.14 +0.27 / -0.32 + +log evidence: -302.40 max log likelihood: -295.16 +residual scatter: 0.656 sigma +``` + +Read it back the way stage 6 says to: both parameters match the maintainer validation run +in `dataset/sne_cosmology/README.md`; there is no truth row because this is real data; and +the residual scatter below 1 sigma is the diagonal-errors approximation showing its hand — +the published analysis uses the full covariance matrix, the natural next wrap. The +aggregator reloaded the same medians and evidence from `output/` without re-running. + +--- + +## Teacher mode and questions + +Append this footer to **every** stage, verbatim: + +> Ask anything here - "what is a prior?", "why nested sampling?" - and I'll explain before we +> move on. Say "teacher mode" for full explanations at every step. + +- **A question never advances the stage.** Answer it, then re-offer the same stage. Only the + user moving on — or, at stage 5, the go-ahead — advances anything. +- **"Teacher mode" mid-run** switches `modes/teacher.md` on for the rest of the run: more + *why*, explicit assumptions, wiki and HowToFit pointers. The stages are unchanged; the + depth around them grows. + +## Hand-off + +When stage 6 is done, or the user stops: + +- One line: the session now runs in **assistant mode** (or **teacher mode** if chosen), and + the one-stage-per-turn pacing no longer applies. +- Name what is on disk — the wrapper and fit scripts, the fits under `output/`, the figures. +- **Record the wrap** (`af_wrap_likelihood` "Combine", default-yes): the function, its + convention, the validation numbers and where the code lives, into + `wiki/project/profile.md` "Likelihood & model code", so a future session skips the stage-1 + interview. Follow `AGENTS.md` "First-interaction protocol" — **ask before creating + `profile.md`**, and never create it in a maintainer clone. +- Where to go next: [`start-new-project`](../skills/start-new-project.md) to turn the + analysis into a standalone science project repo, + [`af_ingest_paper`](../skills/af_ingest_paper.md) for the domain context that makes prior + choices citable, and the RTD + [Scientific Workflow](https://pyautofit.readthedocs.io/en/latest/overview/scientific_workflow.html) + and + [Statistical Methods](https://pyautofit.readthedocs.io/en/latest/overview/statistical_methods.html) + pages for what comes after one fit. + +## Rules + +- **The published prompt is byte-identical** across `README.md`, this file and any other + surface that carries it — never paraphrase or reflow it. +- **Never fabricate output.** Every number, table and path shown comes from running the + code. If a run fails, say so and debug it (`af_debug_fit_failure`); an invented result is + this mode's one unrecoverable failure. +- **The invariant, restated:** the likelihood code is the user's. The wrapper translates and + nothing else — no re-parametrisation, no guards, no silently added noise constant. Say + what looks wrong; let them decide. +- **All `AGENTS.md` safety invariants apply**, code gate included: on a harness without + hooks, self-enforce with `python autoassistant/audit_skill_apis.py --file