Skip to content

intake reconcile: see a prompt that shipped silently, and one filed twice - #307

Merged
Jammy2211 merged 3 commits into
mainfrom
claude/automind-dashboard-regen-5z7dym
Aug 27, 2026
Merged

Jammy2211 merged 3 commits into
mainfrom
claude/automind-dashboard-regen-5z7dym

Conversation

@Jammy2211

@Jammy2211 Jammy2211 commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Two blind spots in intake reconcile, both found by drift that reached the dashboard as pickable backlog, both invisible to every existing guard. Filed as draft/feature/pyautobrain/intake_reconcile_absence_signal.md (PyAutoMind#353) and draft/feature/pyautobrain/reconcile_duplicate_prompt_signal.md.

Leg 4 — the absence signal (--repo)

Leg 3 tests PRESENCE: "the prompt names things that exist upstream". That is structurally blind to a task which shipped leaving no Mind-side trace while the files it names still exist.

smoke_install_stale_jax_pin.md shipped 2026-08-23 (autolens_workspace_test#266, PR #268) with no active.md entry, no active/ move and no complete/ record. Both guards passed it: lifecycle.py check has no invariant for a prompt that was never active/, and reconcile --repo autolens_workspace_test, pointed straight at the right repo, found 0 suspects of 132. It named smoke_install.sh, which still exists.

What it quoted did not: pip install "jax<0.7" "jaxlib<0.7".

The filters, and how they were arrived at

The first cut had three filters and passed its tests. Then it was run against the live backlog (134 prompts, read at autolens_workspace_test) and produced six hits, all six false — with the loudest the most wrong. Two more filters took that to 0, while the retired prompt, restored into draft/ to reproduce, still fires.

Filter The false positive it kills
the fence carries an explicit source language a traceback or pytest summary goes in a bare fence, and those lines are absent from every repo by construction
the prompt names a file that exists upstream with that extension without an anchor, "absent from what?" has no answer
the line is absent from the whole checkout, not just that file a refactor that moved a line is not the prompt shipping
the prompt mentions the repo being read mge_jit_regression_rebaseline.md is about autolens_workspace_developer and was being read at _test. The Repos: header had this answer all along, while --repo had to be told the target by hand — the open question the filing prompt raised
the anchor is corroborated — at least one quoted line of that kind is still present in the named file proposed code. einstein_radius_jit_native_seed_finder.md quotes 23 lines of a seed finder it wants written; all 23 are absent because none ever existed, and at 101.0 it outscored every true positive

The motivating case passes corroboration exactly: it quoted two lines from smoke_install.sh — the PyAuto install line, still there, and the jax pin, gone.

Feeds upstream_score only, never score: a quoted line can vanish for reasons other than the prompt shipping, so it ranks for review and never retires.

Leg 5 — duplicate candidates (offline)

Every other signal scores a prompt against the completion archive. Nothing scored the live prompts against each other.

That let two filings of one finding both stay live: bug/workspaces/jax_likelihood_pins_stale_by_1e4.md (filed 08-14) and bug/autolens/jax_likelihood_smoke_pins_stale.md (filed 08-19) — same three scripts, same failing smoke gate, five days apart, neither naming the other. The 08-19 copy was verified and retired on 08-26; the 08-14 copy kept rendering as pickable backlog until a drift audit found it on 08-27. Reconstructed from the two records' ## Original prompt sections, the new leg pairs them at 12.0 on the shared script paths.

Measured on the live backlog: 36 pairs → 2.

Filter Why
bare basenames never count; a path named by >4 prompts is dropped start_here.py, modeling.py, no_run.yaml are workspace-wide conventions — sharing one is a convention, not a task. This alone removed 31 of the 36
mutual reference disqualifies a phased parent and its child name each other — a series
a folder index naming both disqualifies the four draft/bug/health_fixes/ prompts were split by cause, so they share failing scripts and none names another; the folder's README.md names all four

Runs on the default offline path — it reads only draft/.

Two bugs found on the way

  • _PATH_RE opened with \b, which cannot match before a leading dot, so .github/workflows/release.yml silently truncated to a path resolving against no file. Dot-directories are where CI recipes live, which is half of what these prompts are about. Caught by a test.
  • The tenant firewall rejected the first two commits: the new comments and the test's section header named a concrete satellite repo in .py files. Genericised rather than allowlisted — growing FIREWALL_ALLOWLIST would mean one more file an adopting fork must rewrite, for a name carrying none of the finding. AGENTS.md keeps the concrete names deliberately (prose is out of the firewall's scope by design). The test file was already violating its own header, which promises every fixture is fictional.

The --repo seam becomes _UpstreamReader — callable for leg 3, .quotes for leg 4, probed with getattr so every pre-existing lambda fake still exercises leg 3 alone.

Testing

15 new tests, mostly refusals: quoted output, moved lines, an unanchored quote, proposed code, the wrong repo, one shared path, declared series, offline, never-writes. 584 pass, tenant firewall OK.

Two of the original positive fixtures were wrong in the same way the false positives were — they quoted only absent lines, which the real prompt never did. The live run caught that; the tests alone would not have.

Neither leg can produce a shipped verdict or retire anything: both stay advisory, as reconcile already promises.

claude added 3 commits August 27, 2026 19:58
…ne filed twice

Two blind spots, both found by drift that reached the dashboard as pickable
backlog, both invisible to every existing guard.

**Leg 4 — the absence signal (`--repo`).** Leg 3 tests PRESENCE: "the prompt
names things that exist upstream". smoke_install_stale_jax_pin.md shipped
2026-08-23 (autolens_workspace_test#266, PR #268) with no Mind-side trace at
all, and both guards passed it — `lifecycle.py check` has no invariant for a
prompt that was never `active/`, and `reconcile --repo autolens_workspace_test`
pointed straight at the right repo found 0 suspects of 132. It named
`smoke_install.sh`, which still exists. What it QUOTED — `pip install "jax<0.7"
"jaxlib<0.7"` — did not. Verified against the real clone: the gone line flags,
the still-present install line in the same fence does not.

Three filters keep it off every prompt that quotes its own evidence, all three
required: an explicit source-language fence (a traceback goes in a bare fence,
and those lines are absent from every repo by construction); a named file that
exists upstream, so "absent from what?" has an answer; and absence from the
whole checkout, so a moved line reads as the refactor it is. Feeds
`upstream_score` only — a quote can vanish for reasons other than shipping.

**Leg 5 — duplicate candidates (offline).** Every other signal scores a prompt
against the completion ARCHIVE; nothing scored live prompts against each other.
That let two filings of the same finding — bug/workspaces/jax_likelihood_pins_
stale_by_1e4.md (08-14) and bug/autolens/jax_likelihood_smoke_pins_stale.md
(08-19), same three scripts, same failing gate — both live until a drift audit
found the survivor nine days later. Reconstructed from the two records'
`## Original prompt` sections, the new leg pairs them at 12.0 on the shared
script paths.

Measured on the live backlog: 36 pairs -> 2. The filter that did almost all of
that is dropping BARE BASENAMES — `start_here.py`, `modeling.py`, `no_run.yaml`
are workspace-wide conventions, and sharing one is a convention, not a task.
Mutual reference and a folder index naming both disqualify a pair: a phased
parent and child, and the four health_fixes/ prompts split by cause whose
README names all four, are declared series rather than accidents.

Also fixes a real parsing bug the tests caught: `_PATH_RE` opened with `\b`,
which cannot match before a leading dot, so `.github/workflows/release.yml`
silently truncated to a path resolving against no file. Dot-directories are
where CI recipes live, which is half of what these prompts are about.

The `--repo` seam becomes `_UpstreamReader` — callable for leg 3, `.quotes` for
leg 4, probed with getattr so every pre-existing lambda fake still exercises
leg 3 alone. 13 new tests, mostly refusals; 582 pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vd6ptkoJsjAJMosPWsCDZz
… its repo

Running leg 4 across the live backlog (134 prompts, read at
autolens_workspace_test) surfaced six hits and every one was false. Two causes,
two filters, 6 -> 0 — while the retired smoke_install_stale_jax_pin.md, restored
into draft/ to reproduce, still fires.

**Proposed code.** einstein_radius_jit_native_seed_finder.md quotes 23 lines of
a JAX-native seed finder it wants WRITTEN. All 23 are absent because none has
ever existed, and at 101.0 it outscored every true positive — the loudest thing
in the output was the most wrong. delaunay_research.md is the same shape.

The fix is to require the anchor to be CORROBORATED: at least one line the
prompt quotes for that extension must still be PRESENT in the file it names. One
present line proves the prompt is talking about this file, in this checkout, and
only then does a sibling line's absence mean something happened. The motivating
case passes it exactly — it quoted two lines from smoke_install.sh, the PyAuto
install line still there and the jax pin gone.

**The wrong repo.** A prompt read against a repo it is not about quotes lines
absent from it by construction. mge_jit_regression_rebaseline.md is about
autolens_workspace_developer and was being read at _test. The prompt that asked
for this leg raised it as an open question ("consider whether the Repos: header
should drive the upstream target"): the header had the answer all along while
--repo had to be told by hand. Now a prompt that never mentions the repo under
test is skipped, with the slug carried on the reader.

The two positive fixtures were wrong in the same way the false positives were —
they quoted only absent lines, which the real prompt did not. Made faithful, and
both new refusals pinned. 584 pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vd6ptkoJsjAJMosPWsCDZz
CI's tenant firewall caught what the tests could not: the new leg-4 comments and
the test's section header named a concrete satellite repo, in .py files. The
framework organs must stay adoptable as a config-diff fork, so an instance fact
outside the declared config surfaces is drift — and growing FIREWALL_ALLOWLIST
instead would mean one more file an adopting fork has to rewrite, for a name
that carries none of the finding.

The shape is the evidence, not the tenant: "a workspace-test repo, issue #266 /
PR #268" says everything the comment needed. The dates, issue and PR numbers are
not instance facts and stay.

AGENTS.md keeps the concrete names deliberately — skills prose and AGENTS.md are
out of the firewall's scope by design (production prompts, never genericised),
and the runnable `--repo` example there is more useful naming a real repo.

The test file was already violating its own header, which promises every fixture
is fictional and that nothing in it names a real prompt or record.

tenant firewall: OK. 584 pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vd6ptkoJsjAJMosPWsCDZz
@Jammy2211
Jammy2211 merged commit 3b4eedf into main Aug 27, 2026
2 checks passed
@Jammy2211
Jammy2211 deleted the claude/automind-dashboard-regen-5z7dym branch August 27, 2026 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants