Skip to content

feat(backends): add experimental herdr-core runtime backend - #9

Open
Profreshor wants to merge 6 commits into
mainfrom
fm/herdr-core-backend
Open

Profreshor wants to merge 6 commits into
mainfrom
fm/herdr-core-backend

Conversation

@Profreshor

@Profreshor Profreshor commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Intent

Heads up, I'm trying the new herdr-core platform that was released today. The subagents you just assigned went to the old server while you're on the new one.

Decision: "Build herdr-core adapter now". The question was whether to build a herdr-core runtime for firstmate so workers launch on the new platform, or keep workers on the old herdr server for now.

What Changed

  • Add the experimental herdr-core (hc) runtime adapter, including backend selection, auto-detection, dependency/version checks, task-terminal ownership labels, and task metadata.
  • Support herdr-core task spawning, supervision, control, and safe relaunch/recovery using the recorded core session and terminal endpoint.
  • Document setup, behavior, verification evidence, and limitations, with a fake-hc regression suite covering the adapter lifecycle.

Risk Assessment

Medium: adds a new experimental runtime backend. Existing backends are unchanged apart from the known-backend lists, and herdr-core is used only when config/backend, FM_BACKEND, --backend herdr-core, or HC_TERMINAL_ID auto-detection selects it. Relaunching a finished hc terminal works: hc get on a finished terminal exits 0 with state=finished on live hc 0.1.13, and the live evidence below shows the owned finished terminal removed and rebound from t1 to t2.

Activation

Merging does not switch this home to herdr-core. After merge, firstmate sets config/backend=herdr-core to activate it.

Worker live check

Run on 2026-10-29 with hc 0.1.13 in an isolated hc core (HC_SESSION=fmhc-live), with a scratch Firstmate home and project; the operator's default core was not touched. Steps run: fm-spawn.sh --backend herdr-core (scout, Claude Haiku), an fm-send steer answered PINEAPPLE, fm-peek, fm-control interrupt (verified=agent-alive), fm-control exit (agent then read dead), relaunch adopting the live shell, relaunch after hc kill + hc rm rebinding to a new terminal in the recorded worktree, and teardown. Exact outputs are in docs/verification/runtime-backends.md#herdr-core.

Pre-existing local test failures

Two local failures were checked against an export of origin/main at 95495de, without this branch:

Test Result on origin/main Cause
tests/fm-test-run.test.sh under LANG=en_US.UTF-8 rc=1, comm: file 2 is not in sorted order from --check-coverage Locale-dependent sort; passes under LC_ALL=C, which CI uses
tests/fm-control-relaunch.test.sh rc=1, not ok - Herdr Pi relaunch should complete (pi registration) pi executable not found on PATH on this host

Both fail identically without this change.

Testing

Focused regression passed. Live Firstmate spawned a real Codex worker into a named hc core with recorded herdr-core metadata, removed a real finished owned terminal and rebound it to a new live terminal in the same worktree, and refused a foreign-labeled terminal. All labs and worktrees were torn down. The change-scope scan found no unrelated hunks.

  • Live validation: ✅ go - 3 of 3 scenarios driven live against the product
Scenario Result Live Evidence
Spawn worker on herdr-core ✅ pass live Live hc spawn/relaunch evidence
Relaunch a finished owned terminal ✅ pass live Live hc spawn/relaunch evidence
Refuse a foreign-labeled terminal ✅ pass live Live foreign-terminal ownership refusal
Evidence: Live hc spawn and finished-terminal rebind

Source: Live hc spawn and finished-terminal rebind

/tmp/fm-lab.hPhFzm
spawned hclive harness=codex kind=scout window=fm-lab-herdr-core-e2e-2194625-28071:t1 worktree=~/.treehouse/01M64SSQ4P7YDDBZ7T38CK6M6J-2f89d1/1/01M64SSQ4P7YDDBZ7T38CK6M6J
--- initial endpoint ---
fm-lab-herdr-core-e2e-2194625-28071:t1
{"address":"t1","argv":["/usr/bin/zsh"],"cols":200,"created_ms":1793311416245,"cwd":"~/.no-mistakes/worktrees/aa96bb817995/01M64SSQ4P7YDDBZ7T38CK6M6J","die_with_parent":false,"exit_code":null,"finished_ms":null,"foreground":{"argv":["node","/usr/bin/codex","--dangerously-bypass-approvals-and-sandbox","--disable","hooks","-c","notify=[\"bash\",\"-c\",\"touch '/tmp/fm-lab.hPhFzm/state/hclive.turn-ended'\"]","⁣FIRSTMATE_OP: v1 launch-brief: # Current worker role contract\nYou are a crewmate: an autonomous worker agent managed by firstmate.\nThis section establishes your current identity before every project or task instruction below and supersedes any conflicting role identity in those instructions.\nDo the assigned work yourself and report only to firstmate; do not adopt a firstmate or secondmate supervisor identity, delegate the task, run fleet supervision, or address the captain.\nYour steering inbox is `/tmp/fm-lab.hPhFzm/state/hclive.inbox`; this exact path belongs to your current task even when it is outside the worktree or under the supervising firstmate home, so read and acknowledge its messages and do not reject it as another home's state.\nNever inspect or change any other home's endpoint namespace; this authorization is limited to the exact task paths named by this brief.\nWhen this task works on Firstmate itself, the repository root `AGENTS.md` (also imported by `CLAUDE.md`) is project content and the supervisor contract for the firstmate managing you: follow this brief instead of that supervisor contract.\nProject instructions still govern the work wherever they do not conflict with this worker identity, including `CONTRIBUTING.md` and `firstmate-coding-guidelines` for Firstmate changes.\nIf the `firstmate-coding-guidelines` skill name does not resolve in this session, read `~/.no-mistakes/worktrees/aa96bb817995/01M64SSQ4P7YDDBZ7T38CK6M6J/.agents/skills/firstmate-coding-guidelines/SKILL.md` instead.\n\n# Task\n## Captain's intent\nExercise a disposable herdr-core worker.\n\n## Firstmate spec\nPermit lifecycle checks."],"cwd":"~/.treehouse/01M64SSQ4P7YDDBZ7T38CK6M6J-2f89d1/1/01M64SSQ4P7YDDBZ7T38CK6M6J","name":"node","pid":2196486},"identity":"ghostty","keep":false,"labels":{"fm_home":"firstmate-e8b89f06","fm_task":"hclive"},"name":null,"notify":false,"parent":null,"pid":2194984,"reason":null,"reported":false,"rm":false,"rows":120,"screen":"exact","signal":null,"state":"live","status":null,"status_ms":null,"template":null,"title":"","turn":0}
--- force terminal into real finished state ---
{"address":"t1","argv":["/usr/bin/zsh"],"cols":200,"created_ms":1793311416245,"cwd":"~/.no-mistakes/worktrees/aa96bb817995/01M64SSQ4P7YDDBZ7T38CK6M6J","die_with_parent":false,"exit_code":1,"finished_ms":1793311426858,"identity":"ghostty","keep":false,"labels":{"fm_home":"firstmate-e8b89f06","fm_task":"hclive"},"name":null,"notify":false,"parent":null,"pid":2194984,"reason":"killed","reported":false,"rm":false,"rows":120,"screen":"exact","signal":null,"state":"finished","status":null,"status_ms":null,"template":null,"title":"","turn":0}
finished-get-exit=0
{"address":"t1","argv":["/usr/bin/zsh"],"cols":200,"created_ms":1793311416245,"cwd":"~/.no-mistakes/worktrees/aa96bb817995/01M64SSQ4P7YDDBZ7T38CK6M6J","die_with_parent":false,"exit_code":1,"finished_ms":1793311426858,"identity":"ghostty","keep":false,"labels":{"fm_home":"firstmate-e8b89f06","fm_task":"hclive"},"name":null,"notify":false,"parent":null,"pid":2194984,"reason":"killed","reported":false,"rm":false,"rows":120,"screen":"exact","signal":null,"state":"finished","status":null,"status_ms":null,"template":null,"title":"","turn":0}
--- relaunch through Firstmate ---
spawned hclive harness=codex kind=scout window=fm-lab-herdr-core-e2e-2194625-28071:t2 worktree=~/.treehouse/01M64SSQ4P7YDDBZ7T38CK6M6J-2f89d1/1/01M64SSQ4P7YDDBZ7T38CK6M6J
rebound-from=fm-lab-herdr-core-e2e-2194625-28071:t1
rebound-to=fm-lab-herdr-core-e2e-2194625-28071:t2
{"address":"t2","argv":["/usr/bin/zsh"],"cols":200,"created_ms":1793311427736,"cwd":"~/.treehouse/01M64SSQ4P7YDDBZ7T38CK6M6J-2f89d1/1/01M64SSQ4P7YDDBZ7T38CK6M6J","die_with_parent":false,"exit_code":null,"finished_ms":null,"foreground":{"argv":["/usr/bin/zsh"],"cwd":"~/.treehouse/01M64SSQ4P7YDDBZ7T38CK6M6J-2f89d1/1/01M64SSQ4P7YDDBZ7T38CK6M6J","name":"zsh","pid":2197180},"identity":"ghostty","keep":false,"labels":{"fm_home":"firstmate-e8b89f06","fm_task":"hclive"},"name":null,"notify":false,"parent":null,"pid":2197180,"reason":null,"reported":false,"rm":false,"rows":120,"screen":"exact","signal":null,"state":"live","status":null,"status_ms":null,"template":null,"title":"","turn":0}
A new version of treehouse is available: v3.1.2 → v3.1.7
Run "treehouse update" to update

🌳 Worktree returned to pool.
teardown hclive complete (window fm-lab-herdr-core-e2e-2194625-28071:t2, worktree ~/.treehouse/01M64SSQ4P7YDDBZ7T38CK6M6J-2f89d1/1/01M64SSQ4P7YDDBZ7T38CK6M6J)
Backlog: hclive just finished (this home keeps no markdown backlog at /tmp/fm-lab.hPhFzm/data/backlog.md). Update /tmp/fm-lab.hPhFzm/data/backlog.md - move hclive to Done, keep Done to the 10 most recent, then re-scan Queued and dispatch only work whose blockers are gone and date is due.
Evidence: Live foreign-terminal ownership refusal

Source: Live foreign-terminal ownership refusal

/tmp/fm-lab.EHHaFD
foreign-terminal=t1
{"address":"t1","agent":"zsh","cwd":"~/.no-mistakes/worktrees/aa96bb817995/01M64SSQ4P7YDDBZ7T38CK6M6J","state":"live","turn":0}
adapter-capture-exit=1
agent-state=missing

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 error
  • 🚨 bin/backends/herdr-core.sh:302 - hc retains an exited task terminal as finished, but this maps to dead. Relaunch treats dead as adoptable, then its first cd is refused because target_ready accepts only live; a normal exited worker therefore cannot relaunch. Route finished terminals through a cleanup-and-rebind transition instead of adoption.

🔧 Fix applied.
3 issues (1 error, 2 warnings) still open:

  • 🚨 bin/fm-spawn.sh:2033 - With set -e active, a live shell terminal (or an already-missing terminal) makes fm_backend_herdr_core_remove_finished return 1, so the script exits before the following case $? can retain adoption/rebind behavior. Capture its status in an if/||-guarded command before branching, so only the helper's status 2 refuses.
  • ⚠️ tests/fm-backend-herdr-core.test.sh:262 - The new regression calls the removal helper and creates a terminal directly; it never executes fm-spawn.sh --relaunch, so it passes despite the failing relaunch wiring above. Exercise a recorded finished endpoint through the relaunch entry point and assert replacement session, terminal, and worktree.
  • ⚠️ docs/herdr-core-backend.md:52 - The guide says every agent-free terminal is adopted, but the required finished-terminal behavior removes the owned finished terminal and rebinds it. Distinguish a live shell (adopt) from a finished terminal (remove and recreate) to keep the operational contract accurate.

🔧 Fix applied.
1 error still open:

  • 🚨 bin/backends/herdr-core.sh:160 - hc get is documented to exit 11 for a finished terminal, but this accepts only 0 or 3 before parsing JSON. Thus an owned finished terminal is classified unreadable; relaunch neither removes nor rebinds it and refuses. Accept exit 11 as a parseable response, and make the fake get return 11 for finished terminals so the regression covers real CLI behavior.
✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 3 of 3 scenarios driven live against the product
Scenario Result Live Evidence
Spawn worker on herdr-core ✅ pass live Live hc spawn/relaunch evidence
Relaunch a finished owned terminal ✅ pass live Live hc spawn/relaunch evidence
Refuse a foreign-labeled terminal ✅ pass live Live foreign-terminal ownership refusal
  • bash tests/fm-backend-herdr-core.test.sh
  • Disposable named hc core: fm-spawn.sh ... --backend herdr-core --harness codex, real hc kill, then fm-spawn.sh --relaunch
  • Disposable named hc core: real foreign-labeled terminal checked through the herdr-core adapter
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Add bin/backends/herdr-core.sh so config/backend=herdr-core launches ship
and scout workers as herdr-core (hc) terminals, with treehouse still the
worktree provider. Each task is one labelled terminal (fm_home, fm_task)
pinned to its recorded hc session; ownership is proven by labels before
any read, send, kill, or remove.

Wire it through backend selection and validation, HC_TERMINAL_ID
auto-detection (ancestry decides when HERDR_ENV is also present), the
hc version gate, task metadata, the control plane (interrupt, exit,
relaunch with proven-absence rebind), busy and liveness classification,
and recorded-inventory discovery. Secondmate spawns are refused.

Document setup, detection, and limits in docs/herdr-core-backend.md and
record the live hc 0.1.13 verification pass.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T22:12:46.990193Z d130c30 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d130c30664

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

local label=$1 cwd=$2 session=${3:-} id home out addr
[ -n "$session" ] || session=$(fm_backend_herdr_core_session)
id=${label#fm-}
home=$(fm_backend_hometag)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Include the operational home in terminal ownership labels

When two independent FM_HOME instances share the same code root and HC_SESSION, this call produces identical ownership labels: fm_backend_hometag hashes FM_ROOT and uses FM_HOME only for the optional secondmate marker. Consequently, matching task IDs cause one home's launch to be refused because the other home's terminal exists, and the ownership check in fm_backend_herdr_core_get accepts that other home's terminal as owned. I reproduced both homes receiving the same tag and the second home accepting the first home's terminal. Include the resolved operational-home path in the discriminator to preserve AGENTS.md's prohibition on claiming another home's work.

AGENTS.md reference: AGENTS.md:L123-L123

Useful? React with 👍 / 👎.

… unknown-backend diagnostic expects the newly registered herdr-core backend. The previously failing assertion now passes during the bootstrap suite; tests/fm-backend-herdr-core.test.sh passed locally. The full bootstrap suite exceeded this environment’s 30s command window after the relevant assertion passed; no unrelated diff hunks were flagged by jev. Risk: only the stale expected diagnostic was changed
@Profreshor
Profreshor force-pushed the fm/herdr-core-backend branch from db210e3 to 94b9583 Compare October 30, 2026 00:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant