Skip to content

feat(mcp): return the data and judgment procedures from the tools - #671

Open
mlieberman85 wants to merge 7 commits into
darnitdevorg:mainfrom
mlieberman85:feat/tool-procedures
Open

mlieberman85 wants to merge 7 commits into
darnitdevorg:mainfrom
mlieberman85:feat/tool-procedures

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Stacked on #670. Review the last 3 commits: the spec change, the procedures, and the skipped list.

Summary

darnit-comply told the agent to "follow the /darnit-data skill" and "follow the /darnit-audit skill". Nothing loads another skill's text, so the comply run improvised data collection from one line and asked questions differently from a darnit-data run (#654).

  • darnit.server.procedures holds the steps once:

    • get_pending_data returns them as procedure;
    • get_judgment_requests returns the judging steps on every page;
    • the baseline confirm_project_data result ends with the next get_pending_data call.

    Calls name only the owner and repo the caller passed, never ones read from the checkout's remote, and hold no candidate value.

  • skipped: a get_judgment_requests page names the controls it left out because they already have a stored PASS candidate, and the procedure says not to request them again. In the audit_openssf_baseline JSON output is about 1 MB and exceeds the MCP client's result limit #644 e2e runs the agent asked for them again and got empty pages.

  • Skills:

    • The skills defer to the procedures, and a test checks that no skill tells the agent to follow another one.
    • Comply and remediate pass owner and repo to remediate_audit_findings.
  • Spec: 5.4 and 7.x.

Refs #654.

Type of Change

  • New feature (non-breaking change adding functionality)

Framework Changes Checklist

  • Updated framework spec (docs/architecture/framework-design.md)
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

Testing

  • uv run pytest tests/ on the whole rebased stack (18bd63e): 5286 passed, 18 skipped
  • e2e: see the table below
  • uv run ruff check .

End-to-end

darnit-e2e: headless /darnit:darnit-audit openssf-baseline through Claude Code. Confirm tools are disallowed and the repository is named in the prompt.

darnit darnit repo wtf-frontend cosign-installer
main (2026-10-20; cosign-installer on 603bdda) 387s, 44 turns 415s, 47 turns 297s
#653 only (2026-10-21) 299s, 57 turns 250s, 55 turns 185s, 39 turns
full stack, rebased on 8070949 (2026-10-23) 282s, 50 turns, 30 judgments 145s, 23 turns, 15 judgments 141s, 30 turns
  • Results: per-control results match main. The only differences are controls the model judged, and the effect of operator-side answers on wtf-frontend, which needs the repository named. wtf-frontend also shows QA-03.01 and QA-07.01 passing because their platform settings changed on GitHub.
  • Stored candidates skew the comparison. A control with a stored PASS candidate for unchanged evidence is skipped, so run-to-run times depend on how many are stored. darnit's HEAD moved to 8070949, which invalidated its stored candidates, so it was judged in full (30 controls).
  • Denials:
    • darnit: one page of get_judgment_requests was larger than Claude Code's tool-result limit and was saved to a file. The agent's attempt to read it with Bash was denied. This is the oversized single-request follow-up.
    • wtf-frontend: the agent tried to run zizmor itself, and that was denied.

AI assistance

  • No AI assistance was used
  • AI assistance was used

Claude Code (Claude Opus) wrote the change, tests and this description; I reviewed them and ran the e2e.


Drafted with AI assistance (Claude), reviewed and filed by me. Measurements quoted were executed and verified.

Marc-cn
Marc-cn previously approved these changes Oct 29, 2026

@Marc-cn Marc-cn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed its own three commits. One source for the procedure text, calls name only the repository and checkout the caller passed (placeholders otherwise, never a detected value), confirm_project_data ends with the next call, skipped lists controls with a stored candidate. One leftover from #670: a PENDING llm_judgment control with no llm_consultation is still absent from requests, remaining and skipped, so an agent cannot tell why it never shows up. Rare, but listing it (with why) would close the gap. Good to merge in order.

A coding agent gets what it judges from get_judgment_requests: each
control's prompt, hints, and files, with one shared files map holding
exactly the content submit_judgment checks citations against, paged
under 40,000 bytes. Summary rows of a PENDING control carry
pending.kind and a candidate summary (5.4).

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
The json audit output is about 1 MB, because every control waiting on a
judgment carries its own copy of its files and all of its evidence. An
MCP client refuses a tool result that large, so the comply flow spilled
it to a file and parsed it with shell tools.

get_judgment_requests audits the checkout the same way submit_judgment
does and returns only the prompt, hints, evidence digest, and file keys
per control, with each file once in a shared map. Pages stay under
40,000 bytes; 'remaining' names the controls for the next call.
Controls with a stored PASS candidate are skipped unless include_judged
is set. It records nothing.

Summary rows of a PENDING control now carry pending.kind, and a
candidate's model, reasoning, and number of cited excerpts, so a client
can tell a judgment request from a candidate without the json output.

Refs darnitdevorg#644.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
…tput

darnit-audit and darnit-comply read summary rows, then page through
get_judgment_requests and cite from its files. The '~164K' size claims
in the skills and the audit tool description are gone.

Refs darnitdevorg#644.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
…erver

darnitdevorg#630 removed create_server_from_dict. The two tests that check the
get_judgment_requests registration now write a minimal framework TOML
and call create_server, as darnitdevorg#630 did for its own tests.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
get_pending_data and get_judgment_requests carry a 'procedure': the
ordered steps for the result, with calls that name only the repository
the caller named. A confirm_project_data result on a server with
get_pending_data ends with the next get_pending_data call. The text has
one home in the framework, so skills and clients defer to it.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
darnit-comply told the agent to "follow the /darnit-data skill" and
"follow the /darnit-audit skill", but nothing loads another skill's
text, so the comply run improvised data collection from one line and
asked questions differently from a darnit-data run.

darnit.server.procedures holds the steps once. get_pending_data returns
them as 'procedure' (show candidates labelled unconfirmed, ask every
question, per-question skip, one confirm_project_data call with
accept_candidates filled only after a yes, the next call, the review of
stored values, the summary), get_judgment_requests returns the judging
steps on every page, and the baseline confirm_project_data result ends
with the next get_pending_data call. Calls name only the owner and repo
the caller passed, never ones read from the checkout's remote, and hold
no candidate value.

The skills defer to the procedures instead of restating them, and no
skill tells the agent to follow another one (a test checks). The comply
and remediate skills pass owner and repo to remediate_audit_findings,
which needs them to find operator-side confirmations (darnitdevorg#612).

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
A page now names, in 'skipped', the controls left out because they
already have a stored PASS candidate, and the judging procedure says
not to request them again. In the darnitdevorg#644 e2e runs the agent called the
tool again with those ids and got empty pages back.

Refs darnitdevorg#644.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants