Conversation
…triggers gen_spec_v2 is an alternative to gen_spec that answers questions v1's schema cannot express: which system variables a rule depends on, whether it must be checked before or after the tool runs, what tool lookups or chat history it needs, and what the user has to supply before it can be enforced at all. Output format follows smith's spec_generation byte-for-byte (fixed key order, omit-when-empty), so specs are interchangeable with that corpus. New per-item fields: trigger pre_tool | post_tool requires system_vars / tool_history / message_history pending_for_user missing_tool | missing_var | clarification resolved_by_user the answers, once given conflicts contradictions across the spec set debug tool_info, archive (rejected items + why), notes Four entry points, because the caller knows when a complete set exists: generate_guard_specs_v2 one spec per tool, nothing else generate_spec_conflicts_v2 conflicts across a complete set generate_guard_specs_v2_full both generate_guard_examples_v2 re-run only the examples stage System variables come from a dict or a sys_var.json path. Values of any shape are kept (nested mappings and lists included); the agent's own action catalog (action_list, action_description) is excluded as it describes tools, not the acting user. The rendered block gives the model the exact access path (input.extensions.subject.<name>) and domain, and code drops any variable the model invents but the file does not declare. Codegen path: specs_v2_to_v1 adapts v2 to the v1 schema gen_py already consumes, deriving skip = pending_for_user OR message_history OR post_tool OR system_vars. gen_py is unchanged. Reference grounding (refmatch) snaps each quoted reference back to the exact policy-document substring via a normalized-to-original offset map, with a difflib fallback. An item whose references ground in nothing is archived rather than kept, so a rule quoted from a tool docstring cannot masquerade as policy. Tests: 273 pass and 13 skip on a fresh checkout (the 13 need the employee ground-truth corpus under tests/data/, which is not committed). e2e coverage mirrors v1's calculator variants (functions, methods, langchain, OpenAPI dict), plus tau2 airline and an opt-in v1-vs-v2 guard-set delta report (-m delta). Known issue: on tau2 airline, a one-sentence policy can still bind to tools it does not govern (over-attachment). review_votes=5 is the current mitigation; see docs/superpowers/evidence/2026-08-11-gen-spec-v2-test-baseline.md. Backward compatibility: v1's loader accepts v2 files while ignoring the v2 fields, which yields all skip=False — point v2 at its own work_dir. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ithout it Removing tests/data/ from the commit changes what a fresh clone measures: 273 passed + 13 skipped rather than 488 passed. Record both, and how to restore the corpus. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lone The 13 corpus tests skipped on a fresh checkout, which meant the byte-parity, reference-grounding and codegen-identifier gates were not running for anyone who had not copied smith's employee output into tests/data/ by hand. tests/examples/employee_mini/ is that corpus at minimum size: six of the 28 ground-truth specs, copied byte for byte, chosen so between them they cover 70 of the 71 distinct key paths in the full set — both sides of the adapter's skip decision, all three pending types, both triggers, tool_history, message_history, conflicts present and absent, and the empty spec. Real generator output rather than hand-written fixtures, so these gates keep testing what a model actually produces. Three derivations, documented in the directory's README: the six-spec subset, guidance.txt truncated to the rules those specs quote (seven unreferenced rule bullets dropped), and global.json's conflict targets pruned to the items the subset contains. The five test files now read this slice unconditionally, so requires_corpus is gone and tests/data/ is gitignored — the full 28-spec example and its benchmarks belong to the evaluate-toolguard project. Verified with tests/data/ absent: 332 passed, 0 skipped, 0 failed. The gates run fewer cases than the full corpus (27 references rather than 95, 6 fixtures rather than 28) but gate the same behaviors; both counts are in the baseline doc. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A policy document that quotes a literal -- obtain the user's explicit
confirmation ("yes") -- makes models escape the inner quotes in one field
and leave them bare in another, in the same reply. The payload is complete
and brace-balanced; only the parse fails.
The retry loop could never fix that. It re-sent an unchanged message list,
so the model had no new information and reproduced the same mistake on every
attempt: five attempts were one attempt billed five times. Make the retry a
repair turn instead -- the rejected reply and the parser's own error text go
back to the model, which knows which quote was content. That covers every
malformation class, not just quotes.
Two consequences of fixing the loop:
- The backoff sleep between attempts is gone. It bought nothing for a
formatting error; rate limits and timeouts are already retried in
LitellmModel._generate, a layer down.
- A model that answers a correction with a byte-identical reply is abandoned
at once rather than burning the remaining attempts on a proven dead end.
Only once the model's own attempts are spent does the parser guess, by
escaping bare quotes inside string values with a string-aware scan. That
guess is fallible and cannot be made otherwise -- a content quote followed
by a comma is indistinguishable from a delimiter, and guessing wrong can
silently shorten a value:
{"a": "say "x", "b": "y"} -> {'a': 'say "x', 'b': 'y'}
which parses cleanly with the tail of 'a' gone. In a guard spec a truncated
rule reads as well-formed while encoding half a rule, so the salvage runs
last and logs a warning naming the repair count.
Candidate extraction is now brace-balanced and string-aware, so a brace
inside a string value no longer ends the object early, and an unfenced reply
stops at its own closing brace rather than swallowing trailing prose.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run that quietly produces 31 specs for 33 tools ships two tools unguarded, and the shortfall is indistinguishable from a smaller request: we found the two missing tools by counting files in the output directory. Add SpecGenerationError, shared by both generators. It carries both sides of the outcome -- the specs that generated cleanly and were written, and the ToolFailure entries naming those that did not -- so one except clause works across v1 and v2. v1 fanned out with a bare asyncio.gather, so the first exception discarded every other tool's finished spec: 33 tools lost to one unparseable response. Isolate each tool, then raise once at the end. v2 already isolated per tool, but recorded the loss only in a logger.error and returned a short list. Its ToolErrorPolicy enum was also declared and never wired to the field, which was a bare str -- so on_tool_error="skipp" silently meant "raise". Wire the enum up and add a third policy: skip return the partial list, as before raise abort at the first failure raise_at_end finish and write every other tool, then raise (new default) raise_at_end is the default because a missing guard spec is a security outcome, not a logging detail. Nothing that succeeded is lost -- the files are written before the raise -- so a failure still costs only its own tool. This changes public behaviour: callers relying on a partial list must pass on_tool_error="skip". test_a_failing_tool_does_not_abort_the_others tracked the old default and now asserts isolation via the exception's .specs. The field uses Field(default=...) rather than a bare member assignment because the pre-commit mypy hook runs without the project installed, so it cannot resolve the StrEnum compat shim and reads the member as a plain str. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_calculator, test_gen_spec_v2_codegen and test_guard_set_delta all
failed to collect with "No module named 'examples'". The package is not
missing -- it is tests/examples -- but they imported it as a top-level
name, which only works with tests/ itself on sys.path. It is not: tests/
has an __init__.py, so pytest walks past it and inserts the repo root,
which makes tests.examples importable but not examples.
Import them as tests.examples.calculator.inputs instead of adding tests/
to sys.path, which would also expose buildtime, runtime, data and tmp as
top-level names and leave a shadowing hazard for anyone later writing
"import runtime".
Add the empty tests/examples/__init__.py that was the only gap in the
chain, so the import resolves through regular packages rather than a PEP
420 namespace portion wedged between two of them.
Collection goes from 357 tests with 3 errors to 367 with none.
Note this un-hides test_guard_set_delta, which is marked @pytest.mark.delta
and documented in pyproject as opt-in ("two full LLM spec runs"). Nothing
deselects it -- addopts has no -m filter -- so it only stayed unrun because
this import was broken, and a plain "pytest tests" now includes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.