Repository navigation
Release 0.6.0 - #147
Merged
Merged
Release 0.6.0#147
Conversation
Add load_healthbench_scenarios to build HealthBench scenarios at run time (PR #93) # Conflicts: # simpleaudit/__init__.py # simpleaudit/scenarios/__init__.py
Twenty-four audit passes over the grading core, judges, targets, experiment caching, tracing and docs. Every confirmed finding is fixed at its root with a regression test; AUDIT.md records each finding, fix, verification result, the pass log and 22 documented remaining risks. Highlights: - Security: remove a committed Open WebUI API key default from examples/audit_openwebui_rag.py (rotation is still required); redact credentials in span attributes, span names, header text, tool events and URLs without hiding evidence the judge grades; fail-closed OTLP auth; auditor no longer inherits judge credentials across providers. - Grading: verdict parsing and severity normalisation hardened (stray keys, think tags, off-ladder values, contradictions); SingleTurnAuditor and ModelAuditor now grade the same input the same way (documents, default prompts, validation order, scenario_meta, empty ASR transcripts). - Experiment cache: content-based fingerprints (judge logic, scenario text, dates, attachment bytes), safe quarantine of stale runs, resume re-runs only errored scenarios, case-insensitive label collisions. - Tracing: 16 MiB body cap on HTTP and gRPC receivers, lowercase OTLP ids, corrected exporter env-var docs. - Docs/examples: corrected README claims and the inverted custom-judge rubric; examples no longer crash on ERROR results. Verification: 2396 passed, 2 skipped; `ruff check --select E9,F821` clean.
Pass 25 (partial, Haiku review of pass-24 changes): file:// URIs with percent-escapes were fingerprinted by path, so a regenerated clip could reuse stale cached runs. Decode the path as the ASR target does. Records pass 25 in AUDIT.md; the two-clean-pass criterion is still open.
AUDIT.md is an internal working log and must not be published.
- Redact secrets in span/tool text that is not strict JSON (Python repr, truncated or trailing JSON), env assignments (*_PASSWORD=) and curl -u. - Redact URL passwords containing / or @; treat pwd/pass/jwt/sid/session_id query parameters as secrets. - Resume with adaptive_reruns off sets aside saved adaptive reruns instead of silently dropping them from the combined results. - Inspect severity_counts counts between-step values toward the more severe. - Local attachments with # or ? in the name are typed by their real extension. - An off-ladder default-judge severity records parse_error and the reason.
- Checklist: duplicate entries that agree on status keep the one whose quote verifies, so the grade no longer depends on entry order; violated optional items are not listed as issues. - Saved results get the usual umask mode instead of mkstemp's 0600. - Redact provider API keys (sk-, ghp_, AKIA, xox, AIza, hf_) in free span and tool text; config.json redaction covers connection strings, pwd, dsn and signing keys. - A fully cached experiment is not reported cancelled; a combined results file listing models this run omits is set aside instead of overwritten.
Found while testing PR #143 live against an OpenAI-compatible endpoint: - AuditExperiment(max_parallel_reps=N) runs independent repetitions of run_scenario_reps concurrently (default 1 keeps the old behaviour); live, 4 reps went from 26 s to 7 s with results in rep order. - ModelAuditor.aclose() closes the HTTP clients it created; sync run() and each experiment repetition call it, so clients no longer close on a dead event loop ('Event loop is closed' tracebacks, sockets held open). - An auditor probe that is empty only after <think> stripping is retried under max_retries instead of failing the scenario at once.
…ps every attribute parse_otlp_json and the protobuf path dropped span events and the status message, so why a span failed (OpenTelemetry records exceptions as an 'exception' event) never reached the store or the judge. Both are now kept (status_message, events) and pass through normalize_span. summarize_for_judge cut each span's JSON at 2,000 characters from the front, so a long input history pushed later attributes such as the model output out of view. It now shortens long values first (max_chars_per_value, halved until the span fits) so every attribute stays visible, and shows a failed span's status message and exception.
main had moved 91 commits past dev (release 0.5.0, 0.5.1, new packs and judges). dev's only own commit was the healthbench-loader merge, whose content is already on main through PR #93, so this merge takes main's version of simpleaudit/__init__.py. Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
The main branch ruleset requires this check, so feature branches and forks have to go through dev first.
simpleaudit.calibration scores a judge's stored verdicts against a reference that is independent of it: human labels typed in the terminal with label(), or the same transcripts re-graded by a judge from another model family. Any number of labels works; every rate carries its n and an interval, and the report says when the count is only indicative. Adds krippendorff_alpha to simpleaudit.stats, pinned to the published worked example. rejudge() now records the new judge model in the result metadata instead of carrying the source judge's, so a rejudged file can serve as a reference.
Krippendorff's alpha corrects for chance from the observed category frequencies, so when nearly every reference item is pass it reads low at near-perfect agreement: 98 pass and 2 low with 97 matches gives 0.39. Gwet's AC2 estimates chance agreement from how evenly the levels are used and reads 0.99 on the same data. The report now prints both, with the reference level counts between them, and leaderboard() ranks by AC2. gwet_ac in simpleaudit.stats handles any number of raters, missing ratings, and identity, linear or quadratic weights; the tests pin it to the two-rater closed form and to Bennett's S under uniform marginals.
Audit and remediation: 26 passes of fixes and regression tests
Resolve summarize_for_judge: keep per-value shortening and failure lines from this branch, and redact attributes and failure text as dev does.
reframing.rejudge_async: take dev's metadata handling (superset of this branch's; calibration tests pass). stats: union of imports.
fix(tracing): keep span events and status messages; judge summary keeps every attribute
Add judge calibration against human labels or a cross-family judge
chore(release): bump SimpleAudit to 0.6.0
Contributor
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 0.6.0: merges dev into main.
CI green on dev; unit suite 2447 passed; live end-to-end tests against Qwen3.8-27B passed.