Repository navigation
bench: seeded-vulnerability security bench and scorer - #1973
Merged
Merged
Conversation
12 synthetic diffs each seeding one known vulnerability plus 4 safe look-alike decoys, keys.json with expected CWE, file:line, marker token and CWE equivalents, and security_bench_score.py scoring any arms on precision, recall (bootstrap CI via rival_eval.boot) and decoy false positives. Normalizes CWE ids and paths, collapses duplicate findings, scores file+line when a tool gives no CWE, and fails closed (exit 2) on a missing arm dir or findings file, malformed findings, invalid keys or duplicate arm names. Offline; no model calls. Tests: 16/16 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #1968. It carries the same change as one squashed commit on current main, plus the
size-budget-raise:lines that the size ratchet required and that failed #1968. The full description and review history are on #1968.Tests: privacy clean, size-ratchet pass, plus the PR's own suites (see the commit message).
🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z