Repository navigation
bench: seeded-vulnerability security bench and scorer - #1968
Closed
remigiusz-antczak wants to merge 2 commits into
Closed
remigiusz-antczak wants to merge 2 commits into
remigiusz-antczak wants to merge 2 commits into
Conversation
12 synthetic diffs each seeding one known vulnerability plus 4 safe look-alike decoys, keys.json with expected CWE, file:line, marker token and CWE equivalents, and security_bench_score.py scoring any arms on precision, recall (bootstrap CI via rival_eval.boot) and decoy false positives. Normalizes CWE ids and paths, collapses duplicate findings, scores file+line when a tool gives no CWE, and fails closed (exit 2) on a missing arm dir or findings file, malformed findings, invalid keys or duplicate arm names. Offline; no model calls. Tests: 16/16 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z
Collaborator
Author
|
Superseded by #1973 (merged). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a seeded-vulnerability bench so the security review path can be scored on precision and recall against a rival security-review tool, instead of on judge preference.
scripts/eval-fixtures/security-seeded/: 12 small synthetic diffs, one seeded vulnerability each, plus 4 look-alike decoys that are safe. The seeded classes are SQL injection, XSS, SSRF, path traversal, missing authorization, a hardcoded secret, unsafe deserialization, command injection, open redirect, IDOR, TOCTOU and weak crypto.keys.jsonmaps each diff to its CWE and line, with a short list of equivalent parent and child CWEs. All code and data are fictional, and the fake secret uses an obviously fake shape.scripts/security_bench_score.pyscores each arm's findings (<diff-id>.json, a list of{file, line, cwe}):scripts/test_security_bench_score.py(16 tests) is wired intoscripts/test-ci-gates.sh. It checks that each keyed line holds the vulnerable statement and covers the tolerance boundary, normalization, equivalence and the fail-closed paths.No paid run yet. The next step is one capped run of each arm on the 16 diffs, then a scored comparison.
Tests: 16/16 (test_security_bench_score.py), run on the commit's own tree. Independent review: 0🔴; the 9🟡 and 2🔵 are fixed here.
🤖 Generated with Claude Code
https://claude.ai/code/session_014QNFFPWVcPUbeuvwM5fk2Z