You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Eval] Score injection_guard through the official BrowseSafe++ SDK (browsesafe-pp) and prepare leaderboard predictions #119
#87 / draft PR #117 build our own BrowseSafe++ train-pair converter (scripts/build_browsesafe_plus_set.py) and score injection_guard with s1a run injection_guard. Perplexity has since published an official SDK and leaderboard for the benchmark (announcement, 2026-11-02, paper arXiv:2610.17457, quick start):
pip install "browsesafe-pp[eval] @ git+https://github.com/perplexityai/browsesafe-plus-plus.git" (not on PyPI yet).
bs.load(split="train", view=..., n_pairs=..., filters=...) yields attack then control per pair. A detector is predict(request) -> bs.Prediction(injected=True|False|None), where None means abstain.
bs.evaluate reports precision, recall, control-page FPR, paired correctness, errors and abstentions, with one attempt per page.
Test-split scores only come from the leaderboard Space. browsesafe predict ... --view html --view markdown --view accessibility writes a predictions JSONL that a signed-in HF account uploads (one submission per account per week, reviewed by maintainers).
Without an adapter we can't check that our converter's numbers match the reference scorer. We also can't produce a test-split number at all, because the test inputs are leaderboard-only and #87 correctly forbids scraping them.
Proposed work
A small adapter (for example s1a/evals/browsesafe_pp.py or under scripts/) that exposes predict(request). It wraps the existing injection_guard decision on request.content with the same 6,000-character clip (MAX_CHARS) the runtime uses. Map the act band to injected=True and the allow band to False. Map the uncertain band to None (abstain), and say so in the docs, because that is a choice and affects the comparison.
Optional dependency only. Nothing in core CI imports browsesafe_pp. Gate it behind an extra or uv run --with.
Leaderboard prep, not submission. Document the browsesafe predict command for all three views. Uploading to the Space under someone's HF account is a maintainer decision and stays out of the PR.
Acceptance criteria
Offline tests with a fake request object cover band-to-prediction mapping, clipping and error-to-failure handling (no network, no SDK install needed in core CI).
docs/benchmarks.md has a BrowseSafe++ SDK subsection with the exact commands, SDK commit, dataset revision and abstain mapping. The scores table says N/A until a run with HF_TOKEN + TYPESAFE_API_KEY exists, and no numbers are invented.
Test inputs are never vendored, cached in the repo or used for anything except browsesafe predict (they are licensed for evaluation only).
Activity