Repository navigation
fix: remove broken doc references and annotate unread threshold field - #94
Open
SushilPoudel2005 wants to merge 1 commit into
Open
SushilPoudel2005 wants to merge 1 commit into
SushilPoudel2005 wants to merge 1 commit into
Conversation
The module docstring in case_protocol.py cited docs/scoring-protocol.md and docs/case-design.md, neither of which exists in the repository. Remove that sentence to prevent confusing contributors who follow the dead links. Add a comment above VISUAL_SSIM_PASS noting that each checkpoint in visual-suite.json carries a threshold field that the scorer does not currently read -- the constant applies uniformly across all checkpoints. This documents the intent gap from issue TIGER-AI-Lab#24 for a future follow-up. Applied across all 70 task-bundled copies of case_protocol.py, which are identical and contain no task-specific logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: remove broken doc references and annotate unread threshold field
Refs #24
What this changes
Problem 1 — Broken docstring references
The module docstring in
case_protocol.pyends with:Neither file exists in the repository. I tried to follow both links while
reading the codebase and hit dead ends. This PR removes that sentence.
The paragraph it closes is complete without it.
Problem 2 — Undocumented intent gap for
thresholdEvery checkpoint in every task's
visual-suite.jsoncarries athresholdfield. Across the benchmark there are 9,505 checkpoints at
0.99and 287 at0.99994(verified with a quick grep). The scorer does not read the field —it always uses the module-level constant
VISUAL_SSIM_PASS = 0.99. The 287higher-threshold checkpoints silently run the wrong bar at evaluation time.
This PR adds a three-line comment above
VISUAL_SSIM_PASSto make the gapvisible. It doesn't wire in the threshold yet, because doing that correctly
requires the result
visualsitems to carry checkpoint IDs (they currentlydon't), which is a schema change. I'd rather surface the gap clearly and get
direction before touching the result schema.
What's not in this PR
The
budgetssubfields beyondbuild_timeout_secinevaluation-contract.json(
cpus,memory_mb,startup_timeout_sec,storage_mb) are present in all71 contracts and never enforced. I left those out deliberately — enforcement
requires infrastructure-level hooks I can't reach from
case_protocol.py.Question for maintainers
For both the per-checkpoint
thresholdand the unused budget fields (cpus,memory_mb,startup_timeout_sec,storage_mb): would you prefer to removethe unused fields and clean up the contracts now, or keep them as forward-looking
hooks and wire in enforcement in a follow-up? I can extend this PR in either
direction within the next few days if you have a preference.
Scope and approach
All 70 task-bundled copies of
case_protocol.pyare byte-for-byte identical(verified with md5). There is no single canonical source in
src/; the fixwas applied uniformly to all copies with a small Python script.
Checks run locally:
uv run ruff check scripts src tests— passuv run ruff format --check scripts src tests— passuv run python scripts/check_tasks.py --code-only— all 70 tasks passNote:
tasks/is excluded from ruff viaextend-exclude = ["tasks"]inruff.toml, so the modified files are not linted. The changes aredocumentation and comments only, with no runtime logic.
🤖 Generated with Claude Code