Skip to content

fix: remove broken doc references and annotate unread threshold field - #94

Open
SushilPoudel2005 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
SushilPoudel2005:fix/24-sealed-contract-fields
Open

SushilPoudel2005 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
SushilPoudel2005:fix/24-sealed-contract-fields

Conversation

@SushilPoudel2005

Copy link
Copy Markdown

fix: remove broken doc references and annotate unread threshold field

Refs #24

What this changes

Problem 1 — Broken docstring references

The module docstring in case_protocol.py ends with:

See docs/scoring-protocol.md and docs/case-design.md.

Neither file exists in the repository. I tried to follow both links while
reading the codebase and hit dead ends. This PR removes that sentence.
The paragraph it closes is complete without it.

Problem 2 — Undocumented intent gap for threshold

Every checkpoint in every task's visual-suite.json carries a threshold
field. Across the benchmark there are 9,505 checkpoints at 0.99 and 287 at
0.99994 (verified with a quick grep). The scorer does not read the field —
it always uses the module-level constant VISUAL_SSIM_PASS = 0.99. The 287
higher-threshold checkpoints silently run the wrong bar at evaluation time.

This PR adds a three-line comment above VISUAL_SSIM_PASS to make the gap
visible. It doesn't wire in the threshold yet, because doing that correctly
requires the result visuals items to carry checkpoint IDs (they currently
don't), which is a schema change. I'd rather surface the gap clearly and get
direction before touching the result schema.

What's not in this PR

The budgets subfields beyond build_timeout_sec in evaluation-contract.json
(cpus, memory_mb, startup_timeout_sec, storage_mb) are present in all
71 contracts and never enforced. I left those out deliberately — enforcement
requires infrastructure-level hooks I can't reach from case_protocol.py.

Question for maintainers

For both the per-checkpoint threshold and the unused budget fields (cpus,
memory_mb, startup_timeout_sec, storage_mb): would you prefer to remove
the unused fields and clean up the contracts now, or keep them as forward-looking
hooks and wire in enforcement in a follow-up? I can extend this PR in either
direction within the next few days if you have a preference.

Scope and approach

All 70 task-bundled copies of case_protocol.py are byte-for-byte identical
(verified with md5). There is no single canonical source in src/; the fix
was applied uniformly to all copies with a small Python script.

Checks run locally:

  • uv run ruff check scripts src tests — pass
  • uv run ruff format --check scripts src tests — pass
  • uv run python scripts/check_tasks.py --code-only — all 70 tasks pass

Note: tasks/ is excluded from ruff via extend-exclude = ["tasks"] in
ruff.toml, so the modified files are not linted. The changes are
documentation and comments only, with no runtime logic.

I used Claude Code (AI-assisted) to help explore the codebase and draft
this fix. The analysis and final edits are mine.

🤖 Generated with Claude Code

The module docstring in case_protocol.py cited docs/scoring-protocol.md
and docs/case-design.md, neither of which exists in the repository.
Remove that sentence to prevent confusing contributors who follow the
dead links.

Add a comment above VISUAL_SSIM_PASS noting that each checkpoint in
visual-suite.json carries a threshold field that the scorer does not
currently read -- the constant applies uniformly across all checkpoints.
This documents the intent gap from issue TIGER-AI-Lab#24 for a future follow-up.

Applied across all 70 task-bundled copies of case_protocol.py, which
are identical and contain no task-specific logic.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant