Repository navigation
Run contrib SSA YAML in Contrib shard 3 to stop shard 1 timeouts - #10556
Merged
hua7450 merged 1 commit intoOct 31, 2026
Merged
Conversation
Contrib other-shard-1 runs the shared contrib batch, the SSA files one at a time, and the contrib Python tests. SSA grew to 12 files and 27 minutes, and the shard hit its 60-minute timeout on release and PR runs. Shard 3 took 6 to 11 minutes, so SSA's unchanged per-file, one-worker route moves there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Full Suite - Contrib (other-shard-1)keeps hitting its 60-minute timeout, on release runs and on PRs. This moves the contrib SSA YAML route from that shard toother-shard-3, the lightest Contrib shard. It doesn't add a runner or change any test.Why shard 1 times out
Shard 1 runs three things one after another: the shared per-subdir contrib batch, the contrib SSA YAML files one file at a time, and the contrib Python tests. The SSA route had 3 files on Oct 27, all from #10400. #10483 added 3 owner's draw files on Oct 29 and #10484 added 6 PIA ratio lever files on Oct 30, so it now has 12:
Each SSA file is its own subprocess and runs alone (
POLICYENGINE_BATCH_WORKERS=1,--workers 1); the 12 files peaked at 4.1 to 12.2 GB on run 41123038987, so they can't run two at a time on a 16 GB runner.The timeout cancelled shard 1 in 7 of the 14 release runs on Oct 30 (from 04:37 to 20:25 UTC), and at least 4 PR runs the same day (for example 41067837131, 41069882211, 41075000562, 41091546450). Over the same PR runs,
other-shard-3took 6 to 11 minutes.What changes
Makefile: the SSA command moves, unchanged, fromtest-yaml-structural-other(shard 1) to the end oftest-yaml-structural-other-shard-3, afterrefundable_credit_conversion. Both run per-file with one worker, so only one heavy subprocess is resident at a time. The shared batch still excludesssa, so the files run once.test_make_test_matches_ci.py: the SSA route guard now looks for the route in shard 3. It also checks that shard 1 has no SSA invocation, and that CI runs shard 3's target.other-shard-3totest-yaml-structural-other-shard-3, andmake teststill runs the route once.Expected from Oct 30's timings: shard 1 drops by SSA's ~27 minutes, and shard 3 gains them; each should take roughly 35 to 40 minutes. Contrib's wall time is still set by its longest shard (
states-shard-1, about 44 minutes on run 41123038987), so it shouldn't grow.Checks
make -n test-yaml-structural-otherruns only the shared batch, withssaexcluded;make -n test-yaml-structural-other-shard-3runsrefundable_credit_conversion, then SSA per-file withPOLICYENGINE_BATCH_WORKERS=1;make -n testcontains the SSA route once.ruff formatandruff checkpass on the changed test file.SSA keeps growing; at about 2.3 minutes per file, shard 3 has room for roughly 9 more files before it nears the timeout.
axiom: n/a: CI routing and workflow guard changes only.
🤖 Generated with Claude Code