Skip to content

Run contrib SSA YAML in Contrib shard 3 to stop shard 1 timeouts - #10556

Merged
hua7450 merged 1 commit into
PolicyEngine:mainfrom
hua7450:ci-move-contrib-ssa-to-shard-3
Oct 31, 2026
Merged

hua7450 merged 1 commit into
PolicyEngine:mainfrom
hua7450:ci-move-contrib-ssa-to-shard-3

Conversation

@hua7450

@hua7450 hua7450 commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Full Suite - Contrib (other-shard-1) keeps hitting its 60-minute timeout, on release runs and on PRs. This moves the contrib SSA YAML route from that shard to other-shard-3, the lightest Contrib shard. It doesn't add a runner or change any test.

Why shard 1 times out

Shard 1 runs three things one after another: the shared per-subdir contrib batch, the contrib SSA YAML files one file at a time, and the contrib Python tests. The SSA route had 3 files on Oct 27, all from #10400. #10483 added 3 owner's draw files on Oct 29 and #10484 added 6 PIA ratio lever files on Oct 30, so it now has 12:

Step in shard 1 Release run 40586701799 (Oct 27) Release run 41123038987 (Oct 30)
Shared per-subdir batch 16.7 min 17.1 min
Contrib SSA, per-file (3 → 12 files) 7.0 min 27.1 min
Contrib Python tests 21.2 min cut off after 15.7 min
Job about 46 min cancelled at 60 min

Each SSA file is its own subprocess and runs alone (POLICYENGINE_BATCH_WORKERS=1, --workers 1); the 12 files peaked at 4.1 to 12.2 GB on run 41123038987, so they can't run two at a time on a 16 GB runner.

The timeout cancelled shard 1 in 7 of the 14 release runs on Oct 30 (from 04:37 to 20:25 UTC), and at least 4 PR runs the same day (for example 41067837131, 41069882211, 41075000562, 41091546450). Over the same PR runs, other-shard-3 took 6 to 11 minutes.

What changes

  • Makefile: the SSA command moves, unchanged, from test-yaml-structural-other (shard 1) to the end of test-yaml-structural-other-shard-3, after refundable_credit_conversion. Both run per-file with one worker, so only one heavy subprocess is resident at a time. The shared batch still excludes ssa, so the files run once.
  • test_make_test_matches_ci.py: the SSA route guard now looks for the route in shard 3. It also checks that shard 1 has no SSA invocation, and that CI runs shard 3's target.
  • The workflows don't change: both matrices already map other-shard-3 to test-yaml-structural-other-shard-3, and make test still runs the route once.

Expected from Oct 30's timings: shard 1 drops by SSA's ~27 minutes, and shard 3 gains them; each should take roughly 35 to 40 minutes. Contrib's wall time is still set by its longest shard (states-shard-1, about 44 minutes on run 41123038987), so it shouldn't grow.

Checks

  • make -n test-yaml-structural-other runs only the shared batch, with ssa excluded; make -n test-yaml-structural-other-shard-3 runs refundable_credit_conversion, then SSA per-file with POLICYENGINE_BATCH_WORKERS=1; make -n test contains the SSA route once.
  • ruff format and ruff check pass on the changed test file.
  • Not run locally: the guard tests and the Contrib shards. This PR's own Contrib run measures the new shard 1 and shard 3 times.

SSA keeps growing; at about 2.3 minutes per file, shard 3 has room for roughly 9 more files before it nears the timeout.

axiom: n/a: CI routing and workflow guard changes only.

🤖 Generated with Claude Code

Contrib other-shard-1 runs the shared contrib batch, the SSA files one at a
time, and the contrib Python tests. SSA grew to 12 files and 27 minutes, and
the shard hit its 60-minute timeout on release and PR runs. Shard 3 took 6 to
11 minutes, so SSA's unchanged per-file, one-worker route moves there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hua7450
hua7450 merged commit 3f52254 into PolicyEngine:main Oct 31, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant