[MaxGhenis]

Fixes #2770

Reproducer: policyengine_uk/tests/code_health/test_ci_test_shards.py

The Python shards use almost all of the 75-minute job limit, so a slow runner can cancel a job whose tests pass. This adds a seventh shard, the fix test-suite.yaml documents.

Why

• Nine successful test runs started on 31 October (EDT) took 57 to 73 minutes a shard.
• In run 41191216150, first attempt, the limit cancelled shard 1/6 with its tests at 99%, and Result failed on the cancelled job.
• shard_durations.toml puts the listed slow tests at 134 minutes of worker time per shard on six shards and 115 on seven. The balanced split gives each of the seven shards 115 minutes.

What changed

• test-suite.yaml: the matrix gains a 7, and a new paragraph above the job records the numbers above.
• pr_code_changes.yaml, Makefile, docs/engineering/skills/testing.md and docs/book/validation/ubuntu-vs-macos.md: "six" and K/6 become "seven" and K/7.
• A changelog fragment.

The job name and --shard read strategy.job-total, and the required checks name only Test / Result and Lint, so no ruleset changes.

Checks

• test_ci_test_shards.py and test_ci_runner_variables.py pass: 126 tests.
• Run as --shard K/7 for K = 1 to 7, the same files select 19, 23, 16, 15, 13, 22 and 18 tests, which sum to 126, and all pass.

axiom: n/a: CI configuration only

🤖 Generated with Claude Code

[juaristi22]

Reviewed bbfb7c1f36552e55d78ea273c3e177ace244aeff. Complete review; no findings or material evidence gaps.

The seventh matrix entry feeds the dynamic job names and --shard K/7; Result still reconciles all shards against the complete collected test list. PR, merge-queue and release callers retain their existing gates. Live required contexts remain Test / Result and Lint.

Validation: 126 focused CI-health tests passed. All seven PR and release shards and their Result checks passed at this head. Historical speed claims were outside this review; the corrected local harness and its initial setup error are recorded separately.

[MaxGhenis]

Ready for hub review
Merge-train rehearsal at main 2da212a

All 30 PR heads matched the SHAs in the brief (git rev-parse refs/pr/<n>). Nothing was pushed. Every merge is in a local clone only.

1. Each PR against main alone

git merge-tree --write-tree --name-only 2da212abc refs/pr/<n>: all 30 clean: #2744, #2771, #2647, #2340, #2757, #2758, #2760, #2763, #2768, #2769, #2772, #2779, #2780, #2788, #2794, #2807, #2811, #2813, #2776, #2786, #2818, #1969, #2131, #2761, #2756, #2781, #2787, #2279, #2765, #2670.

2. The train

Local branch at 2da212abc, git merge --no-ff --no-edit refs/pr/<n> in table order. Steps 29 and 30 are the other authors' PRs.

Step	PR	Result	Commit	Tree
1	#2744	merged	0ac91fd17850c9c08384cebf80a8ec29c59b9306	21989da22996cbac2bcf2d4393661f992787a391
2	#2771	merged	3427ce2083666054db2382fbced95ddce8e27f2b	c3dcca84b34400ec050fba20eac34f83b1dfadac
3	#2647	merged	f1348f6d4679123cc5723e2e03c0d74dc0a9a32f	c1e4c4c78aa99a584d4b6a4d602b053c395eb785
4	#2340	merged	530acc65aa6f9aa7a4130131f164f4bc107327b0	849aafa35351adf285eb9d18731e1b6bc9700c60
5	#2757	merged	71b3405e64d92ed4af741339a1c0de6460b7c4f1	c655d7dbb799443a731df9378bb8f8581c8618e2
6	#2758	merged	f59d0c7512cbd20424509bdc01e8cf941d1f1d4c	70f277a9e4375fb1af83c74f8f81330714e8b5ef
7	#2760	merged	7af4b17841121e59a5b83c378ca5d3c6134e67c1	85697c6f73c5fd77c7b2cd187ead5f81be6a830d
8	#2763	merged	0024632088c38c0bfe68cc528182d69c1ad40fc2	fe7b87a6f40a05ec6c7559a50a552ecfd3ae8a06
9	#2768	merged	971c56f378c7c5d7485a4773b9021b937508d72d	3abd4f59e271f7bcef6cbd33b9018827ad92a372
10	#2769	merged	4911569c435f8c012eca44752bf23ab5d55df3d3	40409ebf7b2da0bec3eb414e023c4a731fc5c156
11	#2772	merged	a4f9c73cfe4b15cf8256aaaef3f74f35275ad75c	9c46fd7d12066a5dabf478ab24035f51b6834a0a
12	#2779	merged	43aeb932ce680115d4385d368224cb33ad5d1fba	f9be6ceed25a2b9e5b3936ff6681fcead9c21f08
13	#2780	merged	1a0ba9a3101028b2433273b8af9bc70b3b6e6a14	8f5407342893ee8c5e14d7b47f321ca33525edbd
14	#2788	merged	fa8dc3ebef86a417fe4d813d22a2a1f281fd5c61	4b00f5657e0272b1026d909e4efe72554efaefd9
15	#2794	merged	3e86847eacd1322edbf7c3a614f913432b9a955c	2f73b7e64050e40e53468b76b99134577e77476c
16	#2807	merged	cd93ba07d02d96869dd92ea8eeb8c61f7a85e8b5	68da11d1b92fe51de358c8bc0c0c93b769b57f6b
17	#2811	merged	7c1ca44908bb7468ccaa841c62ea15303dfdc147	ab9f3d1255efbc0c42594763f640203e5860d3d8
18	#2813	merged	750deeefdbc8c60cef8183b346c0c14339b3413b	dc69796c120a0f9ace6093b554eb4097afb6a2f3
19	#2776	merged	2520800d28b95e0f7f2d0a13f9486761c40a12d1	2d338e8f5ebec41592f8b180b595a46ad3250c21
20	#2786	merged	ad5eed5d966e284ad596463dd2698e34896f6ed6	1f9059e931d8c71ab099ec8e3794371cfc13a70d
21	#2818	merged	316951359f46eb6d49ece57a6a83088b0bbaaf52	ff6233f31f77928022660339b048c6b2128ff70a
22	#1969	merged	6d0457f7d21b0c2df959da23838fb1ee67b43715	eba538ebd4253573e8fdcb8ab1974a79e5f7a04d
23	#2131	skipped: docs/book/assumptions/growthfactors.md	–	–
24	#2761	merged	371f8068bb355ebca8b6f2d5b1ecd04920c1893b	b572c7d52ee4dc2c310b4fba60674718d3d327fa
25	#2756	merged	2cd50e7da31d05064a1d3f984ccf577e627b7366	0d916ce4a31a2874dfc15557b967cf4b01ba3061
26	#2781	merged	d73f9b044b03939fa2c3888b1c8ce0a6b744876e	2bf0bbed34e18811958c7e0725a09b01da6243b4
27	#2787	merged	58cbdc0aa536f4b46de8ee6458a8c0ed04f6aadb	f66abad429e59b5a505cd6a4a8bd4502261f899a
28	#2279	merged	467a867efcfe7edf181c9b6d4addb49a141ff7d3	e8f18a45a25edbfe1370e1ebc23941b79bcae057
29 (other 1)	#2765	skipped: policyengine_uk/utils/supplied_inputs.py	–	–
30 (other 2)	#2670	merged	4274f8b541408a59378f4ab2ebaf0c8537e4cfce	635c67471e8f5a5c533ae62c1aa0b9efa2b97784

Skipped PRs and the earlier merged PR they clash with (git merge-tree of the skipped PR against each merged PR):

• Value share-like capital by asset type for the sale-expense deduction #2131 (step 23) conflicts in docs/book/assumptions/growthfactors.md, only against Bring the growth factors page into line with the uprating map #2779 (step 12). refs/pr/2131 contains Value means-tested capital less 10% sale expenses and secured debt #1969's head 3a3241cf, and Value means-tested capital less 10% sale expenses and secured debt #1969 merged cleanly at step 22. No landing note says how to resolve this, so it stays skipped.
• Add a trading allowance claim status, so allowance reforms reach claimants without turnover data #2765 (other author) conflicts in policyengine_uk/utils/supplied_inputs.py, only against Value means-tested capital less 10% sale expenses and secured debt #1969 (step 22).

No conflict was resolved by hand. #2771 merged cleanly at step 2.

Final train tree: commit 4274f8b541408a59378f4ab2ebaf0c8537e4cfce, tree 635c67471e8f5a5c533ae62c1aa0b9efa2b97784. uv.lock had no conflict. uv sync --frozen --extra dev exited 0 (3m06s).

3. Tests on the final train tree

HUGGING_FACE_TOKEN and POLICYENGINE_UK_DEFAULT_DATASET were unset. No run skipped a test, so no test skipped for want of a dataset. No run reported an xfail or an error. Memory held at -n 4.

Run	Exit	Passed	Failed	Skipped	Xfailed	Wall
uv run policyengine-core test policyengine_uk/tests/policy -c policyengine_uk	1	5690	6	0	0	824s (13:44); pytest summary "in 797.99s"
uv run pytest -q -o xfail_strict=true -n 4 policyengine_uk/tests/code_health	0	3129	0	0	0	308s

Item 3 files: each was run alone as uv run pytest -q -o xfail_strict=true -n 4 <file>, with a 30-minute cap.

Which files. The brief's two-dot command, git diff --name-only 2da212abc refs/pr/<n>, lists 294 files. That includes files that differ only because several PR branches are behind main: #2744 is 134 commits behind, and #2761 is 35 behind. Most of the extra files come from #2340 (277 files) and #2647 (173). I ran the 74 files that the merged PRs themselves add or change, from the three-dot diff git diff --name-only 2da212abc...refs/pr/<n>, with conftest.py and code_health/ taken out. The other 220 files from the two-dot list were not run; see the end of this comment.

File (under policyengine_uk/tests/)	Exit	Passed	Failed	Skipped	Xfailed	Wall
reform_withheld_main_reference	5	0	0	0	0	22s
test_aggregate_impact_guard	0	63	0	0	0	29s
test_aggregate_impact_partition	0	17	0	0	0	32s
test_aggregate_impact_reforms	0	53	0	0	0	55s
test_aggregate_impact_withheld_reasons	0	25	0	0	0	38s
legacy_award_work_reference	5	0	0	0	0	29s
test_uc_capital_allocation	0	1	0	0	0	29s
test_benefit_cap_lha_scheme_split_properties	0	7	0	0	0	341s
test_benefit_cap_maternity_allowance_properties	0	5	0	0	0	36s
test_benefit_cap_reduction_applied_properties	0	5	0	0	0	545s
test_benefit_cap_specified_benefits_properties	0	46	0	0	0	492s
test_benefit_cap_uc_and_hb_properties	0	1	0	0	0	77s
test_capital_share_components	124	13	0	0	0	1813s
test_capital_valuation_properties	0	18	0	0	0	328s
test_carer_overlapping_benefits_properties	0	4	0	0	0	117s
test_carers_allowance_earnings_taper	0	68	0	0	0	464s
test_child_and_adult_definitions_properties	0	21	0	0	0	438s
test_child_benefit_charge_rounding_properties	0	9	0	0	0	58s
test_corporate_receipts_statistics	0	141	0	0	0	32s
test_disable_simulated_benefits	0	49	0	0	0	232s
test_economic_assumption_horizon	0	11	0	0	0	46s
test_energy_bills_rebate_council_tax_reduction_properties	0	8	0	0	0	241s
test_esa_income_abolition_properties	0	31	0	0	0	104s
test_esa_income_held_outside_award	0	3	0	0	0	33s
test_esa_income_pension_age_properties	1	15	4	0	0	448s
test_esa_income_universal_credit_claim_properties	0	2	0	0	0	141s
test_esa_permitted_work_nlw_reforms	0	5	0	0	0	72s
test_gambling_duty_base_projection	0	23	0	0	0	131s
test_housing_benefit_wtc_disregard	0	55	0	0	0	106s
test_housing_benefit_wtc_disregard_segments	0	18	0	0	0	41s
test_import_obr_forecasts	0	127	0	0	0	95s
test_income_support_eligibility_properties	0	4	0	0	0	167s
test_income_support_gate_reads_claimant_or_partner_awards	0	13	0	0	0	285s
test_income_tax_step_6_step_7_properties	0	21	0	0	0	55s
test_isa_balances	0	2	0	0	0	21s
test_legacy_award_readers_properties	0	20	0	0	0	234s
test_legacy_award_work_properties	0	4	0	0	0	133s
test_legacy_capital_tariff_properties	0	6	0	0	0	25s
test_legacy_closure_scope	0	21	0	0	0	50s
test_means_test_income_properties	0	2	0	0	0	201s
test_mixed_age_couple_properties	0	1	0	0	0	952s
test_ni_parameter_scales	0	106	0	0	0	195s
test_other_investment_income_household_income_properties	0	1	0	0	0	36s
test_oxford_council_tax_reduction_universal_credit_properties	0	7	0	0	0	190s
test_pension_age_projection_invariants	0	22	0	0	0	18s
test_pension_credit_reported_capital_properties	0	4	0	0	0	250s
test_pension_credit_uprating	0	35	0	0	0	50s
test_pension_credit_young_person_income_support	0	10	0	0	0	61s
test_population_vintage	0	95	0	0	0	18s
test_program_registry	0	119	0	0	0	19s
test_savings_credit_only_receipt_properties	0	6	0	0	0	223s
test_savings_starting_rate_indexation	0	26	0	0	0	84s
test_scottish_carer_supplement_uprating	0	7	0	0	0	121s
test_sdlt_dated_schedule_properties	0	56	0	0	0	28s
test_sdlt_purchase_schedules	0	29	0	0	0	21s
test_september_cpi_uprating	0	17	0	0	0	38s
test_september_cpi_uprating_properties	0	8	0	0	0	20s
test_state_pension_components	0	33	0	0	0	269s
test_state_pension_reported_weeks	1	4	7	0	0	87s
test_statutory_paternity_pay_income_tax_properties	0	8	0	0	0	141s
test_stock_capital_periods	0	2	0	0	0	22s
test_stored_household_totals	1	41	1	0	0	229s
test_structural_reform_periods	0	67	0	0	0	448s
test_threshold_indexation	0	195	0	0	0	46s
test_uc_advance_repayment_properties	0	8	0	0	0	196s
test_uc_company_override_properties	0	11	0	0	0	111s
test_uc_company_owner_properties	0	7	0	0	0	265s
test_uc_deduction_flag_properties	0	10	0	0	0	329s
test_uc_deductions	0	24	0	0	0	98s
test_uc_legacy_closure_properties	0	4	0	0	0	167s
test_uc_property_capital	0	65	0	0	0	266s
test_uc_property_capital_mock	0	26	0	0	0	53s
test_uc_tax_free_childcare_choice	0	20	0	0	0	101s
test_uc_work_related_groups_properties	0	14	0	0	0	247s

Notes on that table:

• Exit 5 ("no tests ran"): aggregate_impact/reform_withheld_main_reference.py and legacy_award_work_reference.py hold no test_ functions, so pytest collected nothing from them.
• Exit 124 (stopped at 30 minutes): test_capital_share_components.py, with 13 of its 14 nodes passed. I then ran the nodes alone. ::test_share_like_capital_keeps_the_supplied_aggregate passed in 147s. ::test_capital_matches_the_original_lists was still running at the 30-minute cap (exit 124, 1810s, nothing finished), so this is the test the file run had reached. The same node, run in worktrees with -n 2 and the two runs in parallel, passed at main in 1583.86s and at Value means-tested capital less 10% sale expenses and secured debt #1969's head (3a3241cf) in 1650.03s. Only Value means-tested capital less 10% sale expenses and secured debt #1969 touches this file. I have no train-tree result for this node beyond "did not finish in 30 minutes".
• No run reached the 90-minute cap.

4. Failures and where each starts

Method: each step was checked out as a detached worktree at its recorded merge commit. Tests ran there with the final train's .venv, using PYTHONPATH=<worktree>, so dependencies come from the final uv.lock. Pytest nodes ran with -n 2 so the main run kept its CPUs. For a clash, I built pair trees: main merged with each earlier PR, then that tree merged with the PR that adds the test, using git merge-tree and git commit-tree, all local.

A. tests/policy/baseline/finance/benefit/family/esa_income/esa_income_pension_age.yaml: 6 YAML cases

The file is not on main; #2807 adds it.

• Case 4, "the same mixed-age couple without the saving keep income-related ESA and get no Pension Credit": esa_income_eligible@2026: [0.] differs from 1.0
• Case 5, "a saved mixed-age couple that does not claim Pension Credit keep income-related ESA": esa_income@2026: [1178.0823] differs from 5000.0
• Case 6, "a single claimant aged 60 keeps income-related ESA": esa_income_eligible@2026: [0.] differs from 1.0
• Case 12, "the couple's Pension Credit does not bar a member outside the couple…": esa_income@2026: [1178.0823] differs from 5000.0
• Case 13, "where both members of a mixed-age couple report income-related ESA…": esa_income_eligible@2026: [0.] differs from 1.0
• Case 14, "where only the older member of an unsaved mixed-age couple has the support component…": esa_income@2026: [0.] differs from 10000.0

B. tests/test_esa_income_pension_age_properties.py: 4 nodes

The file is not on main; #2807 adds it.

• test_a_parameter_reform_keeps_the_age_and_take_up_conditions: line 691, assert [0.0] == [5000]
• test_the_screen_reads_no_pension_credit_condition_without_a_couple_claim: line 1141, assert [False, False, True, False] == [False, False, True, True]
• test_couple_esa_follows_pension_credit_worked_out_without_it: line 364 calls esa_screen, and the error is raised in tests/legacy_award_work_reference.py:254 (couple_report_closed): assert all(not 65 <= adult["age"] < 70 for adult in couple) gives AssertionError: [{'age': 58, …}, {'age': 65, …}]
• test_no_branch_opens_without_a_couple_that_could_be_on_pension_credit: line 469, assert [0.0, 0.0, 0.…75390625, 0.0] == [5000, 5000, 0, 5000, 0]

A and B: fail from step 18 (#2813). All 10 pass at #2807's head (1d6df1a4), at step 16 (YAML) and at step 17 (YAML and pytest). All 10 fail at step 18. #2813's head has no copy of either file. A pair tree of main + #2807 + #2813 alone (4a941b6b) reproduces all 10 failures: YAML 6 failed and 18 passed, pytest 4 failed. This is a clash between #2813 and #2807. Files both PRs touch:
policyengine_uk/variables/gov/dwp/esa_income.py, policyengine_uk/variables/gov/dwp/is_on_income_related_esa.py, policyengine_uk/variables/gov/dwp/_legacy_award_payee.py, policyengine_uk/tests/legacy_award_work_reference.py.

C. tests/test_state_pension_reported_weeks.py::test_full_rate_reporters_have_no_add_on_in_any_year[2024–2030]: 7 nodes

The file is not on main; #2818 adds it. All seven fail at line 115, basic_state_pension … == pytest.approx(WEEKS_IN_YEAR * rates["BASIC"], abs=PENNY):
2024 obtained 8801.0, expected 8814.0 · 2025 9161.867 vs 9175.4 · 2026 9600.619 vs 9614.8 · 2027 9975.056 vs 9989.790 · 2028 10314.229 vs 10329.464 · 2029 10572.108 vs 10587.725 · 2030 10836.404 vs 10852.411.

Fails from step 21 (#2818). It passes at #2818's head (54cffcec). The file does not exist at step 20, and the nodes fail at steps 21, 22, 25, 28 and 30. Pair trees of #2818 with main alone, and with main plus each of steps 1–20 in turn, all pass except #2340 (7 failed). This is a clash between #2818 and #2340. The two PRs' three-dot diffs share no file.

D. tests/test_stored_household_totals.py::test_stored_household_totals_grow_past_the_projection_horizon

The file is not on main; #2781 adds it. It fails at line 222, assert capital[year][0] == pytest.approx(6 * savings[year][0], rel=1e-5): obtained 5800.0, expected 6000.0 ± 0.06.

Fails from step 26 (#2781). It passes at #2781's head (9def3670) and on main + #2781. The file does not exist at step 25, and the node fails at steps 26, 27, 28 and 30. Pair trees of #2781 with main plus each merged PR from steps 1–25 pass except #1969 (step 22). This is a clash between #2781 and #1969. File both PRs touch: policyengine_uk/data/uprating_indices.yaml.

None of these three failing files is on main, so there was no run at main for them. I fixed nothing and give no reason for any failure.

LAST_INDEX_YEAR

• main has no LAST_INDEX_YEAR. Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761's head sets it to 2074, not 2039: its commit 84b1dbf4 "Check the index to the end of the growth file" replaced 2039. Derive economic assumption index horizon from the growth series #2756's head has no such constant.
• On the train tree, at step 24 (Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761), step 25 (Derive economic assumption index horizon from the growth series #2756) and the final tree, utils/import_obr_forecasts.py:597 reads LAST_INDEX_YEAR = 2074. The train tree does not hold the literal 2039.
• On the final tree, growth_horizon(CountryTaxBenefitSystem().parameters.gov.economic_assumptions.yoy_growth) printed 2074, and import_obr_forecasts.LAST_INDEX_YEAR printed 2074. There is no mismatch on the train tree.
• No test compares import_obr_forecasts.LAST_INDEX_YEAR with growth_horizon. I searched with grep: the only LAST_INDEX_YEAR names in tests are local constants in test_september_cpi_uprating.py and test_pension_credit_uprating.py, and OLD_LAST_INDEX_YEAR = 2039 in test_economic_assumption_horizon.py.
• Mutant: at the final train commit, in a worktree, I set LAST_INDEX_YEAR = 2039 (the script asserted one occurrence). I ran test_import_obr_forecasts.py, test_gambling_duty_base_projection.py and test_economic_assumption_horizon.py (-n 2): 1 failed, 160 passed. The mutant was killed by test_import_obr_forecasts.py::test_the_index_check_reaches_the_end_of_the_growth_file, at pytest.raises(ValueError, match="in 2044"), which gave Failed: DID NOT RAISE ValueError (line 1215). I reverted the file afterwards, and git status --porcelain printed nothing.

Not done, and why

• Value share-like capital by asset type for the sale-expense deduction #2131 and Add a trading allowance claim status, so allowance reforms reach claimants without turnover data #2765 were not merged. Both conflict, as above, and no landing note covers them, so their tests were not run on the train.
• 220 Python files from the brief's two-dot git diff list were not run. They are files that the stale-based PR branches (Model the 25p State Pension age addition at 80 #2340, Report the benefit cap reduction actually applied to Universal Credit #2647, Count 16-year-olds outside education as UC and Pension Credit qualifying young persons until the 1 September after their 16th birthday #2279, Read a period after a quoted reproducer id as the sentence's end #2744, Gate supplied UC latent deduction rates on uc_has_deduction #2760, Detect undeclared tax movement at any baseline level and count non-finite responses as movement #2758, Correct programme agencies and national coverage #2757, Project the gambling duty bases with the OBR's October 2026 betting and gaming forecast #2761, …) differ on only because they are behind main; the PRs themselves don't change them. Running them would have taken several more hours at the rate seen here.
• test_capital_share_components.py::test_capital_matches_the_original_lists has no train-tree result within 30 minutes; see the timings above.
• Bisection ran with the final train's environment, not each step's own uv.lock.
• No failure was rerun to get a pass. The pair-tree runs are separate builds, not reruns.

---

Generated by Claude Code