Skip to content

fix(test-impact): let a build drop the suites only some runners register - #2684

Merged
DeusData merged 2 commits into
mainfrom
fix/runner-conditional-suites
Oct 18, 2026
Merged

DeusData merged 2 commits into
mainfrom
fix/runner-conditional-suites

Conversation

@DeusData

Copy link
Copy Markdown
Owner

Smart CI: a build may drop the suites only some runners register

Since TEST_IMPACT_MODE=gate (2026-10-17 22:10Z), a PR whose selection gets narrowed fails its x86 Linux test leg before any test runs:

FAIL: the test selection names suites this build does not register: coverage_setup_success coverage_setup_uncertain

Seen on #2679 and #2680. Only that leg consumes a selection; every other leg runs every test, so their results stand.

Cause

The selection comes from the coverage runner's map. tests/test_main.c registers four suites only under #if defined(CBM_TEST_COVERAGE) && !defined(_WIN32). scripts/run-tests-parallel.sh tolerates a selected suite the build lacks only when optional-suites.txt names it, and selection.py lists a suite there only when the answer marks it whole with CONDITIONAL. The model recorded a condition around a RUN_TEST registration, but not around the runner's RUN_SELECTED_SUITE, so these suites reached the leg as required. A real answer for #2679's head showed it: of the four, only coverage_children (whose own registrations are conditional) carried CONDITIONAL.

Fix

  • cbm_test_runner_suite_t.conditional: every site that registers the suite sits under a preprocessor condition. A suite also registered outside any condition runs in every build and keeps the strict contract.
  • After every whole-suite path (setup unknown/hit, triggers, inventory, rejected artifact) and the per-test path, a selected suite whose runner registration is conditional runs whole with CONDITIONAL. selection.py then lists it as optional, and a build without it drops it with a notice, exactly as its full run never runs it. The leg's guard is unchanged: a selected suite that no condition explains still fails the run.

Verification

  • RED first: with the two .c changes reverted, test_selection_conditionally_registered_suite_is_conditional_on_every_path (seven selection paths, plus an unconditional control) and test_model_marks_runner_suites_registered_under_a_condition fail on their assertions; with them, test_impact passes 89/89.
  • Real input: the fixed engine on fix(python): resolve a base class spelled through a root-binding import #2679's head against 354ccecb marks all four coverage suites CONDITIONAL (before: one), and selection.py's conditional_suites lists all four as optional (before: coverage_children only).
  • Affected suites green (the test_impact family, runner_isolation).
  • cppcheck 2.20, clang-format, the contract steps (scripts/test.sh --contracts-only) and the security audit are clean.

After the merge

A rerun of an affected PR replays its frozen merge commit, and select-tests builds its engine from that checkout, so it would rerun the old engine. Affected PRs need a new merge commit: merge main into the branch, or push again.

The diagnosis and the fix come from the graph-quality session, which found it independently; this PR carries their commit unchanged, plus a fixture correction (the selection test's coverage map did not parse).

…ONAL on every selection path

A runner suite registered only under a preprocessor condition (the four
coverage_* suites under `#if defined(CBM_TEST_COVERAGE) && !defined(_WIN32)`
in tests/test_main.c) is absent from builds without that condition. The
selection marked such a suite CONDITIONAL only on the per-registration path,
and only when a registration of its own was conditional and contributed a
reason. Every suite-level path (setup unknown or hit, suite-body, changed and
rule triggers, inventory, rejected artifact) made the suite whole without it,
and the runner registration's own condition was never recorded.

PR CI on the x86 Linux gcc leg then failed with "the test selection names
suites this build does not register: coverage_setup_success
coverage_setup_uncertain" (#2679, #2680): the suite-body trigger and an
incomplete setup row selected both suites whole without CONDITIONAL, so
selection.py left them out of optional-suites.txt.

The model now records whether a runner registration sits under a condition
(conditional only when every site of the suite is), and the selection marks
any selected conditionally registered suite whole with CONDITIONAL after every
whole-suite path. Unconditionally registered suites keep the strict contract.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
… map

A coverage map must cover every function it lists, and five of the seven
variants covered only function 1, so the map failed to parse and the test
stopped at its setup ("map is NULL") with the fix applied or not. The
changed function (0) is now covered by the setup row of a suite no runner
registers, so alpha's own rows stay quiet and each variant still reaches
alpha by exactly one path. RED with the selection fix reverted (alpha
lacks CONDITIONAL), green with it.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData merged commit c23f162 into main Oct 18, 2026
29 checks passed
DeusData added a commit that referenced this pull request Oct 18, 2026
…lection fix #2684)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData added a commit that referenced this pull request Oct 18, 2026
…art CI selection fix #2684)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData added a commit that referenced this pull request Oct 18, 2026
…Smart CI selection fix #2684)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData added a commit that referenced this pull request Oct 18, 2026
… up the Smart CI selection fix #2684)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData added a commit that referenced this pull request Oct 18, 2026
…Smart CI selection fix #2684)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData added a commit that referenced this pull request Oct 18, 2026
…the Smart CI selection fix #2684)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
timothybrush pushed a commit to timothybrush/codebase-memory-mcp that referenced this pull request Oct 19, 2026
…ed-class (pick up ef90708: origin/main incl. the Smart CI selection fix DeusData#2684)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant