Skip to content

ci: test-windows-guards intermittently dies in setup — ASCII baseline did not index, nodes:None #1952

Closed

Description

@ilyabrykau-orca

ci: test-windows-guards intermittently dies in its own setup — "ASCII baseline did not index", nodes:None

Status: root cause found and fix open — PR #1958. index_and_count read node/edge counts exclusively from list_projects, whose stats columns are published asynchronously (on some venues never within a one-shot MCP session); index_repository's own response carries the synchronous counts. Reproduced deterministically off-CI: the project row arrives with no stats fields while the index response reports nodes=12,edges=21 and definition queries return 5 — exactly the nodes: None + definition_nodes: 5 signature below. #1958 gates on the index response and retries the setup steps once; #1961 additionally makes the fixture polyglot (TS+Go, #1959).

Type: CI flake report (harness setup, not a product regression — the same commits pass on retrigger).

Signature

tests/windows/test_non_ascii_path.py exits 2 before any guard runs:

SETUP FAIL: ASCII baseline did not index: {'name': 'C-Users-runneradmin-cbm-windows-guards-root-…-ascii_repo', 'nodes': None, 'edges': None, 'definition_nodes': 5}

The baseline fixture is the harness's own 2-file TypeScript repo (make_fixture: math.ts + main.ts). definition_nodes: 5 with nodes: None means the MCP one-shot indexed and counted definitions but the node/edge counts came back empty — the guard's index_and_count gate (base.get("error") or not base.get("nodes")) then aborts setup.

Occurrences today (2026-08-30, all on PR-CI runs of Go-gated diffs; the fixture is TS)

PR run outcome
#1907 33307147558 FAIL — passed on retrigger of the identical tree (33319… sibling wave)
#1944 33314735547 / job 99270211759 FAIL, same signature
#1936 33319503142 / job 99288826006 FAIL (its round-1 failure was a different job — test_subprocess.c:688 ASSERT(ready) on the CLANG64 shard — which then passed)
#1947 33323962011 / job 99296338445 FAIL
#1949 33324761792 / job 99304765726 FAIL

Same-day control group: #1939 and #1941 (other authors) passed the job first try; #1913/#1915/#1931/#1937/#1938 waves passed it.

Stronger signal: whole-harness environment failures in one job

#1936's round-2 job shows the setup failing at EVERY stage, not just the baseline index:

SETUP FAIL: ASCII baseline did not index: {'…', 'nodes': None, 'edges': None, 'definition_nodes': 5}
SETUP FAIL: permanent daemon did not start for the crash check
SETUP FAIL: index did not run:
SETUP FAIL: ASCII control did not index via CLI:

The permanent daemon not starting at all points at the runner environment (spawn/AV/socket), not at the indexed fixture. #1944 hit the job twice consecutively (round 2).

Why it matters

Fork contributors cannot rerun failed jobs (Must have admin rights), so every flake costs a full no-op force-push wave to retrigger ~35 checks. Five hits in one day suggests the failure rate is high enough to gate honest PRs on runner luck.

Possible directions (from reading the harness, not verified on a Windows box)

  • index_and_count already collects cache log tails when projects comes back empty — the same diagnostics on the nodes: None path would show whether the one-shot's summary JSON was truncated or the store query raced the publish.
  • A bounded retry of the baseline setup step (the fixture is byte-identical by construction) would de-flake the gate without weakening the non-ASCII comparisons it exists for.

Happy to PR the retry/diagnostics if that direction is acceptable.

Activity

  1. CaptainMittens commented on Aug 31, 2026

    @CaptainMittens
    Contributor

    Another occurrence, plus a symptom I do not think is recorded here yet: this flake also prints a false regression, and that is probably worse than the setup abort.

    I read the test-windows-guards log for every failure of that job on PR CI between 2026-08-29T23:09Z and 2026-08-31T22:23Z — 9 failures out of 47 runs where the job ran. All 9 carry your nodes: None / definition_nodes: 5 setup signature. All 9 also print:

    RED (tests/windows/test_daemon_stability.py): section_<name> failed
    RED (tests\windows\test_daemon_stability.py) - REGRESSION: a fixed Windows bug is broken again
    

    The section named changes from run to run:

    Section named in the RED Runs
    section_cold_storm 4
    section_churn_stability 1
    section_crash_recovery 1
    section_hook_fail_open 1
    section_start_status_port 1
    section_stop_refuses_busy 1

    One environment fault therefore appears as six different "a fixed Windows bug is broken again" messages. A reviewer who sees that line on a contributor's PR has fair reason to read it as a real regression in that PR, and the changing section name hides the common cause.

    A surviving daemon looks like the link. In run 33443815210 the guard reports:

    the active CBM daemon runs a different build than this binary
    CBM could not start because the active account daemon uses a different cache directory
    

    The setup dies, leaves a daemon from a different build and a different cache directory running, and each later daemon-dependent guard then measures that daemon instead of a clean one. If #1958 retries the setup steps while that daemon is still up, it may be worth stopping the daemon before the retry.

    Two notes on the occurrence table:

    • Not only Go-gated diffs. Two of the 9 are not Go: run 33443815210 on fix/cypher-reject-trailing-tokens (a C change in the Cypher parser, fix(cypher): refuse a query the parser did not fully read #1875) and run 33376934088 on ci/codeql-gate-budget.
    • The Go skew is real but I could not explain it: Go branches 7/23, everything else 2/24, Fisher exact p = 0.072. I tested two explanations and both failed — job concurrency (runs that started alone failed more, not less) and Go runs indexing more slowly (Go guard jobs average 807s against 859s for the rest). I would treat it as unexplained rather than as a lead.

    The newest occurrence is 2026-08-31T22:23Z, so the fault still reproduces with #1958 unmerged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stability/performanceServer crashes, OOM, hangs, high CPU/memorywindowsWindows-specific issues

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions