Skip to content

A WPF test hang now names its step and says whether the stranded body returned (#6632) - #6636

Merged
erikdarlingdata merged 5 commits into
devfrom
fix/6632-wpf-plan-withheld-hang
Oct 23, 2026
Merged

erikdarlingdata merged 5 commits into
devfrom
fix/6632-wpf-plan-withheld-hang

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Fixes #6632.

Claude posting for Erik Darling.

This is the brief's fallback outcome. The wait is not named and the cause is not fixed. I could not name the call the STA thread blocks in from the code and the logs. This PR adds step names to the test body and a stranded-body record to the host, so the next CI hang says which call it stopped in, and whether it was slow or blocked for good. Keep #6632 open until that report comes in.

Why

StatementScrubViewerPlanWithheldTests.APlanWithheldWhole_ShowsTheWithheldSentenceInTheViewer_NotAParseError hit the 60 s hang guard in CI. The #6216 report said "the body named no step", one dispatcher operation still running, and the thread in Wait (Executive).

What the logs show:

What the code and the evidence rule out:

  • A managed lock, a SemaphoreSlim, a static constructor's class-init lock, a GC wait, or an idle dispatcher. All of these are user-mode waits, which ProcessThread.WaitReason reports as UserRequest, and a GC suspension shows as Suspended. Executive is KWAIT_REASON 0 or 7: a wait inside the kernel, such as a synchronous I/O completion, a file object's lock, or a driver.
  • A second body on the shared thread. The Lite and Darling WPF tests take one gate so parallel STA classes stop racing a shared WPF cache (#5559) #5567 gate admits one body at a time, and the report counts one operation started.
  • The host's drain. A drain stuck in Dispatcher.Yield would leave the thread idle in GetMessage (UserRequest) with no operation running. The report showed one operation running.
  • The control's withheld path. LoadPlan on the marker returns before any await or Task.Run. Clear() sets element properties only, and Cleanup() unsubscribes one static event.
  • Defender on-access scanning a DLL as it loads. The runner image's Configure-WindowsDefender.ps1 sets DisableRealtimeMonitoring and DisableIOAVProtection, and excludes C:\ and D:\.

The leading hypothesis is unproven: WPF's first start in the process is I/O-bound and runs past 60 s under the Guard pass's load. The 47 ms of CPU fits that. The host can't confirm it, because once the guard fires it marks the thread stranded for good and never learns whether the body would have returned.

What changes

All three files are test code. Nothing in PerformanceMonitor.Ui or any product changes.

The next Guard-stage hang will then show:

  1. the step it stopped in;
  2. through the ViewerPlanCorrectionsDataStartTests failures that follow in the same pass, whether the body was slow or blocked.

Test plan

  • Darling.Tests builds with 0 errors. Its 6 warnings are in DarlingMcpNamedTokenTests.cs and ChangeWatchDetectionLiveTests.cs, which are already on dev and untouched here. Lite.Tests builds with 0 warnings; it compiles StaTestThread.cs through a link.
  • New behaviour test: StaTestThreadTests.ALaterRefusal_SaysWhetherTheStrandedBodyReturned_AndHowLongAfterItStarted.
    • RED with origin/dev's StaTestThread.cs planted: Assert.Contains() Failure: Sub-string not found (Total 1, Failed 1).
    • GREEN at head.
  • StaTestThreadTests, StatementScrubViewerPlanWithheldTests, StaThreadCensusTests, ViewerPlanCorrectionsDataStartTests and every *Census* class: Total 520, Failed 0, Skipped 1.
  • Lite.Tests *StaTestThread*, *PlanViewer* and *Census*: Total 49, Failed 0.
  • Full suite and pre-push.sh: not run, per the brief. CI decides.
  • No test forces the hang, because the wait is not named. The step names and the stranded-body record are what the next occurrence will report.

What the coordinator should double-check

CHANGELOG entry

None: a test-only PR.

Considered, not added

  • Writing a minidump of the test process when the guard fires (createdump ships with the runtime), uploaded as a CI artifact, would give the stuck thread's native stack. It is larger work: an artifact step in build.yml and a dump that can run to hundreds of MB. It is worth doing only if the step names and the return record still leave the wait unnamed.
  • Warming WPF before the first guarded body. Without a named cause this would hide the hang rather than fix it, and it would change what the Guard pass measures.

@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) October 23, 2026 14:31
Comment on lines +264 to +265
_ = finished.ContinueWith(
_ => Interlocked.Exchange(ref _stuckReturnedAfterMs, Math.Max(1, Environment.TickCount64 - began)),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking nit on precision: _stuckReturnedAfterMs is stamped when finished (the InvokeAsync task) completes, which is the whole Job — body and drain (CloseLeftoverWindows + Dispatcher.Yield(ApplicationIdle)) — not the instant the body itself returns. The field's own doc comment at line 175-176 ("how long after it started the body ... did return") and the PR description ("records whether that body returns later, and how long after it started") both describe it as a body-return timestamp.

In the scenario this PR is chasing — a cold first WPF start under load — the drain right after a cold dispatcher's first-ever operations could itself take non-trivial time, so the reported "N ms" slightly overstates how long the body alone took. More importantly, if the drain itself ever got stuck (not the body), FailIfStuck() would keep reporting "has not returned yet" even though the body genuinely did return — the two are conflated.

The new test's own comment at line 147 ("The record is written when the host's job finishes, just after its drain") shows this is understood, just not surfaced in the public-facing doc/PR text. Might be worth a one-line clarification in the field comment and/or the refusal message, since the whole point of this PR is precise wording for whoever reads the next CI hang report.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

  • VERIFIED — "All three files are test code. Nothing in PerformanceMonitor.Ui or any product changes." — gh pr diff --name-only lists only Darling/Darling.Tests/{StaTestThread.cs,StaTestThreadTests.cs,StatementScrubViewerPlanWithheldTests.cs}.
  • VERIFIED — "the host's drain ... now names its own step" — StaTestThread.cs:304-306, Step("the host's drain after the body: ...") added inside Job before CloseLeftoverWindows().
  • VERIFIED — "the host records whether that body returns later, and how long after it started" and the two refusal sentences — StaTestThread.cs:175-177,255-266,328-339 (_stuckReturnedAfterMs, the ContinueWith, FailIfStuck's two message branches), matched exactly by StaTestThreadTests.cs:143,150 ("has not returned yet (#6632)", regex That body did return, \d+ ms after it started).
  • VERIFIED — "No test asserted the old words [the removed 'stuck inside a body that never returned' wording]" — grepped the whole repo for that phrase and STA test thread/WPF test body did not return; the only hits are the source file itself and no .yml/.md depends on the old wording.
  • VERIFIED — "it compiles StaTestThread.cs through a link" (Lite.Tests) — Lite.Tests/Lite.Tests.csproj:69, <Compile Include="..\Darling\Darling.Tests\StaTestThread.cs" Link="StaTestThread.cs" />. This is also why no separate Lite.Tests change was needed for parity: the same file ships to both suites.
  • UNVERIFIED — the RED/GREEN result for the new test, the build/warning counts, and the Darling.Tests/Lite.Tests run totals ("Total 520, Failed 0" etc.) — this tree is Linux and read-only; the suites are WPF (Windows-only) and nothing here can build or run them.
  • UNVERIFIED — the CI forensics (job ids, runner image, ProcessThread.WaitReason samples, CPU-ms figures) — these are claims about external CI logs, not about this diff's code.

Riskiest lines

  1. Darling/Darling.Tests/StaTestThread.cs:264-266 — the continuation that stamps _stuckReturnedAfterMs fires on completion of the whole Job (body + drain), not on the body's own return, so the reported duration (and, in a drain-hangs scenario, the returned/not-returned verdict itself) folds in the drain — see inline comment. Covered by ALaterRefusal_SaysWhetherTheStrandedBodyReturned_AndHowLongAfterItStarted, but that test's own in-code comment ("just after its drain") shows it's asserting the as-built, drain-inclusive behavior rather than a stricter body-only one — not a gap in the test, just a looser contract than the field's doc comment implies.
  2. Darling/Darling.Tests/StaTestThread.cs:333-338 — FailIfStuck's sentinel logic (returnedAfter > 0 vs. "not returned yet") is the only thing standing between a correct slow-vs-stuck verdict and a misleading one once the thread is permanently stranded. Covered by the same new test (asserts both branches' exact wording).
  3. Darling/Darling.Tests/StaTestThreadTests.cs:148 — the SpinWait.SpinUntil poll for the async stamp to appear is the one place a flaky false-negative could sneak in (CI trap: "Our wrong test is a bug we introduced, never a flake"). It polls up to 10s with no fixed sleep, which is the right shape; not a finding.

Traps near this diff

  • "Lite and Darling ship in parity ... a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding" — does not apply here: StaTestThread.cs is the one file both suites compile (via the Link= include in Lite.Tests.csproj:69), so this change reaches Lite.Tests automatically with no counterpart edit needed.
  • "Ruling (#5833): tests are load-bearing ... never answer a trap with a test that reads source text or asserts how code is spelled" — does not apply: ALaterRefusal_SaysWhetherTheStrandedBodyReturned_AndHowLongAfterItStarted (StaTestThreadTests.cs:129-156) is a real behaviour test — it blocks a real body on a ManualResetEventSlim, lets the real hang guard fire, releases it, and asserts on the live refusal text produced by that run, not on the source of StaTestThread.cs.
  • "Our wrong test is a bug we introduced, never a flake" — does not apply: no new test here is failing; it's a new passing test plus pure diagnostic instrumentation (Step(...) calls) added to an existing test, with no logic change to that test's assertions (StatementScrubViewerPlanWithheldTests.cs).

Divergence (blind read vs PR body)

The PR body describes the stranded-body record as tracking "whether that body returns later, and how long after it started" — phrased as a body-return timestamp. The implementation actually timestamps completion of the body's dispatcher operation as a whole, which includes the post-body drain (CloseLeftoverWindows + Dispatcher.Yield(ApplicationIdle)); the new test's own in-code comment acknowledges this ("The record is written when the host's job finishes, just after its drain") but the PR body does not mention it. Everything else in the body matches the diff on inspection; this is the only point where the author's description is less precise than the code.

Reviewer checklist

  • Build and run Darling.Tests/Lite.Tests on a Windows WPF-capable machine to confirm the stated RED→GREEN result for the new test and the reported pass counts (520/0, 49/0), since this tree cannot build or execute WPF tests.

@claude

claude Bot commented Oct 23, 2026

Copy link
Copy Markdown

Follow-up pass (bacf84e..4774017): the only content in this range is the merge commit 477401708 itself, which brings in 0a6121e1c ("Fix the store script's owner list for the route functions #6593 changed", #6644) from dev. That commit is base-branch content merged into this PR's branch, not an author change on this PR — it only touches Darling/tools/provision-roles.sql's config.update_notification_route/config.set_notification_route_destinations signatures and adds config.set_notification_route_agent, unrelated to the WPF plan-withheld hang fix this PR is about. No commit in this range interacts with that file or with the prior WPF test changes, so per the merge rule there is nothing new to review here.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: LGTM

Claims

No new factual claims to grade: this range (bacf84e..4774017) contains only the merge commit 477401708 ("Merge branch 'dev' into fix/6632-wpf-plan-withheld-hang") and the dev-branch commit 0a6121e1c it brings in. There is no PR-body text or commit message in this range making a claim about the diff's behavior.

Riskiest lines

No lines to assess — the only diff content (Darling/tools/provision-roles.sql) is base-branch (dev) content merged in via 0a6121e1c, not an author change on this PR, and no commit in this range interacts with it.

Traps near this diff

No traps apply — there is no author change in this range to check against the traps file.

Divergence (blind read vs PR body)

None — no new claims in this range to compare against the diff.

Reviewer checklist

  • None beyond the prior pass's checklist (build/run Darling.Tests/Lite.Tests on Windows), since this range adds no new code to the WPF fix.

@erikdarlingdata
erikdarlingdata merged commit 473e634 into dev Oct 23, 2026
34 of 35 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/6632-wpf-plan-withheld-hang branch October 23, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant