Skip to content

fix: stop compute log capture deadlocking a command at exit - #1165

Merged
kvlonge merged 3 commits into
mainfrom
fix/sqb-test-hang
Oct 19, 2026
Merged

kvlonge merged 3 commits into
mainfrom
fix/sqb-test-hang

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Why

The DuckDB E2E (CLI commands) job hung until its 45-minute limit in two runs. Both times the
compile in the audit meta test that must fail on a non-JSON factory meta value never exited; one
or two other commands in the same runs also never finished.

A stack dump from a local reproduction (about 1 in 5,000 runs under load) showed a deadlock in
compute log capture. A late log record from a background thread was written to a capture already
marked complete. The storage built the rejection while still holding its lock, and building it
imports Python, which can release the GIL. The main thread, pruning the logs, then waited for the
lock while holding the GIL, so neither thread could continue.

Changes

  • Compute log storage releases its lock before raising the rejection for a write to an inactive
    capture, and documents that no Python may run while the lock is held.
  • An integration test forces that exact interleaving (a slow first import of the storage
    exceptions while the main thread prunes). Without the fix it deadlocks every time.
  • run_sqb in the E2E helpers gives each command ten minutes. A command still running then has its
    process group sent SIGABRT with PYTHONFAULTHANDLER set, so the failing test shows every
    Python thread's stack and the job continues. Standard input ends at once when a test passes
    none, which matches the /dev/null stdin xdist workers already have. Tests for the helper join
    the CLI shard.

Verification

  • New storage test fails (deadlock) before the fix and passes after.
  • Focused suites: compute log unit and integration tests, Recall integration tests, the audit
    meta, upstream function and janitor E2E files, and the shared helper tests.
  • 4,000 stress runs of the failing compile on the fixed build with no hang.
  • make check-ci via the verify hook, ruff, ty, fensu and the public hygiene scan.

…hed capture

A write that reached a capture already marked complete built its rejection while still holding
the storage lock. Building it imports Python, which can release the GIL; the main thread then
pruning the logs waited for that lock while holding the GIL, so neither could continue and the
command never exited. The lock is now released before the rejection is raised.
…alling the job

run_sqb now gives each command ten minutes. A command still running then has its process group
aborted with Python's fault handler enabled, so the failing test reports every thread's stack,
and the group is killed. Standard input ends at once when a test passes none.
@kvlonge
kvlonge enabled auto-merge (squash) October 19, 2026 21:01
@kvlonge
kvlonge merged commit 21727f6 into main Oct 19, 2026
44 checks passed
@kvlonge
kvlonge deleted the fix/sqb-test-hang branch October 19, 2026 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant