Skip to content

fix: match interrupted connections by the connection, not a reusable id - #1214

Merged
kvlonge merged 2 commits into
mainfrom
fix/interrupted-connection-identity
Oct 22, 2026
Merged

kvlonge merged 2 commits into
mainfrom
fix/interrupted-connection-identity

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Why

A deadline cancellation marked the interrupted connection by id(connection) in a process-wide set. Only a recovering connection pool's health check cleared marks, so a connection that never returned to such a pool left its mark behind. Once that connection was collected, a later connection allocated at the same address was health-checked as if it had been interrupted, running an extra probe statement and publishing its statement events. This made the scenario capture pipeline's connection-lifecycle unit test fail intermittently in CI. Separately, the 1000-model build memory guard sat within 0.2-0.5% of its cap on main, so unrelated branches failed it at random.

Changes

  • InterruptedConnections keeps a weak reference to each marked connection. Objects that can't be weakly referenced get a strong one. A check reports only when the mark still refers to the same connection, and each check still clears the mark.
  • The 1000-model build memory guard goes from 1,728 to 1,760 MiB: main measures 1,778-1,808 MB on CI runners (about +/-15 MB noise) against the old 1,812 MB cap.
  • No change for connections that really were interrupted: they are still checked once and replaced when unusable.
  • Unit tests cover single reporting with and without weak references, and a new connection reusing a collected interrupted connection's identity. That last test fails without the fix.

Verification

  • Forcing a mark onto the capture pipeline's connection reproduces the CI failure: two extra statement events.
  • Focused execution-limit, scheduling, pipeline, load and executor integration tests pass (517).
  • ruff, ty and fensu are clean, make check-ci passes, and the hygiene scan is clean.

Deadline cancellation marked a connection by `id(connection)` in a
process-wide set that only a recovering pool's health check cleared. A
mark left on a connection that never returned to such a pool outlived
it, and a later connection allocated at the same address was then
health-checked as if interrupted, running an extra probe statement.

Each mark now holds a weak reference to its connection, or a strong
one when the connection cannot be weakly referenced, and a check
reports only when the mark still refers to that same connection.
Main runs at 1,778-1,808 MB on CI runners against a 1,812 MB cap, so unrelated branches fail the guard at random (about +/-15 MB runner noise). 1,760 MiB leaves about 31 MB above the highest CI value seen.
@kvlonge
kvlonge enabled auto-merge (squash) October 22, 2026 06:38
@kvlonge
kvlonge merged commit e1a2850 into main Oct 22, 2026
37 checks passed
@kvlonge
kvlonge deleted the fix/interrupted-connection-identity branch October 22, 2026 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant