Skip to content

Retry a Jupyter kernel start that loses a port to another socket (bd-e87d5tmn) - #1190

Merged
cscheid merged 2 commits into
mainfrom
bugfix/bd-e87d5tmn-jupyter-kernel-startup-flakes
Nov 12, 2026
Merged

cscheid merged 2 commits into
mainfrom
bugfix/bd-e87d5tmn-jupyter-kernel-startup-flakes

Conversation

@cscheid

@cscheid cscheid commented Nov 12, 2026

Copy link
Copy Markdown
Member

Problem

Since #1173 (one Jupyter kernel per document), CI on main sometimes fails Jupyter e2e tests. The python kernel dies during startup with this error:

zmq.error.ZMQError: Address already in use (addr='tcp://127.0.0.1:38829')

There have been two failures so far, in two different tests:

  • run 43235574747: parallel_documents_do_not_interleave_on_a_kernel
  • run 43118290398: render_leaves_no_kernel_behind

Cause

reserve_ports gets the kernel's ports from runtimelib::peek_ports. That function binds port 0, records the port and closes the listener. The kernel binds the port only after it starts, and in between any socket can take it. This race is a known TOCTOU, documented in runtimelib.

#1173 made the race much easier to hit:

  • Kernel starts within a process now run concurrently. While one kernel starts, the shell/iopub connect for it keeps retrying, and every retry takes a fresh ephemeral source port. One of those can be a port that was peeked for another kernel but not yet bound. The reserved_ports set doesn't cover this case.
  • Other nextest processes start kernels too, and the in-process reservation can't see them.
  • A kernel per document means more kernel starts in total.

Fix

The kernel has to bind the ports itself, so no reservation scheme can close the gap. Instead, the start is retried:

  • JupyterError::is_port_conflict() recognizes a kernel that exited during startup with an EADDRINUSE message on stderr. It matches the POSIX, MSVC-CRT and Winsock wordings.
  • start_kernel makes up to 3 attempts, each with freshly peeked ports. Any other startup failure is still returned immediately.

#1173 already reports a kernel that dies during startup right away instead of after a 30s timeout, so a retry costs only the kernel's startup time.

The full assessment is in claude-notes/plans/2026-11-12-jupyter-kernel-port-race.md.

Tests

The fake kernel has a new mode, QUARTO_FAKE_KERNEL_FAIL_FIRST=<n>:<path>. It counts its spawns in <path> and exits with the ipykernel port-conflict message on the first n spawns. New tests cover three cases:

  • one port conflict, then success (2 spawns)
  • conflicts on every attempt, so it gives up after 3 spawns
  • a startup failure that isn't a port conflict is not retried (1 spawn)

There are also unit tests for the message matching. The existing exit-at-startup test now uses a message that isn't a port conflict.

With the attempt count set to 1, the first two new tests fail.

cargo xtask verify --skip-hub-build passes locally on macOS. I couldn't cross-check quarto-core for Windows with clippy from macOS, because tree-sitter's C build fails for x86_64-pc-windows-msvc. The change uses no platform-specific APIs.

🤖 Generated with Claude Code

cscheid and others added 2 commits November 12, 2026 11:14
…e87d5tmn)

The daemon peeks a kernel's ports (bind port 0, close) and the kernel
binds them only once up; in between any socket can take one -- a
concurrently starting kernel's connect retries, or another process. Since
per-document kernels (#1173) start concurrently, CI hits this as
"zmq.error.ZMQError: Address already in use" in the Jupyter e2e tests.

No reservation can close that gap, so retry the start on fresh ports (up
to 3 attempts) when the kernel exits reporting EADDRINUSE. Other startup
failures still fail at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@posit-snyk-bot

posit-snyk-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@cscheid
cscheid merged commit 71f9e94 into main Nov 12, 2026
15 checks passed
@cscheid
cscheid deleted the bugfix/bd-e87d5tmn-jupyter-kernel-startup-flakes branch November 12, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants