Repository navigation
Retry a Jupyter kernel start that loses a port to another socket (bd-e87d5tmn) - #1190
Merged
Merged
Conversation
…e87d5tmn) The daemon peeks a kernel's ports (bind port 0, close) and the kernel binds them only once up; in between any socket can take one -- a concurrently starting kernel's connect retries, or another process. Since per-document kernels (#1173) start concurrently, CI hits this as "zmq.error.ZMQError: Address already in use" in the Jupyter e2e tests. No reservation can close that gap, so retry the start on fresh ports (up to 3 attempts) when the kernel exits reporting EADDRINUSE. Other startup failures still fail at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since #1173 (one Jupyter kernel per document), CI on
mainsometimes fails Jupyter e2e tests. The python kernel dies during startup with this error:There have been two failures so far, in two different tests:
parallel_documents_do_not_interleave_on_a_kernelrender_leaves_no_kernel_behindCause
reserve_portsgets the kernel's ports fromruntimelib::peek_ports. That function binds port 0, records the port and closes the listener. The kernel binds the port only after it starts, and in between any socket can take it. This race is a known TOCTOU, documented in runtimelib.#1173 made the race much easier to hit:
reserved_portsset doesn't cover this case.Fix
The kernel has to bind the ports itself, so no reservation scheme can close the gap. Instead, the start is retried:
JupyterError::is_port_conflict()recognizes a kernel that exited during startup with anEADDRINUSEmessage on stderr. It matches the POSIX, MSVC-CRT and Winsock wordings.start_kernelmakes up to 3 attempts, each with freshly peeked ports. Any other startup failure is still returned immediately.#1173 already reports a kernel that dies during startup right away instead of after a 30s timeout, so a retry costs only the kernel's startup time.
The full assessment is in
claude-notes/plans/2026-11-12-jupyter-kernel-port-race.md.Tests
The fake kernel has a new mode,
QUARTO_FAKE_KERNEL_FAIL_FIRST=<n>:<path>. It counts its spawns in<path>and exits with the ipykernel port-conflict message on the firstnspawns. New tests cover three cases:There are also unit tests for the message matching. The existing exit-at-startup test now uses a message that isn't a port conflict.
With the attempt count set to 1, the first two new tests fail.
cargo xtask verify --skip-hub-buildpasses locally on macOS. I couldn't cross-check quarto-core for Windows with clippy from macOS, because tree-sitter's C build fails forx86_64-pc-windows-msvc. The change uses no platform-specific APIs.🤖 Generated with Claude Code