Repository navigation
fix: bound MCP connect by one deadline and cancel failed quantize workers - #812
Open
vishal-bala wants to merge 4 commits into
Open
vishal-bala wants to merge 4 commits into
vishal-bala wants to merge 4 commits into
Conversation
vishal-bala
force-pushed
the
refactor/asyncio-timeout-taskgroup
branch
from
November 3, 2026 12:09
626fe56 to
ced2cc1
Compare
vishal-bala
marked this pull request as ready for review
November 11, 2026 09:35
vishal-bala
force-pushed
the
refactor/asyncio-timeout-taskgroup
branch
from
November 11, 2026 10:20
7902a5e to
41d6767
Compare
_connect_redis_client wrapped the connection and the first INFO in separate asyncio.wait_for calls, so a slow server could hold startup for up to twice runtime.startup_timeout_seconds before failing. Both now run under one asyncio.timeout block. The raised exception is still TimeoutError, which map_exception already handles. A client that connected but then failed or timed out on INFO was never closed; it is now closed before the error propagates.
async_multi_worker_quantize ran its workers under asyncio.gather, which raised the first worker's exception while the others kept reading from and writing to Redis after the caller had been told the run failed. It now runs them in an asyncio.TaskGroup, so the siblings are cancelled and have closed their connections before the error propagates. Callers keep the old contract. The TaskGroup's ExceptionGroup is unwrapped and the first worker exception is raised as is; any others that failed in the same instant are logged. Without that, the async executor's report would read 'unhandled errors in a TaskGroup' instead of the worker's own message. A cancelled worker stops at an awaited pipeline call, and every backup write between those awaits is synchronous, so its shard records only whole batches. Resume re-reads an unrecorded dump batch and re-converts an unrecorded quantize batch from the stored originals, which is idempotent. Tests cover both cancellation points and the resume.
The MCP guide now names the three startup steps that runtime.startup_timeout_seconds bounds, since the native-hybrid probe runs without one. The migration guide says cancellation applies with --async and that the sync executor lets sibling workers finish. The shared-deadline test drops a wall-clock bound that the TimeoutError already proves, and the simultaneous-failure test pins that the first worker's error is the one raised.
make test passes --log-level=CRITICAL, which hid the ERROR records the simultaneous-failure test counts. The test now sets the level itself.
vishal-bala
force-pushed
the
refactor/asyncio-timeout-taskgroup
branch
from
November 13, 2026 10:35
41d6767 to
84430e9
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
With Python 3.11 as the floor, two async paths can use the standard library's structured primitives in place of hand-rolled timing and fan-out, and both had a behaviour bug the primitives fix. MCP startup gave the Redis connection and the first
INFOa timeout each, so a slow server could hold startup for up to twiceruntime.startup_timeout_secondsbefore it failed. Async multi-worker quantization ran its workers underasyncio.gather, which reports the first worker's failure while the other workers keep reading and writing Redis after the caller has been told the run failed.Changes
One startup deadline for connect and INFO
RedisVLMCPServer._connect_redis_clientnow runs the connection and theINFO servercheck inside a singleasyncio.timeout(timeout)block. The exception is still the builtinTimeoutError, whichmap_exceptioninredisvl/mcp/errors.pyalready maps toBACKEND_UNAVAILABLE, andstartup()still tears down and re-raises as before. A client that connected but then failed or timed out onINFOused to be dropped without being closed. It is now closed before the error propagates. Reading the index and building the vectorizer keep their ownwait_fortimeouts, because each is a single await.Sibling workers are cancelled on the first failure
async_multi_worker_quantizeruns its workers in anasyncio.TaskGroup. When one worker raises, the group cancels the rest, and each worker'sfinallycloses its Redis connection before the error reaches the caller.The public exception contract is unchanged. The group's
ExceptionGroupis unwrapped, and the first failing worker's exception is raised as is, outside theexceptblock so its traceback and cause stay intact. Any other workers that failed in the same instant are logged at error level. Without the unwrap,AsyncMigrationExecutor.applywould record "unhandled errors in a TaskGroup (1 sub-exception)" in the migration report instead of the worker's own message, because it formats the exception intoMigrationValidation.errors.A cancelled worker leaves a backup shard that resume can use. Cancellation can only land on an awaited pipeline call, and every backup write between those awaits is synchronous, so a shard records only whole batches. A dump batch interrupted mid-read is not recorded, and resume reads it again. A quantize batch interrupted after Redis applied some of its
HSETs is not marked done, and resume converts it again from the backup's stored originals, which writes the same bytes.Minor changes
docs/user_guide/how_to_guides/mcp.mddocuments which startup stepsruntime.startup_timeout_secondsbounds.docs/user_guide/how_to_guides/migrate-indexes.mdsays that with--asynca failing worker cancels the others, while the sync executor lets them finish.AsyncMigrationExecutor.applydocstring and thequantize.pymodule docstring nameasyncio.TaskGroupinstead ofasyncio.gather.Notes
The sync twin,
multi_worker_quantize, still lets sibling threads finish their shards after one fails, because a running thread cannot be cancelled; it is unchanged here. Either way no shard is still being written when the failure is reported.This PR is stacked on the StrEnum migration (#811), which is stacked on the Python 3.10 drop (#810), and needs both merged first. It shares no files with the StrEnum change; it sits above it only because the stack is linear.
Reviewer Brief
Start here:
redisvl/migration/quantize.py,async_multi_worker_quantize. The unwrap after theTaskGroupblock decides what every async migration caller sees when a worker fails.Where it could fail
redisvl/migration/quantize.py:593: only the first of several simultaneous worker failures is raised; check that logging the others, rather than raising a group, is acceptable for callers that want every error.redisvl/mcp/server.py:815: if the deadline fires insideRedisConnectionFactory._get_aredis_connection(itsCLIENT SETINFOcall is the first network round trip), no client has been returned yet, so nothing is closed here; this predates the PR.Not verified: cluster mode for either path (
_async_worker_quantizeopens a plainredis.asyncio.from_urlclient, which predates this PR); redis-py versions other than 8.1.0, the one installed locally, for how a cancelled pipeline tidies up its connection.Beyond CI: the four new regression tests fail against the code before the change (both cancellation tests see no sibling cancelled, the simultaneous-failure test logs nothing, and the deadline test does not raise).
Reviewed
b4255a63(rebased unchanged from56e6b43b), fixes in95010ceaandced2cc1f· Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docs, Redis behaviourNote
Medium Risk
Touches MCP startup/teardown and async migration quantization fan-out; failures now stop sibling workers, but behavior changes could affect long-running migrations and startup timeout expectations.
Overview
MCP startup now applies a single
asyncio.timeoutaround Redis connect and the firstINFO servercall in_connect_redis_client, so slow backends fail within oneruntime.startup_timeout_secondsinstead of nearly doubling it. Clients that connect but fail onINFOare closed before the error propagates.Async multi-worker quantization switches from
asyncio.gathertoasyncio.TaskGroupinasync_multi_worker_quantize: the first worker failure cancels siblings (each still closes its Redis client infinally), unwrapsExceptionGroupso callers get the first worker’s original exception, and logs any additional simultaneous failures. Partial backup shards remain resumable because only whole batches are recorded.Docs clarify what
startup_timeout_secondscovers for MCP, and that--asyncmigrations cancel remaining quantize workers on failure (sync lets them finish). Tests add coverage for the shared connect/INFO deadline and for cancel/resume behavior during async multi-worker quantize.Reviewed by Cursor Bugbot for commit 84430e9. Bugbot is set up for automated code reviews on this repo. Configure here.