Skip to content

fix: bound MCP connect by one deadline and cancel failed quantize workers - #812

Open
vishal-bala wants to merge 4 commits into
mainfrom
refactor/asyncio-timeout-taskgroup
Open

vishal-bala wants to merge 4 commits into
mainfrom
refactor/asyncio-timeout-taskgroup

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

With Python 3.11 as the floor, two async paths can use the standard library's structured primitives in place of hand-rolled timing and fan-out, and both had a behaviour bug the primitives fix. MCP startup gave the Redis connection and the first INFO a timeout each, so a slow server could hold startup for up to twice runtime.startup_timeout_seconds before it failed. Async multi-worker quantization ran its workers under asyncio.gather, which reports the first worker's failure while the other workers keep reading and writing Redis after the caller has been told the run failed.

Changes

One startup deadline for connect and INFO

RedisVLMCPServer._connect_redis_client now runs the connection and the INFO server check inside a single asyncio.timeout(timeout) block. The exception is still the builtin TimeoutError, which map_exception in redisvl/mcp/errors.py already maps to BACKEND_UNAVAILABLE, and startup() still tears down and re-raises as before. A client that connected but then failed or timed out on INFO used to be dropped without being closed. It is now closed before the error propagates. Reading the index and building the vectorizer keep their own wait_for timeouts, because each is a single await.

Sibling workers are cancelled on the first failure

async_multi_worker_quantize runs its workers in an asyncio.TaskGroup. When one worker raises, the group cancels the rest, and each worker's finally closes its Redis connection before the error reaches the caller.

The public exception contract is unchanged. The group's ExceptionGroup is unwrapped, and the first failing worker's exception is raised as is, outside the except block so its traceback and cause stay intact. Any other workers that failed in the same instant are logged at error level. Without the unwrap, AsyncMigrationExecutor.apply would record "unhandled errors in a TaskGroup (1 sub-exception)" in the migration report instead of the worker's own message, because it formats the exception into MigrationValidation.errors.

A cancelled worker leaves a backup shard that resume can use. Cancellation can only land on an awaited pipeline call, and every backup write between those awaits is synchronous, so a shard records only whole batches. A dump batch interrupted mid-read is not recorded, and resume reads it again. A quantize batch interrupted after Redis applied some of its HSETs is not marked done, and resume converts it again from the backup's stored originals, which writes the same bytes.

Minor changes

  • docs/user_guide/how_to_guides/mcp.md documents which startup steps runtime.startup_timeout_seconds bounds.
  • docs/user_guide/how_to_guides/migrate-indexes.md says that with --async a failing worker cancels the others, while the sync executor lets them finish.
  • The AsyncMigrationExecutor.apply docstring and the quantize.py module docstring name asyncio.TaskGroup instead of asyncio.gather.

Notes

The sync twin, multi_worker_quantize, still lets sibling threads finish their shards after one fails, because a running thread cannot be cancelled; it is unchanged here. Either way no shard is still being written when the failure is reported.

This PR is stacked on the StrEnum migration (#811), which is stacked on the Python 3.10 drop (#810), and needs both merged first. It shares no files with the StrEnum change; it sits above it only because the stack is linear.

Reviewer Brief

Start here: redisvl/migration/quantize.py, async_multi_worker_quantize. The unwrap after the TaskGroup block decides what every async migration caller sees when a worker fails.

Where it could fail

  • redisvl/migration/quantize.py:593: only the first of several simultaneous worker failures is raised; check that logging the others, rather than raising a group, is acceptable for callers that want every error.
  • redisvl/mcp/server.py:815: if the deadline fires inside RedisConnectionFactory._get_aredis_connection (its CLIENT SETINFO call is the first network round trip), no client has been returned yet, so nothing is closed here; this predates the PR.

Not verified: cluster mode for either path (_async_worker_quantize opens a plain redis.asyncio.from_url client, which predates this PR); redis-py versions other than 8.1.0, the one installed locally, for how a cancelled pipeline tidies up its connection.

Beyond CI: the four new regression tests fail against the code before the change (both cancellation tests see no sibling cancelled, the simultaneous-failure test logs nothing, and the deadline test does not raise).

Reviewed b4255a63 (rebased unchanged from 56e6b43b), fixes in 95010cea and ced2cc1f · Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docs, Redis behaviour


Note

Medium Risk
Touches MCP startup/teardown and async migration quantization fan-out; failures now stop sibling workers, but behavior changes could affect long-running migrations and startup timeout expectations.

Overview
MCP startup now applies a single asyncio.timeout around Redis connect and the first INFO server call in _connect_redis_client, so slow backends fail within one runtime.startup_timeout_seconds instead of nearly doubling it. Clients that connect but fail on INFO are closed before the error propagates.

Async multi-worker quantization switches from asyncio.gather to asyncio.TaskGroup in async_multi_worker_quantize: the first worker failure cancels siblings (each still closes its Redis client in finally), unwraps ExceptionGroup so callers get the first worker’s original exception, and logs any additional simultaneous failures. Partial backup shards remain resumable because only whole batches are recorded.

Docs clarify what startup_timeout_seconds covers for MCP, and that --async migrations cancel remaining quantize workers on failure (sync lets them finish). Tests add coverage for the shared connect/INFO deadline and for cancel/resume behavior during async multi-worker quantize.

Reviewed by Cursor Bugbot for commit 84430e9. Bugbot is set up for automated code reviews on this repo. Configure here.

@vishal-bala
vishal-bala added this pull request to stack #769 November 2, 2026 09:17
@vishal-bala
vishal-bala force-pushed the refactor/asyncio-timeout-taskgroup branch from 626fe56 to ced2cc1 Compare November 3, 2026 12:09
@vishal-bala vishal-bala added the auto:patch Increment the patch version when merged label Nov 6, 2026
@vishal-bala
vishal-bala marked this pull request as ready for review November 11, 2026 09:35
@vishal-bala
vishal-bala force-pushed the refactor/asyncio-timeout-taskgroup branch from 7902a5e to 41d6767 Compare November 11, 2026 10:20
Base automatically changed from refactor/str-enum to main November 13, 2026 10:35
_connect_redis_client wrapped the connection and the first INFO in
separate asyncio.wait_for calls, so a slow server could hold startup for
up to twice runtime.startup_timeout_seconds before failing. Both now run
under one asyncio.timeout block. The raised exception is still
TimeoutError, which map_exception already handles.

A client that connected but then failed or timed out on INFO was never
closed; it is now closed before the error propagates.
async_multi_worker_quantize ran its workers under asyncio.gather, which
raised the first worker's exception while the others kept reading from
and writing to Redis after the caller had been told the run failed. It
now runs them in an asyncio.TaskGroup, so the siblings are cancelled and
have closed their connections before the error propagates.

Callers keep the old contract. The TaskGroup's ExceptionGroup is
unwrapped and the first worker exception is raised as is; any others
that failed in the same instant are logged. Without that, the async
executor's report would read 'unhandled errors in a TaskGroup' instead
of the worker's own message.

A cancelled worker stops at an awaited pipeline call, and every backup
write between those awaits is synchronous, so its shard records only
whole batches. Resume re-reads an unrecorded dump batch and re-converts
an unrecorded quantize batch from the stored originals, which is
idempotent. Tests cover both cancellation points and the resume.
The MCP guide now names the three startup steps that
runtime.startup_timeout_seconds bounds, since the native-hybrid probe
runs without one. The migration guide says cancellation applies with
--async and that the sync executor lets sibling workers finish.

The shared-deadline test drops a wall-clock bound that the TimeoutError
already proves, and the simultaneous-failure test pins that the first
worker's error is the one raised.
make test passes --log-level=CRITICAL, which hid the ERROR records the
simultaneous-failure test counts. The test now sets the level itself.
@vishal-bala
vishal-bala force-pushed the refactor/asyncio-timeout-taskgroup branch from 41d6767 to 84430e9 Compare November 13, 2026 10:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:patch Increment the patch version when merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant