Skip to content

refactor(llm): unify parallel embedding fetch with bounded concurrency - #424

Open
lplalbert wants to merge 2 commits into
apache:mainfrom
lplalbert:refactor/unify-parallel-embedding-fetch
Open

lplalbert wants to merge 2 commits into
apache:mainfrom
lplalbert:refactor/unify-parallel-embedding-fetch

Conversation

@lplalbert

@lplalbert lplalbert commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Two divergent parallel-embedding implementations for the same job:

  • utils/embedding_utils.get_embeddings_parallel — batch 500, no concurrency bound, despite its docstring promising "a semaphore to limit maximum concurrency"
  • BuildSemanticIndex._get_embeddings_parallel — batch 1000, Semaphore(10), sync method via run_in_executor

build_gremlin_example_index.py carried a FIXME for this split ("we need keep the logic same with build_semantic_index.py"). The unbounded helper is a hazard on large inputs: 100k chunks -> 200 batch requests all in flight at once.

What this changes

One bounded implementation (get_embeddings_parallel) shared by all three index builds:

  • the helper now acquires a Semaphore(10) per batch (the bound BuildSemanticIndex already used), honoring the docstring contract that already existed
  • BuildSemanticIndex delegates to the helper; its private copy is removed and the FIXME is retired
  • order preservation is unchanged (asyncio.gather)

Behavior deltas (for review)

  • Semantic vid index batches 500 instead of 1000 — progress-bar granularity only; embedding models batch internally (32/10 texts per API call)
  • Semantic vid index fetches through the shared helper, which runs executor-wrapped sync get_texts_embeddings (see the review follow-up below)
  • Gremlin-example and vector-index builds (already on the helper) gain the concurrency bound

Review follow-up (0722aac)

imbajin's review caught a lifecycle regression in the unification: the helper ran batches through embedding.async_get_texts_embeddings, while the index operators call it through asyncio.run once per update — a fresh loop per update against an instance-owned async client whose pooled keep-alive connections outlive the first loop. The second nonempty update on one instance could fail with RuntimeError: Event loop is closed and leave the new vertices unindexed; all three consumers of the helper shared the failure. The helper now retains the synchronous executor path: each batch runs get_texts_embeddings on the loop's executor, which holds no loop-bound state, so client and loop lifetimes no longer need to match. Order, batching (500) and the in-flight bound (10) are unchanged; the async fetch methods stay on the embedding API for direct use.

Validation (executed locally)

  • SKIP_EXTERNAL_SERVICES=true uv run --extra dev pytest -q hugegraph-llm/src/tests -> 1292 passed, 12 skipped, 2 failed (after 0722aac; 1290 passed at the first push) — the two test_external_config.py failures (test_saved_complete_connection_survives_deployment_upgrade, test_single_file_mount_fallback_and_existing_mode) also fail on clean origin/main (verified via stash), pre-existing and unrelated
  • Red-green: new test_bounds_concurrent_batches fails before the fix (30 batches in flight vs bound 10) and passes after; the updated semantic-index tests fail against the old private implementation (4 failed) and pass after; after 0722aac, test_two_updates_on_one_instance_keep_indexing is red before with RuntimeError: Event loop is closed on the second update and green after, and test_fetch_avoids_loop_bound_async_clients pins the fetch to the sync client
  • uv run --extra dev ruff format --check + ruff check clean on changed files
  • uv run --extra dev ty check hugegraph-llm/src/hugegraph_llm reports 265 diagnostics before and after 0722aac (none introduced)
  • Pre-existing, out of scope: test_build_vector_index.py/test_gremlin_example_index_query.py emit a "coroutine never awaited" RuntimeWarning (their patched async helper returns a coroutine the also-mocked asyncio.run discards) — reproduced on clean origin/main

Refs #107

🤖 Generated with Claude Code

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: get_embeddings_parallel now caps in-flight batches at 10 and BuildSemanticIndex delegates to it; the OpenAI and Ollama async paths match their sync counterparts, and the new bound test fails on the base helper and passes here.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 7.0/10 · Confidence: high

Blocking: yes

Summary

Reusing the semantic index with a persistent async embedding client can fail on a subsequent update.

Evidence

The changed call creates a fresh event loop for each run while provider instances retain their async clients. A local HTTP/1.1 keep-alive reproduction using httpx succeeded on the first asyncio.run and raised RuntimeError: Event loop is closed on the second.

Validation notes

Controlled HTTP transport reproduction and static caller review; no live OpenAI or Ollama integration was run.

if added_vids:
vids_to_process = self._extract_names(added_vids) if all_pk_flag else added_vids
added_embeddings = asyncio.run(self._get_embeddings_parallel(vids_to_process))
added_embeddings = asyncio.run(get_embeddings_parallel(self.embedding, vids_to_process))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Keep the async client on a single event loop

When the same semantic-index operator or embedding instance handles another nonempty update, asyncio.run creates a new loop after closing the first one, but the OpenAI/Ollama async client retains its HTTP connection pool. With a reusable keep-alive connection, the second request can fail with RuntimeError: Event loop is closed, leaving the new vertices unindexed. I reproduced this lifecycle with an httpx.AsyncClient and a local HTTP/1.1 keep-alive server. Please keep the client and event loop lifetimes aligned (or retain the synchronous executor path), and cover two updates using the same instance.

@lplalbert lplalbert Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Root cause: the shared fetch ran every batch through embedding.async_get_texts_embeddings, so the instance-owned async client (and its pooled keep-alive connections) outlived the per-update loop that asyncio.run creates and closes. The same failure applied to all three consumers of the helper (BuildSemanticIndex, BuildGremlinExampleIndex, BuildVectorIndex).

Structural fix (0722aac): retained the synchronous executor path. Each batch now runs get_texts_embeddings on the loop's executor — the route BuildSemanticIndex used before this PR. The synchronous clients hold no loop-bound state, so client and loop lifetimes no longer need to match across updates. Order, batching (500) and the in-flight bound (10) are unchanged; the async fetch methods stay on the embedding API for direct use.

Validation:

  • test_two_updates_on_one_instance_keep_indexing: two nonempty updates on one BuildSemanticIndex instance through the real helper. Red before 0722aac with RuntimeError: Event loop is closed on the second update; both updates now index their new vertices. The embedding fake reproduces the reported lifecycle (one pooled connection bound to the loop that first used it) — it models the transport binding, not a live HTTP keep-alive exchange, so it does not establish real-client behavior.
  • test_fetch_avoids_loop_bound_async_clients pins the fetch to the sync client.
  • Full suite: 1292 passed, 12 skipped, 2 failed — the same two pre-existing test_external_config.py Windows mode-bit failures as on main. ruff format/check clean on touched files; ty check hugegraph_llm 265 diagnostics before and after (zero new).

lplalbert and others added 2 commits November 16, 2026 10:27
BuildSemanticIndex._get_embeddings_parallel duplicated get_embeddings_parallel
with divergent batching (1000 vs 500) and concurrency (Semaphore(10) vs none),
and the shared helper's docstring promised a semaphore it never had. Every index
build now shares one bounded implementation:

- get_embeddings_parallel caps in-flight batches at 10 (the bound
  BuildSemanticIndex already used), so a large input cannot fan out one
  request per batch
- BuildSemanticIndex delegates to the shared helper; its private copy is
  removed and the build_gremlin_example_index.py FIXME is resolved
- order preservation across batches is unchanged (asyncio.gather)

Behavioral deltas for the semantic vid index: batches of 500 instead of 1000
(progress granularity only; embedding models batch internally per API call) and
async_get_texts_embeddings instead of executor-wrapped sync get_texts_embeddings
- the Ollama/OpenAI implementations build sync and async clients from the same
config with identical batching and ordering semantics.

Tests: new tests/utils/test_embedding_utils.py pins order, batching, empty
input, error propagation and the concurrency bound (red before the fix: all 30
batches in flight at once). Semantic-index tests follow the unified batching
and patch the shared entry point (red before the fix, green after).

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ient

Review finding (imbajin, apache#424): index operators run get_embeddings_parallel
through asyncio.run() once per update, so each nonempty update gets a fresh
event loop after the previous one is closed, while the OpenAI/Ollama async
client kept by the embedding instance retains its HTTP connection pool. With a
reusable keep-alive connection, the second update reuses pooled state bound to
the closed loop and fails with RuntimeError: Event loop is closed, leaving the
new vertices unindexed. The same failure applied to every consumer of the
shared helper (BuildSemanticIndex, BuildGremlinExampleIndex, BuildVectorIndex).

The fetch path now retains the synchronous executor route BuildSemanticIndex
used before this PR: each batch runs get_texts_embeddings on the loop's
executor. The synchronous clients hold no loop-bound state, so client and loop
lifetimes no longer need to match across updates. Order, batching (500) and
the in-flight bound (10) are unchanged; the async fetch methods remain on the
embedding API for direct use.

Tests: test_two_updates_on_one_instance_keep_indexing covers two nonempty
updates on one BuildSemanticIndex instance through the real helper (red
before: RuntimeError: Event loop is closed on the second update, via an
embedding fake whose pooled connection is bound to the loop that first used
it). test_fetch_avoids_loop_bound_async_clients pins the fetch to the sync
client. The tracking fake now records from both paths, keeping the order,
concurrency-bound and error-propagation tests valid for either route.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@lplalbert
lplalbert force-pushed the refactor/unify-parallel-embedding-fetch branch from d1d8ead to 0722aac Compare November 16, 2026 02:39

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: get_embeddings_parallel caps in-flight batches at 10 and runs each batch through sync get_texts_embeddings on the loop's executor, so BuildSemanticIndex, BuildVectorIndex and BuildGremlinExampleIndex share one bounded, loop-agnostic path; the earlier event-loop-lifetime comment is fixed and covered by test_two_updates_on_one_instance_keep_indexing.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 7.5/10 · Confidence: high

Blocking: yes

Summary

Synchronous executor batches can continue issuing requests after a peer batch fails.

Evidence

A controlled standard-library executor reproduction showed ongoing work completing before asyncio.run returned. Source inspection confirms OpenAI batches issue sequential requests; no live provider was exercised.

Validation notes

Controlled executor reproduction and static source review; no live embedding provider tested.

# Callers run this helper on a fresh loop per update, while the embedding
# instance (and its clients) outlives those loops. The synchronous client
# holds no loop-bound state, so repeated runs on one instance stay safe.
result = await loop.run_in_executor(None, embedding.get_texts_embeddings, batch)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Stop remaining batch work after a failure

If one batch fails while other batches are running, asyncio.run() waits for its executor threads to finish; cancelling the futures cannot stop their synchronous work. For Gremlin and vector indexing this replaces cancellable async requests, and an OpenAI batch can continue issuing up to 50 sequential requests after the overall operation has failed, delaying the error and incurring unnecessary API usage. Please make remaining batches cooperatively stop on failure or use cancellable requests with a client lifecycle scoped to the event loop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants