Repository navigation
Conversation
5c63ea7 to
1b9241e
Compare
|
Rebased onto latest ontogeny (e3b98ec). The branch previously carried stale |
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: batching the neighbour query shares the edge limit, dedup() and max_items cap across each batch of 10 vids, so the subgraph drops most neighbours of every vid after the first. Wrapping the per-vid traversal in local(...) keeps one round-trip per batch with per-vid results.
| batch = matched_vids[start : start + batch_size] | ||
| gremlin_query = VID_QUERY_NEIGHBOR_TPL.format( | ||
| keywords=f"'{matched_vid}'", | ||
| keywords=", ".join(f"'{vid}'" for vid in batch), |
There was a problem hiding this comment.
Important: bothE().limit({edge_limit}), dedup() and .limit({max_items}) in VID_QUERY_NEIGHBOR_TPL are global steps, so the vids in one batch share a single edge budget and dedup set; on TinkerGraph 3.5.1 with one edge label, 10 vids with 8 neighbours each return 8 paths, all from the first vid, instead of 80. Wrap the traversal after g.V(...) in local(...) so each vid keeps its own limits and dedup, and add a test that checks the generated query keeps that per-vid scope.
There was a problem hiding this comment.
Thanks for catching this — it was a real behaviour regression, not just an edge case.
Fixed in e3409f6. The traversal after g.V(...) is now wrapped in local(__. ...), so each matched vid runs the inner repeat ... path ... limit(max_items) in its own scope. bothE().limit(edge_limit), dedup() and .limit(max_items) are now per vid instead of sharing one edge budget / dedup set / cap across the batch, while the query count is still 1 + ceil(N / 10) (N=25: 4 execs, unchanged).
Generated query shape:
g.V('v1', ..., 'v10')
.local(
__.repeat(
bothE('edge').limit(edge_limit).otherV().dedup()
).times(max_deep).emit()
.simplePath()
.path()
.by(project('label', 'id', 'props')...)
.by(project('label', 'inV', 'outV', 'props')...)
.limit(max_items)
)
.toList()
Semantics are back to the old per-vid loop: each vid contributes its own paths, capped independently by max_items, with no cross-vid contamination of the edge limit or dedup() set.
Tests added in src/tests/nodes/test_graph_query_node.py, both asserting the generated query keeps the per-vid scope:
test_neighbor_query_wraps_per_vid_traversal_in_local_scope—bothE(,dedup()and both.limit(steps are insidelocal(...), with the global.toList()outside.test_neighbor_query_keeps_limits_inside_local_not_global— nolimit(/dedup()leaks ahead of.local(, and both limits are scoped inside it.
Verification:
ruff check <both files> -> All checks passed
ruff format --check <both files> -> 2 files already formatted
PYTHONPATH=src pytest src/tests/nodes -q -> 11 passed
Branch was also rebased onto current upstream/ontogeny, so the diff is now just the two files above.
1b9241e to
e3409f6
Compare
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: the branch sits on open #391, so merging it lands that PR's 68 commits and 383 files in ontogeny. The local(...) change resolves the earlier shared-budget comment: on HugeGraph 1.8.0 the batched query returns the same paths, in the same order, as the per-vid loop.
| # Number of matched vids grouped per single gremlin ``exec`` for the neighbour | ||
| # traversal. One round-trip per group instead of one per matched vid; the batch | ||
| # stays small to keep each query's work bounded. | ||
| NEIGHBOR_QUERY_BATCH_SIZE = 10 |
There was a problem hiding this comment.
There was a problem hiding this comment.
@bitflicker64 Done — rebased the two llm commits onto ontogeny and force-pushed perf/gremlin-neighbor-batching. The PR diff now contains only those 2 commits and graph_query_node.py + its test (test_graph_query_node.py).
The batched neighbour query ran repeat/bothE().limit()/dedup()/limit(max_items) as global steps, so all vids in a batch shared one edge budget, one dedup set and one result cap; on a batch of 10 vids only the first vid's neighbours survived. Wrap the traversal after g.V(...) in local(__. ...) so each vid keeps its own limits and dedup while still issuing one exec per batch. Add tests asserting the generated query keeps those steps inside the local() scope.
e3409f6 to
fb2dee7
Compare
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Score: 9/10. Summary: Wrapping the neighbour traversal in local(...) keeps edge_limit, dedup() and max_items per vid while cutting exec calls from 1 + N to 1 + ceil(N/10), and the branch holds only graph_query_node.py and its test on ontogeny. The new tests in test_graph_query_node.py pass in the llm CI job.
imbajin
left a comment
There was a problem hiding this comment.
Score: 8.0/10 · Confidence: medium
Blocking: no
Summary
No new issue found in the per-vertex batching change. The earlier shared-limit and branch-history findings are addressed in this diff.
Evidence
Reviewed the traversal, call path, existing discussion and server ID lookup; the LLM CI job passed for this head.
Validation notes
The added tests mock Gremlin execution; I did not independently run a live server or validate other server versions and high-load behavior.
Closes #429
Summary
The id-to-match subgraph branch queried neighbours with one
gremlin().exec()per matched vid, so a large id set cost one server round-trip per id. Group the ids into batches ofNEIGHBOR_QUERY_BATCH_SIZE(10) and issue one exec per batch, concatenating each batch's["data"]inmatched_vidsorder.Changes
nodes/hugegraph_node/graph_query_node.py— batch the per-vid neighbour traversal; the full-vertexVERTEX_QUERY_TPLexec and the property-match branch are unchanged.tests/nodes/test_graph_query_node.py— covers batch count (5 vids → 2 execs, 25 vids →1 + ceil(25/10)), single vid, empty set, id split across batches, and cross-batch path ordering.Query count
For N matched vids:
1 + Nexecs →1 + ceil(N / 10)execs (e.g. N=25: 26 → 4).Test plan
From
hugegraph-llm:Result: new tests pass, 9 passed in
src/tests/nodes, ruff clean.Notes
perf/query-links-batch @215f248.VID_QUERY_NEIGHBOR_TPLends with one global.limit(max_items), so in a batch the cap is shared across the group instead of per vid. Ordering is preserved, but results can be a subset of the old per-vid union when the cap is hit. This mirrors the property-match branch, which already applies one global limit across all keywords.