Repository navigation
fix(llm): escape vertex IDs in Gremlin queries - #383
atharrva01 wants to merge 5 commits into
Conversation
|
hi @imbajin , this pr fixed Gremlin query failures caused by apostrophes in vertex IDs by adding proper string escaping and regression tests. |
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: At this head the PR adds gremlin_str_literal() but no production code calls it. The latest commit (baf1bcc, "test(llm): accept list form in fake gremlin endpoint") puts the three call sites changed in 3f7b54c back to their main form, so IDs like 1:O'Brien still break the queries and the three new regression tests fail. The helper itself looks correct for Groovy single-quoted strings, and the tests in 3f7b54c are a good way to pin the behaviour. Evidence: read the full diff at baf1bcc against merge-base 84ee9c4, plus git show 3f7b54c and git show baf1bcc. Ran uv run pytest on tests/nodes/test_graph_query_node.py, tests/utils/test_graph_index_utils.py, tests/operators/index_op/test_semantic_id_query.py and tests/utils/test_hugegraph_utils.py on Python 3.11: 3 failed, 13 passed at baf1bcc (each failure is ServerError: startup failed: unterminated string literal from FakeGremlinEndpoint); 16 passed with the hugegraph_llm sources of 3f7b54c and the tests of baf1bcc. The CI workflows on this head are waiting for maintainer approval, so they have not run yet.
baf1bcc to
955d270
Compare
|
hi @bitflicker64, I accidentally committed the main versions of those three files in the last commit. Fixed now, all three call sites use the helper again and the tests pass. I kept the list handling in the fake endpoint so the node test fails for the right reason on main. |
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The serializer can change numeric vertex IDs into string IDs and leaves control characters unescaped, so affected Gremlin lookups fail. Evidence: exact-head static review of the GraphRAG ID flow and Groovy literal handling.
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: The escaping is correct: every literal gremlin_literal emits parses back to the original value in Groovy 2.5 and 4, and a real HugeGraph 1.7.0 server finds ids that contain apostrophes, backslashes, $, tabs, double quotes or newlines. With apostrophe and newline ids, get_vertex_details, _exact_match_vids and the vid path of _subgraph_query all return the expected vertices. Keeping numeric ids numeric has a side effect, though. The neighbor query now returns paths for numeric-id vertices, and _process_vertex cannot format them. On a numeric-id graph, graph RAG now loses the vertex context that main returns (inline comment). Evidence: full diff 84ee9c4..fbd4743 and the surrounding GraphQueryNode, SemanticIdQuery, get_vertex_details and vector-index code. The four touched test files pass (26 passed), and the CI unit selection passes (549 passed, 3 skipped). With the three production files reset to 84ee9c4, the 4 new call-site tests fail, so they do pin the fix. The emitted literals were round-tripped through groovy-2.5.14, 2.5.23 and 4.0.25. The PR's own code paths were run against hugegraph/server 1.7.0 (Gremlin 3.5.1, RocksDB) on a throwaway container: g.V('123') finds 0 vertices and g.V(123) finds 1, and the numeric-id regression is reproduced at this head but not with main's graph_query_node.py. All 24 latest-head checks pass, including llm-hugegraph-boundary and llm-core-smoke. None of them builds a numeric-id graph, so none of them reaches this path.
imbajin
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The production Gremlin literal escaping and numeric-ID formatting look correct; the added fake endpoint parser rejects one valid literal shape and leaves that case untested. Evidence: exact-head static review of FakeGremlinEndpoint.exec.
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Score: 10/10. Summary: gremlin_literal is correct for every place this PR uses it, numeric ids stay numeric, and the numeric-id path formatting in _process_vertex now works.
Evidence: read the diff fbd4743..142fdf2 and the full PR diff against 84ee9c4. The four touched test files pass on Python 3.11 (30 passed). With line 197 of graph_query_node.py put back to node_str = matched_str, test_subgraph_query_keeps_numeric_vertex_ids_numeric fails with TypeError: can only concatenate str (not "int") to str, so the test pins the fix. I also round-tripped 3037 literals from gremlin_literal (apostrophes, backslashes, literal \u0027 text, $, ${...}, double quotes, CR/LF, NUL, DEL, U+2028, emoji, CJK, plus 3000 random mixes) through Groovy 2.5.23 (the version HugeGraph 1.7.0 ships) and Groovy 4.0.25. Each one parsed to a single String equal to the input. All 24 CI checks pass at 142fdf2, and the new tests run in llm-unit-contract. No live HugeGraph run at this head; the run against HugeGraph 1.7.0 was at fbd4743, and the delta since then is one str() call plus tests.
Earlier findings:
- My earlier thread, call sites reverted at baf1bcc: fixed. The branch was rewritten and all three call sites use the helper from 3f7b54c on.
- My earlier thread,
TypeErrorin_process_vertexfor numeric ids: fixed in 63005af, and the test now feeds two paths that start at123. - @imbajin, numeric ids turned into strings, and unescaped control characters: fixed in fbd4743.
- @imbajin, fake endpoint regex cut off quoted ids: fixed in 142fdf2.
Possible follow-up, outside this PR and not counted: the edge labels (graph_query_node.py:350) and the property key in PROPERTY_QUERY_NEIGHBOR_TPL are still quoted by hand in the same templates. This PR does not touch those lines. Running them through gremlin_literal too would make the templates consistent.
|
Thanks for the thorough review! @imbajin @bitflicker64 , Agreed on the edge labels and property key, I'll do those in a separate PR so this one stays as is. |
|
hi @imbajin, gentle ping on this one. bitflicker64 approved it and CI is green. Is there anything else you'd like me to change before it can be merged? |
142fdf2 to
9c94f31
Compare
|
Rebased onto latest main. The only conflict was in @imbajin, could you take a look when you have a moment? |
pytest's default prepend import mode collides when two test modules share a basename in directories without __init__.py. The in-flight Gremlin-escaping work (apache#383/apache#385) adds its own test_graph_query_node.py, so rename this module to test_graph_query_vertex_knowledge.py (the behavior it covers) instead of keeping a duplicate name. Co-Authored-By: Claude Code <noreply@anthropic.com>
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Score: 10/10. Summary: The rebase leaves the production hunks unchanged, so gremlin_literal still escapes string ids and keywords, keeps numeric ids numeric, and _process_vertex formats numeric ids; the full hugegraph-llm suite passes and the new call-site tests fail against main's sources.
imbajin
left a comment
There was a problem hiding this comment.
Score: 8.1/10 · Confidence: medium
Blocking: no
Summary
No new actionable finding in this revision. The earlier numeric-ID and cached-path issues are addressed: integer IDs remain numeric, and cached IDs are converted to strings before concatenation.
Evidence
Checked the literal helper and its GraphQueryNode, SemanticIdQuery and get_vertex_details call sites. Isolated execution of the exact-head helper passed eight cases covering quotes, backslashes, control characters and numeric versus string IDs. Current LLM, quality, CodeQL and CI Result checks are successful.
Validation notes
The helper check uses Python literal round trips, not a real Groovy parser or HugeGraph query. No live server/provider test was independently run. A current-head Pull Request Labeler run also has startup_failure; the successful core checks above do not mean every workflow passed.
pytest's default prepend import mode collides when two test modules share a basename in directories without __init__.py. The in-flight Gremlin-escaping work (apache#383/apache#385) adds its own test_graph_query_node.py, so rename this module to test_graph_query_vertex_knowledge.py (the behavior it covers) instead of keeping a duplicate name. Co-Authored-By: Claude Code <noreply@anthropic.com>
I found that vertex IDs containing apostrophes, like
1:O'Brien, break Gremlin queries and silently cause graph RAG to lose its graph context. The same issue can also cause/rag/graphto return a 500.I added a
gremlin_str_literal()helper to escape backslashes and apostrophes, and applied it wherever IDs or keywords are interpolated into Gremlin queries. Plain IDs behave as before.Tests: Added regression tests for the operator, node, and
get_vertex_details. All three fail onmainand pass with this fix. Module tests pass and Ruff is clean.Note: Live HugeGraph testing wasn't performed. Apostrophe removal during keyword extraction remains out of scope for this PR.
Before / After: screenshots below.


Before
After