Skip to content

fix(llm): escape vertex IDs in Gremlin queries - #383

Open
atharrva01 wants to merge 5 commits into
apache:mainfrom
atharrva01:fix/llm-escape-gremlin-vertex-ids
Open

atharrva01 wants to merge 5 commits into
apache:mainfrom
atharrva01:fix/llm-escape-gremlin-vertex-ids

Conversation

@atharrva01

Copy link
Copy Markdown

I found that vertex IDs containing apostrophes, like 1:O'Brien, break Gremlin queries and silently cause graph RAG to lose its graph context. The same issue can also cause /rag/graph to return a 500.

I added a gremlin_str_literal() helper to escape backslashes and apostrophes, and applied it wherever IDs or keywords are interpolated into Gremlin queries. Plain IDs behave as before.

Tests: Added regression tests for the operator, node, and get_vertex_details. All three fail on main and pass with this fix. Module tests pass and Ruff is clean.

Note: Live HugeGraph testing wasn't performed. Apostrophe removal during keyword extraction remains out of scope for this PR.

Before / After: screenshots below.
Before
image
After
image

@atharrva01

Copy link
Copy Markdown
Author

hi @imbajin , this pr fixed Gremlin query failures caused by apostrophes in vertex IDs by adding proper string escaping and regression tests.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: At this head the PR adds gremlin_str_literal() but no production code calls it. The latest commit (baf1bcc, "test(llm): accept list form in fake gremlin endpoint") puts the three call sites changed in 3f7b54c back to their main form, so IDs like 1:O'Brien still break the queries and the three new regression tests fail. The helper itself looks correct for Groovy single-quoted strings, and the tests in 3f7b54c are a good way to pin the behaviour. Evidence: read the full diff at baf1bcc against merge-base 84ee9c4, plus git show 3f7b54c and git show baf1bcc. Ran uv run pytest on tests/nodes/test_graph_query_node.py, tests/utils/test_graph_index_utils.py, tests/operators/index_op/test_semantic_id_query.py and tests/utils/test_hugegraph_utils.py on Python 3.11: 3 failed, 13 passed at baf1bcc (each failure is ServerError: startup failed: unterminated string literal from FakeGremlinEndpoint); 16 passed with the hugegraph_llm sources of 3f7b54c and the tests of baf1bcc. The CI workflows on this head are waiting for maintainer approval, so they have not run yet.

@atharrva01
atharrva01 force-pushed the fix/llm-escape-gremlin-vertex-ids branch from baf1bcc to 955d270 Compare October 15, 2026 12:41
@atharrva01

Copy link
Copy Markdown
Author

hi @bitflicker64, I accidentally committed the main versions of those three files in the last commit. Fixed now, all three call sites use the helper again and the tests pass.

I kept the list handling in the fake endpoint so the node test fails for the right reason on main.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The serializer can change numeric vertex IDs into string IDs and leaves control characters unescaped, so affected Gremlin lookups fail. Evidence: exact-head static review of the GraphRAG ID flow and Groovy literal handling.

@atharrva01
atharrva01 requested a review from imbajin October 15, 2026 15:28

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The escaping is correct: every literal gremlin_literal emits parses back to the original value in Groovy 2.5 and 4, and a real HugeGraph 1.7.0 server finds ids that contain apostrophes, backslashes, $, tabs, double quotes or newlines. With apostrophe and newline ids, get_vertex_details, _exact_match_vids and the vid path of _subgraph_query all return the expected vertices. Keeping numeric ids numeric has a side effect, though. The neighbor query now returns paths for numeric-id vertices, and _process_vertex cannot format them. On a numeric-id graph, graph RAG now loses the vertex context that main returns (inline comment). Evidence: full diff 84ee9c4..fbd4743 and the surrounding GraphQueryNode, SemanticIdQuery, get_vertex_details and vector-index code. The four touched test files pass (26 passed), and the CI unit selection passes (549 passed, 3 skipped). With the three production files reset to 84ee9c4, the 4 new call-site tests fail, so they do pin the fix. The emitted literals were round-tripped through groovy-2.5.14, 2.5.23 and 4.0.25. The PR's own code paths were run against hugegraph/server 1.7.0 (Gremlin 3.5.1, RocksDB) on a throwaway container: g.V('123') finds 0 vertices and g.V(123) finds 1, and the numeric-id regression is reproduced at this head but not with main's graph_query_node.py. All 24 latest-head checks pass, including llm-hugegraph-boundary and llm-core-smoke. None of them builds a numeric-id graph, so none of them reaches this path.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The production Gremlin literal escaping and numeric-ID formatting look correct; the added fake endpoint parser rejects one valid literal shape and leaves that case untested. Evidence: exact-head static review of FakeGremlinEndpoint.exec.

Comment thread hugegraph-llm/src/tests/utils/mock.py Resolved
@atharrva01
atharrva01 requested a review from imbajin October 17, 2026 12:58

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 10/10. Summary: gremlin_literal is correct for every place this PR uses it, numeric ids stay numeric, and the numeric-id path formatting in _process_vertex now works.

Evidence: read the diff fbd4743..142fdf2 and the full PR diff against 84ee9c4. The four touched test files pass on Python 3.11 (30 passed). With line 197 of graph_query_node.py put back to node_str = matched_str, test_subgraph_query_keeps_numeric_vertex_ids_numeric fails with TypeError: can only concatenate str (not "int") to str, so the test pins the fix. I also round-tripped 3037 literals from gremlin_literal (apostrophes, backslashes, literal \u0027 text, $, ${...}, double quotes, CR/LF, NUL, DEL, U+2028, emoji, CJK, plus 3000 random mixes) through Groovy 2.5.23 (the version HugeGraph 1.7.0 ships) and Groovy 4.0.25. Each one parsed to a single String equal to the input. All 24 CI checks pass at 142fdf2, and the new tests run in llm-unit-contract. No live HugeGraph run at this head; the run against HugeGraph 1.7.0 was at fbd4743, and the delta since then is one str() call plus tests.

Earlier findings:

  • My earlier thread, call sites reverted at baf1bcc: fixed. The branch was rewritten and all three call sites use the helper from 3f7b54c on.
  • My earlier thread, TypeError in _process_vertex for numeric ids: fixed in 63005af, and the test now feeds two paths that start at 123.
  • @imbajin, numeric ids turned into strings, and unescaped control characters: fixed in fbd4743.
  • @imbajin, fake endpoint regex cut off quoted ids: fixed in 142fdf2.

Possible follow-up, outside this PR and not counted: the edge labels (graph_query_node.py:350) and the property key in PROPERTY_QUERY_NEIGHBOR_TPL are still quoted by hand in the same templates. This PR does not touch those lines. Running them through gremlin_literal too would make the templates consistent.

@atharrva01

Copy link
Copy Markdown
Author

Thanks for the thorough review! @imbajin @bitflicker64 , Agreed on the edge labels and property key, I'll do those in a separate PR so this one stays as is.

@atharrva01

Copy link
Copy Markdown
Author

hi @imbajin, gentle ping on this one. bitflicker64 approved it and CI is green. Is there anything else you'd like me to change before it can be merged?

@atharrva01
atharrva01 force-pushed the fix/llm-escape-gremlin-vertex-ids branch from 142fdf2 to 9c94f31 Compare November 9, 2026 17:07
@atharrva01

Copy link
Copy Markdown
Author

Rebased onto latest main. The only conflict was in test_hugegraph_utils.py, next to the new schema-backup tests. The code is unchanged. The llm unit suite passes locally on Python 3.14 (1304 passed) and ruff is clean. #385 is rebased on top of this too.

@imbajin, could you take a look when you have a moment?

lplalbert pushed a commit to lplalbert/hugegraph-ai that referenced this pull request Nov 10, 2026
pytest's default prepend import mode collides when two test modules
share a basename in directories without __init__.py. The in-flight
Gremlin-escaping work (apache#383/apache#385) adds its own test_graph_query_node.py,
so rename this module to test_graph_query_vertex_knowledge.py (the
behavior it covers) instead of keeping a duplicate name.

Co-Authored-By: Claude Code <noreply@anthropic.com>

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 10/10. Summary: The rebase leaves the production hunks unchanged, so gremlin_literal still escapes string ids and keywords, keeps numeric ids numeric, and _process_vertex formats numeric ids; the full hugegraph-llm suite passes and the new call-site tests fail against main's sources.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 8.1/10 · Confidence: medium

Blocking: no

Summary

No new actionable finding in this revision. The earlier numeric-ID and cached-path issues are addressed: integer IDs remain numeric, and cached IDs are converted to strings before concatenation.

Evidence

Checked the literal helper and its GraphQueryNode, SemanticIdQuery and get_vertex_details call sites. Isolated execution of the exact-head helper passed eight cases covering quotes, backslashes, control characters and numeric versus string IDs. Current LLM, quality, CodeQL and CI Result checks are successful.

Validation notes

The helper check uses Python literal round trips, not a real Groovy parser or HugeGraph query. No live server/provider test was independently run. A current-head Pull Request Labeler run also has startup_failure; the successful core checks above do not mean every workflow passed.

lplalbert pushed a commit to lplalbert/hugegraph-ai that referenced this pull request Nov 15, 2026
pytest's default prepend import mode collides when two test modules
share a basename in directories without __init__.py. The in-flight
Gremlin-escaping work (apache#383/apache#385) adds its own test_graph_query_node.py,
so rename this module to test_graph_query_vertex_knowledge.py (the
behavior it covers) instead of keeping a duplicate name.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants