[pcbeingused333]

Follow-up to #831: same helper, same class of bug (tokens compared after escaping).

Motivation

FullTextQueryHelper._tokenize_and_escape_query escapes each token and only then compares it with the stopword set and the text_weights keys. Escaping rewrites punctuation, so don't becomes don\'t, which is not in the English stopword list (don't is), and state-of-the-art becomes state\-of\-the\-art, which no longer matches a weight set for state-of-the-art. Result: TextQuery("don't panic", ...) keeps don\'t in the query, and weights on hyphenated or apostrophe words are silently ignored. TextQuery and HybridQuery share the helper.

Changes

• The token is normalised (strip, lowercase) and checked against stopwords and weights in its raw form; it is escaped afterwards. The weight suffix is attached to the escaped token.
• tests/unit/test_query_types.py: "don't panic" now produces @body:(panic), and a weight on state-of-the-art is applied to state\-of\-the\-art.

Release Notes

Stopwords containing punctuation (e.g. don't) are now removed from text queries, and weights on words with punctuation are now applied.

Reviewer Brief

Start here: redisvl/utils/full_text_query_helper.py, _tokenize_and_escape_query.

Where it could fail: output for tokens without punctuation is unchanged; tokens with punctuation that were previously (wrongly) kept now get dropped if they are stopwords, which changes the query string for those inputs.

Not verified: a live Redis search; the unit tests ran with --confcutdir=tests/unit (no Docker Redis).

Reviewed: commit 1f24503, pre-PR review by Claude (Opus 5.5); change written with Claude (Sonnet 5.5).

Beyond CI: with the helper restored to main the 2 new tests fail; with the fix all of test_query_types.py and test_text_query_weights.py pass (79 passed).

Prepared with the help of an AI coding assistant (Claude).

---

Note

Low Risk
Localized change to query-string construction with targeted unit tests; only affects tokens containing punctuation that were previously mishandled.

Overview
Fixes full-text tokenization so stopword filtering and text weights are evaluated on normalized raw tokens, with Redis escaping applied only afterward. Previously, escaping ran first (e.g. don't → don\'t, hyphenated phrases backslash-escaped), so punctuation-bearing stopwords were not removed and weighted keys on those tokens did not match.

FullTextQueryHelper._tokenize_and_escape_query now strips/lowercases tokens, drops stopwords and attaches weights using the unescaped form, then escapes each surviving token before building the OR query string. Behavior for plain tokens is unchanged; queries that include apostrophe or hyphen stopwords/weights now produce the intended Redis syntax.

Unit tests cover "don't panic" dropping the stopword and weighted state-of-the-art applying on the escaped token.

^{Reviewed by Cursor Bugbot for commit 1f24503. Bugbot is set up for automated code reviews on this repo. Configure here.}