Add calculated predicate search plumbing - #6721
Conversation
Expose calculated predicates through the search protocol and evaluate them safely against warmed Tantivy fast fields.
| SegmentComponent::Delete, | ||
| ]; | ||
| let mut files = HashSet::new(); | ||
| for segment_meta in index_meta.segments { |
There was a problem hiding this comment.
I don't understadn what this is about?
You were hit by the new tantivy plugin change that the list_files method does not exist anymore? I think there is another utility to do that computation no?
There was a problem hiding this comment.
It seems like we DO need to modify tantivy.
| SegmentComponent::Postings, | ||
| SegmentComponent::Positions, | ||
| SegmentComponent::Terms, | ||
| SegmentComponent::Store, |
| BYTES = 7; | ||
| IP_ADDR = 8; | ||
| JSON = 9; | ||
| CUSTOM = 10; |
There was a problem hiding this comment.
do we need this in protobuf land? Shouldn't we just return an error or panic before?
| int32 priority = 21; | ||
|
|
||
| // Predicate expression evaluated by Tantivy against fast fields on each leaf. | ||
| optional CalculatedPredicate calculated_predicate = 22; |
There was a problem hiding this comment.
I don't understand how this can make sense with the existing query_ast.
| checksum = "1b5d307320b3181d6d7954e663bd7c774a838b8220fe0593c86d9fb09f498b4b" | ||
| dependencies = [ | ||
| "gimli", | ||
| "gimli 0.32.3", |
There was a problem hiding this comment.
can you go through the added dependencies and double check if they are justfiied or not pulled at all?
| [[package]] | ||
| name = "cranelift-assembler-x64-meta" | ||
| version = "0.134.4" | ||
| source = "registry+https://github.com/rust-lang/crates.io-index" |
There was a problem hiding this comment.
that one is likely unnecessary for instance. This is not a change to do in quicwkit, but in tantivy but still very important to follow up.
| ] | ||
|
|
||
| [[package]] | ||
| name = "wasmtime-internal-core" |
There was a problem hiding this comment.
im surprised we need that. do we need to disable things in cranelift? (again, in tantivy)
| } | ||
| } | ||
|
|
||
| message CalculatedPredicateFuncCall { |
There was a problem hiding this comment.
The query ast is expressed using json today. I don't think it we need to change this?
2 participants
Summary
Test plan
cargo test -p quickwit-search calculated_predicate --libcargo test -p quickwit-search test_metadata_count_request_with_calculated_predicate --libcargo +nightly fmt --all