Skip to content

Add calculated predicate search plumbing #6721

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Mallets wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add calculated predicate search plumbing #6721

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
314 changes: 296 additions & 18 deletions quickwit/Cargo.lock
View file
Open in desktop

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion quickwit/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,8 @@ quickwit-storage = { path = "quickwit-storage" }
quickwit-telemetry-exporters = { path = "quickwit-telemetry-exporters" }
quickwit-transport = { path = "quickwit-transport" }

tantivy = { git = "https://github.com/quickwit-oss/tantivy/", rev = "86641f7", default-features = false, features = [
tantivy = { git = "https://github.com/quickwit-oss/tantivy/", rev = "c661f6a", default-features = false, features = [
"jitexpr",
"lz4-compression",
"mmap",
"quickwit",
Expand Down
24 changes: 19 additions & 5 deletions quickwit/quickwit-directories/src/hot_directory.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ use serde::{Deserialize, Serialize};
use tantivy::directory::error::OpenReadError;
use tantivy::directory::{FileHandle, FileSlice, OwnedBytes};
use tantivy::error::DataCorruption;
use tantivy::index::SegmentComponent;
use tantivy::{Directory, HasLen, Index, IndexReader, ReloadPolicy, TantivyError};

use crate::{CachingDirectory, DebugProxyDirectory};
Expand Down Expand Up @@ -459,11 +460,24 @@ impl Directory for HotDirectory {

fn list_index_files(index: &Index) -> tantivy::Result<HashSet<PathBuf>> {
let index_meta = index.load_metas()?;
let mut files: HashSet<PathBuf> = index_meta
.segments
.into_iter()
.flat_map(|segment_meta| segment_meta.list_files())
.collect();
let segment_components = [
SegmentComponent::Postings,
SegmentComponent::Positions,
SegmentComponent::Terms,
SegmentComponent::Store,
SegmentComponent::FastFields,
SegmentComponent::FieldNorms,
SegmentComponent::Delete,
];
let mut files = HashSet::new();
for segment_meta in index_meta.segments {

@fulmicoton fulmicoton Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understadn what this is about?

You were hit by the new tantivy plugin change that the list_files method does not exist anymore? I think there is another utility to do that computation no?

@fulmicoton fulmicoton Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like we DO need to modify tantivy.

for segment_component in &segment_components {
let path = segment_meta.relative_path(segment_component.clone());
if index.directory().exists(&path)? {
files.insert(path);
}
}
}
files.insert(Path::new("meta.json").to_path_buf());
files.insert(Path::new(".managed.json").to_path_buf());
Ok(files)
Expand Down
1 change: 1 addition & 0 deletions quickwit/quickwit-doc-mapper/src/doc_mapper/field_mapping_entry.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -783,6 +783,7 @@ fn deserialize_mapping_type(
let json_options: QuickwitJsonOptions = serde_json::from_value(json)?;
Ok(FieldMappingType::Json(json_options, cardinality))
}
Type::Custom => bail!("custom fields are not supported in quickwit yet."),
}
}

Expand Down
1 change: 1 addition & 0 deletions quickwit/quickwit-doc-mapper/src/doc_mapper/field_mapping_type.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ fn primitive_type_to_str(primitive_type: &Type) -> &'static str {
Type::Facet => {
unimplemented!("Facets are not supported by quickwit at the moment.")
}
Type::Custom => "custom",
}
}

Expand Down
1 change: 1 addition & 0 deletions quickwit/quickwit-doc-mapper/src/doc_mapper/tantivy_val_to_json.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,7 @@ pub fn tantivy_value_to_json(value: TantivyValue) -> JsonValue {
.expect("Invalid datetime is not allowed."),
TantivyValue::Facet(facet) => JsonValue::String(facet.to_string()),
TantivyValue::Bytes(bytes) => BinaryFormat::Base64.format_to_json(&bytes),
TantivyValue::Custom(bytes) => BinaryFormat::Base64.format_to_json(&bytes),
TantivyValue::IpAddr(ip_v6) => {
let ip_str = if let Some(ip_v4) = ip_v6.to_ipv4_mapped() {
ip_v4.to_string()
Expand Down
18 changes: 14 additions & 4 deletions quickwit/quickwit-indexing/src/actors/packager.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ use quickwit_directories::write_hotcache;
use quickwit_doc_mapper::NamedField;
use quickwit_doc_mapper::tag_pruning::append_to_tag_set;
use quickwit_proto::search::{ListFieldsEntry, ListFieldsMetadata, ListFieldsType};
use tantivy::index::FieldMetadata;
use tantivy::index::{FieldMetadata, SegmentComponent};
use tantivy::schema::{FieldType, Type};
use tantivy::{InvertedIndexReader, ReloadPolicy, SegmentMeta};
use tokio::runtime::Handle;
Expand Down Expand Up @@ -187,15 +187,24 @@ fn list_split_files(
scratch_directory: &TempDirectory,
) -> io::Result<Vec<PathBuf>> {
let mut split_files = vec![scratch_directory.path().join("meta.json")];
let segment_components = [
SegmentComponent::Postings,
SegmentComponent::Positions,
SegmentComponent::Terms,
SegmentComponent::Store,

@fulmicoton fulmicoton Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above.

SegmentComponent::FastFields,
SegmentComponent::FieldNorms,
SegmentComponent::Delete,
];

// list the segment files
for segment_meta in segment_metas {
for relative_path in segment_meta.list_files() {
for segment_component in &segment_components {
let relative_path = segment_meta.relative_path(segment_component.clone());
let filepath = scratch_directory.path().join(relative_path);
if filepath.try_exists()? {
// If the file is missing, this is fine.
// segment_meta.list_files() may actually returns files that
// may not exist.
// Segment metadata may reference optional files that do not exist.
split_files.push(filepath);
}
}
Expand Down Expand Up @@ -361,6 +370,7 @@ fn tantivy_type_to_list_field_type(typ: Type) -> ListFieldsType {
Type::Json => ListFieldsType::Json,
Type::Str => ListFieldsType::Str,
Type::U64 => ListFieldsType::U64,
Type::Custom => ListFieldsType::Custom,
}
}

Expand Down
4 changes: 4 additions & 0 deletions quickwit/quickwit-proto/build.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,10 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
)
.type_attribute("PartialHit.sort_value", "#[derive(Copy)]")
.type_attribute("SortByValue", "#[derive(Ord, PartialOrd)]")
.type_attribute("CalculatedPredicate", "#[derive(Hash, Eq)]")
.type_attribute("CalculatedPredicateExpr", "#[derive(Hash, Eq)]")
.type_attribute("CalculatedPredicateExpr.node", "#[derive(Hash, Eq)]")
.type_attribute("CalculatedPredicateFuncCall", "#[derive(Hash, Eq)]")
.type_attribute("SearchRequest", "#[derive(Hash, Eq)]")
.type_attribute("PartialHit", "#[derive(Hash, Eq)]")
.out_dir("src/codegen/quickwit")
Expand Down
78 changes: 78 additions & 0 deletions quickwit/quickwit-proto/protos/quickwit/search.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,10 +200,85 @@ enum ListFieldsType {
BYTES = 7;
IP_ADDR = 8;
JSON = 9;
CUSTOM = 10;

@fulmicoton fulmicoton Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this in protobuf land? Shouldn't we just return an error or panic before?

}

// -- Search -------------------

// A predicate expression evaluated against fast-field values on each leaf
// segment. The expression is lowered to tantivy::query::CalculatedPredicateQuery
// by the search leaf.
message CalculatedPredicate {
CalculatedPredicateExpr expr = 1;
}

message CalculatedPredicateExpr {
oneof node {
CalculatedPredicateLiteral literal = 1;
// Fast-field name read by the calculated predicate query.
string variable = 2;
CalculatedPredicateFuncCall func_call = 3;
}
}

message CalculatedPredicateLiteral {
oneof value {
int64 int_value = 1;
uint64 uint_value = 2;
// IEEE-754 bits for an f64 literal. This keeps SearchRequest Eq/Hash-safe.
fixed64 double_value_bits = 3;
string string_value = 4;
bool bool_value = 5;
}
}

message CalculatedPredicateFuncCall {

@fulmicoton fulmicoton Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The query ast is expressed using json today. I don't think it we need to change this?

enum Function {
FUNCTION_UNSPECIFIED = 0;
FUNCTION_ABS = 1;
FUNCTION_AND = 2;
FUNCTION_CEIL = 3;
FUNCTION_CONCAT = 4;
FUNCTION_ADD = 5;
FUNCTION_DIVIDE = 6;
FUNCTION_EQ = 7;
FUNCTION_FLOOR = 8;
FUNCTION_GT = 9;
FUNCTION_GT_EQ = 10;
FUNCTION_IF = 11;
FUNCTION_INT_MOD = 12;
FUNCTION_LEFT = 13;
FUNCTION_LT = 14;
FUNCTION_LT_EQ = 15;
FUNCTION_IS_NOT_NULL = 16;
FUNCTION_IS_NULL = 17;
FUNCTION_LOWER = 18;
FUNCTION_MAX = 19;
FUNCTION_MIN = 20;
FUNCTION_MULTIPLY = 21;
FUNCTION_NEQ = 22;
FUNCTION_NOT = 23;
FUNCTION_OR = 24;
FUNCTION_POW = 25;
FUNCTION_SQRT = 26;
FUNCTION_REGEXP_EXTRACT = 27;
FUNCTION_REGEXP_LIKE = 28;
FUNCTION_RIGHT = 29;
FUNCTION_ROUND = 30;
FUNCTION_SPLIT_AFTER = 31;
FUNCTION_SPLIT_BEFORE = 32;
FUNCTION_SUBTRACT = 33;
FUNCTION_SUBSTRING = 34;
FUNCTION_SUBSTRING_COUNT = 35;
FUNCTION_TEXT_JOIN = 36;
FUNCTION_TRIM = 37;
FUNCTION_UPPER = 38;
}

Function function = 1;
repeated CalculatedPredicateExpr args = 2;
}

message SearchRequest {
// Index ID patterns
repeated string index_id_patterns = 1;
Expand Down Expand Up @@ -280,6 +355,9 @@ message SearchRequest {
// Scheduling priority for leaf search execution. Negative values are allowed,
// and lower values have higher priority. Callers that omit it get priority 0.
int32 priority = 21;

// Predicate expression evaluated by Tantivy against fast fields on each leaf.
optional CalculatedPredicate calculated_predicate = 22;

@fulmicoton fulmicoton Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand how this can make sense with the existing query_ast.

}

enum CountHits {
Expand Down
Loading
Loading