Add pco-inspired lightweight numeric encodings - #9544
Conversation
Signed-off-by: "Will Manning" <will@willmanning.io>
Signed-off-by: "Will Manning" <will@willmanning.io>
Signed-off-by: "Will Manning" <will@willmanning.io>
Signed-off-by: "Will Manning" <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
Merging this PR will regress 1 benchmark
|
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -3.8% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.016x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (0.982x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
File Size Changes (17 files changed, -45.4% overall, 2↑ 15↓)
Totals:
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.975x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 2↑ 2↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
File Size Changes (3 files changed, -46.5% overall, 0↑ 3↓)
Totals:
|
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.007x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 1↑ 2↓)
duckdb / parquet / ns (1.014x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -44.6% overall, 0↑ 201↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 3↑ 6↓)
duckdb / parquet / ns (0.999x ➖, 3↑ 5↓)
File Size Changes (49 files changed, -45.6% overall, 11↑ 38↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.981x ➖, 1↑ 0↓)
File Size Changes (17 files changed, -45.4% overall, 2↑ 15↓)
Totals:
|
|
please don't add them to the core edition yet |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.987x ➖, 6↑ 3↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
File Size Changes (3 files changed, -49.5% overall, 0↑ 3↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.024x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.048x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.144x ➖, 2↑ 2↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 3↓)
duckdb / parquet / ns (1.002x ➖, 2↑ 2↓)
File Size Changes (201 files changed, -41.6% overall, 0↑ 201↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.945x ➖, 3↑ 0↓)
datafusion / parquet / ns (1.034x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.061x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.018x ➖, 0↑ 0↓)
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
random-access / parquet / ns (1.003x ➖, 0↑ 0↓)
random-access / lance / ns (0.993x ➖, 0↑ 0↓)
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.942x ➖, 5↑ 0↓)
vortex / vortex-file-compressed / bytes (0.954x ➖, 4↑ 0↓)
vortex / vortex-file-compressed / ratio (0.946x ➖, 9↑ 0↓)
vortex / parquet / ns (0.999x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
|
Signed-off-by: Will Manning <will@willmanning.io>
Summary
Add native
BlockResidual,OrderedFloat, andFloatQuantarrays for numeric data that Pco compresses well but ALP and ALP-RD do not. These arrays retain fast scalar access and use composable Vortex children. The selectors use cheap sample analysis and calibrated access-cost factors. The new arrays belong to the draftcore:2026.08.4edition.Changes
Results
The final benchmark compared 17 datasets with five timed iterations per operation. The corpus included GloVe
f32and OpenAI-on-C4f64embeddings.