Repository navigation
[C++] Apply field predicates to aligned table query rows - #1026
jixuan1989 wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The scalar fallback can lose matching NULL rows and return nullable Boolean rows outside the time filter.
3 open issues
What changed in this PR
Addresses #1025 by evaluating FIELD predicates in C++ aligned table queries while preserving the unfiltered fast path.
Changes:
- Adds typed, NULL-aware row predicates and legacy callback dispatch.
- Evaluates predicates before OFFSET/LIMIT and advances rejected rows.
- Adds tests for types, NULLs, pagination, and callback errors.
| File | Description |
|---|---|
cpp/test/reader/table_field_filter_test.cc |
Adds predicate and query regression tests. |
cpp/src/reader/filter/table_field_filter.h |
Defines borrowed typed row views and predicate API. |
cpp/src/reader/filter/table_field_filter.cc |
Implements typed and legacy predicate dispatch. |
cpp/src/reader/block/single_device_tsblock_reader.h |
Declares FIELD evaluation helper. |
cpp/src/reader/block/single_device_tsblock_reader.cc |
Integrates filtering into scalar row processing. |
Review effort: Balanced
| } | ||
| } | ||
| if (can_multi) { | ||
| if (can_multi && field_filter == nullptr) { |
There was a problem hiding this comment.
Confirmed. The aligned value page's time statistics exclude NULL entries, so they cannot rule out candidate timestamps in the shared time page. I reproduced this with (t0,42),(t1,NULL),(t2,NULL) and a time window [t1,t2]: the prior PR returned 0 rows, while Java and the corrected C++ return 4 across two devices, on either writer's file. Commit 249b0d5 uses only shared time-page statistics for this conservative page check. Missing time statistics keep the page. The new test runs with row/batch-1/batch-4 and parallel on/off.
The revision also fixes the test-only TAG predicate leak behind the two Linux ASan CI failures: the query borrows this predicate, so it now has stack ownership. The old test reproduced 480 leaked bytes in six allocations; the revised Linux ASan/UBSan full suite passes with leak detection enabled.
Validation for 249b0d5: all 24 added regression configurations fail on the old PR head and pass after the fix; 74 focused, 979 Windows Release, 1000 all-codec Linux, and 999 Linux ASan/UBSan full-suite tests pass. Twenty added NULL-boundary Java/C++ own/cross-reader results match. Whole-PR changed executable lines: 87/89 (97.75%); review-only: 12/12 (100%). Query evidence retains the original eight cases and adds 30 native FIELD selectivity/projection cases with verified full results, open/metadata/first-row/full timing and process peak memory; comparison boundaries and unmeasured scope are documented.
Revision report · Source, raw test and performance evidence
AI disclosure: This analysis, implementation, validation and reply were produced by OpenAI Codex.
AI 使用说明：本分析、修改、验证和回复由 OpenAI Codex 完成。
| } | ||
| } | ||
| if (can_multi) { | ||
| if (can_multi && field_filter == nullptr) { |
There was a problem hiding this comment.
Confirmed. The BOOLEAN NULL scalar branch skipped the timestamp predicate and could emit an out-of-window NULL before FIELD evaluation and OFFSET/LIMIT. Commit 249b0d5 checks that timestamp, backs off the provisional row, and continues without consuming a value for a NULL. The two-device reproduction previously returned 6 rows for the window; revised C++ and Java return the same 4, with both Java-written and C++-written inputs. The test also verifies callback timestamps and OFFSET/LIMIT in all six row/batch/parallel configurations.
Validation for 249b0d5: all 24 added regression configurations fail on the old PR head and pass after the fix; 74 focused, 979 Windows Release, 1000 all-codec Linux, and 999 Linux ASan/UBSan full-suite tests pass. Twenty added NULL-boundary Java/C++ own/cross-reader results match. Whole-PR changed executable lines: 87/89 (97.75%); review-only: 12/12 (100%). Query evidence retains the original eight cases and adds 30 native FIELD selectivity/projection cases with verified full results, open/metadata/first-row/full timing and process peak memory; comparison boundaries and unmeasured scope are documented.
Revision report · Source, raw test and performance evidence
AI disclosure: This analysis, implementation, validation and reply were produced by OpenAI Codex.
AI 使用说明：本分析、修改、验证和回复由 OpenAI Codex 完成。
| } | ||
| } | ||
| if (can_multi) { | ||
| if (can_multi && field_filter == nullptr) { |
There was a problem hiding this comment.
Confirmed. A sole all-NULL aligned FIELD has a zero-count value index and no value context, but still has shared timestamps that must reach a NULL-aware predicate. Commit 249b0d5 uses the existing time-only context specifically for that aligned, zero-value-count, non-null-FIELD-filter case. The empty-result shortcut remains for non-aligned or other time-filtered-out value queries. Before the revision the reproduction returned 0 rows; revised C++ and Java return 28, or the same 2 after a time window and OFFSET/LIMIT, on both writers' files. Always-false, legacy-NULL rejection, and wholly outside-window controls are covered too.
Validation for 249b0d5: all 24 added regression configurations fail on the old PR head and pass after the fix; 74 focused, 979 Windows Release, 1000 all-codec Linux, and 999 Linux ASan/UBSan full-suite tests pass. Twenty added NULL-boundary Java/C++ own/cross-reader results match. Whole-PR changed executable lines: 87/89 (97.75%); review-only: 12/12 (100%). Query evidence retains the original eight cases and adds 30 native FIELD selectivity/projection cases with verified full results, open/metadata/first-row/full timing and process peak memory; comparison boundaries and unmeasured scope are documented.
Revision report · Source, raw test and performance evidence
AI disclosure: This analysis, implementation, validation and reply were produced by OpenAI Codex.
AI 使用说明：本分析、修改、验证和回复由 OpenAI Codex 完成。
A FIELD predicate must see rows whose timestamps lie in the selected time range even when the column's non-NULL values all lie outside it. Replace the all-values-NULL check in the empty-context shortcut with an aligned-source check so such devices fall through to the time-only context instead of returning an empty result.
decode_tv_batch<int32_t> replaced this loop in apache#823, leaving the function uncalled. Its NULL branch also predates the time-filter fix applied to the other decode paths in this series.
|
Thanks for your pr 🌹 . Follow-up review fixes on top of the two commits above: 1. Empty-context shortcut missed all-NULL-in-range rows ( The committed version keyed the empty-context shortcut on The fix relaxes the check to New test 2. Removed dead per-value INT32 decode (
Verification: field-filter tests 80/80, targeted reader regressions 160/160, full C++ suite 1229 passed / 0 failed. |
|
Confirmed both changes in your follow-up. A whole-column positive value count does not establish any non-NULL value in the selected range, so the aligned shared-time fallback is needed here. The new six-configuration regression fails on the old library and passes at Local checks: 80 permanent focused cases, 24 extra boundary cases, and three actual non-aligned gap cases pass. Windows Release (codecs off, same retained baseline configuration): 985 passed, 3 skipped, 10 disabled. All-codec Linux combined ASan/UBSan focused/extra checks pass; reader scope is 343 passed, 2 external-fixture skips, 9 disabled. Forty-eight final-C++/Java own/cross reads match times, order and typed values. Maintainer-delta coverage is 4/4 executable lines (100%) and 5/6 branches (83.33%); affected reader files overall are 79.81% / 55.92%. Query acceptance remains NOT cleared. The frozen protocol retained 68 initial cases and independently confirmed seven initial flags, with 20,928 fully consumed/checked formal queries. Three remain higher: The broader sanitizer runs are not clean. An unchanged TS2DIFF signed overflow aborts the Java-fixture read; byte-identical old/new headers reproduce it on the same valid raw page. A separately filtered remainder then hits an unaligned read in the unchanged pinned LZOkay dependency. Failed logs are preserved, with no suppression. Reader-scope passes above must not be read as a full sanitizer pass. Writing was not retimed for this reader-only follow-up. 中文说明： 两处修改的功能判断已由上述测试确认，但查询门槛仍未通过：三个复核项分别上升约 0.83%、0.90%、1.43%，尚未证明代价必要。原生 NULL 路径/旧正确应用层参考最高 1.7273 倍，两者工作不同，不能据此单独归因于引擎。完整 sanitizer 检查也因未改动的 TS2DIFF/LZOkay 问题中止，原始失败日志保留，不能以 reader 专项通过替代全量通过。中英文警告和提交范围已置于 PR 正文顶部。 Bilingual report · Sources, all raw results and hash-verified evidence parts. Earlier measurements remain under their original head/scope. The independent agent found no unresolved actionable source issue; the complete different-model final review gate is not passed. No production code was authored/pushed by Codex in this follow-up. AI disclosure: This analysis, local testing, documentation and reply were produced by OpenAI Codex. The two reviewed production commits were authored by maintainer ColinLee/ColinLeeo. AI 使用说明： 本分析、本地测试、说明和回复由 OpenAI Codex 完成；被验证的两次生产代码提交由维护者 ColinLee/ColinLeeo 编写。 |
|
After validation, the sanitizer issues are confirmed. The current sanitizer checks require stricter tracking. The related failures originate from TS2DIFF and the third-party LZOkay dependency, and will be tracked in separate issues and PRs. |
|
The current implementation is correct, but I do not think the row-oriented path is the right final execution model for aligned data. For aligned series, all columns already share the same ordered time axis. The existing time-filter path also provides the necessary scan-pruning foundation: device/chunk/page statistics are used to determine what actually needs to be decoded. We should extend the same idea to FIELD filtering:
This avoids per-row timestamp merging and This batch-oriented design is also intended to address the performance regressions found in the current validation: it preserves aligned batch decoding and avoids the overhead of the per-row merge and filter path. I plan to implement this as a follow-up rather than continuing to optimize the current per-row loop. Keeping this PR focused on correctness also makes the change easier to review. |
|
Agreed to keep this PR focused on correctness and handle aligned batch execution in the follow-up you described. At One API boundary to preserve: The existing frozen regression evidence is useful for the follow-up: shared timestamps must retain all-NULL selected ranges; value-only statistics must not discard those rows; NULL/binary/primitive values must retain their current semantics and lifetime; OFFSET/LIMIT count only selected rows across batches/devices; callback/decode errors remain terminal. I rechecked the Windows and Linux XML records for the six configurations of the NULL-range, all-NULL, typed-value, post-filter OFFSET/LIMIT and callback-error tests. These are existing passes on the unchanged head, not new execution or validation of the proposed batch implementation. I also recorded your confirmation of the sanitizer failures as separate follow-up work. The published baseline reproductions and failed logs remain intact, and the full-sanitizer status stays not clean. Neither follow-up has a linked implementation in these comments yet. Performance status remains unchanged: the published warm-cache MSVC comparisons retain +0.826%, +0.902%, and +1.433% confirmed median latency signals. Their necessity is unproven, so query acceptance is not cleared. Future batch measurements should compare fully checked equivalent queries with the current correct native path, and report the application-reference projection/materialization boundary separately. No new timing, source revision, or acceptance pass is claimed here. 中文说明： 同意保持当前 PR 的正确性范围，将对齐数据批量解码、掩码合并和选择后物化作为后续实现。源码确实显示 FIELD 过滤绕过共享多列/批量快路径，但这不能直接解释每个小幅旧新延迟差异，也不能预先证明后续提速。当前任意行回调没有引用列清单或表达式树，后续需要明确接口或兼容回退，并保留 NULL 共享时间轴、类型/字节与借用期、过滤后 OFFSET/LIMIT、错误终止等行为。已核对既有冻结测试 XML，没有重新执行测试，也没有验证尚未提交的批量实现。sanitizer 两项按维护者意见分别跟踪，完整检查仍未通过。暖缓存 MSVC 的 +0.826%、+0.902%、+1.433% 查询信号和必要性未证明的结论保留，性能门槛仍未通过；没有新增计时或生产代码修改。 Frozen bilingual report and limitations AI disclosure: This source/test-evidence assessment, tracking and reply were prepared by OpenAI Codex. The proposed follow-up implementation is the maintainer's plan and has not been validated here. AI 使用说明： 本源码与既有测试证据核对、跟踪及回复由 OpenAI Codex 完成；后续实现为维护者提出的计划，本回复未对其作已实现或已通过验证的声明。 |
Add AlignedFieldFilter (a columnar predicate tree) and SelectionMask (a row bitmask), and thread them through the aligned chunk/block readers. Single-threaded flow: page statistics (may_match/all_match) prune pages that cannot match or mark pages that fully match; only filter-referenced columns are decoded first; output columns are decoded lazily and only for pages with at least one matching row; the predicate is evaluated once per page into a bitmask, and the scatter loop only tests bits. NOT is pushed down to leaf comparisons so NULL rows keep SQL three-valued semantics, get_data_type_size is no longer called on string columns, and query() error paths free the field filter.
|
This PR now also includes a columnar/batch FIELD-predicate path for aligned table queries (commit
|
Updated: committed mask bulk copy and C++/Java selectivity benchmarkMeasured on October 22, 2026, after committing and pushing Exact source and implementation
Data and queries
Measurement and validation30 scenarios × 2 implementations × 5 fresh processes × 5 timed queries = 1,500 new timed samples. None of the previous working-tree or paired-experiment timings are merged into this matrix. Each process first executes one discarded exhaustive timestamp/order/row-count/projected-value validation query and two further discarded warm-up queries. Every timed query reopens its reader and verifies exact row count plus timestamp and all projected-value checksums. All scenarios pass; checksums match across readers and writer files. Timing includes file open, metadata/query construction, scan/filter, consumption of selected timestamps and every projected value, and result/reader close. JVM startup, file generation, exhaustive validation and warm-up are excluded. Both use 1,024-row output blocks. The statistic is the median of five process medians, each over five timed queries. Ratio = Java reference median / C++ median; >1 means C++ is faster. Cases are deterministically shuffled and implementation order alternated; measured processes do not overlap. This is a warm-OS-cache microbenchmark, not a cold-cache or tail-latency result. C++-written TsFile
Java-written TsFile
For non-empty results, C++ was 2.66×–6.43× faster than this Java reference in this matrix. For 8 projected FIELDs / 100% accepted, C++ vs Java was 60.109 vs 242.057 ms (4.03×) on the C++-written file, and 61.416 vs 226.072 ms (3.68×) on the Java-written file. The larger 0%-match ratios reflect C++ page-statistics pruning versus the Java reference scanning data before rejecting rows. They are not isolated decoding-speed ratios. This is dense INT64 data and this predicate/projection matrix on one host, not a universal performance guarantee or a before/after regression-acceptance result. Separate paired bulk-copy experimentA separate earlier paired experiment used the same checked benchmark and C++-written file to compare the pre-bulk correctness-fixed library and this exact final library. Three fresh process pairs per case, five timed queries after exhaustive validation and two warm-ups; order was balanced. It found 8 FIELDs / 100% accepted: 142.848 → 56.521 ms (2.53×) and 50% accepted: 122.115 → 77.474 ms (1.58×). This supporting experiment has 465 samples including its Java reference check and is retained separately in Correctness and formattingThe final implementation adds 48 tests relative to Reproducible evidence and update historyFull report, exact protocol, source drivers and reproduction instructions · 1,500 new raw timing samples · All 300 measurement-process outputs · Validation output. This edits the existing performance comment to reflect the pushed implementation, rather than appending new results to stale tables. The original pre-bulk working-tree evidence remains unchanged for traceability; no old timing samples are merged into the new matrix. The supporting paired bulk-copy experiment is separately retained in the new evidence. AI disclosure: Implementation, tests, benchmark drivers, execution, result verification, analysis and this comment were prepared by OpenAI Codex. |
Copy selected bitmap runs with dense fixed-width bulk appends while preserving NULL/string fallbacks, duplicate projections, post-filter row ranges, and source-block cursor ownership. Keep page-plan decode mode stable across TsBlocks, retain NaN rows when floating statistics cannot prove a match, and evaluate missing aligned columns as NULL. Add focused and integration regression coverage.
This candidate is local and unsubmitted. Public PR head remains 候选仅在本地，尚未提交，公开PR仍为 The new frozen96-case four-version round completed13,584 children and43,776 formal plus3,136 warmup queries, with18 predeclared one-shot confirmations. Fifteen comparisons against6f retain increases: floating callbacks up to+2.94%, narrow integer callbacks up to+1.86%, sparse noFIELD batch+0.62%. The largest pointwise95 ratio interval is1.02149–1.03652. No repeated publicfcab/prior11b increase was found in this bounded round; this is not equivalence. Finite zero-hit floating queries are0.3363–0.3485×publicfcab in initial blocks and0.9881–1.0089×prior11b; no old-round ratios were divided or pooled. Necessary extra cost remains unproved; gates remainfalse. Both actual independent reviewers ( 新冻结的96场景四版本计时完成13,584进程，正式43,776次及预热3,136次查询，18项各一次预定确认。相对6f的15项比较保留上升：浮点同回调最高+2.94%、窄投影整数回调最高+1.86%、无FIELD稀疏批查询+0.62%；最大项逐项95比率区间1.02149–1.03652。本次有界轮次未确认相对公开fcab/上一版11b的重复上升，不表示等效。有限值零命中浮点初测延迟为公开fcab的0.3363–0.3485倍、11b的0.9881–1.0089倍，没有跨旧轮求比率或池化。尚未证明额外成本必要，门槛保持false。独立agent和另一实际模型（ Changed-line/branch coverage for fcab→local2525 is97.92%/71.90%; wholePR1180→local2525 remains85.64%/62.57%; selected16 affected files80.81%/55.65%. Coverage accumulates3ce/2525 test runs on260 byte-identical production inputs; it is not a cleared isolated final-test profile. Final test pass counts use fresh final XML. Warm cache, PLAIN/default timeTS_2DIFF/UNCOMPRESSED generated data; original8 retain old configuration and lack phases, per-query quantiles and memory. Cold cache/P99/compressed/general nonaligned FIELD/parallel scaling/query-only memory remain unmeasured. Reader-only writes were not retimed; earlier PR1018/1024 gaps remain. Old45d and all prior samples stay intact; no extra timing rounds were added. The first spotless missingPATH raw log was accidentally overwritten and is explicitly disclosed in the report. fcab→本地2525的修改行/分支覆盖率97.92%/71.90%；整个PR1180→本地2525仍为85.64%/62.57%，所选16文件80.81%/55.65%。覆盖率累积3ce/2525两个测试提交在260项逐字节相同生产输入上的运行，不是清零后最终测试提交的独立profile；最终测试通过数来自全新最终XML。生成数据为暖缓存、值PLAIN/时间默认TS_2DIFF/UNCOMPRESSED；原8项保持旧配置，仍缺分项、单次分位数和内存。冷缓存/P99/压缩/广泛非对齐FIELD/并行扩展/查询专属内存仍未测，reader-only未重测写入，前两个PR的缺口未补齐。旧45d及全部旧样本保持原样，没有增加计时轮次。首次spotless缺PATH失败的原始日志被意外覆盖，报告已明确披露。 Current bilingual report / 当前中英文报告 · All raw evidence and both final reviews / 全部原始证据及两份最终审阅 Earlier diagnostics / 先前诊断
Current-head CI logs and bilingual diagnosis / 当前提交CI日志及中英文判断 Maintainer ColinLee's new public commit 维护者ColinLee的新公开提交 Remaining limits:6 schema-only/LIMIT0 malformedOR checks return E_OK/empty rather than strict expected E_INVALID_ARG;12valid controls pass. Fresh full Linux sanitizer exits in unchanged pinned LZOkay; no full pass. Current11b→fcab changed-line/branch coverage91.67%/70.27%; wholePR1180→fcab84.26%/61.56%, selected affected files80.54%/55.41%. Conservative negated statistics and preexisting duplicate lazy-mask evaluation are source observations, not proof of complete slowdown causality or necessity. ValuesPLAIN/timeTS_2DIFF/UNCOMPRESSED for generated matrices; original8 keep original configuration. Cold cache/P99/compressed/general nonaligned FIELD/parallel scaling remain unmeasured. Writes were not retimed for this reader-only revision. Historical local45d stays unmerged/unpushed, without transferring its results tofcab. 剩余限制：空表/LIMIT0的6项畸形OR仍返回E_OK/空结果，未按严格期望返回E_INVALID_ARG；12个合法控制通过。新全量Linux sanitizer仍在未修改的固定LZOkay中止，不能称总通过。本轮11b→fcab修改行/分支覆盖91.67%/70.27%；整个PR1180→fcab为84.26%/61.56%，所选受影响文件80.54%/55.41%。保守取反统计及原有lazy-mask重复求值是源码观察，未证明完整退化因果或必要性。生成矩阵的值为PLAIN、时间TS_2DIFF、UNCOMPRESSED；原8项保留原配置。冷缓存/P99/压缩/广泛非对齐FIELD/并行扩展仍未测；仅reader修订没有重测写入吞吐。历史本地45d仍未合入或推送，其结果不转移到fcab。 Current bilingual report / 当前中英文报告 · All raw evidence and both final reviews / 全部原始证据与两份最终审阅 Earlier diagnostics / 先前诊断Following up the batch update: I checked public11b against the earlier6f source and retained the failed f5 reproductions. The large-page continuation crash and NaN EQ/NE pruning are fixed. Three issues remain on11b: ordered floating A local repair passed1,112 Windows tests,207 focused plus369 reader Linux ASan/UBSan/LSan checks, and1,896 Java/C++ complete-file queries with an independently checked full-row oracle. Two independent local-source/evidence review sessions checked the findings;this does not mean the submission gate passed. I have not pushed the repair: its frozen60-case before/after query experiment still confirms regressions (including+2.56% structured-row vs11b and up to+5.05% narrow legacy callback vs6f). Full sanitizer is also not clean because of unchanged pinnedLZOkay UB;local repair changed-line coverage is89.47%. All samples/failures are retained. I am minimizing a concrete unused-mask allocation and filling the directTsBlock regression gap;that allocation is not a demonstrated explanation of the legacy callback slowdown. 已确认大页续读和NaN等于/不等于裁剪修复。11b仍有浮点NOT漏行、畸形OR通过FULL_PASS绕过验证，以及手动destroy未释放新掩码vector的问题；专项LSan报告16152字节/2019次分配。14项新增配置在修复前失败。三种写入文件均复现40个NOT差异：32例丢NaN值行，8例NaN字面量丢全部非NULL行，负控进程均正常退出。 本地修订通过Windows1112项、Linux207专项和369reader的ASan/UBSan/LSan，以及1896次双向Java/C++完整文件查询；独立审阅还补正了验证脚本的完整行数和精确值断言。尚未推送源码：固定60场景查询复核仍有退化，最高示例为相对11b+2.56%、相对6f旧回调+5.05%，必要性未证明；全量仍因原LZOkay UB中止，本地修改行覆盖率89.47%。不会用其它更快场景抵销这些失败，也不会追加轮次找通过。 Report / 报告 · Sources, raw outputs, counters and all timing samples / 复现源码、实际输出、计数和全部样本 CI setup failures / CI 环境阻塞I checked all seven newly failed Ubuntu jobs at11b. Six fail during dependency installation because Examples: C++ Release/ASan setup, Python3.9 setup, Python3.11 formatter setup. The two workflow files are unchanged between6f,11b and the local repair. Related formatter provisioning work exists in #918,but it does not modify these workflow setup calls. I have not changed shared CI workflows or rerun the same unsupported setup. 已逐一核对7项Ubuntu失败：6项在安装依赖时因缺少
This candidate passed 1,112 Windows tests, scoped Linux ASan/UBSan/LSan runs with 207 and 369 passing tests, and 1,896 complete-file Java/C++ queries. Local repair line/branch coverage is 91.14%/72.25%; whole-PR changed-line/branch coverage is 84.81%/62.78%, and coverage of the selected affected production files is 80.65%/55.72%. Full sanitizer testing was not rerun; the earlier failure in unchanged LZOkay code remains. Both independent source/evidence reviewers conclude 该候选已通过 1,112 项 Windows 测试、分别有 207 和 369 项通过的 Linux 专项 ASan/UBSan/LSan 检查，以及 1,896 次完整文件 Java/C++ 查询。本地修补行/分支覆盖率为 91.14%/72.25%；整个 PR 修改行/分支覆盖率为 84.81%/62.78%，所选受影响生产文件的行/分支覆盖率为 80.65%/55.72%。全量 sanitizer 未重跑，先前未修改的 LZOkay 代码失败记录仍保留。两位独立源码及证据审阅者均判定 Latest bilingual report / 最新中英文报告 · Scoped diagnostics and all timing records / 分项诊断及全部计时记录 The earlier AI disclosure: OpenAI Codex wrote and ran the validation harnesses and this reply; public maintainer commits retain their Git authorship. AI 使用说明： 本验证驱动、本回复及本地验证由OpenAI Codex编写和执行；公开维护者提交保留其Git作者归属。 AI disclosure: OpenAI Codex authored this local unsubmitted repair, validation harnesses and reply, and ran the reported validation. Public maintainer commits retain their Git authorship. AI 使用说明： 本地未提交修补、验证驱动及本回复由OpenAI Codex编写，本地验证由该AI代理执行；公开维护者提交保留Git作者归属。 |
Negating an ordered FLOAT/DOUBLE comparison flipped the operator (NOT (x < 5) -> x >= 5). NaN makes every ordered comparison false, so the negation must be true for NaN rows while the flipped form is false and dropped them; when the literal itself was NaN, every non-NULL row was lost. Ordered floating-point leaves now keep the operator and record the negation, and evaluation computes !(value <op> literal) for non-NULL rows. Page statistics no longer prove a full-page match or a skip for such leaves. A composite node missing an operand could be classified as a full-page match through its remaining operand. Page statistics then skipped evaluation, so a malformed OR silently returned rows (and did not terminate in batch mode). all_match() now requires both operands, and build_page_plan() rejects a structurally invalid filter with E_INVALID_ARG before classifying pages. AlignedChunkReader is placement-new'd and torn down through destroy() without running its destructor, so the page-plan filter state added for FIELD filtering leaked its backing buffers. destroy() now releases filter_column_mask_ and the current page mask. Tests cover NOT over all six operators and five literals (including NaN and both infinities) for FLOAT and DOUBLE, with NULL rows from a device that lacks the float columns; and a malformed OR that was previously accepted.
This candidate is local and unsubmitted. Public PR head remains
fcab7437. OpenAI Codex authored local2525a557from the maintainer head: NaN-safe ordered statistics shortcuts, whole-tree checks before empty-table/LIMIT0 returns, and mask lifecycle work. These repairs have not been applied to the public branch. Against frozen fcab, seven regression groups fail; local final Windows focused214/full1119 and Linux ASan/UBSan/LSan focused214/reader376 pass. Both evaluators and24 independent file-query controls pass;1,896 standard Java/C++ file queries plus2,160 floating and2,752 integer preflight queries and ten Java scans pass. Generated four-version C++ writer products are byte-identical. Full Linux sanitizer still fails at unchanged pinned LZOkay; no full pass is claimed.候选仅在本地，尚未提交，公开PR仍为
fcab7437。 OpenAI Codex从维护者提交编写本地2525a557，恢复NaN安全的有序统计快捷路径、在空表/LIMIT0返回前检查整树，并修正掩码生命周期；这些修补尚未应用到公开分支。冻结fcab上7组回归失败，本地最终Windows专项214/全量1119及Linux ASan/UBSan/LSan专项214/reader376通过；两条求值器与24项独立文件查询控制通过，1,896次标准Java/C++查询、2,160次浮点及2,752次整数预检查询、10次Java全读均通过；四版C++写入产物字节一致。Linux全量sanitizer仍在未修改的固定LZOkay失败，不能称总通过。The new frozen96-case four-version round completed13,584 children and43,776 formal plus3,136 warmup queries, with18 predeclared one-shot confirmations. Fifteen comparisons against6f retain increases: floating callbacks up to+2.94%, narrow integer callbacks up to+1.86%, sparse noFIELD batch+0.62%. The largest pointwise95 ratio interval is1.02149–1.03652. No repeated publicfcab/prior11b increase was found in this bounded round; this is not equivalence. Finite zero-hit floating queries are0.3363–0.3485×publicfcab in initial blocks and0.9881–1.0089×prior11b; no old-round ratios were divided or pooled. Necessary extra cost remains unproved; gates remainfalse. Both actual independent reviewers (
gpt-6.1-sol,gpt-6-astra, runtime-verified) completed final source/evidence review and concludeNOT_READY_TO_PUSH. No Copilot review was requested.新冻结的96场景四版本计时完成13,584进程，正式43,776次及预热3,136次查询，18项各一次预定确认。相对6f的15项比较保留上升：浮点同回调最高+2.94%、窄投影整数回调最高+1.86%、无FIELD稀疏批查询+0.62%；最大项逐项95比率区间1.02149–1.03652。本次有界轮次未确认相对公开fcab/上一版11b的重复上升，不表示等效。有限值零命中浮点初测延迟为公开fcab的0.3363–0.3485倍、11b的0.9881–1.0089倍，没有跨旧轮求比率或池化。尚未证明额外成本必要，门槛保持false。独立agent和另一实际模型（
gpt-6.1-sol、gpt-6-astra，runtime身份已核实）完成最终源码及证据审阅，均判定NOT_READY_TO_PUSH；没有请求Copilot审阅。Changed-line/branch coverage for fcab→local2525 is97.92%/71.90%; wholePR1180→local2525 remains85.64%/62.57%; selected16 affected files80.81%/55.65%. Coverage accumulates3ce/2525 test runs on260 byte-identical production inputs; it is not a cleared isolated final-test profile. Final test pass counts use fresh final XML. Warm cache, PLAIN/default timeTS_2DIFF/UNCOMPRESSED generated data; original8 retain old configuration and lack phases, per-query quantiles and memory. Cold cache/P99/compressed/general nonaligned FIELD/parallel scaling/query-only memory remain unmeasured. Reader-only writes were not retimed; earlier PR1018/1024 gaps remain. Old45d and all prior samples stay intact; no extra timing rounds were added. The first spotless missingPATH raw log was accidentally overwritten and is explicitly disclosed in the report.
fcab→本地2525的修改行/分支覆盖率97.92%/71.90%；整个PR1180→本地2525仍为85.64%/62.57%，所选16文件80.81%/55.65%。覆盖率累积3ce/2525两个测试提交在260项逐字节相同生产输入上的运行，不是清零后最终测试提交的独立profile；最终测试通过数来自全新最终XML。生成数据为暖缓存、值PLAIN/时间默认TS_2DIFF/UNCOMPRESSED；原8项保持旧配置，仍缺分项、单次分位数和内存。冷缓存/P99/压缩/广泛非对齐FIELD/并行扩展/查询专属内存仍未测，reader-only未重测写入，前两个PR的缺口未补齐。旧45d及全部旧样本保持原样，没有增加计时轮次。首次spotless缺PATH失败的原始日志被意外覆盖，报告已明确披露。
Current bilingual report / 当前中英文报告 · All raw evidence and both final reviews / 全部原始证据及两份最终审阅
Earlier publicfcab and local45d evidence / 公开fcab及本地45d的先前证据
Current-head CI logs and bilingual diagnosis / 当前提交CI日志及中英文判断
Maintainer ColinLee's new public commit
fcab7437fixes ordered FLOAT/DOUBLE NOT with NaN by complementing only nonNULL comparisons in both evaluators, guards malformed nonempty aligned plans, and releases owned mask storage during manual destroy. Valid-query evidence: Windows1104full/199focused passed; Linux ASan/UBSan/LSan199focused+361reader passed;1896complete-file Java/C++ queries plus864floating C++ queries and6Java full scans passed. Two actual independent reviewers (gpt-6.1-sol,gpt-6-astra, verified runtime identities) completed source/protocol/final-evidence reviews and concludeNOT_READY_TO_PUSH.维护者ColinLee的新公开提交
fcab7437在两条求值路径中仅对非NULL比较结果取反，修复FLOAT/DOUBLE有序NOT与NaN语义，保护非空对齐计划的畸形树检查，并在手动destroy时释放掩码所有权存储。合法查询证据：Windows全量1104/专项199通过；Linux ASan/UBSan/LSan专项199及reader361通过；1896次完整文件Java/C++查询、864次浮点C++查询和6次Java全量扫描通过。两个实际独立审阅会话（gpt-6.1-sol、gpt-6-astra，runtime身份已核实）完成源码、协议和最终证据审阅，均判定NOT_READY_TO_PUSH。Remaining limits:6 schema-only/LIMIT0 malformedOR checks return E_OK/empty rather than strict expected E_INVALID_ARG;12valid controls pass. Fresh full Linux sanitizer exits in unchanged pinned LZOkay; no full pass. Current11b→fcab changed-line/branch coverage91.67%/70.27%; wholePR1180→fcab84.26%/61.56%, selected affected files80.54%/55.41%. Conservative negated statistics and preexisting duplicate lazy-mask evaluation are source observations, not proof of complete slowdown causality or necessity. ValuesPLAIN/timeTS_2DIFF/UNCOMPRESSED for generated matrices; original8 keep original configuration. Cold cache/P99/compressed/general nonaligned FIELD/parallel scaling remain unmeasured. Writes were not retimed for this reader-only revision. Historical local45d stays unmerged/unpushed, without transferring its results tofcab.
剩余限制：空表/LIMIT0的6项畸形OR仍返回E_OK/空结果，未按严格期望返回E_INVALID_ARG；12个合法控制通过。新全量Linux sanitizer仍在未修改的固定LZOkay中止，不能称总通过。本轮11b→fcab修改行/分支覆盖91.67%/70.27%；整个PR1180→fcab为84.26%/61.56%，所选受影响文件80.54%/55.41%。保守取反统计及原有lazy-mask重复求值是源码观察，未证明完整退化因果或必要性。生成矩阵的值为PLAIN、时间TS_2DIFF、UNCOMPRESSED；原8项保留原配置。冷缓存/P99/压缩/广泛非对齐FIELD/并行扩展仍未测；仅reader修订没有重测写入吞吐。历史本地45d仍未合入或推送，其结果不转移到fcab。
Current bilingual report / 当前中英文报告 · All raw evidence and both final reviews / 全部原始证据与两份最终审阅
Historical evidence through public11b and unpushed local45d / 公开11b及未推送本地45d的历史证据
This candidate passed 1,112 Windows tests, scoped Linux ASan/UBSan/LSan runs with 207 and 369 passing tests, and 1,896 complete-file Java/C++ queries. Local repair line/branch coverage is 91.14%/72.25%; whole-PR changed-line/branch coverage is 84.81%/62.78%, and coverage of the selected affected production files is 80.65%/55.72%. Full sanitizer testing was not rerun; the earlier failure in unchanged LZOkay code remains. Both independent source/evidence reviewers conclude
NOT_READY_TO_PUSH. The mask-allocation guard has not been shown to explain the legacy callback regressions.该候选已通过 1,112 项 Windows 测试、分别有 207 和 369 项通过的 Linux 专项 ASan/UBSan/LSan 检查，以及 1,896 次完整文件 Java/C++ 查询。本地修补行/分支覆盖率为 91.14%/72.25%；整个 PR 修改行/分支覆盖率为 84.81%/62.78%，所选受影响生产文件的行/分支覆盖率为 80.65%/55.72%。全量 sanitizer 未重跑，先前未修改的 LZOkay 代码失败记录仍保留。两位独立源码及证据审阅者均判定
NOT_READY_TO_PUSH；尚无证据证明掩码分配优化能解释旧回调的性能退化。Latest bilingual report / 最新中英文报告 · Scoped diagnostics and all timing records / 分项诊断及全部计时记录
The earlier
55db175dmeasurements and findings for public head11bfc35cbelow retain their original source scopes. 以下55db175d的旧测量和公开提交11bfc35c的问题记录保留各自的原始提交范围及失败证据。The local repair passed Windows 1,112 tests and Linux ASan/UBSan/LSan focused207 plus reader369 tests;1,896 complete-file Java/C++ queries match an independent complete oracle. However, its confirmed dense width4/100%-hit row query is +2.56% vs11b, and narrow low-hit legacy callbacks are up to+5.05% vs6f. Other confirmed increases are retained. Necessity has not been demonstrated; no performance exception is claimed. Full sanitizer still aborts in unchanged pinned LZOkay; local11b→55db changed-line/branch coverage is89.47%/68.78%.
本地修订的Windows1112项及Linux207专项、369reader检查在ASan/UBSan/LSan下通过，1896次完整文件Java/C++查询与独立完整期望一致。但确认的4列/100%命中行查询相对11b 慢2.56%，窄投影低命中旧回调相对6f 最高慢5.05%，其它确认退化均保留。必要性未证明，不使用性能例外；全量sanitizer仍在未修改的LZOkay中止。本地11b→55db修补差异的行/分支覆盖率为89.47%/68.78%。
Bilingual report / 中英文报告 · Complete retained diagnostics / 完整保留的诊断证据
The batch commits f5d1243/11bfc35c were authored by the maintainer. The repair is local AI-authored work; source submission remains blocked by unresolved validation. Earlier measurements below retain their original commits and scopes.
批量实现f5d1243a/11bfc35c由维护者编写；本地修订由AI编写，源分支仍须满足验证门槛。下面早期记录仅适用于其原提交和范围。
Historical validation before the current batch implementation / 当前批量实现之前的历史验证
Fixes #1025.
null_w4_p1_b1024_tag0_parallel0dense_w4_p10_b1024dense_w4_p100_b1024Current behavior and maintainer follow-up
The PR executes FIELD predicates on aligned table rows, including NULL rows that must reach the predicate before OFFSET/LIMIT. Maintainer ColinLeeo's follow-up adds two commits,
f2e157ed3and6f4a971e; these were authored by the maintainer, not Codex.249b0d51aand passes on the new head. The non-aligned empty-context shortcut and hard-error handling remain.decode_tv_batch<int32_t>. The used non-alignedChunkReaderhelper remains.Local validation against frozen
249b0d51a/6f4a971ee:当前行为和维护者修正
本 PR 使对齐表数据逐行执行 FIELD 谓词，包括应先进入谓词、再应用 OFFSET/LIMIT 的 NULL 行。维护者上述两次提交修正“整列有值、所选时间范围全 NULL”的漏行，并删除无调用的私有辅助函数；实际 INT32 解码走既有模板，非对齐保护及硬错误传播保留。
新增范围回归在旧库六种配置全部失败，在新提交全部通过。本地永久专项 80 项、额外边界 24 项和非对齐间隙 3 项通过；Windows Release 985 项通过、3 项跳过、10 项禁用。Linux 六压缩库的 ASan/UBSan 专项和额外用例通过，reader 范围 343 项通过、2 项外部夹具跳过、9 项禁用；两个扩大检查仍中止，不能替代为全量 sanitizer 通过。
48 次最终 C++/Java 自身或交叉读取逐项一致；Java 使用时间过滤后应用层 IS NULL/OFFSET/LIMIT，不宣称原生 FIELD 执行路径相同。旧、新 C++ 两种规模的 NULL 文件字节一致，不宣称 Java/C++ 整文件一致。本次维护者修改行覆盖率 100%、分支 83.33%，受影响 reader 文件合计为 79.81%/55.92%，不是全库覆盖率。
查询包含 68 项初测、七项一次性独立复核，正式完整核对 20,928 次，另有 1,456 次预热，涵盖不同投影、命中率和 NULL 范围；仅测暖缓存、MSVC Release、无压缩数据，冷缓存、压缩查询、广泛非对齐 FIELD 和通用并行扩展仍未测。原始数据和异常样本保留。此次仅改 reader，没有重测写入吞吐。另一模型最终证据审阅尚未运行，完整提交前独立审阅门槛未通过；Codex 没有新增生产代码推送。
Current bilingual validation report · Raw sources, results and evidence download instructions
Historical implementation and validation through 249b0d5 / 历史提交说明与验证
Fixes #1025.
With a one-row aligned table (
time=1, v=42), an always-false FIELD predicate currently returns the row without invoking the predicate. This change makes Java and C++ return zero rows and invoke their callbacks once; the null-filter and always-true controls still return the row.Why the current paths are wrong
init_internal()enables the aligned bulk path even whenfield_filteris non-null. That path copies decoded batches directly into the result and never evaluates the predicate. The scalar merge path also has an unfinished FIELD-filter condition: with a non-null filter, it neither appends nor advances the current row, so sparse selections can loop indefinitely.How the change preserves row semantics
get_current_value(), so reusing that accessor for several FIELDs would silently bind the wrong values. The unfiltered shared-vector and bulk-copy paths stay enabled.E_OKplusaccepted=falsemeans reject; a nonzero status is an error.RET_FAIL(expr)assignsretand testsret != E_OK, so callback/read failures return their original error and the existingTableResultSetfailure latch prevents resuming a partially consumed read as EOF.Predicate API and boundaries
Add
TableFieldFilter::satisfy_row(const TableFieldRow&, bool&)without changingFilter's vtable. Its views are valid only during the callback. This avoids treating FIELDs as TAG string segments: the existingsatisfyRow(int, segments)callback is for TAGs and also narrows timestamps to 32 bits.Ordinary
Filtersubclasses remain usable for one selected integer/Boolean or string-like FIELD through their existing value overloads. NULL does not match these legacy predicates. Legacy FLOAT/DOUBLE or multiple-FIELD shapes returnE_NOT_SUPPORT; use the typed row callback for those shapes and NULL-aware predicates. TAG filtering remains separate. Unprojected FIELD loading, predicate pushdown/vectorization, and new C/Python wrapper APIs are outside this change.Review revision: aligned NULLs and time predicates
All three Copilot findings were reproduced against prior PR head
e2e9c3786f9753ec5d5c1b813fd2f1fb0d71d5b8. Four new cases × six row/batch/parallel configurations failed before the revision (24/24) and pass afterward.(t0,42),(t1,NULL),(t2,NULL), a query restricted to[t1,t2]must still present both NULL rows to a NULL-aware FIELD predicate.The before/after C++ results on both Java-written and C++-written files are:
IS NULLIS NULLTwenty revised C++/Java own/cross-reader results match exactly. Ten prior-PR C++ reads are retained, including eight mismatches. Java performs native time filtering and projected reads, followed by application-level
IS NULLand OFFSET/LIMIT; this is a result oracle, not a native Java FIELD callback performance baseline. Full source and actual row outputs are in the revision evidence.Validation after review
Base:
1180b13675e11a0230708af2cc70d037e0e12623; previous PR head:e2e9c3786f9753ec5d5c1b813fd2f1fb0d71d5b8; revised head:249b0d51a6bfe202185aa21b794ff21ccdfafcf3.cpp/src/reader/aligned_chunk_reader.cccpp/src/reader/block/single_device_tsblock_reader.cccpp/src/reader/block/single_device_tsblock_reader.hcpp/src/reader/filter/table_field_filter.cccpp/src/reader/filter/table_field_filter.hWindows optional-codec fixture generation still has the documented untouched-baseline failure (336 matching assertion values), so compressed-file compatibility is validated by the all-codec Linux build. The earlier CRLF fixture-manifest normalization and unchanged 135 TsFile hashes remain documented in the original report. Explicit
"time"projection remains an existing table-schema limitation; ordinary implicit timestamps and TAG-only projection are tested.Query performance after review
Query and write scope are reported separately. This revision changes reader paths and test ownership only; no writer, encoder, Java source or on-disk format changed. Existing write results are retained, and writing was not retimed for this reader revision.
Existing eight query cases: develop base versus revised head. Same verified 300,000-row × 8 INT64 PLAIN/UNCOMPRESSED files, same MSVC Release static configuration, eight fully consumed/checksummed queries per process, two warmups then 25 alternating process pairs, CPU affinity mask 4. The p95 below is process time for eight queries, not individual-request p95. Timing includes open/setup/read/checksum/cleanup; no cold-cache claim. The original initial and 300-pair confirmation measurements remain linked and preserved.
all_rowall_batchtime_rowtime_batchoffset_rowoffset_batchsparse_rowsparse_batchFIELD selectivity and projection: prior PR head versus revised head. Thirty cases cover 1/4/8 projected FIELDs, 0/1/10/50/100% accepted rows and row/batch-1024 APIs. The identical dense file has 40,000 rows × 8 INT64 PLAIN/UNCOMPRESSED fields. Prior PR head correctly executes the same native callback on these non-NULL inputs, so it is a valid baseline for this revision's filtering cost. The upstream ignored/hanging implementation is not used as an equivalent filtering baseline.
Two warmups and 25 alternating process pairs per case, four queries/process, CPU affinity mask 4; every cell, returned row count, checksum and 160,000 callbacks/process are checked. Raw per-query arrays provide file-open time, metadata/query-plan time, first-row time (unavailable at 0% results), complete query latency and throughput; peak working set is the whole process, not incremental query memory. Per-query p50/p95 use 100 queries per variant/case clustered in 25 processes; bootstrap ratios resample the 25 process pairs. No P99 or controlled cold-cache result is claimed.
fields1_accept0_rowfields1_accept0_batchfields1_accept1_rowfields1_accept1_batchfields1_accept10_rowfields1_accept10_batchfields1_accept50_rowfields1_accept50_batchfields1_accept100_rowfields1_accept100_batchfields4_accept0_rowfields4_accept0_batchfields4_accept1_rowfields4_accept1_batchfields4_accept10_rowfields4_accept10_batchfields4_accept50_rowfields4_accept50_batchfields4_accept100_rowfields4_accept100_batchfields8_accept0_rowfields8_accept0_batchfields8_accept1_rowfields8_accept1_batchfields8_accept10_rowfields8_accept10_batchfields8_accept50_rowfields8_accept50_batchfields8_accept100_rowfields8_accept100_batchFull metrics and every initial sample, including outliers, are retained in
field-query-performance.jsonandexisting-query-performance.json. NULL/page/chunk/device and parallel boundaries are covered by correctness tests; the new FIELD timing matrix is dense, one-device, single-reader, uncompressed. It does not measure a TAG selectivity matrix, FIELD-filtered sparse/NULL latency, non-aligned FIELD performance, compressed-query performance, or parallel scaling. Those scopes remain explicitly unmeasured. Original and revised query comparisons do not prove universal Java/C++ performance equivalence.No case in either new dataset has its paired-median 95% interval wholly above 1. The measured scope shows no statistically clear slowdown; this does not establish an absolute absence of regressions or a p95 non-inferiority guarantee. Individual higher p95 samples and outliers are preserved in the tables and raw data.
Review revision validation report · Revision sources, scripts, raw logs and measurements
Original validation report · Original evidence, including all initial and confirmation performance samples
Authorship clarification: the original implementation and validation were written by OpenAI Codex; subsequent maintainer commits, including
fcab7437, retain their Git authorship. The original required disclosure is preserved below.作者说明：原始实现及验证由OpenAI Codex编写；后续维护者提交（含
fcab7437）保留各自Git作者归属。以下保留原有规定的AI说明。Authorship: OpenAI Codex wrote the original PR and this unsubmitted local repair; maintainer commits including publicfcab retain their Git authorship. The required original disclosure follows.
作者说明：OpenAI Codex编写原PR和本次未提交本地修补；包括公开fcab在内的维护者提交保留Git作者归属。以下保留规定的原AI说明。
AI disclosure: The implementation, tests and description in this pull request were written by an AI coding agent (OpenAI Codex), which also performed the reported local validation.
AI 使用说明： 本 PR 的代码修改、测试和正文由 AI 编程代理（OpenAI Codex）编写，文中报告的本地验证也由该 AI 代理执行。