Skip to content

[Test] Expand cross-language compatibility coverage across scales and nullable BOOLEAN columns - #1021

Draft
ColinLeeo wants to merge 6 commits into
apache:developfrom
ColinLeeo:codex/full-compatibility-matrix
Draft

ColinLeeo wants to merge 6 commits into
apache:developfrom
ColinLeeo:codex/full-compatibility-matrix

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

The previous compatibility fixtures cover selected codecs with 300-row or NULL-page samples. They do not exercise all implemented common codecs, large datasets, or Java's default BOOLEAN RLE encoding alongside other columns.

This expands the matrix to 47 implemented common type/encoding pairs, six common compression algorithms (UNCOMPRESSED, SNAPPY, GZIP, LZ4, ZSTD, LZMA2), and 32 / 4,096 / 1,000,000 rows. Each core fixture includes a separate boolean_value column and independent, fixed-seed 10% NULL masks for the main value and BOOLEAN columns. Dedicated legacy NULL-page cases retain their structural NULL patterns. Java uses its default BOOLEAN RLE; C++ uses its implemented BOOLEAN PLAIN writer. Both readers verify actual chunk codecs, values, NULL positions, timestamps, and column alignment.

  • 846 core fixtures per writer, plus 72 existing NULL-page fixtures.
  • Java additionally retains 36 explicit max_point_number=0 fixtures: 954 Java files and 918 C++ files in total.
  • Compare manifest coverage sets so missing combinations cannot silently pass.
  • Enumerate the implemented intersection verified against both encoder/decoder factories, excluding enum-only placeholders. Actual fixture generation and readback exercise every included combination.
  • Shard CI by compression and enable SNAPPY/GZIP. Run the target Surefire method once, and use 4,096-row pages for scale cases while retaining 64-row NULL-page regressions.

BOOLEAN RLE support (#1019) has merged into develop. The newly exposed codec defects are addressed independently in #1022 (DICTIONARY embedded NUL bytes) and #1023 (SPRINTZ extreme transitions, FIRE arithmetic, and block boundaries). This matrix PR remains draft until those fixes merge.

Baseline failures (before the independent fixes)

The complete 32-row matrix reproduces existing failures across all six compressors:

  • SPRINTZ INT32/INT64: extreme-value transitions decode incorrectly in both directions.
  • SPRINTZ FLOAT/DOUBLE: negative zero loses its sign bit in both directions.
  • DICTIONARY TEXT/STRING: C++ truncates dictionary strings containing embedded NUL bytes.

Million-row shard checks also reproduce these failures. This draft preserves the failing cases so they remain visible as merge blockers. The failing inputs are retained; fixes are provided in #1022 and #1023.

Validation

  • Complete fixture generation passed: 954 Java files (295,239,744 rows) and 918 C++ files (283,190,208 rows).
  • Full Java-to-C++ audit: all 954 fixtures attempted; 846 passed, 108 failed (72 SPRINTZ + 36 DICTIONARY fixtures).
  • Full C++-to-Java audit: all 918 fixtures attempted; 846 passed, 72 failed (SPRINTZ fixtures).
  • Failures reproduce across 32, 4,096 and 1,000,000 rows and all six compressors. No other failure categories were observed.
  • Java/C++ Spotless checks, workflow YAML parsing, and git diff --check passed.

Independent fix validation

  • [Java][CPP] Preserve SPRINTZ extreme transitions and block boundaries #1023 passes all 72 SPRINTZ combinations in each direction (four numeric types × six compressors × three scales), including 1,000,000 rows and deterministic 10% NULLs.
  • With the regenerated SPRINTZ fixtures and both fixes applied, both full audits pass: Java-to-C++ 954/954 files, C++-to-Java 918/918 files. All original 108/72 failures are eliminated.
  • The original audit results above describe the unfixed develop baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant