Describe the bug, including details regarding any error messages, version, and platform.
The Variant spec requires the field ids in an object's header to be sorted by the
UTF-8 byte order of the field names, so a reader can binary-search them.
VariantBuilder sorted the fields — and Variant.getFieldByKey binary-searched them —
using String.compareTo, which orders by UTF-16 code units, not UTF-8 bytes.
The two orderings are identical for all field names in the Basic Multilingual Plane, but
they diverge for names containing supplementary-plane characters (U+10000 and above):
String.compareTo orders a leading high surrogate (0xD800–0xDBFF) before code points in
U+E000..U+FFFF, whereas UTF-8 byte order (and the spec) orders them after. Consequences:
- An object parquet-java builds with such keys has field ids sorted in an order that
violates the spec.
- A spec-compliant reader (e.g. the Apache Arrow C++/Rust/Go Variant readers) binary-searching
that object can fail to find fields.
- Conversely, parquet-java's own binary search fails to find a supplementary-plane key in an
object produced by a spec-compliant writer.
The bug only surfaces when an object both contains a supplementary-plane key and is large
enough to take the binary-search path, so it has gone unnoticed.
Component(s)
Core
Describe the bug, including details regarding any error messages, version, and platform.
The Variant spec requires the field ids in an object's header to be sorted by the
UTF-8 byte order of the field names, so a reader can binary-search them.
VariantBuildersorted the fields — andVariant.getFieldByKeybinary-searched them —using
String.compareTo, which orders by UTF-16 code units, not UTF-8 bytes.The two orderings are identical for all field names in the Basic Multilingual Plane, but
they diverge for names containing supplementary-plane characters (U+10000 and above):
String.compareToorders a leading high surrogate (0xD800–0xDBFF) before code points inU+E000..U+FFFF, whereas UTF-8 byte order (and the spec) orders them after. Consequences:
violates the spec.
that object can fail to find fields.
object produced by a spec-compliant writer.
The bug only surfaces when an object both contains a supplementary-plane key and is large
enough to take the binary-search path, so it has gone unnoticed.
Component(s)
Core