Skip to content

Variant field names not being ordered correctly via UTF-8 byte order #3735

Description

@rayokota

Describe the bug, including details regarding any error messages, version, and platform.

The Variant spec requires the field ids in an object's header to be sorted by the
UTF-8 byte order of the field names, so a reader can binary-search them.
VariantBuilder sorted the fields — and Variant.getFieldByKey binary-searched them —
using String.compareTo, which orders by UTF-16 code units, not UTF-8 bytes.

The two orderings are identical for all field names in the Basic Multilingual Plane, but
they diverge for names containing supplementary-plane characters (U+10000 and above):
String.compareTo orders a leading high surrogate (0xD800–0xDBFF) before code points in
U+E000..U+FFFF, whereas UTF-8 byte order (and the spec) orders them after. Consequences:

  • An object parquet-java builds with such keys has field ids sorted in an order that
    violates the spec.
  • A spec-compliant reader (e.g. the Apache Arrow C++/Rust/Go Variant readers) binary-searching
    that object can fail to find fields.
  • Conversely, parquet-java's own binary search fails to find a supplementary-plane key in an
    object produced by a spec-compliant writer.

The bug only surfaces when an object both contains a supplementary-plane key and is large
enough to take the binary-search path, so it has gone unnoticed.

Component(s)

Core

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions