fix(hll): preserve estimator state across union transformations

[jaideeppyne]

Problem

HllUnion reconstructs HLL register arrays when it copies an HLL-mode input into an empty union, downsamples that input, or converts an HLL8 result to HLL4 or HLL6. These operations change the representation but not the logical sketch.

Before this change, rebuilding the registers also rebuilt or invalidated estimator metadata. A single-input union could therefore switch from the HIP estimator to the composite estimator, and result-type conversion could preserve an estimate only conditionally while still using confidence bounds for the wrong estimator mode.

HIP depends on the ordered history of register updates. The composite estimate and KxQ cache depend only on the current registers. Treating the HIP accumulator and out-of-order flag as independently mutable fields made it possible to construct inconsistent combinations and made correctness depend on setter order.

Solution

• Model the selected estimate atomically as either Hip(accumulator) or Composite, while keeping register-derived KxQ state separate.
• Name the internal component Estimator rather than HipEstimator, because it owns both estimation strategies.
• Preserve that estimate state when copying, downsampling, or changing the HLL4/HLL6/HLL8 encoding of the same logical sketch.
• Invalidate HIP only when registers from independent update histories are merged.
• Restore serialized estimator state in one operation instead of through order-dependent setters.

This preserves the Java and C++ invariant that estimator state must accompany the registers for an isomorphic copy or downsample.

Reference implementations

• Java restores both fields after downsampling.
• C++ preserves both fields in copy_or_downsample.

Observable behavior

• Copying a single HLL-mode sketch into an empty union preserves its exact estimate and, when lg_k is unchanged, its confidence bounds.
• Downsampling a single HLL-mode sketch preserves its estimate; bounds still reflect the smaller resulting lg_k.
• HLL4, HLL6, and HLL8 result encodings now return identical estimates and confidence bounds for the same union state.
• Merging independent HLL histories still switches to the composite estimator.
• Coupon-mode iteration-order behavior is unchanged and remains outside this change.

Tests

The regression coverage exercises all three source encodings and all three result encodings in HIP mode, result conversion after a composite-mode merge, single-input downsampling, and repeated-input stability.

Validated with:

• cargo x check
• cargo x test
• cargo x lint

The original issue was found with a Rust/C++ differential harness. The initial patch and harness were developed with Claude Code and manually verified; the estimator-state refactor and final review were developed with Codex and verified with the repository's full test and lint workflows.

[Copilot]

Pull request overview

This PR fixes an HLL union correctness issue where merging an HLL-mode sketch into an empty HllUnion rebuilt registers via bulk operations, inadvertently invalidating the HIP estimator and causing estimates/bounds to use the out-of-order (composite) estimator path even after restoring the HIP accumulator.

Changes:

• Preserve both HIP accumulator and the estimator “out-of-order” validity flag when HllUnion::update initializes its gadget from an array-mode sketch.
• Add internal APIs to read/write the out-of-order flag on array sketches to enable faithful state carry-over.
• Update and extend integration tests to pin expected estimator/bounds behavior, and add a user-visible changelog entry.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
datasketches/src/hll/union.rs	Copies HIP accumulator and out-of-order flag during copy/downsample initialization to preserve estimator semantics when unioning into an empty gadget.
datasketches/src/hll/array8.rs	Exposes internal setter/getter for the estimator out-of-order flag on Array8.
datasketches/src/hll/array6.rs	Exposes internal getter for the estimator out-of-order flag on Array6.
datasketches/src/hll/array4.rs	Exposes internal getter for the estimator out-of-order flag on Array4.
tests-integration/tests/hll_test/union.rs	Adjusts idempotency assertions and adds tests to ensure a single-sketch union reproduces estimates/bounds.
CHANGELOG.md	Documents the user-visible HllUnion estimator/bounds behavior fix.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[tisonkun]

Good catch! Thanks for your contribution @jaideeppyne

I pushed one follow-up to refactor the EstimateState to leverage Rust's type system.

Merging ...