Native MergeRows can panic or produce wrong masks - arrow 59 and_not mishandles sliced arrays #7596

Describe the bug

MergeRowsStream builds the "not matched" masks with arrow::compute::kernels::boolean::and_not (merge_rows.rs:625-626). In arrow 59.x and_not mixes up the offsets of its inputs when one of them is a slice (apache/arrow-rs#10698, fixed in apache/arrow-rs#10699 for 60.0.0, not backported). Comet main resolves arrow to 59.3.2, so the bug is still there.

After the full outer join one marker column can be a slice with a non-zero offset. Then the task either panics with offset + len out of bounds, or gets wrong masks with no error and applies inserts and deletes to the wrong rows. We hit the panic on Iceberg MERGE with spark.comet.exec.mergeRows.enabled=true (Comet 1.2.0, Spark 4.0.2); with the flag off the jobs pass.

Steps to reproduce

// arrow = "=59.3.2"
use arrow::array::{Array, BooleanArray};
use arrow::compute::kernels::boolean::{and, and_not, not};

fn mask(len: usize, f: fn(usize) -> bool) -> BooleanArray {
    BooleanArray::from((0..len).map(f).collect::<Vec<_>>())
}

fn main() {
    let left = mask(3000, |i| i % 3 != 0).slice(0, 1000);
    let right = mask(2000, |i| i % 5 == 0).slice(1000, 1000);
    let expected = and(&left, &not(&right).unwrap()).unwrap();
    let actual = and_not(&left, &right).unwrap();
    let wrong = (0..1000).filter(|&i| actual.value(i) != expected.value(i)).count();
    println!("wrong values: {wrong} of 1000"); // 59.x: 533, 60.1: 0
}

Expected behavior

and_not(a, b) gives the same result as and(a, not(b)), and native MergeRows does not panic.

Additional context

Until Comet moves to arrow 60 (#7249), replacing the two calls with and(a, &not(b)?) fixes it.

Willingness to contribute

I can contribute a fix for this bug independently