Repository navigation
Conversation
…n its header claims A required column spins in ReadBatch and Skip when its page header claims more values than the page holds: the decoder runs out, returns zero values and no error, and the page never reaches its end. The decoders that take their value count from the encoded data are DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY and BYTE_STREAM_SPLIT. Optional columns consume their definition levels, so they do not spin. Return an error when a read of a nonempty batch makes no progress. This matches the check in Arrow C++ TypedColumnReaderImpl::ReadBatch. Skip fails the same way, where C++ would fail on the next ReadBatch of the same page.
There was a problem hiding this comment.
🟡 Changes recommended
The reported SeekToRow hang lacks a regression test exercising real page repositioning and verifying short-page errors through both the column reader and pqarrow.RecordReader.
1 open issue
What changed in this PR
This PR prevents Parquet column readers from looping indefinitely when a required page contains fewer values than its header claims.
Changes:
- Return errors when nonempty read or skip operations make no progress.
- Add regression tests for short pages, decoder-error preservation, null-only pages, and zero-sized reads.
| File | Description |
|---|---|
parquet/file/column_reader.go |
Adds no-progress guards and a shared short-page error. |
parquet/file/column_reader_short_page_internal_test.go |
Adds bounded decoder helpers and malformed-page regression tests. |
Review effort: Balanced
…n its header claims SeekToRow reaches the no-progress guard through skipRows. The test pager now repositions to its page on a seek, and the test page carries a first row index, as the real page reader provides, so the row runs the seek path instead of a no-op.
zeroshade
left a comment
There was a problem hiding this comment.
Thanks. I wrote ten int32 values in one DELTA_BINARY_PACKED page and set the delta header's count to 2: on main ReadBatch, Skip and SeekToRow on a required column never return, and here each returns the error, with ReadBatch also returning the two values it could read. It also merges cleanly with #1560, and both sets of tests pass together.
Possible follow-up, not for this PR: on an optional or repeated column the same page reads as 10 levels and 2 values with a nil error. That matches C++ and is unchanged here, but the missing values go unreported.
|
@hcrosse looks like the tests in CI are failing due to an issue in this PR |
|
CI failed here on That is an existing bug in the pure Go BYTE_STREAM_SPLIT decoder, which your test exposes: it panics when asked for zero values, while the AVX2/NEON paths return early. I missed it in review because I only ran the default build. #1579 fixes the decoder, and with it applied your test passes under Nothing to change in this PR. Once #1579 is merged, could you rebase onto main so CI runs against the fix? I'll merge when it's green. |
|
@zeroshade Thanks for fixing the decoder. I'll update this branch from main once #1579 is merged. |

Rationale for this change
On a malformed required page that claims more values than it holds,
ReadBatch,SkipandSeekToRowloop forever, including seeks throughpqarrow'sRecordReader. On main,pqarrow.ReadTablereturns an error.I reproduced this on v18.8.0 and main, with Go 1.26.10 on macOS arm64. I wrote ten required
int32values in one uncompressedDELTA_BINARY_PACKEDpage and set the delta header's count to 2:On both versions,
ReadBatch(10)did not return within five seconds.What changes are included in this PR?
readBatchInPageandskipRowsreturn an error when a nonempty batch makes no progress. The read guard matches C++:https://github.com/apache/arrow/blob/976779b2aac587f51b87c3bd2e6ae4c36f75ab6c/cpp/src/parquet/column_reader.cc#L1666-L1674
Skipfails immediately. C++ fails on the nextReadBatch. As in C++, optional columns are unchanged. The guard adds one comparison per batch.Are these changes tested?
TestReadBatchPageHoldsFewerValuesThanHeader,TestReadBatchInPageReportsNoProgressandTestSkipPageHoldsFewerValuesThanHeaderfail on main.TestShortPageGuardKeepsDecoderErrorandTestShortPageGuardLeavesConsistentPagesAlonecheck decoder errors, null pages and zero sized requests. All five pass with this change.Full
go test -race ./parquet/...passed on linux/amd64 with pinned test data, and focused tests andgolangci-lint2.13.2 passed on macOS arm64.Are there any user-facing changes?
Yes.
ReadBatch,ReadBatchInPage,SkipandSeekToRowreturn errors for these pages. There is no API change.