Skip to content

fix(parquet/file): return an error when a page holds fewer values than its header claims - #1565

Open
hcrosse wants to merge 2 commits into
apache:mainfrom
hcrosse:parquet-delta-count-spin
Open

hcrosse wants to merge 2 commits into
apache:mainfrom
hcrosse:parquet-delta-count-spin

Conversation

@hcrosse

@hcrosse hcrosse commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

On a malformed required page that claims more values than it holds, ReadBatch, Skip and SeekToRow loop forever, including seeks through pqarrow's RecordReader. On main, pqarrow.ReadTable returns an error.

I reproduced this on v18.8.0 and main, with Go 1.26.10 on macOS arm64. I wrote ten required int32 values in one uncompressed DELTA_BINARY_PACKED page and set the delta header's count to 2:

// delta header: block size 128, 4 miniblocks, 10 values
i := bytes.Index(data, []byte{0x80, 0x01, 0x04, 0x0a})
if i < 0 {
	panic("delta header not found")
}
data[i+3] = 0x02

rdr, _ := file.NewParquetReader(bytes.NewReader(data))
col, _ := rdr.RowGroup(0).Column(0)
vals := make([]int32, 10)
total, n, err := col.(*file.Int32ColumnChunkReader).ReadBatch(10, vals, nil, nil)

On both versions, ReadBatch(10) did not return within five seconds.

What changes are included in this PR?

readBatchInPage and skipRows return an error when a nonempty batch makes no progress. The read guard matches C++:

https://github.com/apache/arrow/blob/976779b2aac587f51b87c3bd2e6ae4c36f75ab6c/cpp/src/parquet/column_reader.cc#L1666-L1674

Skip fails immediately. C++ fails on the next ReadBatch. As in C++, optional columns are unchanged. The guard adds one comparison per batch.

Are these changes tested?

TestReadBatchPageHoldsFewerValuesThanHeader, TestReadBatchInPageReportsNoProgress and TestSkipPageHoldsFewerValuesThanHeader fail on main. TestShortPageGuardKeepsDecoderError and TestShortPageGuardLeavesConsistentPagesAlone check decoder errors, null pages and zero sized requests. All five pass with this change.

Full go test -race ./parquet/... passed on linux/amd64 with pinned test data, and focused tests and golangci-lint 2.13.2 passed on macOS arm64.

Are there any user-facing changes?

Yes. ReadBatch, ReadBatchInPage, Skip and SeekToRow return errors for these pages. There is no API change.

…n its header claims

A required column spins in ReadBatch and Skip when its page header claims more values than the page holds: the decoder runs out, returns zero values and no error, and the page never reaches its end.

The decoders that take their value count from the encoded data are DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY and BYTE_STREAM_SPLIT. Optional columns consume their definition levels, so they do not spin.

Return an error when a read of a nonempty batch makes no progress. This matches the check in Arrow C++ TypedColumnReaderImpl::ReadBatch. Skip fails the same way, where C++ would fail on the next ReadBatch of the same page.
@hcrosse
hcrosse marked this pull request as ready for review November 14, 2026 03:12
@hcrosse
hcrosse requested a review from zeroshade as a code owner November 14, 2026 03:12
Copilot AI balanced review requested due to automatic review settings November 14, 2026 03:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The reported SeekToRow hang lacks a regression test exercising real page repositioning and verifying short-page errors through both the column reader and pqarrow.RecordReader.

1 open issue
What changed in this PR

This PR prevents Parquet column readers from looping indefinitely when a required page contains fewer values than its header claims.

Changes:

  • Return errors when nonempty read or skip operations make no progress.
  • Add regression tests for short pages, decoder-error preservation, null-only pages, and zero-sized reads.
File Description
parquet/​file/​column_reader.go Adds no-progress guards and a shared short-page error.
parquet/​file/​column_reader_short_page_internal_test.go Adds bounded decoder helpers and malformed-page regression tests.

Review effort: Balanced

Comment thread parquet/file/column_reader.go Resolved
…n its header claims

SeekToRow reaches the no-progress guard through skipRows. The test pager now repositions to its page on a seek, and the test page carries a first row index, as the real page reader provides, so the row runs the seek path instead of a no-op.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I wrote ten int32 values in one DELTA_BINARY_PACKED page and set the delta header's count to 2: on main ReadBatch, Skip and SeekToRow on a required column never return, and here each returns the error, with ReadBatch also returning the two values it could read. It also merges cleanly with #1560, and both sets of tests pass together.

Possible follow-up, not for this PR: on an optional or repeated column the same page reads as 10 levels and 2 values with a nil error. That matches C++ and is unchanged here, but the missing values go unreported.

@zeroshade

Copy link
Copy Markdown
Member

@hcrosse looks like the tests in CI are failing due to an issue in this PR

@zeroshade

Copy link
Copy Markdown
Member

CI failed here on TestReadBatchPageHoldsFewerValuesThanHeader/byte_stream_split, in the jobs that run with -tags noasm:

panic: runtime error: index out of range [0] with length 0
  encoding.decodeByteStreamSplitBatchWidth4InByteOrderDefault  byte_stream_split_little_endian.go:42

That is an existing bug in the pure Go BYTE_STREAM_SPLIT decoder, which your test exposes: it panics when asked for zero values, while the AVX2/NEON paths return early. I missed it in review because I only ran the default build. #1579 fixes the decoder, and with it applied your test passes under noasm.

Nothing to change in this PR. Once #1579 is merged, could you rebase onto main so CI runs against the fix? I'll merge when it's green.

@hcrosse

hcrosse commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

@zeroshade Thanks for fixing the decoder. I'll update this branch from main once #1579 is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants