Repository navigation
fix(parquet/internal/encoding): don't panic decoding zero BYTE_STREAM_SPLIT values without SIMD - #1579
Open
zeroshade wants to merge 1 commit into
Open
fix(parquet/internal/encoding): don't panic decoding zero BYTE_STREAM_SPLIT values without SIMD#1579zeroshade wants to merge 1 commit into
zeroshade wants to merge 1 commit into
Conversation
…_SPLIT values without SIMD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
ByteStreamSplitDecoder.Decodefor 4- and 8-byte types panics withindex out of range [0] with length 0when there is nothing left to decode, but only when the pure Go implementation is used (-tags noasm, or a CPU without AVX2/NEON). The pure Go functions take&out[0]of an empty slice. The AVX2 and NEON wrappers already return early for zero values, so the default build on amd64 and arm64 returns0, nil.A column reader hits this when a page holds fewer values than its header claims and it asks the decoder for the rest.
What changes are included in this PR?
The pure Go width-4 and width-8 decode functions (little- and big-endian) return early for zero values, like the SIMD wrappers.
Are these changes tested?
Yes,
TestByteStreamSplitDecodeExhausteddecodes past the end of a page for int32 and int64. It panics without the fix under-tags noasm.Are there any user-facing changes?
No API changes. Decoding an exhausted BYTE_STREAM_SPLIT page no longer panics in builds without SIMD.