Skip to content

feat(bigquery-jdbc): add job-stream reader for Arrow results - #15048

Open
keshavdandeva wants to merge 2 commits into
jdbc/arrow-support-featurefrom
feat/jdbc-arrow-job-stream-reader
Open

keshavdandeva wants to merge 2 commits into
jdbc/arrow-support-featurefrom
feat/jdbc-arrow-job-stream-reader

Conversation

@keshavdandeva

Copy link
Copy Markdown
Contributor

b/582677690

Adds a reader for a query job's _default stream, so results can be read through the Storage Read API without creating a read session. Nothing calls it yet; Task 4 (b/582676785) wires it into statement execution.

Changes

  • Build the stream name from the job reference (project, location, job ID).
  • Send the Arrow options on every ReadRows request. A resumed request is the first request with only the offset changed.
  • The reader forwards each message's row_count and decides nothing. BigQueryArrowResultSet.loadNextBatch() fails when a batch doesn't hold the rows its message reports.
  • Any stream error ends iteration: the next call to next() returns false, and getters throw SQLException until then.

Behaviour changes on the existing HTAPI path

  • A row_count that differs from the rows in the batch is now an error.
  • After a stream error, next() returns false. Before, it read on from the following message.

Testing

  • make lint and make unittest: 1280 tests, 0 failures, 1 skipped in a class this PR doesn't touch.
  • 16 new unit tests. The result-set tests use real Arrow batches.

@keshavdandeva
keshavdandeva requested review from a team as code owners November 17, 2026 00:30

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements support for reading query job results directly from BigQuery job streams, rather than relying solely on read sessions. Key changes include updating BigQueryArrowBatchWrapper to track reported row counts, enhancing BigQueryArrowResultSet to validate row counts and handle stream errors more robustly, and adding functionality to BigQueryStatement to construct and read from job streams. Extensive unit tests were added to verify the new stream reading logic, error handling, and row count validation. I have no feedback to provide as there were no review comments.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant