IO: Absorb upsert PyArrow logic into io/pyarrow.py - #3813
Open
qzyu999 wants to merge 1 commit into
Open
Conversation
Move PyArrow-specific table operations (joins, group_by, duplicate detection, row comparison) from table/upsert_util.py into pyiceberg/io/pyarrow.py. The upsert_util module now delegates to helpers in io/pyarrow rather than importing pyarrow directly. This consolidates PyArrow logic behind the io/pyarrow module boundary, which is a precondition for the decomposition proposed in apache#3737. Fixes apache#3812 (PR A)
This was referenced Aug 19, 2026
rambleraptor
left a comment
rambleraptor
left a comment
Collaborator
There was a problem hiding this comment.
This is just a simple code reorg, so it looks fine.
I think it's good to consolidate our PyArrow logic, especially since PyArrow is an optional dependency.
| Or, | ||
| ) | ||
| from pyiceberg.io.pyarrow import ( | ||
| _upsert_get_rows_to_update, |
Collaborator
There was a problem hiding this comment.
These shouldn't be private if we're importing them elsewhere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Move PyArrow-specific table operations (joins, group_by, duplicate detection, row comparison) from
table/upsert_util.pyintopyiceberg/io/pyarrow.py. Theupsert_utilmodule now delegates to helpers inio/pyarrowrather than importingpyarrowdirectly.Changes
pyiceberg/io/pyarrow.py:_upsert_unique_keys,_upsert_has_duplicate_rows,_upsert_get_rows_to_updatepyiceberg/table/upsert_util.py: removed directimport pyarrow/from pyarrow import compute, now imports helpers frompyiceberg.io.pyarrowWhat stays the same
from pyiceberg.table.upsert_util import ...imports continue to workWhy
This consolidates PyArrow logic behind the
io/pyarrowmodule boundary.table/upsert_util.pywas the only file inpyiceberg/table/with a top-levelimport pyarrowand inline PyArrow compute operations (joins, aggregations, etc.). Moving these operations intoio/pyarrow.pymeans all runtime PyArrow usage routes through one module, which is a precondition for the decomposition proposed in #3737.See #3812 for the full audit and tracking.
Related