HDDS-16314. Filter out blocks still in use before enqueuing overwritten key versions in S3 MPU complete - #11135
Open
peterxcli wants to merge 2 commits into
Open
HDDS-16314. Filter out blocks still in use before enqueuing overwritten key versions in S3 MPU complete#11135peterxcli wants to merge 2 commits into
peterxcli wants to merge 2 commits into
Conversation
…en key versions in S3 MPU complete S3MultipartUploadCompleteRequest enqueued the overwritten old key versions to deletedTable without filterOutBlocksStillInUse, unlike OMKeyCommitRequest (HDDS-9146). If the overwritten key shares any block with the newly completed MPU key (e.g. older version groups retained from a versioning- enabled overwrite history), KeyDeletingService would physically delete blocks still referenced by the live key table entry, causing data loss. Apply the same filter in the MPU complete overwrite path; the shared base class covers both the OBS/LEGACY and FSO layouts.
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request fixes a data-loss risk in the Ozone Manager S3 multipart upload (MPU) complete path when overwriting an existing key: it now filters out blocks that are still referenced by the newly committed key before enqueueing old key versions into deletedTable, aligning MPU-complete behavior with the existing key-commit overwrite protection.
Changes:
- Apply
filterOutBlocksStillInUse(omKeyInfo, oldKeyVersionsToDelete)inS3MultipartUploadCompleteRequest#validateAndUpdateCachebefore adding old versions to the delete list. - Update overwrite expectations in the existing MPU-complete success test (no delete-table entry when the overwritten key has no blocks after filtering).
- Add a new unit test that asserts shared blocks are not enqueued for deletion (and wire it to run for both LEGACY/OBS and FSO via helper overrides).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCompleteRequest.java | Adds the in-use block filtering step before enqueueing overwritten key versions for deletion in MPU complete. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCompleteRequest.java | Adds regression coverage for shared-block overwrite and updates overwrite delete-table expectations. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadCompleteRequestWithFSO.java | Overrides test helpers so the new shared-block test runs correctly under FSO bucket layout. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
peterxcli
marked this pull request as ready for review
August 28, 2026 09:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
When an S3 multipart upload completes, add all old key versions to deletedTable, even those that were already deleted, and delete them by KeyDeletingService. If the key has any
ContainerBlockID, add them to deletedTable and delete them by KeyDeletingService. If the key has noContainerBlockID, do not add them to deletedTable.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16314
How was this patch tested?
testOverwrittenKeySharedBlocksNotEnqueuedForDeletioninTestS3MultipartUploadCompleteRequest, with FSO helper overrides so it runs in bothTestS3MultipartUploadCompleteRequestandTestS3MultipartUploadCompleteRequestWithFSO. It completes an MPU over a key table entry that shares one block with the committed part and holds one block of its own, then asserts the deletedTable contains only the old-only block while the completed key still references the shared one. The test fails without the production change.TestS3*unit tests in ozone-manager pass (175 tests), andcheckstyle:checkpasses on the module.Generated-by: Claude Code (claude-fable-5)