Skip to content

fix(s3): abort unknown-length multipart upload when the request body fails or the putObject future is cancelled - #7566

Open
joviegas wants to merge 4 commits into
masterfrom
joviegas/tn-cancel-issue-fix
Open

joviegas wants to merge 4 commits into
masterfrom
joviegas/tn-cancel-issue-fix

Conversation

@joviegas

@joviegas joviegas commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation and Context

This PR fixes two bugs in multipart putObject (multipartEnabled(true)) when the content length is unknown.

1. The request body fails, and the SDK still completes the upload

  • If the request body fails while the SDK handles a CreateMultipartUpload or UploadPart response, the SDK fails the putObject future. It still sends CompleteMultipartUpload with the parts uploaded so far. When that request succeeds, S3 stores a partial object at the key. The SDK must abort the upload instead and leave the object at the key unchanged.
  • This happens because onError sets isDone, the same flag onComplete sets. The check that sends CompleteMultipartUpload never reads the failure flag. The CreateMultipartUpload callback also sends parts 1 and 2 without checking for a failure. A failure before the upload id is known sends no abort.

2. The caller cancels, and the SDK leaves the parts in S3

  • If the caller cancels the putObject future between part uploads, the SDK sends neither CompleteMultipartUpload nor AbortMultipartUpload. S3 keeps the parts, and bills their storage, until a lifecycle rule or a manual abort removes them. The SDK must abort the upload instead.
  • This happens because the cancel listener never sends an abort. The abort happens as a side effect of a failing part. With no part in flight, nothing sends it.

Modifications

  • UnknownContentLengthAsyncRequestBodySubscriber now uses the state model of KnownContentLengthAsyncRequestBodySubscriber (Fix S3TransferManager upload pause hanging and racing completion #7516). One State field (RUNNING, COMPLETING, FAILING) replaces the failureActionInitiated and completedMultipartInitiated flags.
  • The first failure or cancel moves the state from RUNNING to FAILING. CompleteMultipartUpload needs a move from RUNNING to COMPLETING, so a failed upload never completes. When the state moves to FAILING, the SDK sends AbortMultipartUpload. If the upload id is not known yet, the abort waits for the CreateMultipartUpload callback to deliver it. After a failure, the SDK starts no new UploadPart and closes the part bodies it does not upload.
  • onSubscribe now rejects a null subscription and cancels a second subscription instead of the first (as Reactive Streams rules 2.13 and 2.5 require).

Testing

Added unit tests

AI tools

  • I wrote this pull request description myself, not with an AI tool
  • I have reviewed all changes in this pull request myself, including any generated by AI tools

License

  • I confirm that this pull request can be released under the Apache 2 license

@joviegas
joviegas requested a review from a team as a code owner November 18, 2026 02:45
@joviegas
joviegas force-pushed the joviegas/tn-cancel-issue-fix branch 2 times, most recently from 3594960 to 37f9e49 Compare November 19, 2026 04:24
@joviegas
joviegas force-pushed the joviegas/tn-cancel-issue-fix branch from 37f9e49 to c0ae095 Compare November 19, 2026 05:18
@joviegas
joviegas requested a review from ianbotsf November 19, 2026 17:43
@@ -191,6 +208,10 @@ public void onNext(CloseableAsyncRequestBody asyncRequestBody) {
CompletableFutureUtils.forwardExceptionTo(returnFuture, createMultipartUploadFuture);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This forwarding (and the same one in MultipartUploadHelper.createMultipartUpload) defeats abortWhenUploadIdKnown() when the CreateMultipartUpload response hasn't arrived yet:

  1. The body fails, or the caller cancels. returnFuture completes exceptionally.
  2. That failure is forwarded here, so createMultipartUploadFuture fails too. Because BaseAsyncClientHandler forwards it to the API call, the HTTP request is also cancelled.
  3. The callback takes the throwable != null branch and calls uploadIdFuture.completeExceptionally(...).
  4. abortWhenUploadIdKnown() uses thenAccept, so it never runs.

If S3 had already handled the request, the upload stays open with no parts. The SDK never learns its id, so the caller can only find it by listing the bucket's uploads. Zero parts means no storage cost, but this contradicts the description: "If the upload id is not known yet, the abort waits for the CreateMultipartUpload callback to deliver it."

Suggestion: don't forward a returnFuture failure into the CreateMultipartUpload future on this path. Let the call finish, and let the existing success branch (state != RUNNING -> abort through uploadIdFuture) clean up. The cost is that the client waits for one extra response after a cancel.

if (t != null) {
s.cancel();
MultipartUploadHelper.cancelingOtherOngoingRequests(futures, t);
if (tryMarkFailing()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The known-length subscriber also aborts when CompleteMultipartUpload is in flight:

if (tryMarkFailing() || state.get() == State.COMPLETING) {

Here, a cancel during COMPLETING loses the CAS, so no abort is sent. CompleteMultipartUpload keeps going and can store the object at the key while the caller gets a CancellationException. The abort would race the complete, but it gives the caller the same best-effort outcome as the known-length path. Can these two match? If the difference is intended, a short comment saying why would help.

/**
* Models a CreateMultipartUpload whose response has already arrived, so a failure forwarded to it is ignored.
*/
private static final class RespondedCreateMultipartUploadFuture extends CompletableFuture<CreateMultipartUploadResponse> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RespondedCreateMultipartUploadFuture ignores completeExceptionally. That means CMPU_RESPONSE_PENDING_IN_CALLBACK (line 66) really models a response that has already arrived. No test covers a CreateMultipartUpload that is still pending when the body fails or the caller cancels, which is the case in my comment on line 208 of UnknownContentLengthAsyncRequestBodySubscriber. Could you add FailurePoint and CancelPoint cases that use a plain CompletableFuture, and assert that one abort is sent once the response arrives? Those cases fail on the current head.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants