Skip to content

fix(bigtable): remove shadow attemptSettings from MutateRowsAttemptCallable - #15045

Open
igorbernstein2 wants to merge 2 commits into
mainfrom
fix-mutaterows-remove-shadow-attempt-settings
Open

igorbernstein2 wants to merge 2 commits into
mainfrom
fix-mutaterows-remove-shadow-attempt-settings

Conversation

@igorbernstein2

Copy link
Copy Markdown
Contributor

Summary

MutateRowsAttemptCallable previously maintained a shadow this.attemptSettings field initialized via retryAlgorithm.createFirstAttempt(null) that was only updated in handleAttemptError and never in handleAttemptSuccess, separate from externalFuture.getAttemptSettings() managed by GAX's BasicRetryingFuture.

This caused three issues:

  1. Desynchronization across partial successes: When partial entry failures (handleAttemptSuccess) were followed by an RPC-level failure (handleAttemptError), this.attemptSettings lagged behind the actual attempt count tracked by externalFuture.
  2. Per-call RetrySettings overrides ignored on RPC-level errors: Because this.attemptSettings was created and advanced with RetryingContext = null, per-call RetrySettings overrides (such as a larger maxAttempts on ApiCallContext) were ignored in handleAttemptError, prematurely marking entries as permanent failures and aborting retries at the stub's default maxAttempts.
  3. Inconsistent isRetryable() and tracer reporting on retry exhaustion: Evaluating TimedAttemptSettings inside handleAttemptError flipped MutateRowsException.isRetryable() to false on the final attempt (inconsistent with handleAttemptSuccess and GAX's ApiException.isRetryable() contract, and causing BasicRetryingFuture to record tracer.attemptPermanentFailure instead of tracer.attemptFailedRetriesExhausted).

Changes

  • Remove the duplicate this.attemptSettings field from MutateRowsAttemptCallable.
  • In handleAttemptError, check result-based retryability (retryAlgorithm.getResultAlgorithm()) with callContext, leaving TimedAttemptSettings (maxAttempts and totalTimeout) tracking and enforcement to the parent RetryingFuture.
  • Add unit tests in MutateRowsAttemptCallableTest covering partial entry failure followed by RPC-level failure and per-call RetrySettings (maxAttempts) overrides.

@igorbernstein2
igorbernstein2 requested review from a team as code owners November 16, 2026 22:34

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the retry mechanism in MutateRowsAttemptCallable by replacing RetryAlgorithm<MutateRowsRequest> with RetryAlgorithm<MutateRowsAttemptResult> and delegating retry decisions to shouldRetryBasedOnResult. It also adds new tests to verify partial success handling and per-call retry settings overrides. The feedback suggests adding a defensive null check for retryAlgorithm in the constructor, appending throws Exception to the new test method signature, and applying a timeout with specific exception handling on the .get() call to prevent tests from hanging.

this.currentRequest = Preconditions.checkNotNull(originalRequest, "currentRequest");
this.callContext = Preconditions.checkNotNull(callContext, "callContext");
this.retryableCodes = Preconditions.checkNotNull(retryableCodes, "retryableCodes");
this.retryAlgorithm = retryAlgorithm;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add a null check for retryAlgorithm in the constructor to enforce defensive programming and maintain consistency with other parameters.

Suggested change
this.retryAlgorithm = retryAlgorithm;
this.retryAlgorithm = Preconditions.checkNotNull(retryAlgorithm, "retryAlgorithm");

}

@Test
public void perCallRetrySettingsMaxAttemptsOverrideTest() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add throws Exception to the method signature to support throwing checked exceptions (such as TimeoutException or InterruptedException) from the .get() call with a timeout.

Suggested change
public void perCallRetrySettingsMaxAttemptsOverrideTest() {
public void perCallRetrySettingsMaxAttemptsOverrideTest() throws Exception {
References
  1. In Java test methods that declare throws Exception, it is acceptable to let InterruptedException propagate.

Comment on lines +546 to +551
Throwable actualError = null;
try {
retryingCallable.futureCall(request, perCallContext).get();
} catch (Throwable t) {
actualError = t.getCause();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Use a timeout on the .get() call to prevent the test from hanging indefinitely if the asynchronous execution fails or deadlocks. Additionally, catch ExecutionException specifically so that other exceptions (like TimeoutException) can propagate and fail the test clearly.

Suggested change
Throwable actualError = null;
try {
retryingCallable.futureCall(request, perCallContext).get();
} catch (Throwable t) {
actualError = t.getCause();
}
Throwable actualError = null;
try {
retryingCallable.futureCall(request, perCallContext).get(10, java.util.concurrent.TimeUnit.SECONDS);
} catch (ExecutionException t) {
actualError = t.getCause();
}
References
  1. Avoid using very short timeouts (such as 200ms) when waiting for asynchronous events or mock server requests in tests, as they can cause flakiness in busy CI/CD environments. Use a larger timeout (e.g., 2000ms) to ensure reliability.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant