Skip to content

fix: honor read.partial.row.timeout.ms as the readRows watchdog timeout - #4629

Merged
mutianf merged 1 commit into
googleapis:mainfrom
mutianf:partial-row-timeout-watchdog
Sep 2, 2026
Merged

fix: honor read.partial.row.timeout.ms as the readRows watchdog timeout#4629
mutianf merged 1 commit into
googleapis:mainfrom
mutianf:partial-row-timeout-watchdog

Conversation

@mutianf

@mutianf mutianf commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

google.bigtable.grpc.read.partial.row.timeout.ms has never had any effect on a read.

It was handed to gax as the retry settings' rpcTimeout, but the read path always put a timeout on the ApiCallContext, and gax only applies rpcTimeout when the context carries none of its own:

// ServerStreamingAttemptCallable#call
if (!rpcTimeout.isZero() && context.getTimeoutDuration() == null) {
  context = context.withTimeoutDuration(rpcTimeout);
}

Since scanTimeouts.attemptTimeout always resolves (10 min default), the context timeout was always present and the key was always discarded.

What this changes

Wire it to readRowsSettings.setWaitTimeout(), which is the timeout the key was named for: how long a stream may go without receiving a response before it is cancelled and retried, reset by every response. That is a bound on the gap between responses, not on the attempt.

This matters for sparse filtered scans over large tables, which can legitimately traverse a lot of non-matching rows between results. A large export hit the 5 minute default watchdog on ten consecutive attempts and failed the job; there was no supported way to raise it.

The attempt timeout moves off the ApiCallContext and onto the retry settings, where it no longer suppresses rpcTimeout. The gRPC deadline stays — on the scan path PaginatedRowResultScanner reuses a single context across segment fetches and Deadline.after() is absolute, so it bounds the scanner's whole lifetime rather than one ReadRows.

No new configuration key, and no deprecation.

Why it is not a breaking change

The wait timeout is applied raise-only: a value below the 5 minute client default is ignored. Because the key never reached the wire before, honoring a short value outright would newly cancel reads for anyone who happens to have one set.

Non-breakingness was verified differentially rather than by argument: 30 configurations were run against this tree and against the pre-change tree, comparing both the gax settings and the deadline actually observed server-side by a fake BigtableImplBase reading io.grpc.Context.current().getDeadline().

  • Wire deadline: 1 difference in 30.
  • Settings: 10 differences in 510 comparisons (30 configs × 17 properties) — 9 are the intended waitTimeout raises, the 10th is the same case below.
  • totalTimeout, maxAttempts, retry delays and the gRPC CallOptions deadline: identical in all 30.
  • Unary reads, mutations and sampleRowKeys: unaffected.

The rule governing the deadline, before and after, is min(attemptTimeout ?: 10m, operationTimeout ?: ∞). The partial row timeout does not appear in it at all.

The one difference is bigtable.read.rpc.attempt.timeout.ms=0. GrpcCallContext.withTimeoutDuration nulls a zero duration, so the old code left the context empty and gax applied the partial row timeout as the attempt deadline — the single configuration where the key ever reached the wire. The new code sets maxRpcTimeout=PT0S, which gax's !rpcTimeout.isZero() guard skips, so there is no attempt deadline. Permissive direction only; the read stays bounded by the 12h total timeout and the watchdog.

Beam

Exposes --bigtableReadPartialRowTimeoutMs on the SequenceFile export template. Note that a single attempt is separately capped by --bigtableReadRpcAttemptTimeoutMs (10 min default), so that needs raising too for a gap longer than that.

Tests

TestBigtableHBaseVeneerSettings covers the mapping and the raise-only clamp. Verified by mutation: deleting the clamp fails those tests, and dropping the setInitialRpcTimeout/setMaxRpcTimeout block makes the server observe veneer's 30 minute default instead of 10 minutes.

google.bigtable.grpc.read.partial.row.timeout.ms was handed to gax as the
retry settings' rpcTimeout, but the read path always put a timeout on the
ApiCallContext, and gax only applies rpcTimeout when the context carries none
of its own (ServerStreamingAttemptCallable#call). The key was silently
discarded and had no effect on any read.

Wire it to readRowsSettings.setWaitTimeout() instead, which is the timeout it
was named for: how long a stream may go without receiving a response before it
is cancelled and retried, reset by every response. This lets a sparse filtered
scan survive long stretches of non-matching rows, which is what prompted the
change -- a large export hit the 5 minute default watchdog on ten consecutive
attempts and failed the job.

Applied raise-only: a value below the 5 minute client default is ignored.
Because the key never reached the wire before, honoring a short value outright
would newly cancel reads for anyone who happens to have one set.

The attempt timeout moves off the ApiCallContext and onto the retry settings,
where it no longer suppresses rpcTimeout. The gRPC deadline stays: on the scan
path PaginatedRowResultScanner reuses a single context across segment fetches
and Deadline.after() is absolute, so it bounds the scanner's whole lifetime
rather than one ReadRows.

Verified non-breaking by differential testing over 30 configurations. The
deadline observed server-side is identical before and after in all of them
except bigtable.read.rpc.attempt.timeout.ms=0, where the old code let the
partial row timeout through as the attempt deadline and the new code sets
none. That direction is permissive only, and the read stays bounded by the
total timeout and the watchdog.

Also exposes --bigtableReadPartialRowTimeoutMs on the SequenceFile export
template.

Change-Id: Ic39d1572503786b083f38b9b5dd2ddaa0e512508
@mutianf
mutianf requested a review from a team as a code owner September 2, 2026 02:54
@product-auto-label product-auto-label Bot added size: m Pull request size is medium. api: bigtable Issues related to the googleapis/java-bigtable-hbase API. labels Sep 2, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Bigtable HBase client's timeout handling to correctly distinguish between the watchdog wait timeout (the gap between consecutive responses) and the RPC attempt timeout. Specifically, it configures the watchdog wait timeout on the readRowsSettings (ensuring it only raises the default 5-minute threshold) and maps the attempt timeout to the retry settings rather than the ApiCallContext. It also exposes this configuration in the Beam import templates and adds comprehensive unit tests. The reviewer identified a potential issue where the watchdog timeout could be lowered below the 5-minute default if defaultWaitTimeout is null, and provided a code suggestion to enforce the minimum threshold.

Comment on lines +791 to +795
Duration defaultWaitTimeout = readRowsSettings.getWaitTimeout();
Duration responseTimeout = operationTimeouts.getResponseTimeout().get();
if (defaultWaitTimeout == null || responseTimeout.compareTo(defaultWaitTimeout) > 0) {
readRowsSettings.setWaitTimeout(responseTimeout);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If defaultWaitTimeout is null, the code currently sets the wait timeout to responseTimeout without any lower-bound check. However, the client default watchdog timeout is 5 minutes. If responseTimeout is less than 5 minutes, this would lower the watchdog timeout, violating the "raise-only" contract described in the Javadoc and PR description. To prevent this, we should default to a minimum of 5 minutes when defaultWaitTimeout is null.

      Duration defaultWaitTimeout = readRowsSettings.getWaitTimeout();
      Duration responseTimeout = operationTimeouts.getResponseTimeout().get();
      Duration minWaitTimeout = defaultWaitTimeout != null ? defaultWaitTimeout : Duration.ofMinutes(5);
      if (responseTimeout.compareTo(minWaitTimeout) > 0) {
        readRowsSettings.setWaitTimeout(responseTimeout);
      }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default timeout is 5 minutes.

@mutianf
mutianf merged commit cbd7dc7 into googleapis:main Sep 2, 2026
17 of 18 checks passed
@mutianf
mutianf deleted the partial-row-timeout-watchdog branch September 2, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigtable Issues related to the googleapis/java-bigtable-hbase API. size: m Pull request size is medium.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants