fix: honor read.partial.row.timeout.ms as the readRows watchdog timeout - #4629
Conversation
google.bigtable.grpc.read.partial.row.timeout.ms was handed to gax as the retry settings' rpcTimeout, but the read path always put a timeout on the ApiCallContext, and gax only applies rpcTimeout when the context carries none of its own (ServerStreamingAttemptCallable#call). The key was silently discarded and had no effect on any read. Wire it to readRowsSettings.setWaitTimeout() instead, which is the timeout it was named for: how long a stream may go without receiving a response before it is cancelled and retried, reset by every response. This lets a sparse filtered scan survive long stretches of non-matching rows, which is what prompted the change -- a large export hit the 5 minute default watchdog on ten consecutive attempts and failed the job. Applied raise-only: a value below the 5 minute client default is ignored. Because the key never reached the wire before, honoring a short value outright would newly cancel reads for anyone who happens to have one set. The attempt timeout moves off the ApiCallContext and onto the retry settings, where it no longer suppresses rpcTimeout. The gRPC deadline stays: on the scan path PaginatedRowResultScanner reuses a single context across segment fetches and Deadline.after() is absolute, so it bounds the scanner's whole lifetime rather than one ReadRows. Verified non-breaking by differential testing over 30 configurations. The deadline observed server-side is identical before and after in all of them except bigtable.read.rpc.attempt.timeout.ms=0, where the old code let the partial row timeout through as the attempt deadline and the new code sets none. That direction is permissive only, and the read stays bounded by the total timeout and the watchdog. Also exposes --bigtableReadPartialRowTimeoutMs on the SequenceFile export template. Change-Id: Ic39d1572503786b083f38b9b5dd2ddaa0e512508
There was a problem hiding this comment.
Code Review
This pull request updates the Bigtable HBase client's timeout handling to correctly distinguish between the watchdog wait timeout (the gap between consecutive responses) and the RPC attempt timeout. Specifically, it configures the watchdog wait timeout on the readRowsSettings (ensuring it only raises the default 5-minute threshold) and maps the attempt timeout to the retry settings rather than the ApiCallContext. It also exposes this configuration in the Beam import templates and adds comprehensive unit tests. The reviewer identified a potential issue where the watchdog timeout could be lowered below the 5-minute default if defaultWaitTimeout is null, and provided a code suggestion to enforce the minimum threshold.
| Duration defaultWaitTimeout = readRowsSettings.getWaitTimeout(); | ||
| Duration responseTimeout = operationTimeouts.getResponseTimeout().get(); | ||
| if (defaultWaitTimeout == null || responseTimeout.compareTo(defaultWaitTimeout) > 0) { | ||
| readRowsSettings.setWaitTimeout(responseTimeout); | ||
| } |
There was a problem hiding this comment.
If defaultWaitTimeout is null, the code currently sets the wait timeout to responseTimeout without any lower-bound check. However, the client default watchdog timeout is 5 minutes. If responseTimeout is less than 5 minutes, this would lower the watchdog timeout, violating the "raise-only" contract described in the Javadoc and PR description. To prevent this, we should default to a minimum of 5 minutes when defaultWaitTimeout is null.
Duration defaultWaitTimeout = readRowsSettings.getWaitTimeout();
Duration responseTimeout = operationTimeouts.getResponseTimeout().get();
Duration minWaitTimeout = defaultWaitTimeout != null ? defaultWaitTimeout : Duration.ofMinutes(5);
if (responseTimeout.compareTo(minWaitTimeout) > 0) {
readRowsSettings.setWaitTimeout(responseTimeout);
}There was a problem hiding this comment.
Default timeout is 5 minutes.
google.bigtable.grpc.read.partial.row.timeout.mshas never had any effect on a read.It was handed to gax as the retry settings'
rpcTimeout, but the read path always put a timeout on theApiCallContext, and gax only appliesrpcTimeoutwhen the context carries none of its own:Since
scanTimeouts.attemptTimeoutalways resolves (10 min default), the context timeout was always present and the key was always discarded.What this changes
Wire it to
readRowsSettings.setWaitTimeout(), which is the timeout the key was named for: how long a stream may go without receiving a response before it is cancelled and retried, reset by every response. That is a bound on the gap between responses, not on the attempt.This matters for sparse filtered scans over large tables, which can legitimately traverse a lot of non-matching rows between results. A large export hit the 5 minute default watchdog on ten consecutive attempts and failed the job; there was no supported way to raise it.
The attempt timeout moves off the
ApiCallContextand onto the retry settings, where it no longer suppressesrpcTimeout. The gRPC deadline stays — on the scan pathPaginatedRowResultScannerreuses a single context across segment fetches andDeadline.after()is absolute, so it bounds the scanner's whole lifetime rather than oneReadRows.No new configuration key, and no deprecation.
Why it is not a breaking change
The wait timeout is applied raise-only: a value below the 5 minute client default is ignored. Because the key never reached the wire before, honoring a short value outright would newly cancel reads for anyone who happens to have one set.
Non-breakingness was verified differentially rather than by argument: 30 configurations were run against this tree and against the pre-change tree, comparing both the gax settings and the deadline actually observed server-side by a fake
BigtableImplBasereadingio.grpc.Context.current().getDeadline().waitTimeoutraises, the 10th is the same case below.totalTimeout,maxAttempts, retry delays and the gRPCCallOptionsdeadline: identical in all 30.The rule governing the deadline, before and after, is
min(attemptTimeout ?: 10m, operationTimeout ?: ∞). The partial row timeout does not appear in it at all.The one difference is
bigtable.read.rpc.attempt.timeout.ms=0.GrpcCallContext.withTimeoutDurationnulls a zero duration, so the old code left the context empty and gax applied the partial row timeout as the attempt deadline — the single configuration where the key ever reached the wire. The new code setsmaxRpcTimeout=PT0S, which gax's!rpcTimeout.isZero()guard skips, so there is no attempt deadline. Permissive direction only; the read stays bounded by the 12h total timeout and the watchdog.Beam
Exposes
--bigtableReadPartialRowTimeoutMson the SequenceFile export template. Note that a single attempt is separately capped by--bigtableReadRpcAttemptTimeoutMs(10 min default), so that needs raising too for a gap longer than that.Tests
TestBigtableHBaseVeneerSettingscovers the mapping and the raise-only clamp. Verified by mutation: deleting the clamp fails those tests, and dropping thesetInitialRpcTimeout/setMaxRpcTimeoutblock makes the server observe veneer's 30 minute default instead of 10 minutes.