Prevent malformed retry delays from poisoning Nexus completions

[Quinn-With-Two-Ns]

What changed?

Validate the retry delay is a valid proto duration.

Why?

Prevent malformed retry delays from poisoning Nexus completions.

How did you test it?

• built
• run locally and tested manually
• covered by existing tests
• added new unit test(s)
• added new functional test(s)

Potential risks

Potentially users could have been sending an invalid proto duration, unclear how exactly, and now we would fail their request.

---

Note

Medium Risk
Request validation on the activity-failure path may reject clients that previously sent invalid duration payloads; behavior change is intentional but could surface latent bad callers.

Overview
RespondActivityTaskFailed and RespondActivityTaskFailedById now reject failures whose ApplicationFailureInfo.NextRetryDelay is not a valid google.protobuf.Duration, including nested causes in the failure chain.

A new validateActivityFailureNextRetryDelays helper runs CheckValid() on any set delay and returns InvalidArgument with a clear message instead of accepting malformed values that could break downstream handling (e.g. Nexus completions).

Unit tests cover edge cases (range, sign mismatch, nested invalid delay); functional tests confirm both respond-by-token and respond-by-ID paths leave the activity RUNNING when validation fails and still allow a valid failure afterward.

^{Reviewed by Cursor Bugbot for commit b7bb95b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Quinn-With-Two-Ns]

For the record the bug report from Codex is as follow

ApplicationFailureInfo.NextRetryDelay is a google.protobuf.Duration. Binary protobuf and gRPC can carry a structurally invalid duration, such as {seconds: 1, nanos: -1}, but protobuf JSON rejects it. RespondActivityTaskFailed validates the failure variant and byte size without validating this nested duration, and the standalone activity reads it with AsDuration without checking CheckValid. If the failure is terminal, the malformed duration remains in the persisted failure.

Nexus completion later converts the entire Temporal failure with protojson.Marshal. That conversion rejects the duration before either the internal or system callback invoker is constructed, so the target Nexus operation never receives the already-committed activity outcome. The callback task returns an internal processing error and remains scheduled for queue retries, or eventually reaches the DLQ when that policy is enabled; retrying cannot repair the persisted failure.

Evidence:

• RespondActivityTaskFailed checks only that the failure has ApplicationFailureInfo, then enforces byte-size limits before forwarding it to History; it never validates NextRetryDelay: service/frontend/workflow_handler.go:1849, service/frontend/workflow_handler.go:1882
• Standalone activity failure handling immediately calls AsDuration and, for a terminal failure, preserves the original failure: chasm/lib/activity/activity.go:630, chasm/lib/activity/statemachine.go:262
• Temporal-to-Nexus failure conversion serializes the complete failure with protobuf JSON, whose well-known-type encoder validates google.protobuf.Duration: common/nexus/failure.go:168
• GetNexusCompletion converts the persisted terminal failure and turns a conversion error into Internal: chasm/lib/activity/activity.go:555
• Callback execution loads the Nexus completion before constructing or invoking either transport and returns that error directly: chasm/lib/callback/component.go:84, chasm/lib/callback/tasks.go:122
• Unexpected task errors retry until the configured DLQ threshold, if any, without mutating the malformed activity outcome: service/history/queues/executable.go:613

Impact:

1. A buggy or malicious worker reports a terminal standalone activity failure with a small but invalid NextRetryDelay duration.
2. The frontend accepts the binary gRPC request, and History closes the activity with that failure.
3. The activity's Nexus callback cannot serialize its completion arguments, so no request reaches the target namespace.
4. The backing Nexus operation remains running until its own timeout, or indefinitely if it has none, while the source callback repeatedly fails or is sent to the DLQ.