diff --git a/docs/migration-1.x-to-2.x.md b/docs/migration-1.x-to-2.x.md new file mode 100644 index 00000000..3b8ad36b --- /dev/null +++ b/docs/migration-1.x-to-2.x.md @@ -0,0 +1,367 @@ +# Migrating from 1.x to 2.x + +`2.x` is a breaking major release. Every change is a bug fix or brings Python to +parity with the JavaScript and Java SDKs. The changes most likely to touch your +code are the typed, per-operation **error hierarchy**, the +**serialize/deserialize round trip on the first run**, and the new fail-fast +**default `map` / `parallel` completion** (which completes after the first +observed failure and stops scheduling pending items; already-started items are +not cancelled, so with unlimited concurrency all items may already be running - +set `max_concurrency` if you need to bound that). Opt back into process-all with +`CompletionConfig.all_completed()`. + +There is no compatibility shim: removed names (for example `CallableRuntimeError`) +are gone with no alias. If you are not ready to migrate, stay on `1.x`. + +> Instrumentation plugins are out of scope here. The experimental `plugins=` hook +> already existed in `1.x`; `2.x` adds opt-in auto-discovery and `PluginLoadError`. +> The plugin interface itself also changed incompatibly (hook signatures changed, +> enums moved, and `InvocationEndInfo.status` is now required), so plugin authors +> should expect to update. Because plugins are opt-in and still evolving, they are +> documented with the plugin/OpenTelemetry feature rather than in this guide. + +## Porting to 2.0 + +Each change below lists what changed and what you must do. The ones most likely +to touch your code are the [typed error hierarchy](#callableruntimeerror-and-friends-removed), +the [first-run serialize/deserialize round trip](#first-run-serializedeserialize-round-trip), +and the [fail-fast `map` / `parallel` default](#completionconfigall_completed-tolerates-all-failures). +See [Finding affected code](#finding-affected-code) for a grep checklist. + +### `CallableRuntimeError` and friends removed + +`CallableRuntimeError`, `UserlandError`, and +`CallableRuntimeErrorSerializableDetails` are gone; typed per-operation errors +replace them. + +Catch `StepError`, `InvokeError`, `ChildContextError`, or `WaitForConditionError` +(or the base `DurableOperationError`) instead of `CallableRuntimeError`. See +[Error handling](#error-handling-the-biggest-change) for the full hierarchy. + +### `CallbackError` moved out of the termination tree + +`CallbackError` is no longer a termination reason, and graded subtypes were +added. + +Remove any `termination_reason == TerminationReason.CALLBACK_ERROR` check (the +enum member is gone). Optionally catch `CallbackTimeoutError`, +`CallbackExternalError`, or `CallbackSubmitterError`. See +[Callbacks](#callbacks). + +### `BatchResult.throw_if_error()` raises typed errors + +It no longer raises `CallableRuntimeError`. + +Replace `except CallableRuntimeError` with `ChildContextError` (ordinary item +failure), `SerDesError` (item serialize/deserialize failure), and +`BatchCompletionError` (custom `should_complete` failed the batch with no item +error). `ChildContextError` and `BatchCompletionError` share the base +`DurableOperationError`, but `SerDesError` does not, so catch +`(DurableOperationError, SerDesError)` or list all three: + +```python +# 1.x +except CallableRuntimeError: + ... + +# 2.x +except (DurableOperationError, SerDesError): + ... +``` + +See [map / parallel](#map--parallel) for a full example. + +### Serdes failures surface as `SerDesError`, not `ExecutionError` + +`SerDesError` is a direct child of `DurableExecutionsError`, not `ExecutionError`. + +If you caught serdes failures with `except ExecutionError`, catch `SerDesError` +(or `DurableExecutionsError`) instead: + +```python +# 1.x +except ExecutionError: + ... + +# 2.x +except SerDesError: + ... +``` + +### First-run serialize/deserialize round trip + +`step`, child contexts, `map`/`parallel`, and `wait_for_condition` now round-trip +their result through the serdes on the first run, returning +`deserialize(serialize(x))` - the same canonical value replay returns. + +If you relied on the raw pre-serialization object, use the deserialized shape +instead (or make your `SerDes` round-trip identity). Ensure `wait_for_condition` +`initial_state` is serializable by the configured serdes. For a transient serdes +failure, raise the new `RetryableSerDesError` (retries) instead of `SerDesError` +(permanent). See [Serialize/Deserialize round trip](#serializedeserialize-round-trip). + +### Empty-string payloads are preserved + +`1.x` dropped empty-string (`""`) payloads when serializing; `2.x` keeps them. + +A step, invoke, or child result of `""` that surfaced as `None` (dropped payload) +in `1.x` now surfaces as `""`. If you treated an absent payload as `None`, handle +`""` explicitly. + +### `InvokeConfig.timeout` / `timeout_seconds` removed + +Both fields are gone. + +Remove them. Enforce any timeout inside the invoked function or as a separate +timer. + +### `map` / child batching names removed + +Removed: `ItemBatcher`, `ItemsPerBatchUnit`, `BatchedInput`, `TerminationMode`, +`StepFuture`, `MapConfig.item_batcher`, `ChildConfig.item_serdes`; also +`ChainedInvokeFailedToStartType`, `ChainedInvokeTimeoutType`, and +`ChainedInvokeStopType` (from `lambda_service`). + +Remove all uses. Replace `ChildConfig.item_serdes` with `ChildConfig.serdes`. + +### Config validation moved to construction and call time + +`MapConfig`, `ParallelConfig`, and `CompletionConfig` now validate arguments at +construction (for example `max_concurrency=0` or `min_successful=0` raise +`ValidationError`). `min_successful > total` is validated at the +`map()`/`parallel()` call, not at construction. + +Wrap config construction, and the `map()`/`parallel()` call, in +`try/except ValidationError` when inputs are external. + +### `CompletionConfig.all_completed()` tolerates all failures + +It now actually tolerates every failure, and the default `map` / `parallel` +completion config is now fail-fast. + +If you hand-built the old all-`None` config, use the factory instead. To preserve +1.x process-all behavior, pass it explicitly: + +```python +# 2.x - keep processing every item even when some fail +MapConfig(completion_config=CompletionConfig.all_completed()) +ParallelConfig(completion_config=CompletionConfig.all_completed()) +``` + +### `BatchResult.all` omits never-started branches + +`total_count` and positional iteration differ for early-completed batches, +because never-started branches are no longer included. + +If you index `.all` by original position or expect `total_count` to include +unstarted branches, update that logic. + +### `summary_generator` output moved into an envelope + +For `map`/`parallel`, a custom `summary_generator` output is now stored under a +`"summary"` key in an SDK-owned envelope; it no longer replaces the checkpoint +payload. `ChildConfig.summary_generator` is unchanged: its output is still +checkpointed verbatim. + +If you parse `map`/`parallel` summary payloads from execution history, read the +`"summary"` key from the envelope. Child-context summary consumers need no change. + +### `WaitDecision` and wait timeouts removed + +`WaitDecision` is gone, along with `WaitStrategyConfig.timeout` and +`WaitStrategyConfig.timeout_seconds`. + +Use `WaitForConditionDecision` (`stop_polling()` / `continue_waiting(delay)`). + +### `wait_for_condition` raises on exhaustion + +It raises `WaitForConditionError` when it exhausts `max_attempts`. + +Catch `WaitForConditionError` instead of inspecting the returned state. + +### Finding affected code + +Grep for the removed and changed names before upgrading: + +```bash +rg -n "CallableRuntimeError|UserlandError|CallableRuntimeErrorSerializableDetails" . +rg -n "CallbackError|CALLBACK_ERROR" . +rg -n "InvokeConfig\(|\.timeout_seconds" . +rg -n "WaitDecision|WaitStrategyConfig\(|item_batcher|ItemBatcher|ItemsPerBatchUnit" . +rg -n "TerminationMode|BatchedInput|StepFuture|ChildConfig\(" . +rg -n "ChainedInvoke|except ExecutionError" . +``` + +## Error Handling (the biggest change) + +In `1.x` nearly every user-land failure surfaced as one `CallableRuntimeError`, +so a failed step was indistinguishable from a failed invoke or child branch. `2.x` +raises a specific type per operation, all under a new base `DurableOperationError`. +Inspect the failure through its fields, not `__cause__`: `error_type`, `message`, +`data`, and `stack_trace`. Do not rely on `__cause__` being the original +exception - the SDK reconstructs a `DurableOperationError` stand-in carrying +those same fields (on both the first run and replay, for determinism), so the +original type is not preserved (a `ValueError` does not stay a `ValueError`) and +custom attributes are lost. + +For `StepError`, `InvokeError`, `ChildContextError`, and `WaitForConditionError`, +`error_type` is the name of the error that escaped your code (e.g. `"ValueError"`). +The graded callback errors are different: `CallbackTimeoutError`, +`CallbackExternalError`, and `CallbackSubmitterError` are constructed without the +originating `error_type`, so `error_type` is the callback class name, not the +underlying cause. Use the specific callback exception type (and `message` / +`data` / `stack_trace`) to distinguish those. + +```python +# 1.x +from aws_durable_execution_sdk_python.exceptions import CallableRuntimeError +try: + result = context.step(charge_card, name="charge") +except CallableRuntimeError as e: + context.logger.error("something failed: %s", e.message) + +# 2.x +from aws_durable_execution_sdk_python import StepError, DurableOperationError +try: + result = context.step(charge_card, name="charge") +except StepError as e: # or `except DurableOperationError` to catch any operation + context.logger.error("charge step failed: %s", e.message) +``` + +New types, all exported from the package root: `DurableOperationError` (base), +`StepError`, `InvokeError`, `ChildContextError`, `WaitForConditionError`, +`CallbackError` (+ `CallbackExternalError`, `CallbackTimeoutError`, +`CallbackSubmitterError`), plus `SerDesError` (now exported) and +`RetryableSerDesError`. `SerDesError` stays a direct child of +`DurableExecutionsError`; `RetryableSerDesError` is a retryable `InvocationError`. + +### Callbacks + +`context.wait_for_callback(...)` returns the payload directly and raises the +callback error from the call itself (there is no `callback.result()`): + +```python +from aws_durable_execution_sdk_python import ( + CallbackError, CallbackTimeoutError, CallbackSubmitterError, +) +try: + payload = context.wait_for_callback(submit_approval, name="approval") +except CallbackTimeoutError: + ... # timeout / heartbeat expiry +except CallbackSubmitterError: + ... # the submitter step failed +except CallbackError as e: # external + internal + context.logger.error("callback failed: %s", e.message) +``` + +### map / parallel + +`throw_if_error()` can raise three types: `ChildContextError` for the ordinary +item/branch failure, `SerDesError` if an item result failed to serialize or +deserialize, and `BatchCompletionError` when a custom `should_complete` predicate +marked the batch failed with no failed item (see the completion-predicate section +below). `ChildContextError` and `BatchCompletionError` share the base +`DurableOperationError`, but `SerDesError` does not, so catch +`(DurableOperationError, SerDesError)` or list all three explicitly. + +```python +from aws_durable_execution_sdk_python import ( + ChildContextError, SerDesError, BatchCompletionError, +) + +result = context.map(items, process_item) +try: + result.throw_if_error() +except (ChildContextError, SerDesError, BatchCompletionError): + for err in result.get_errors(): # every failed item's ErrorObject + context.logger.error("%s: %s", err.type, err.message) +``` + +## Serialize/Deserialize Round Trip + +`1.x` returned the raw in-memory result on the first run but the deserialized +result on replay, so a non-identity custom `SerDes` produced different values. +`2.x` round-trips (`serialize` then `deserialize`) on the first run for `step`, +child contexts, `map`/`parallel`, and `wait_for_condition` (which also feeds the +deserialized state to the wait strategy). This makes first-run behavior match +replay for deterministic, reversible custom serdes. If your code depended on +the raw pre-serialization object on the first run, switch to the deserialized +shape (or make the serdes round-trip identity). This also surfaces genuine +serialization bugs on the first run instead of later on replay. + +`invoke` and `wait` are unaffected. `wait_for_callback` is implemented via a +child context, so its result is serialized and deserialized before it returns: +do not rely on callback-result object identity. The enclosing child context uses +the default (extended-type) serdes, not `WaitForCallbackConfig.serdes`, so the +value your callback deserializer returns must itself be serializable by the +default serdes; otherwise the child raises `SerDesError`. + +`wait_for_condition` also round-trips `initial_state` through the serdes before +the first check, so `initial_state` must now be serializable by the configured +serdes. Custom serdes should serialize polling state to a non-empty string; an +empty string checkpoint payload is currently treated as no stored polling state +on resume, so the operation may restart from `initial_state`. + +## New in 2.x: Custom Completion Predicate (Optional) + +`2.x` adds a `should_complete` predicate to `CompletionConfig`, giving `map` and +`parallel` full control over when a batch completes early. This is a new feature, +not a breaking change - no action is required unless you adopt it. + +```python +from aws_durable_execution_sdk_python import complete_batch, continue_batch +from aws_durable_execution_sdk_python.config import CompletionConfig + +config = CompletionConfig( + should_complete=lambda status: ( + complete_batch() if status.success_count >= 2 else continue_batch() + ) +) +``` + +The predicate receives a `CompletionStatus` snapshot (counts plus per-item +statuses) and returns a `CompletionDecision` - `continue_batch()`, or +`complete_batch(CompletionOutcome.SUCCEEDED)` / `complete_batch(CompletionOutcome.FAILED)` +(the outcome defaults to `SUCCEEDED`). A `FAILED` outcome marks the whole batch +failed; `throw_if_error()` then raises `BatchCompletionError` (a +`DurableOperationError` subtype) even when no individual item failed. Individual +item/branch failures still surface as `ChildContextError`. Notes: + +- It cannot be combined with `min_successful` or the `tolerated_failure_*` + fields; doing so raises `ValidationError` at construction. +- The predicate runs before any branch is scheduled (`completed_count == 0`) + and on suspension state changes. At those points, unscheduled item statuses + are `None`; handle missing statuses explicitly so the predicate does not + fail the batch or complete it before useful work starts. +- The predicate must be deterministic, side-effect-free, and monotonic: once a + progress snapshot returns `complete_batch(outcome)`, every later snapshot + containing that progress must return `complete_batch(outcome)` with the same + `CompletionOutcome`. Replaying an already-completed batch uses the + checkpointed decision, but a mid-run resume re-runs the batch live and + re-evaluates the predicate as completed branches replay, possibly in a + different order. +- New exports: `complete_batch`, `continue_batch`, `CompletionStatus`, + `CompletionDecision`, `CompletionOutcome`, `CompletionItemStatus`, + `BatchItemStatus`, `BatchCompletionError`. + +## New in 2.x: Attempt Number in Contexts (Optional) + +`StepContext` and `WaitForConditionCheckContext` now expose an `attempt` field +(the current attempt number, starting at 1). Read it inside a step or a +`wait_for_condition` check to branch on the retry count. The SDK injects these +contexts, so normal usage needs no change. But `attempt` is a required +dataclass field with no default: if you construct these contexts directly (in +tests, fixtures, or wrappers), you must now pass `attempt` or construction fails +with `TypeError`. + +## Recommended Validation After Upgrading + +1. Build and run your test suite against `2.x`, and grep for the removed names above. +2. Trigger a failure in a `step`, an `invoke`, and a `map`/`parallel` branch; + confirm you catch `StepError`, `InvokeError`, and `ChildContextError`. +3. Exercise a `wait_for_callback` timeout and a submitter-step failure + (`CallbackTimeoutError`, `CallbackSubmitterError`). +4. Exercise a `wait_for_condition` that exhausts its attempts (`WaitForConditionError`). +5. If you use a custom `SerDes`, run a workflow that checkpoints a result, an + error payload, and `wait_for_condition` polling state; confirm first-run + output equals replay output. diff --git a/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/waits.py b/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/waits.py index 3753a1be..b191d201 100644 --- a/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/waits.py +++ b/packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/waits.py @@ -99,7 +99,17 @@ def wait_strategy(result: T, attempts_made: int) -> WaitForConditionDecision: @dataclass(frozen=True) class WaitForConditionConfig(Generic[T]): - """Configuration for wait_for_condition.""" + """Configuration for wait_for_condition. + + Attributes: + wait_strategy: Called after each poll with (state, attempts_made) and + returns a WaitForConditionDecision (continue_waiting or stop_polling). + initial_state: State passed to the first poll. It is round-tripped + through serdes (serialize then deserialize) before the first check, + so it must be serializable by the configured serdes. + serdes: SerDes used to serialize and deserialize the polled state at + each checkpoint. Defaults to the SDK's extended-type serdes when None. + """ wait_strategy: Callable[[T, int], WaitForConditionDecision] initial_state: T