feat(logging): structured logs with request correlation via Powertools

.github/copilot-instructions.md

@@ -18,7 +18,9 @@ Python style
 - Use type aliases for complex types
 - Use built-in generics for type hints
 - Use `logging.getLogger(__name__)`, not print
-- Lazy % formatting in logging: `logger.info("msg %s", var)`
+- Structured logging: constant message + data in `extra`: `logger.info("Message accepted.", extra={"writer": name})`
+- Do not re-log `topic`, `user`, `resource`, `http_method` or the Lambda context; they are bound per request in `src/utils/observability.py`
+- `logger.exception()` only inside an `except` block; outside it pass `exc_info=exc` to `logger.error()`
 - F-strings in exceptions: `raise ValueError(f"Error {var}")`
 - All imports at top of file (never inside functions)
 - Apache 2.0 license header in every .py file (including `__init__.py`)
@@ -27,6 +29,8 @@ Python style
 - Use single backticks in docstrings (`value`), never double backticks (`` ``value`` ``)
 - Do not use `# -----------` separator comments to divide sections
 - End all log messages with a period: `logger.info("Message.")`
+- Every non-2xx response must produce exactly one log line explaining the cause
+- Levels: TRACE payloads, DEBUG steps, INFO request outcomes, WARNING rejected requests, ERROR actionable failures
 
 Patterns
 - `__init__` methods must not raise exceptions; defer validation and connection to first use (lazy init)

DEVELOPER.md

@@ -214,6 +214,49 @@ View container logs in pytest output by increasing log level:
 pytest tests/integration/ -v --log-cli-level=DEBUG
 ```
 
+## Logging Conventions
+
+Logging is structured. `src/utils/observability.py` attaches the AWS Lambda Powertools JSON handler to the root logger, so modules keep using the standard library logger and inherit the format, the Lambda execution context and the request correlation id.
+
+```python
+import logging
+
+logger = logging.getLogger(__name__)
+```
+
+Rules:
+
+- The message is a constant sentence ending with a period. Variable data goes into `extra`, never into the sentence.
+  ```python
+  logger.warning("Request rejected: unknown topic.", extra={"known_topics": sorted(known_topics)})
+  ```
+- Do not add `topic`, `user`, `resource`, `http_method`, `correlation_id` or the Lambda context to `extra`; they are bound once per request by `bind_request_context()` and `append_request_context()`.
+- Every non-2xx response must produce exactly one log line explaining the cause.
+- Every request produces exactly one `INFO` line: `Request completed.`, emitted by `dispatch_request()`. Handlers do not emit their own `INFO` outcome lines; they attach outcome fields with `append_request_context()` (e.g. `writers_ok`, `message_key`, `row_count`) so the completion line carries them.
+- Every failed request produces exactly one `ERROR` record (the aggregated dispatch failure); per-writer failure detail is logged at `WARNING`. This keeps `level = "ERROR"` metric filters counting failures, not log lines. The per-writer `WARNING` carries `exc_info=True`, because the aggregated `ERROR` is emitted outside the `except` block and can no longer reach the traceback.
+- Levels: `TRACE` payloads, `DEBUG` steps, `INFO` request outcomes, `WARNING` rejected requests and soft failures, `ERROR` failures that need action. See the level table in [README](./README.md#logging--correlation).
+- Never log tokens, passwords or full message payloads outside `TRACE`. `TRACE` payload logging goes through `log_payload_at_trace()`, which redacts and size caps the payload.
+- `logger.exception()` is only valid inside an `except` block. Outside one, pass the captured exception: `logger.error("...", exc_info=exc)`.
+- Durations are logged as milliseconds with an explicit key (`duration_ms`, `writer_duration_ms`, `query_duration_ms`).
+
+Assert on structured fields in tests, not on formatted strings:
+
+```python
+def test_rejects_unknown_topic(caplog):
+    caplog.set_level(logging.WARNING)
+    ...
+    assert "Request rejected: unknown topic." == caplog.records[-1].message
+```
+
+Keys bound with `append_request_context()` live on the Powertools formatter rather than on the `LogRecord`. To assert on them, render the record with the Powertools logger (`registered_formatter` exists only there, not on a `logging.getLogger()` instance):
+
+```python
+from src.utils.observability import logger as powertools_logger
+
+payload = json.loads(powertools_logger.registered_formatter.format(caplog.records[-1]))
+assert "run-42" == payload["correlation_id"]
+```
+
 ## Run All Quality Gates
 
 Run Black, Pylint, mypy, unit tests (with coverage), and integration tests in a single command:

README.md

@@ -128,11 +128,52 @@ Supporting configs:
 - `topic_schemas/*.json` – each file contains a JSON Schema for a topic. In the current code these are explicitly loaded inside `event_gate_lambda.py`. (Future enhancement: auto-discover or index file.)
 
 Environment variables:
-- `LOG_LEVEL` (optional) – defaults to `INFO`.
+- `LOG_LEVEL` (optional) – defaults to `INFO`. Accepts `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`. An unknown value falls back to `INFO` and is reported with a warning.
+- `POWERTOOLS_LOG_LEVEL` (optional) – takes precedence over `LOG_LEVEL`.
+- `POWERTOOLS_SERVICE_NAME` (optional) – value of the `service` key in every log line. Defaults to `eventgate`.
+- `TRACE_REDACT_KEYS` (optional) – comma separated message keys redacted from `TRACE` payload logs. Defaults to `password,secret,token,key,apikey,api_key`.
+- `TRACE_MAX_BYTES` (optional) – maximum size of a logged `TRACE` payload. Defaults to `10000`.
 - `CONF_DIR` (optional) – directory containing `config.json` and `access.json`. Defaults to `conf`.
 - `POSTGRES_SECRET_NAME` (optional) – AWS Secrets Manager secret name holding PostgreSQL connection credentials (host, port, database, user, password). Required for Postgres writer and stats reader.
 - `POSTGRES_SECRET_REGION` (optional) – AWS region of the Secrets Manager secret. Must be set together with `POSTGRES_SECRET_NAME`.
 
+## Logging & Correlation
+
+Both lambdas emit structured JSON logs through [AWS Lambda Powertools](https://docs.aws.amazon.com/powertools/python/latest/core/logger/). Every line carries the service name, log level, `function_request_id`, `cold_start` and a `correlation_id`. Static execution facts (`function_name`, `function_memory_size`, `function_arn`) are logged once per container on the cold start line `Lambda execution context.` instead of being repeated on every record.
+
+The correlation id is resolved per request in this order:
+1. The `X-Correlation-ID` request header, when it matches `^[A-Za-z0-9._:-]{1,128}$`.
+2. The `X-Request-ID` request header, with the same constraint.
+3. The API Gateway request id (`requestContext.requestId`).
+
+Callers that already have a run or job id should send it as `X-Correlation-ID` so their logs and EventGate logs can be joined. The resolved id is returned in the `X-Correlation-ID` response header of every response, including errors.
+
+Log levels used by the service:
+
+| Level     | Content                                                                                       |
+|-----------|-----------------------------------------------------------------------------------------------|
+| `TRACE`   | Full message payloads, redacted and size capped. Never enable by default.                      |
+| `DEBUG`   | Configuration loading, lazy initialization, per-writer send attempts, connection reuse.        |
+| `INFO`    | Exactly one `Request completed.` line per request, carrying the outcome fields (`status_code`, `duration_ms`, and e.g. `writers_ok`, `message_key`, `row_count`); cold start initialization. |
+| `WARNING` | Rejected requests (auth, authorization, validation), degraded health, Kafka flush retries, individual writer failures. |
+| `ERROR`   | Partial fan-out failures (one aggregated line per failed request), failed queries, unhandled request errors. |
+
+Every non-2xx response has exactly one log line explaining the cause, and every failed request produces exactly one `ERROR` record — per-writer failure detail is reported at `WARNING`.
+
+Example CloudWatch Logs Insights queries:
+
+```text
+fields @timestamp, level, message, status_code, duration_ms
+| filter correlation_id = "<id>"
+| sort @timestamp asc
+```
+
+```text
+fields @timestamp, topic, user, message
+| filter level = "WARNING" and status_code = 403
+| stats count() by user, topic
+```
+
 ## Local Development & Testing
 
 | Purpose                       | Relative link                                                         |

adr/000-template.md

@@ -0,0 +1,27 @@
+# ADR-XYZ: <ADR Name>
+
+## Status
+
+**<STATUS>** — <DATE>
+
+## Context and Problem Statement
+
+Change me.
+
+## Decision Outcome
+
+Change me.
+
+## Alternatives Considered
+
+Change me.
+
+## Related Tickets
+
+* Ticket 1
+* Ticket 2
+
+## References
+
+* Reference 1
+* Reference 2