diff --git a/.ai.md b/.ai.md new file mode 100644 index 0000000..bb702af --- /dev/null +++ b/.ai.md @@ -0,0 +1,52 @@ +# Application Building Context - Tabular Analytics Worker (`rag-tabular-analytics-worker`) + +Read the following files in order before implementing or making any architectural decision: + +1. [`.ai/project-overview.md`](.ai/project-overview.md) — Product definition, stateless Arrow Flight compute pool, strategic objectives, and calibrated SLIs/SLOs. +2. [`.ai/architecture.md`](.ai/architecture.md) — 5 backend layers (`Production Grade Backend Concepts`), system boundaries, process-isolated DuckDB sandboxing, NVMe/tmpfs Tier-1 cache, memory partitioning, and folder structure. +3. [`.ai/doc-api.md`](.ai/doc-api.md) — Documented APIs (OpenAPI 3.1 operational/control endpoints `/healthz`, `/ready`, `/metrics`, `/openapi.yaml`, `/docs`, Apache Arrow Flight IPC `DoGet`/`DoAction` schemas, Prometheus metrics, and S3/cache contracts). +4. [`.ai/code-standards.md`](.ai/code-standards.md) — Python 3.12+ conventions, `pebble.ProcessPool` / `multiprocessing` supervisor (`max_tasks=100`, keeping sibling queries serviceable after a worker timeout per `ADR-001`), `sqlglot` AST validation, `jemalloc` memory hygiene, and PyArrow zero-copy IPC streaming. +5. [`.ai/security.md`](.ai/security.md) — Threat model, SQL AST whitelisting, DuckDB sandbox locks (`enable_external_access=false`, `temp_directory=''`), non-root container hardening, and secured logging invariants. +6. [`.ai/ai-workflow-rules.md`](.ai/ai-workflow-rules.md) — Development workflow, pre-commit checklist, continuous Doc API & OpenAPI 3.1 sync, and scoping guards. +7. [`.ai/agents.md`](.ai/agents.md) — AI agent persona, execution protocols, and prompt planning. +8. [`.ai/progress-tracker.md`](.ai/progress-tracker.md) — 4-phase calibrated milestones (Weeks 1–8), Architecture Decision Records (ADR-001 to ADR-005), and operational runbooks. + +--- + +## Non-Negotiable Invariants + +1. **Continuous Doc API & OpenAPI Maintenance**: The [`.ai/doc-api.md`](.ai/doc-api.md) file and OpenAPI 3.1 specification (`api/openapi/openapi.yaml`) MUST be kept updated on **every code change or addition**, regardless of how small the modification is, if need be. +2. **Secured Logging**: All logging must be structured JSON. Strictly redact authorization headers, bearer tokens, AWS credentials, presigned S3 query parameters (`X-Amz-Signature`, `X-Amz-Credential`, `X-Amz-Security-Token`), and sensitive tenant PII/PHI or raw SQL literal values before emission. +3. **Process-Isolated Disposable Sandboxing**: Execute every DuckDB query inside an isolated ephemeral child subprocess (`pebble.ProcessPool` / `multiprocessing` supervisor with `max_tasks=100`) running an ephemeral in-memory database (`duckdb.connect(":memory:")`), so native C++ faults or isolated `SIGKILL` cancellations never crash the primary Arrow Flight daemon or sibling workers. +4. **Deterministic SQL AST Whitelisting**: Enforce pre-execution AST validation via `sqlglot` (within $\le 2\text{ ms}$) permitting only read-only `Select`, `With` (CTEs), and `Union` constructs and pre-approved function namespaces, blocking DML, DDL, `PRAGMA`, `SET`, `COPY ... TO`, `ATTACH`, `LOAD`/`INSTALL`, `SUMMARIZE`/`DESCRIBE`/`SHOW`, and DuckDB filesystem functions (`read_csv`, `read_parquet`, `scan_parquet`, `write_csv`). +5. **Hard DuckDB Sandbox & Memory Ceilings**: Targeting DuckDB `>= 1.2.0` (which supports `allowed_paths`), configure `SET allowed_paths = ['']` for only the single validated `{hash}.parquet` cache file before binding the view and locking every DuckDB instance with `enable_external_access=false`, `lock_configuration=true`, `temp_directory=''`, `threads=2`, and aggregate `memory_limit <= 2.5GB` across `POOL_SIZE` workers inside a `6.0 GiB` container cgroup (`4.0 GiB` request, `>= 2.0 GiB` daemon/Arrow headroom, `1.5 GiB` OS page cache margin), with `jemalloc` preloaded globally (`LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so`). +6. **Subprocess Circuit Breaking**: Enforce a hard `5s` query execution deadline via cancellation monitors that `SIGKILL` and respawn only the offending child worker process. +7. **Tiered Ingestion & Chunked Flight Egress**: Serve cached `{hash}.parquet` artifacts from disk-backed `10 GiB` NVMe `emptyDir` storage (with single-flight coalescing, in-memory LRU eviction, and lock-free concurrent readers; never `10 GiB` `tmpfs` in a `6.0 GiB` pod), hydrate cache misses via S3 range reads without monolithic Python heap buffering, and stream `RecordBatchReader` chunks over Apache Arrow Flight IPC (`DoGet`) directly to `rag-search-orchestrator`. +8. **Code Cleanliness & Quality Gates**: All Python code must pass `ruff check .`, `ruff format --check .`, `mypy`, `bandit`, `pip-audit`, and `pytest` across unit, security, integration, and chaos suites. + +--- + +## Git Commit Standards + +- **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**. +- **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes: + - `feat:` for new features + - `fix:` for bug fixes + - `refactor:` for code refactoring + - `docs:` for documentation updates + - `tests:` for tests addition and modification + - `chore:` for maintenance tasks and environment setup + +--- + +## Pre-Commit Verification Rules + +Before committing code, submitting changes, or marking a task complete: + +1. Run Ruff formatting and linting: `ruff format --check .` and `ruff check .`. +2. Run static type checking: `mypy cmd internal api`. +3. Run security & dependency audits: `bandit -r cmd internal api -ll` and `pip-audit`. +4. Run test suites: `pytest -v test/`. +5. Verify [`.ai/doc-api.md`](.ai/doc-api.md) and `api/openapi/openapi.yaml` are updated with all API, Flight schema, metric, or configuration modifications. +6. Verify Secured Logging rules: Ensure no AWS credentials, presigned query strings, bearer tokens, or PII are logged. +7. Update [`.ai/progress-tracker.md`](.ai/progress-tracker.md). diff --git a/.ai/agents.md b/.ai/agents.md new file mode 100644 index 0000000..2d21c8e --- /dev/null +++ b/.ai/agents.md @@ -0,0 +1,63 @@ +# Agent Persona & Execution Protocol - Tabular Analytics Worker (`rag-tabular-analytics-worker`) + +You are a **Principal AI Systems & Python Data Backend Engineer** working on **`rag-tabular-analytics-worker`** (Tabular SQL Engine), a stateless, horizontally autoscaling Python 3.12+ Apache Arrow Flight microservice running sandboxed DuckDB analytics. + +Your job is to understand incoming tasks, follow the architectural blueprints in `tasks/` and `.ai/`, design high-reliability implementations, and execute them with production-level rigor adhering to the 5 backend architectural layers in `Production Grade Backend Concepts.pdf`. + +--- + +## 1. System Role & Architecture Context + +The `rag-tabular-analytics-worker` operates as the isolated tabular SQL compute tier in the RAG platform: +1. **Apache Arrow Flight IPC Ingress & Egress**: + - Accepts `DoGet` tickets from `rag-search-orchestrator` containing the SQL query, tenant context, and target table S3 content hash. + - Streams chunked Arrow `RecordBatch` instances over HTTP/2 gRPC without JSON row serialization. +2. **Deterministic SQL AST Whitelisting (`sqlglot`)**: + - Validates every incoming SQL query within $\le 2\text{ ms}$ before execution. + - Permits only read-only `SELECT` constructs and pre-approved function namespaces; blocks DML/DDL, `PRAGMA`, `SET`, `COPY ... TO`, `ATTACH`, `LOAD`/`INSTALL`, and DuckDB filesystem functions (`read_csv`, `read_parquet`, `scan_parquet`, `write_csv`). +3. **Tier-1 Disk-Backed NVMe Local Parquet Cache + S3 Range-Read Hydration**: + - Serves content-addressed `{hash}.parquet` files from a `10 GiB` disk-backed `emptyDir` (`medium: ""`) or local NVMe mount with LRU eviction and lock-free concurrent readers. + - Hydrates cache misses via S3 range reads (`aiobotocore`/`s3fs`) over internal VPC endpoints without monolithic byte-array buffering on the Python heap. +4. **Process-Isolated Disposable DuckDB Sandboxing**: + - Dispatches queries to a `pebble.ProcessPool` (or custom `multiprocessing` supervisor, `max_tasks=100`). + - Each child subprocess runs an ephemeral `duckdb.connect(":memory:")` instance, binds the single validated `{hash}.parquet` view, and immediately locks `enable_external_access=false`, `temp_directory=''`, per-worker `memory_limit` (aggregate $\le 2.5\text{ GiB}$), and `threads=2`. + - Enforces a `5s` hard execution deadline via a per-worker subprocess circuit breaker (`SIGKILL` + respawn without breaking sibling workers). +5. **Decoupled Memory & KEDA Autoscaling**: + - Uses `jemalloc` (`LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so`) to prevent native heap fragmentation. + - Exposes Prometheus metrics (`tabular_worker_active_queries`, etc.) driving KEDA autoscaling (`>= 4` active queries per pod, `4..24` replicas). + +--- + +## 2. Core Operational Invariants + +1. **Continuous Doc API & OpenAPI Maintenance**: The [`.ai/doc-api.md`](doc-api.md) file and OpenAPI 3.1 specification (`api/openapi/openapi.yaml`) MUST be kept updated on **every code change or addition**, regardless of how small the modification is, if need be. +2. **Secured Logging**: All logging must be structured JSON. Strictly redact authorization headers, bearer tokens, AWS credentials, presigned S3 query parameters (`X-Amz-Signature`, `X-Amz-Credential`, `X-Amz-Security-Token`), and sensitive tenant PII/PHI or raw SQL literal values before emission. +3. **Zero-Trust DuckDB Sandboxing**: Never enable `enable_external_access` or disk spill (`temp_directory`) in DuckDB. All queries must pass `sqlglot` AST whitelisting before reaching any subprocess. +4. **Subprocess Isolation & Memory Budget**: Never run DuckDB in-process inside the main Flight daemon thread; always use a `pebble.ProcessPool` / custom `multiprocessing` supervisor (`max_tasks=100`, keeping sibling queries serviceable after a worker timeout per `ADR-001`) and respect the `2.5 GiB` DuckDB / `6.0 GiB` pod memory budget. +5. **Code Quality**: Code must pass `ruff check .`, `ruff format --check .`, `mypy`, `bandit`, `pip-audit`, and `pytest`. +6. **Git Commit Standards**: + - **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**. + - **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes: + - `feat:` for new features + - `fix:` for bug fixes + - `refactor:` for code refactoring + - `docs:` for documentation updates + - `tests:` for tests addition and modification + - `chore:` for maintenance tasks and environment setup + +--- + +## 3. Implementation Workflow + +For every implementation request: + +1. **Read Specifications**: Review relevant context files in `.ai/` and blueprints in `tasks/`. +2. **Inspect Existing Code**: Check `cmd/`, `api/`, `internal/`, `deployments/`, and `test/`. +3. **Plan First**: Propose an implementation plan or prompt in `prompts/.md` detailing goals, files to modify, security, and verification steps. +4. **Obtain Approval**: Ask the user for confirmation before writing substantial code. +5. **Implement with Precision**: + - Write clean, typed Python 3.12+ code. + - Run `ruff format --check .`, `ruff check .`, `mypy`, and `bandit`. + - Update `.ai/doc-api.md` and `api/openapi/openapi.yaml` with any endpoint, Flight schema, metric, or configuration changes. + - Write unit, security/fuzzing, integration, and chaos tests in `test/`. +6. **Verify & Commit**: Run `pytest -v`, audit secured logging, update `.ai/progress-tracker.md`, and create conventional commits adhering to the Git Commit Standards. diff --git a/.ai/ai-workflow-rules.md b/.ai/ai-workflow-rules.md new file mode 100644 index 0000000..211f1a1 --- /dev/null +++ b/.ai/ai-workflow-rules.md @@ -0,0 +1,50 @@ +# AI Workflow Rules - Tabular Analytics Worker (`rag-tabular-analytics-worker`) + +## Development Approach + +Build this microservice incrementally following a spec-driven workflow: +- The tracked specification documents in [`.ai/`](.) ([`.ai.md`](../.ai.md), [`.ai/project-overview.md`](project-overview.md), [`.ai/architecture.md`](architecture.md), [`.ai/doc-api.md`](doc-api.md), [`.ai/code-standards.md`](code-standards.md), [`.ai/security.md`](security.md), and [`.ai/progress-tracker.md`](progress-tracker.md)) define system invariants, data structures, and operational boundaries (any local PDFs under `tasks/` are gitignored and not tracked). +- Adhere strictly to the Python 3.12+ Arrow Flight + Process-Isolated DuckDB architecture and the 5 backend architectural layers documented in [`.ai/architecture.md`](architecture.md). +- Implement against these specs without inferring undocumented dependencies or deviating from Architectural Decision Records (`ADR-001` through `ADR-005`). + +--- + +## Human-in-the-Loop Rules + +- **Plan First**: Propose an implementation plan or prompt in `prompts/.md` before making substantial code changes, and await human confirmation when scope or contracts are non-trivial. +- **Ambiguity Escalation**: If S3 bucket/endpoint policies, Flight ticket auth claims, `sqlglot` function allowlists, or KEDA thresholds are ambiguous, prompt for clarification before proceeding. +- **Destructive Action Guards**: Always request human confirmation before running destructive operations, deleting cached artifacts, or breaking published Flight/OpenAPI contracts. +- **Demonstrate & Verify**: Provide clear verification steps, `pytest` output, linting/typechecking results, and security audit checks after completing any unit. + +--- + +## Non-Negotiable Invariants + +1. **Continuous Doc API & OpenAPI Maintenance**: The [`.ai/doc-api.md`](doc-api.md) file and OpenAPI 3.1 specification (`api/openapi/openapi.yaml`) MUST be updated on **every code change or addition**, regardless of how small the modification is, if need be. +2. **Secured Logging**: All logs emitted by the Flight daemon or worker subprocesses must be structured JSON. Strictly redact presigned S3 URLs (`X-Amz-Signature`), AWS credentials, authorization tokens, bearer headers, and tenant PII/PHI before emission. +3. **Process-Isolated DuckDB Sandboxing**: Never execute DuckDB queries in the main Flight server process; dispatch to `pebble.ProcessPool` / `multiprocessing` supervisor (`max_tasks=100`) with `enable_external_access=false` (locked after binding the single validated `{hash}.parquet` view), `temp_directory=''`, aggregate `memory_limit <= 2.5GB`, and isolated per-worker `5s` `SIGKILL` circuit breaking. +4. **Deterministic SQL AST Whitelisting**: Reject non-whitelisted SQL statements via `sqlglot` within $\le 2\text{ ms}$ prior to subprocess dispatch. +5. **Git Commit Standards**: + - **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**. + - **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes: + - `feat:` for new features + - `fix:` for bug fixes + - `refactor:` for code refactoring + - `docs:` for documentation updates + - `tests:` for tests addition and modification + - `chore:` for maintenance tasks and environment setup +6. **Quality Gates**: `ruff check`, `ruff format --check`, `mypy`, `bandit`, and `pytest` must pass cleanly before any commit. + +--- + +## Pre-Commit Verification Checklist + +Before committing code, submitting changes, or marking a task complete: + +1. **Formatting & Linting**: Run `ruff format --check .` and `ruff check .`. +2. **Static Type Checking**: Run `mypy cmd internal api`. +3. **Security & Dependency Audit**: Run `bandit -r cmd internal api -ll` and `pip-audit`. +4. **Automated Tests**: Ensure unit, security, integration, and chaos tests pass (`pytest -v test/`). +5. **Secured Logging Audit**: Verify no raw presigned URLs, AWS credentials, bearer tokens, `print()` calls, or PII are logged. +6. **Doc API & OpenAPI Sync**: Confirm [`.ai/doc-api.md`](doc-api.md) and `api/openapi/openapi.yaml` reflect any new or modified endpoints, Flight tickets/actions, metrics, or schemas. +7. **Progress Tracker**: Update [`.ai/progress-tracker.md`](progress-tracker.md) with session notes and completed items. diff --git a/.ai/architecture.md b/.ai/architecture.md new file mode 100644 index 0000000..8842ca7 --- /dev/null +++ b/.ai/architecture.md @@ -0,0 +1,150 @@ +# Architecture Context - Tabular Analytics Worker (`rag-tabular-analytics-worker`) + +## System Stack & Specifications + +| Layer | Technology | Role & Operational Characteristics | +| :--- | :--- | :--- | +| **Language Runtime** | Python 3.12+ (64-bit) | Host Arrow Flight daemon, `sqlglot` AST validation, async S3 hydration, metrics | +| **Memory Allocator** | `jemalloc` (`LD_PRELOAD`) | Mitigates native heap fragmentation across rapid Arrow IPC allocation cycles | +| **IPC & Transport** | Apache Arrow Flight (`pyarrow.flight`) | HTTP/2 gRPC server streaming zero-copy Arrow `RecordBatch` results (`DoGet`, `DoAction`) | +| **SQL Engine** | DuckDB (`duckdb.connect(":memory:")`) | Ephemeral in-memory C++ vectorized analytical engine inside isolated child processes | +| **Worker Pool** | `pebble.ProcessPool` / `multiprocessing` supervisor (`max_tasks=100`) | Disposable subprocess sandboxing with isolated per-worker `5s` `SIGKILL` + respawn (avoids stdlib `BrokenProcessPool`) | +| **SQL Security** | `sqlglot` | Pure-Python deterministic AST whitelisting ($\le 2\text{ ms}$) before subprocess dispatch | +| **Tier-1 Cache** | Disk-backed `10 GiB` NVMe `emptyDir` (`medium: ""`) | Content-addressed `{hash}.parquet` cache with single-flight coalescing, in-memory LRU, & atomic `os.replace` | +| **Object Storage** | AWS S3 (`aiobotocore` / `s3fs`) | Range-read streaming over internal VPC endpoints on cache miss (no monolithic heap buffering) | +| **Ops & Doc API** | HTTP Ops Server + OpenAPI 3.1 | `/healthz`, `/ready`, `/metrics`, `/openapi.yaml`, `/docs` | +| **Autoscaling** | Kubernetes + KEDA `ScaledObject` | Scales `4..24` replicas on `tabular_worker_active_queries >= 4` per pod | + +--- + +## Production-Grade 5-Layer Backend Architecture + +Aligned with `Production Grade Backend Concepts.pdf`: + +### Layer 1: API & Routing +- **Entry Points (`cmd/worker/main.py`)**: + - Initializes 12-factor environment configuration (`internal/config/settings.py`), preloads `ProcessPoolExecutor` child workers, warms/verifies the NVMe/tmpfs cache directory, starts the Apache Arrow Flight gRPC server (`FlightServerBase`) and HTTP operational/OpenAPI server (`/healthz`, `/ready`, `/metrics`, `/openapi.yaml`, `/docs`), and handles `SIGINT`/`SIGTERM` for graceful connection draining. +- **Endpoints & Flight RPCs (`api/flight/`, `internal/flight/`)**: + - **Arrow Flight `DoGet(Ticket)`**: Accepts a JSON/Protobuf-encoded ticket (`sql`, `tenant_id`, `table_hash`, `table_alias`, `request_id`), validates the SQL AST, resolves/hydrates the Parquet table, dispatches to the subprocess pool, and streams zero-copy `RecordBatch` frames. + - **Arrow Flight `DoAction(Action)`**: Supports `cache_warm` (pre-fetch `{hash}.parquet` from S3 into NVMe/tmpfs) and `health_check`. + - **HTTP Operational & OpenAPI 3.1 Endpoints**: `GET /healthz`, `GET /ready`, `GET /metrics`, `GET /openapi.yaml`, `GET /docs`. +- **Middleware & Interception**: + - Request/correlation ID extraction (`X-Request-ID` / Flight call metadata), structured JSON logging with strict secret/PII redaction, and OpenTelemetry span propagation. +- **Input Validation & Sanitization**: + - Pydantic/schema validation on Flight tickets (non-empty `tenant_id`, valid SHA-256 `table_hash`, bounded SQL length) + deterministic `sqlglot` AST whitelisting ($\le 2\text{ ms}$). +- **Rate Limiting & Backpressure**: + - Bounded in-flight concurrency semaphore (`MAX_CONCURRENT_QUERIES`) aligned with `POOL_SIZE` and KEDA scaling target (`tabular_worker_active_queries`), returning gRPC `RESOURCE_EXHAUSTED` when saturated. +- **Idempotency**: + - Content-addressed `{hash}.parquet` local cache keys and read-only `SELECT` execution guarantee idempotent retries from `rag-search-orchestrator`. +- **Sync/Async Event Loop Bridge**: + - `pyarrow.flight.FlightServerBase` runs synchronous C++ gRPC threads; `cmd/worker/main.py` bridges S3 async hydration (`aiobotocore`) via a dedicated `asyncio` event-loop thread (`asyncio.run_coroutine_threadsafe`). + +### Layer 2: Data Persistence & State +- **Stateless Compute Tier (No Transactional DB)**: + - `rag-tabular-analytics-worker` owns no relational database state; permanent schema cataloging and metadata belong to `rag-kb-catalog-service`. +- **Tier-1 Local Artifact Caching (`internal/storage/cache_manager.py`)**: + - Stores content-addressed `{hash}.parquet` files on a disk-backed `10 GiB` NVMe `emptyDir` (`medium: ""`). **Never** mount a `10 GiB` `tmpfs` (`medium: Memory`) inside a `6.0 GiB` cgroup pod (Linux charges `tmpfs` to container RAM, causing `OOMKill`); any dev/fallback `tmpfs` must be capped at $\le 1.0\text{ GiB}$. + - Uses single-flight download coalescing (per-`table_hash` async lock), atomic temp-file-then-rename (`os.replace`) writes, lock-free concurrent readers, and an in-memory LRU index (independent of `noatime` mounts). +- **S3 Range-Read Streaming (`internal/storage/s3_streamer.py`)**: + - Fetches un-cached Parquet artifacts from S3 over internal VPC endpoints using ranged reads (`aiobotocore`/`s3fs`), streaming chunks to disk without monolithic byte-array buffering on the Python heap. +- **Query Bounds & Chunked IPC**: + - Aggregate DuckDB budget (`<= 2.5 GiB` total across `POOL_SIZE` workers, e.g., `memory_limit = floor(2.5 GiB / POOL_SIZE)`) bounds only DuckDB's buffer manager, not total process RSS. Enforce a per-worker process memory boundary (e.g., `resource.setrlimit` `RLIMIT_AS`/`RLIMIT_DATA` in the child or supervisor RSS monitoring with `SIGKILL` + respawn) so DuckDB buffers, complex aggregate/hash state, query result batches, and Python/PyArrow overhead stay within the shared `6.0 GiB` pod cgroup budget, alongside the `5s` hard timeout. + - Subprocesses stream Arrow `RecordBatchReader` chunks over pipes/Unix sockets (no `/dev/shm`, avoiding `SIGBUS`; no `fetch_arrow_table()` full materialization). + +### Layer 3: Architecture & Scale +- **Process-Isolated Disposable Sandboxing (`internal/pool/`)**: + - `pebble.ProcessPool` (or custom `multiprocessing.Process` supervisor) with `max_tasks=100` isolates native DuckDB C++ memory and panics from the host Flight daemon (`ADR-001`). + - `circuit_breaker.py` monitors per-task execution deadlines (`5s`), `SIGKILL`ing only the offending child process and immediately respawning a replacement without breaking sibling workers (`ADR-001`). +- **Memory Partitioning Blueprint (`ADR-003`)**: + - **Pod Cgroup Limit**: `6.0 GiB` (`4.0 GiB` request); CPU `2,000m` request / `4,000m` limit. + - **DuckDB Aggregate Engine Ceiling**: `2.5 GiB` total across active pool workers (`SET memory_limit = floor(2.5GB / POOL_SIZE)` per child). + - **Host Daemon & Arrow Headroom**: $\ge 2.0\text{ GiB}$ reserved for the Python Flight daemon, PyArrow IPC buffers, AST syntax trees, and gRPC TCP frames. + - **OS Page Cache & Buffer Overhead**: `1.5 GiB` margin below the `6.0 GiB` cgroup limit to prevent Linux OOM kills (`SIGKILL` exit 137). +- **High Availability & KEDA Autoscaling (`ADR-004`)**: + - KEDA `ScaledObject` scales `4..24` replicas based on `tabular_worker_active_queries >= 4` per pod across compute-optimized nodes (`c6i.xlarge` / `c7g.xlarge`). +- **Creational Design Patterns**: + - Explicit dependency wiring in `cmd/worker/main.py` for config, cache manager, S3 streamer, AST validator, process pool supervisor, and telemetry registries. + +### Layer 4: Security & Identity +- **Tenant Isolation**: + - Every Flight ticket carries `tenant_id`, `table_hash`, `s3_bucket`, and `s3_key`; `table_resolver.py` validates `s3_bucket` against an allowlist and enforces `s3_key == f"artifacts/{tenant_id}/{table_hash}.parquet"` to prevent cross-tenant confused-deputy reads. +- **Deterministic SQL AST Whitelisting (`ADR-005`, `internal/validator/`)**: + - `sqlglot` permits only read-only `Select`, `With` (CTEs), and `Union` (`UNION`/`INTERSECT`/`EXCEPT`) constructs and pre-approved functions; blocks DML/DDL, `PRAGMA`, `SET`, `RESET`, `COPY ... TO`, `ATTACH`, `DETACH`, `LOAD`, `INSTALL`, `SUMMARIZE`, `DESCRIBE`, `SHOW`, `PIVOT`, and DuckDB filesystem functions (`read_csv`, `read_parquet`, `scan_parquet`, `write_csv`), while relying on the `5s` timeout for Cartesian explosion protection. +- **DuckDB Sandbox Locks (`internal/sandbox/security_profile.py`)**: + - Targeting pinned `duckdb >= 1.2.0` (which supports `allowed_paths`), `duckdb.connect(":memory:")` sets `allowed_paths = ['']` (permitting only the single validated `{hash}.parquet` cache file) before binding the view and immediately locking `enable_external_access=false`, `lock_configuration=true`, `temp_directory=''`, `threads=2`, and per-worker `memory_limit` prior to executing user SQL. +- **Secured Logging & Container Hardening**: + - Structured JSON logs with redaction of credentials, tokens, presigned S3 query params, and PII; non-root container with `jemalloc` preloaded. + +### Layer 5: Observability & Quality +- **Structured Logging**: Machine-parseable JSON logs with `trace_id`, `request_id`, `tenant_id`, `table_hash`, `cache_state`, and `duration_ms`. +- **Prometheus Observability Matrix (`internal/telemetry/metrics.py`)**: + - `tabular_worker_active_queries` (Gauge — primary KEDA metric) + - `tabular_worker_query_duration_seconds` (Histogram partitioned by `warm_cache` vs `cold_s3`) + - `tabular_worker_cache_hit_ratio` (Gauge) + - `tabular_worker_cache_hits_total` / `tabular_worker_cache_misses_total` (Counters for multi-pod PromQL aggregation) + - `tabular_worker_cache_bytes_used` (Gauge) + - `tabular_worker_subprocess_recycles_total` (Counter partitioned by `TIMEOUT`, `MEMORY_LIMIT`, `TASK_LIMIT`) + - `tabular_worker_flight_stream_bytes_total` (Counter) + - `tabular_worker_ast_rejections_total` (Counter partitioned by violated rule type) +- **Distributed Tracing (`internal/telemetry/tracing.py`)**: OpenTelemetry trace decorators across Flight ingress, AST validation, S3 hydration, and subprocess execution. +- **Health Checks & Graceful Shutdown**: `/healthz` (liveness) and `/ready` (readiness verifying process pool & cache mount availability), plus graceful drain on `SIGTERM`. + +--- + +## Repository Folder Structure + +``` +rag-tabular-analytics-worker/ +├── cmd/ +│ └── worker/ +│ └── main.py # Primary Arrow Flight + Ops HTTP daemon entry point +├── api/ +│ ├── flight/ +│ │ ├── schema.py # Flight descriptor schemas, tickets, error mappings +│ │ └── actions.py # Flight action definitions (cache_warm, health_check) +│ └── openapi/ +│ └── openapi.yaml # OpenAPI 3.1 operational & control API specification +├── internal/ +│ ├── config/ +│ │ └── settings.py # Pod configurations, memory ceilings, paths, timeouts +│ ├── flight/ +│ │ ├── server.py # Apache Arrow FlightServerBase implementation +│ │ └── handlers.py # Ticket resolution and subprocess dispatch handlers +│ ├── pool/ +│ │ ├── process_manager.py # ProcessPoolExecutor supervisor with lifetime recycling +│ │ ├── worker_task.py # Child process worker task (executes sandboxed DuckDB) +│ │ └── circuit_breaker.py # 5-second deadline enforcement and process killer +│ ├── sandbox/ +│ │ ├── duckdb_runtime.py # Ephemeral duckdb.connect(':memory:') builder +│ │ └── security_profile.py # DuckDB setting locks (enable_external_access=false) +│ ├── validator/ +│ │ ├── ast_validator.py # Primary sqlglot validation engine +│ │ ├── rules.py # Read-only rules, join limits, banned function sets +│ │ └── exceptions.py # ValidationError hierarchy +│ ├── storage/ +│ │ ├── cache_manager.py # Local NVMe/tmpfs Parquet file cache with LRU eviction +│ │ ├── s3_streamer.py # S3 ranged read fetcher for un-cached Parquet tables +│ │ └── table_resolver.py # Maps query aliases to validated local file descriptors +│ └── telemetry/ +│ ├── metrics.py # Prometheus metrics (tabular_worker_active_queries, etc.) +│ └── tracing.py # OpenTelemetry distributed trace decorators +├── deployments/ +│ ├── docker/ +│ │ └── Dockerfile # Multi-stage image with jemalloc preloaded +│ └── helm/ +│ ├── Chart.yaml +│ ├── values.yaml # 6 GiB RAM limit, NVMe/tmpfs mounts, KEDA config +│ └── templates/ +│ ├── deployment.yaml +│ ├── service.yaml +│ └── scaledobject.yaml # KEDA ScaledObject driven by in-flight queries +├── test/ +│ ├── unit/ # AST validation rules, cache eviction logic +│ ├── integration/ # Arrow Flight IPC streaming with mock S3 and subprocesses +│ ├── security/ # SQL injection, sandbox escape, PRAGMA exploit tests +│ ├── chaos/ # Forced child SIGKILL tests, OOM leak soak tests +│ └── fixtures/ # Normalized test Parquet datasets and queries +├── pyproject.toml +├── Makefile +└── README.md +``` diff --git a/.ai/code-standards.md b/.ai/code-standards.md new file mode 100644 index 0000000..7d75e9b --- /dev/null +++ b/.ai/code-standards.md @@ -0,0 +1,86 @@ +# Code Standards - Tabular Analytics Worker (`rag-tabular-analytics-worker`) + +## General Principles + +- **Stateless Compute & Subprocess Isolation**: Never execute DuckDB queries inside the main Flight daemon process. Always dispatch to a `pebble.ProcessPool` (or custom `multiprocessing.Process` supervisor) with `max_tasks=100` so native C++ faults, memory spikes, or isolated single-worker `SIGKILL` cancellations never crash the Flight server or break sibling workers (avoiding stdlib `BrokenProcessPool`). +- **Fail-Fast on Corrupt or Unsafe Input**: Reject invalid tickets, malformed schemas, or non-whitelisted SQL ASTs within $\le 2\text{ ms}$ using explicit `ValidationError` exceptions mapped to canonical gRPC `INVALID_ARGUMENT` / `PERMISSION_DENIED` status codes. +- **Root-Cause Focus**: Never mask race conditions or slow queries with arbitrary `time.sleep` calls; use bounded locks, atomic file renames (`os.replace`) in the cache, and deterministic `5s` subprocess circuit breakers. + +--- + +## Python 3.12+ & Code Quality Standards + +- **Runtime Target**: Python 3.12+ (64-bit) with `jemalloc` preloaded (`LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so`). +- **Linter, Formatter & Type Checker**: + - Formatter: `ruff format --check .` (and `ruff format .`) + - Linter: `ruff check .` + - Type checker: `mypy cmd internal api` (strict typing on all public and internal boundaries) + - Security scanner: `bandit -r cmd internal api -ll` +- **Dependencies**: + - `duckdb` (sandboxed in-process analytical engine) + - `pyarrow` (Arrow Flight server & zero-copy IPC streaming) + - `sqlglot` (pure-Python deterministic AST parser & whitelist validator) + - `aiobotocore` / `s3fs` (S3 range-read hydration) + - `prometheus-client`, `opentelemetry-api` (observability) +- **Clean Architecture & Dependency Injection**: + - Wire configuration (`settings.py`), cache manager, S3 streamer, AST validator, process pool supervisor, and telemetry registries explicitly in `cmd/worker/main.py`. + - Avoid hidden mutable module-level globals; keep functions small, single-purpose, and well-typed with concise docstrings. + +--- + +## DuckDB Sandbox & SQL AST Invariants + +1. **Mandatory `sqlglot` AST Validation**: + - Every incoming SQL string MUST pass `ast_validator.validate(sql)` before reaching the worker pool. + - Only read-only `Select`, `With` (CTEs), and `Union` (`UNION`/`INTERSECT`/`EXCEPT`) constructs and pre-approved function namespaces are permitted. + - Block DML/DDL (`INSERT`, `UPDATE`, `DELETE`, `DROP`, `ALTER`, `CREATE`), administrative commands (`PRAGMA`, `SET`, `RESET`, `SUMMARIZE`, `DESCRIBE`, `SHOW`, `PIVOT`), disk/extension commands (`COPY ... TO`, `ATTACH`, `DETACH`, `LOAD`, `INSTALL`), and DuckDB file readers/writers (`read_csv`, `read_parquet`, `scan_parquet`, `write_csv`). +2. **Mandatory DuckDB Runtime Locks**: + - Every child worker must open `duckdb.connect(":memory:")`, bind the single validated local `{hash}.parquet` view (`table_resolver.py`), and immediately enforce before running user SQL: + - `SET enable_external_access = false` + - `SET temp_directory = ''` + - `SET memory_limit = ''` (aggregate DuckDB cap $\le 2.5\text{ GiB}$ per pod) + - `SET threads = 2` +3. **Hard Execution Deadline**: + - Every query runs under a `5s` timeout monitored by `circuit_breaker.py`; expired child processes are terminated individually via `SIGKILL` and replaced without disrupting sibling workers. + +--- + +## Memory & Tier-1 Cache Invariants + +1. **Zero Monolithic Heap Buffering & Single-Flight Hydration**: + - Never load entire multi-hundred-MB S3 Parquet objects into Python `bytes` on the heap; use per-`table_hash` single-flight coalescing, stream range-reads directly to temporary files on the disk-backed NVMe `emptyDir` mount, and atomically rename via `os.replace` to `{hash}.parquet` tracked in an in-memory LRU index. + - Never mount a `10 GiB` `tmpfs` (`medium: Memory`) inside a `6.0 GiB` pod (Linux charges `tmpfs` to container RAM); dev/fallback `tmpfs` must be $\le 1.0\text{ GiB}$. +2. **Memory Partitioning**: + - Pod limit: `6.0 GiB` | Aggregate DuckDB limit (`POOL_SIZE * worker_limit`): `2.5 GiB` | Host Daemon & Arrow headroom: $\ge 2.0\text{ GiB}$ | OS page cache margin: `1.5 GiB`. +3. **Chunked Arrow Flight Egress (No Full Table Materialization)**: + - Stream PyArrow `RecordBatch` chunks via `RecordBatchReader` / `FlightDataStream` over pipes or Unix sockets (never `/dev/shm`, and never `fetch_arrow_table()` full materialization) without converting rows to Python dicts or JSON. + +--- + +## Secured Logging Invariants + +- **Structured JSON Logging**: All logs must be emitted as machine-parseable JSON lines. Never use `print()` in production code (`cmd/`, `internal/`, `api/`). +- **Mandatory Correlation IDs**: Include `trace_id`, `request_id`, `tenant_id`, `table_hash`, `cache_state`, and `duration_ms` where applicable. +- **Strict Redaction**: + - Never log AWS credentials (`AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`), presigned S3 query parameters (`X-Amz-Signature`, `X-Amz-Credential`, `X-Amz-Security-Token`), `Authorization` / `Bearer` tokens, or sensitive tenant PII/PHI. + - Log only sanitized S3 bucket/key pairs (`s3_bucket`, `s3_key`) or content hashes (`table_hash`). + - Sanitize internal filesystem paths and stack traces before returning error messages over gRPC/HTTP. + +--- + +## Doc API Mandatory Maintenance + +- **Continuous Synchronization**: Whenever any HTTP endpoint, Arrow Flight ticket/action schema, error mapping, Prometheus metric, or config contract is added or modified, [`.ai/doc-api.md`](doc-api.md) and `api/openapi/openapi.yaml` **MUST** be updated immediately in the same change step, no matter how small the modification is, if need be. + +--- + +## Git Commit Standards + +- **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**. +- **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes: + - `feat:` for new features + - `fix:` for bug fixes + - `refactor:` for code refactoring + - `docs:` for documentation updates + - `tests:` for tests addition and modification + - `chore:` for maintenance tasks and environment setup diff --git a/.ai/doc-api.md b/.ai/doc-api.md new file mode 100644 index 0000000..8283493 --- /dev/null +++ b/.ai/doc-api.md @@ -0,0 +1,134 @@ +# Documented APIs & Contracts - Tabular Analytics Worker (`rag-tabular-analytics-worker`) + +This document is the authoritative specification for all public and internal interfaces of the `rag-tabular-analytics-worker`. + +> [!IMPORTANT] +> **Continuous Doc API & OpenAPI Updates**: This file and `api/openapi/openapi.yaml` MUST be kept synchronized on **every code change or addition**, regardless of how small the modification is, if need be. + +--- + +## 1. OpenAPI 3.1 HTTP Operational & Control Specification + +### Base URL +`http://localhost:8080` (Internal Service / Kubernetes Pod Ops Listener) + +### Endpoints Overview + +| Method | Endpoint | Description | Auth Scope | +| :--- | :--- | :--- | :--- | +| `GET` | `/healthz` | Kubernetes Liveness Probe (event loop & daemon alive) | Public | +| `GET` | `/ready` | Kubernetes Readiness Probe (checks worker pool & NVMe/tmpfs cache mount) | Public | +| `GET` | `/metrics` | Prometheus Metrics Scrape Endpoint | Public / Ops | +| `GET` | `/openapi.yaml` | Raw OpenAPI 3.1 YAML specification (`api/openapi/openapi.yaml`) | Public | +| `GET` | `/docs` | Swagger UI documentation interface | Public | + +--- + +### Detailed HTTP Endpoint Contracts + +#### 1. `GET /healthz` +Verifies the Python host daemon process and async event loop are alive. +- **Response `200 OK`**: +```json +{ + "status": "ok", + "service": "rag-tabular-analytics-worker" +} +``` + +#### 2. `GET /ready` +Verifies the `ProcessPoolExecutor` supervisor has active pre-allocated worker capacity and the Tier-1 NVMe/tmpfs cache directory is mounted and writable. +- **Response `200 OK`**: +```json +{ + "status": "ready", + "checks": { + "process_pool": "healthy", + "cache_mount": "writable" + } +} +``` +- **Response `503 Service Unavailable`** (draining or unhealthy): +```json +{ + "status": "draining", + "checks": { + "drain": "shutting_down" + } +} +``` +or when a dependency check fails: +```json +{ + "status": "not_ready", + "checks": { + "process_pool": "unhealthy", + "cache_mount": "writable" + } +} +``` + +#### 3. `GET /metrics` +Exposes Prometheus text exposition metrics (`text/plain; version=0.0.4`). + +--- + +## 2. Apache Arrow Flight IPC Contract (`api/flight/`) + +### Transport +- **Protocol**: Apache Arrow Flight over HTTP/2 gRPC (`FlightServerBase`). +- **Primary Caller**: `rag-search-orchestrator` (Go). +- **Egress Format**: Zero-copy chunked Arrow `RecordBatch` stream (no JSON row serialization). +- **Distributed Tracing Headers**: Supports W3C `traceparent` and `tracestate` gRPC metadata headers on `DoGet` and `DoAction` for OpenTelemetry context propagation (`trace_id` log correlation). + +### `DoGet(context, ticket)` — Execute Sandboxed Analytical Query +- **Ticket Payload (`api/flight/schema.py`)**: +```json +{ + "request_id": "req_01JXYZ1234567890", + "tenant_id": "tenant-corp-01", + "sql": "SELECT region, SUM(revenue) AS total_rev FROM tbl GROUP BY region", + "table_alias": "tbl", + "table_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "s3_bucket": "rag-tabular-artifacts", + "s3_key": "artifacts/tenant-corp-01/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.parquet" +} +``` +- **Error Mappings**: + - `INVALID_ARGUMENT` (gRPC): Malformed ticket payload or rejected `sqlglot` AST rule violation (`ValidationError`). + - `PERMISSION_DENIED` (gRPC): Cross-tenant `tenant_id`, unallowlisted `s3_bucket`, or canonical `s3_key` mismatch (`TenantIsolationError`). + - `DEADLINE_EXCEEDED` (gRPC): Query exceeded `5s` circuit breaker (`SIGKILL`ed by `circuit_breaker.py`). + - `INTERNAL` (gRPC): Unexpected child subprocess crash/OOM/panic (`SIGKILL`ed) or IPC failure. + - `RESOURCE_EXHAUSTED` (gRPC): Pod concurrency cap (`MAX_CONCURRENT_QUERIES`) saturated (backpressure). + - `NOT_FOUND` (gRPC): Target `{hash}.parquet` artifact missing in both local cache and S3. +- **Tenant & S3 Path Validation**: + - `s3_bucket` MUST match the configured allowlist, and `s3_key` MUST match `artifacts/{tenant_id}/{table_hash}.parquet`. + +### `DoAction(context, action)` (`api/flight/actions.py`) +| Action Type | Request Body (JSON) | Response | +| :--- | :--- | :--- | +| `cache_warm` | `{"tenant_id": "...", "table_hash": "...", "s3_bucket": "...", "s3_key": "..."}` | `{"status": "warmed", "bytes": 1048576}` | +| `health_check` | `{}` | `{"status": "ok", "active_workers": 4}` | + +--- + +## 3. Tier-1 Cache & S3 Storage Conventions + +- **Local Disk-Backed NVMe Path**: `/{s3_bucket}/{tenant_id}/{table_hash}.parquet` (`10 GiB` ceiling, LRU eviction). Cache hits are strictly scoped to the requesting `tenant_id` and `s3_bucket` (or verified against S3 object access before serving) so an artifact warmed by another tenant is never served and missing S3 objects preserve `NOT_FOUND`. +- **S3 Source Path**: `s3:///artifacts//.parquet` (hydrated via S3 range reads over internal VPC endpoints). + +--- + +## 4. Prometheus Observability Contracts (`internal/telemetry/metrics.py`) + +| Metric Name | Type | Labels | Purpose | +| :--- | :--- | :--- | :--- | +| `tabular_worker_active_queries` | Gauge | — | Concurrent in-flight queries (primary KEDA metric, target `>= 4` per pod) | +| `tabular_worker_query_duration_seconds` | Histogram | `cache_state` (`warm_cache`, `cold_s3`) | End-to-end query latency distribution | +| `tabular_worker_cache_hit_ratio` | Gauge | — | Local NVMe Parquet cache hit ratio | +| `tabular_worker_cache_hits_total` | Counter | — | Total local Parquet cache hits (for multi-pod PromQL aggregation) | +| `tabular_worker_cache_misses_total` | Counter | — | Total local Parquet cache misses (for multi-pod PromQL aggregation) | +| `tabular_worker_cache_bytes_used` | Gauge | — | Current local NVMe cache disk utilization | +| `tabular_worker_subprocess_recycles_total` | Counter | `reason` (`TIMEOUT`, `MEMORY_LIMIT`, `TASK_LIMIT`, `CRASH`) | Child worker recycling events | +| `tabular_worker_flight_stream_bytes_total` | Counter | — | Total zero-copy Arrow Flight bytes streamed | +| `tabular_worker_ast_rejections_total` | Counter | `rule` | Rejected SQL statements by violated AST rule | diff --git a/.ai/progress-tracker.md b/.ai/progress-tracker.md new file mode 100644 index 0000000..9df225c --- /dev/null +++ b/.ai/progress-tracker.md @@ -0,0 +1,55 @@ +# Progress Tracker & Implementation Roadmap - Tabular Analytics Worker (`rag-tabular-analytics-worker`) + +## 1. Calibrated Implementation Phases + +This roadmap tracks the implementation milestones detailed in `Tabular Analytics Worker Implementation Plan.pdf`: + +| Phase | Duration | Focus Area | Deliverables & Exit Gates | Status | +| :--- | :--- | :--- | :--- | :--- | +| **Phase 1: Subprocess Sandboxing & AST Validation** | Weeks 1–2 | `sqlglot` AST Security & DuckDB Process Pool | - `sqlglot` AST security filters (read-only `SELECT` grammar, statement bans, column alias checks, $\le 2\text{ ms}$).
- Disposable worker subprocess pool (`pebble.ProcessPool` / custom `multiprocessing` supervisor, `max_tasks=100`, keeping sibling queries serviceable on worker timeout per `ADR-001`) with lifecycle recycling.
- Ephemeral DuckDB `:memory:` runtime (`enable_external_access=false`, `memory_limit=floor(2.5GB / POOL_SIZE)` keeping combined worker limits within `ADR-003`'s `2.5 GiB` aggregate budget, `temp_directory=''`, `threads=2`). | Completed | +| **Phase 2: Disk-Backed NVMe Cache & Arrow Flight Daemon** | Weeks 3–4 | Arrow Flight Server & Tier-1 Storage Hydration | - Apache Arrow Flight server daemon (`FlightServerBase`) handling HTTP/2 gRPC tickets (`DoGet`, `DoAction`) and IPC streams.
- Local Parquet cache on disk-backed `emptyDir` / NVMe (`10 GiB`) with lock-free concurrent readers and LRU eviction.
- S3 range-read ingestion (`aiobotocore`/`s3fs`) avoiding monolithic Python heap buffering.
- OpenAPI 3.1 ops endpoints (`/healthz`, `/ready`, `/metrics`, `/openapi.yaml`, `/docs`). | Completed | +| **Phase 3: Subprocess Telemetry & KEDA Concurrency Integration** | Weeks 5–6 | Circuit Breaking, Metrics & Autoscaling | - Hard `5s` query execution deadlines with automated child `SIGKILL` and recycling.
- Prometheus metrics (`tabular_worker_active_queries`, `tabular_worker_query_duration_seconds`, `tabular_worker_cache_hit_ratio`, `tabular_worker_subprocess_recycles_total`, `tabular_worker_flight_stream_bytes_total`, `tabular_worker_ast_rejections_total`) and OpenTelemetry tracing.
- KEDA `ScaledObject` rules bound to `tabular_worker_active_queries >= 4` (`4..24` replicas). | Completed | +| **Phase 4: Chaos Hardening, Fuzzing & Canary Rollout** | Weeks 7–8 | Security Fuzzing, Soak Tests & Production | - SQL injection fuzz testing (AST bypasses, `PRAGMA`, file access, extension loads).
- Soak testing under $500$ concurrent query streams and $8$-hour `jemalloc` runs ($0$ OOMKills).
- Multi-stage `Dockerfile` (`jemalloc` preloaded), Helm charts, and ArgoCD canary rollout. | Completed | + +--- + +## 2. Architecture Decision Records (ADRs) + +- **ADR-001: Subprocess Worker Isolation**: Adopts disposable worker subprocesses (`pebble.ProcessPool` / custom `multiprocessing` supervisor, `max_tasks=100`) over in-process thread pooling or stdlib `ProcessPoolExecutor` to allow isolated per-worker `SIGKILL` without `BrokenProcessPool` and prevent native DuckDB panics from crashing the Arrow Flight daemon. +- **ADR-002: Dedicated NVMe Local Table Caching**: Implements an explicit disk-backed `10 GiB` NVMe `emptyDir` cache (`medium: ""`, avoiding `10 GiB` `tmpfs` inside a `6.0 GiB` pod) for content-addressed `{hash}.parquet` artifacts with single-flight coalescing, eliminating per-query S3 round-trips and meeting warm-cache latency targets ($P95 \le 45\text{ ms}$). +- **ADR-003: Defensive Memory Budgeting**: Rebalances pod allocations to `6.0 GiB` total container RAM (`4.0 GiB` request) with a `2.5 GiB` aggregate DuckDB limit (`floor(2.5GB / POOL_SIZE)` per child), providing $\ge 2.0\text{ GiB}$ headroom for PyArrow IPC buffers and the Python runtime plus `1.5 GiB` OS page cache margin. +- **ADR-004: In-Flight Concurrency Autoscaling**: Bases KEDA autoscaling rules on in-flight query operations (`tabular_worker_active_queries >= 4`) rather than raw HTTP/2 TCP connection counts. +- **ADR-005: Query-Level AST Whitelisting via `sqlglot`**: Uses `sqlglot` to enforce read-only `SELECT` queries before engine execution, guarding against Text-to-SQL prompt injection risks. + +--- + +## 3. Operational Runbooks + +### Runbook 1: Spikes in Warm-Cache Latency +1. Check the Prometheus metric `tabular_worker_cache_hit_ratio`. +2. If the hit ratio drops below `80%`, inspect NVMe/tmpfs local cache utilization (`tabular_worker_cache_bytes_used`). +3. Verify that the local cache volume has sufficient capacity to retain active tenant working sets without continuous LRU evictions. +4. If cache thrashing occurs, increase the pod's `emptyDir` or NVMe volume allocation in `deployments/helm/values.yaml`. + +### Runbook 2: Elevated Subprocess Recycling Rates +1. Inspect `tabular_worker_subprocess_recycles_total` partitioned by `reason` (`TIMEOUT`, `MEMORY_LIMIT`, `TASK_LIMIT`, `CRASH`). +2. If `TIMEOUT` dominates, analyze slow queries from OpenTelemetry spans to identify non-selective aggregation queries. +3. If `MEMORY_LIMIT` triggers recycles, verify whether upstream prompts are requesting broad queries across wide tables, and tune prompt constraints in `rag-prompt-registry-service`. + +### Runbook 3: AST Rejection Surges +1. Inspect `tabular_worker_ast_rejections_total` grouped by violated `rule`. +2. Determine whether upstream Text-to-SQL generation models are producing unsupported functions or syntax dialects. +3. Update few-shot prompt examples and output schema constraints in `rag-prompt-registry-service` to guide models toward standard DuckDB-compatible read-only syntax. + +--- + +## 4. Git Commit Standards + +- **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**. +- **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes: + - `feat:` for new features + - `fix:` for bug fixes + - `refactor:` for code refactoring + - `docs:` for documentation updates + - `tests:` for tests addition and modification + - `chore:` for maintenance tasks and environment setup diff --git a/.ai/project-overview.md b/.ai/project-overview.md new file mode 100644 index 0000000..9a6657c --- /dev/null +++ b/.ai/project-overview.md @@ -0,0 +1,71 @@ +# Tabular Analytics Worker (`rag-tabular-analytics-worker`) - Project Overview + +## Overview + +The **`rag-tabular-analytics-worker`** is a stateless, horizontally autoscaling Python 3.12+ execution pool within the horizontal compute tier. It executes structured analytical SQL queries over tabular Parquet artifacts extracted by upstream Document AI pools (`rag-document-ai-worker`), running inside sandboxed DuckDB engines with deterministic `sqlglot` AST whitelisting, hard memory ceilings, disabled external/disk access, and zero-copy Apache Arrow Flight IPC streaming to `rag-search-orchestrator`. + +``` +GO SEARCH ORCHESTRATOR (rag-search-orchestrator) + │ + │ 1. Arrow Flight DoGet(Ticket: validated SQL, tenant_id, table_s3_hash) + ▼ +┌───────────────────────────────────────────────────────────────────────────────┐ +│ rag-tabular-analytics-worker (Host Daemon) │ +│ - Apache Arrow FlightServerBase (HTTP/2 gRPC) + Ops HTTP (/healthz, /ready, │ +│ /metrics, /openapi.yaml, /docs) │ +│ - Deterministic sqlglot AST Validator (SELECT-only whitelist, <= 2ms) │ +│ - Tier-1 Local NVMe/tmpfs Parquet Cache ({hash}.parquet, 10 GiB, LRU) │ +│ - ProcessPoolExecutor Supervisor (max_tasks_per_child=100, 5s Circuit Breaker│ +└───────┬───────────────────────────────────────────────────────┬───────────────┘ + │ 2. Cache Miss: S3 Range Read (VPC Endpoint) │ 3. Subprocess Dispatch + ▼ ▼ +┌───────────────────────────────┐ ┌───────────────────────────────┐ +│ S3 / Cloud Object Storage │ │ Ephemeral Worker Subprocess │ +│ - Content-addressed Parquet │ │ - duckdb.connect(':memory:') │ +│ artifacts ({hash}.parquet) │ │ - enable_external_access=false│ +└───────────────────────────────┘ │ - memory_limit='2.5GB' │ + │ - temp_directory='', threads=2│ + └───────────────┬───────────────┘ + │ 4. Arrow RecordBatch + ▼ + Streamed via Flight IPC (Zero-Copy) +``` + +--- + +## Strategic Objectives + +1. **Process-Isolated Disposable Sandboxing**: Isolate DuckDB execution within dedicated ephemeral subprocesses (`max_tasks_per_child=100`) running ephemeral in-memory databases (`:memory:`), guaranteeing that native C++ faults, memory spikes, or query cancellations never crash the primary Flight server daemon. +2. **Deterministic SQL AST Whitelisting**: Enforce pre-execution AST validation via `sqlglot`, blocking DML, DDL, `PRAGMA`s, filesystem extensions, and administrative statements while relying on hard runtime timeouts (`5s`) for join and Cartesian explosion protection. +3. **Tiered Ingestion & Zero-Copy Streaming**: Stream query results directly to Go orchestrators (`rag-search-orchestrator`) via Apache Arrow Flight IPC without JSON conversion, sourcing data from local NVMe/tmpfs caches with S3 range reads. +4. **Decoupled Memory Envelopes**: Enforce explicit memory partitioning across container cgroups (`6.0 GiB` limit), the Python host daemon (`>= 2.0 GiB` headroom), and DuckDB native instances (`2.5 GiB` hard limit) to completely eliminate Linux `SIGKILL` (exit 137) evictions. +5. **Continuous Documentation & Contract Synchronization**: Keep `.ai/doc-api.md` and OpenAPI 3.1 specifications (`api/openapi/openapi.yaml`) continuously synchronized with every code change or addition. + +--- + +## Calibrated Quantitative SLOs & SLIs + +- **Warm-Cache Analytical Query Latency**: $P95 \le 45\text{ ms}$, $P99 \le 120\text{ ms}$ for aggregation, projection, and filter operations over cached tables $\le 1,000,000\text{ rows}$. +- **Cold-Fetch S3 Hydration Query Latency**: $P95 \le 250\text{ ms}$, $P99 \le 400\text{ ms}$ for un-cached Parquet artifacts requiring S3 retrieval over internal VPC endpoints. +- **AST Validation Latency**: All non-whitelisted SQL expressions rejected within $\le 2\text{ ms}$. +- **Arrow Flight Egress Throughput**: Minimum sustained streaming rate of $250\text{ MB/sec}$ per pod for active result-set record batches. +- **Fleet Availability**: $99.9\%$ cluster availability, delegating individual query retries and semantic fallbacks to the upstream Go Search Orchestrator (`rag-search-orchestrator`). +- **Container Stability**: $0$ container `OOMKill`s under sustained maximum concurrency (soak-tested under $500$ concurrent query streams and $8$-hour `jemalloc` runs). + +--- + +## Scope & Exclusions + +### In-Scope +- **Process-Isolated DuckDB Session Pool**: Lifecycle management of ephemeral child worker processes (`pebble.ProcessPool` / `multiprocessing` supervisor, `max_tasks=100`), binding isolated in-memory instances (`duckdb.connect(":memory:")`) locked with `enable_external_access=false`, `temp_directory=''`, aggregate `memory_limit <= 2.5GB` across `POOL_SIZE` workers, and `threads=2`. +- **SQL AST Security Validator**: Deterministic pre-execution AST checks via `sqlglot` permitting only read-only `Select`, `With` (CTEs), and `Union` constructs and pre-approved function namespaces. +- **Tier-1 Artifact Local Storage Cache**: Local caching of content-addressed Parquet artifacts (`{hash}.parquet`) on disk-backed `10 GiB` NVMe `emptyDir` mounts with single-flight coalescing, in-memory LRU eviction, lock-free concurrent readers, and S3 HTTP range-read streaming on cache miss. +- **Arrow Flight IPC Ingress & Egress**: Apache Arrow Flight server daemon accepting tickets, coordinating subprocess dispatch, and streaming chunked Arrow `RecordBatch` instances. +- **Subprocess Circuit Breaking**: Cancellation monitors that `SIGKILL` and respawn child worker subprocesses exceeding execution deadlines (`5s`). +- **Operational HTTP & Telemetry Endpoints**: `/healthz`, `/ready`, `/metrics` (Prometheus), `/openapi.yaml`, and `/docs` documented via OpenAPI 3.1. + +### Exclusions (Explicitly Out-of-Scope) +- **Document Layout & Parquet Extraction**: Document parsing and tabular boundary extraction from raw spreadsheets/PDFs (owned by `rag-document-ai-worker`). +- **Natural Language to SQL Generation**: Generative Text-to-SQL translation and prompt tuning (owned by upstream LLM pipelines via `rag-prompt-registry-service`). +- **Transactional Writes & Long-Term Cataloging**: Permanent schema storage and document metadata updates (owned by `rag-kb-catalog-service` and PostgreSQL). +- **Vector Search & RRF Merging**: Ranking chunk embeddings and merging vector scores (owned by `rag-search-orchestrator` and `rag-inference-serving`). diff --git a/.ai/security.md b/.ai/security.md new file mode 100644 index 0000000..379dcee --- /dev/null +++ b/.ai/security.md @@ -0,0 +1,47 @@ +# Security Context - Tabular Analytics Worker (`rag-tabular-analytics-worker`) + +## Threat Model & Trust Boundaries + +- **Untrusted Text-to-SQL Payloads**: Queries arrive from `rag-search-orchestrator` after upstream LLM Text-to-SQL generation (`rag-prompt-registry-service`). Prompt injection or malformed SQL must be treated as hostile and blocked by deterministic `sqlglot` AST whitelisting ($\le 2\text{ ms}$) before reaching DuckDB. +- **Native Engine Isolation**: DuckDB C++ memory corruption, panics, or runaway cross-joins are isolated inside ephemeral child processes (`pebble.ProcessPool` / `multiprocessing` supervisor, `max_tasks=100`) with an isolated per-worker `5s` `SIGKILL` circuit breaker. +- **Disabled External & Disk Access**: Child DuckDB instances run strictly `:memory:`, bind only the single validated `{hash}.parquet` view, and immediately lock `enable_external_access=false` and `temp_directory=''` before executing user SQL, preventing arbitrary file reads, network exfiltration, or extension loading. +- **Tenant & S3 Path Isolation**: Every Flight ticket includes `tenant_id`, `table_hash`, `s3_bucket`, and `s3_key`. `table_resolver.py` enforces a `s3_bucket` allowlist and canonical `s3_key == f"artifacts/{tenant_id}/{table_hash}.parquet"` to prevent cross-tenant confused-deputy reads (raising gRPC `PERMISSION_DENIED`). +- **Service-to-Service Auth & S3 Read-Only VPC Access**: Arrow Flight connections from `rag-search-orchestrator` use mTLS (or signed service tokens) in addition to `NetworkPolicy`, and worker pods hold least-privilege IAM permissions (`s3:GetObject` only) routed exclusively through internal VPC S3 endpoints. + +--- + +## Secured Logging Invariants + +1. **Structured JSON Telemetry Only**: + - Emit machine-parseable JSON logs containing correlation fields (`trace_id`, `request_id`, `tenant_id`, `table_hash`, `cache_state`, `duration_ms`, `status`). + - Raw `print()` statements are prohibited in `cmd/`, `internal/`, and `api/`. +2. **Credentials, Tokens & Presigned URL Redaction**: + - Strictly redact `Authorization` headers, `Bearer` tokens, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, and presigned S3 query parameters (`X-Amz-Signature`, `X-Amz-Credential`, `X-Amz-Security-Token`). + - Log only sanitized bucket and object keys (`s3_bucket`, `s3_key`) or `{hash}.parquet` identifiers. +3. **PII/PHI & Error Trace Sanitization**: + - Never log raw query result rows, tenant PII/PHI, or unredacted SQL string literals containing sensitive values. + - Error responses returned via Arrow Flight (`gRPC INTERNAL` / `INVALID_ARGUMENT`) or HTTP must sanitize internal stack traces, local mount paths, and cluster hostnames. + +--- + +## Network & Container Hardening + +1. **Multi-Stage Non-Root Container with `jemalloc`**: + - Preload `jemalloc` globally (`LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so`). + - Run as a non-root user, drop all Linux capabilities (`capabilities: drop: ["ALL"]`), and mount only the `10 GiB` NVMe/`tmpfs` `emptyDir` cache path as writable. +2. **Kubernetes Network Policies**: + - **Ingress**: Restricted to `rag-search-orchestrator` on the Arrow Flight HTTP/2 gRPC port and cluster monitoring/probes on the Ops HTTP port (`/healthz`, `/ready`, `/metrics`). + - **Egress**: Restricted to internal VPC S3 endpoints (HTTPS `443`), OpenTelemetry collector, and cluster DNS (`kube-dns`/`CoreDNS` port `53`). General external internet egress is strictly denied. + +--- + +## Git Commit Standards + +- **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**. +- **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes: + - `feat:` for new features + - `fix:` for bug fixes + - `refactor:` for code refactoring + - `docs:` for documentation updates + - `tests:` for tests addition and modification + - `chore:` for maintenance tasks and environment setup diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..832233a --- /dev/null +++ b/.dockerignore @@ -0,0 +1,18 @@ +.git +.github +.venv +.mypy_cache +.pytest_cache +.ruff_cache +__pycache__ +*.pyc +*.pyo +*.pyd +*.egg-info +.ai +prompts +tasks +test +Makefile +*.md +!README.md diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..f6a4489 --- /dev/null +++ b/.env.example @@ -0,0 +1,64 @@ +# Environment configuration for rag-tabular-analytics-worker +# +# IMPORTANT: +# - `.env` is for LOCAL DEVELOPMENT ONLY. +# - Only changes to `.env.prod` affect `deployments/` (synced to +# `deployments/helm/values.yaml` via `python scripts/sync_env_to_deployments.py --sync`). +# +# Copy this file to .env and adjust values as needed. +# OS environment variables take precedence over values defined in .env. + +# Service identity & structured JSON logging +TABULAR_WORKER_SERVICE_NAME=rag-tabular-analytics-worker +TABULAR_WORKER_LOG_LEVEL=INFO + +# Subprocess pool & circuit breaker (ADR-001) +TABULAR_WORKER_POOL_SIZE=4 +TABULAR_WORKER_MAX_TASKS_PER_CHILD=100 +TABULAR_WORKER_QUERY_TIMEOUT_SECONDS=5.0 + +# DuckDB sandbox memory & concurrency limits (ADR-003) +# 2684354560 bytes = 2.5 GiB aggregate across POOL_SIZE workers +TABULAR_WORKER_DUCKDB_TOTAL_MEMORY_BYTES=2684354560 +TABULAR_WORKER_DUCKDB_THREADS=2 +TABULAR_WORKER_ARROW_BATCH_SIZE=8192 + +# SQL AST validator limits (ADR-005) +TABULAR_WORKER_MAX_SQL_LENGTH=8192 +TABULAR_WORKER_MAX_JOINS=5 + +# Arrow Flight & HTTP Ops Server endpoints +TABULAR_WORKER_FLIGHT_HOST=0.0.0.0 +TABULAR_WORKER_FLIGHT_PORT=8815 +TABULAR_WORKER_HTTP_HOST=0.0.0.0 +TABULAR_WORKER_HTTP_PORT=8080 + +# Tier-1 Disk-Backed NVMe Parquet Cache (ADR-002) +# 10737418240 bytes = 10 GiB +TABULAR_WORKER_CACHE_DIR=/var/cache/tabular-worker +TABULAR_WORKER_CACHE_MAX_BYTES=10737418240 + +# S3 VPC Endpoint Streaming & Tenant Allowlist +# Accepts JSON array (e.g. ["bucket-a","bucket-b"]) or comma-separated list +TABULAR_WORKER_S3_BUCKET_ALLOWLIST=[] +TABULAR_WORKER_S3_ENDPOINT_URL= +TABULAR_WORKER_S3_REGION=us-east-1 +TABULAR_WORKER_S3_CHUNK_SIZE_BYTES=8388608 +TABULAR_WORKER_S3_TIMEOUT_SECONDS=4.0 + +# Concurrency & KEDA Backpressure (ADR-004) +TABULAR_WORKER_MAX_CONCURRENT_QUERIES=4 + +# Optional POSIX worker address-space guardrail (bytes; leave empty to disable) +TABULAR_WORKER_WORKER_PROCESS_RLIMIT_BYTES= + +# OpenTelemetry Distributed Tracing +TABULAR_WORKER_OTEL_ENABLED=false +TABULAR_WORKER_OTEL_EXPORTER_ENDPOINT= +TABULAR_WORKER_OTEL_SERVICE_NAME=rag-tabular-analytics-worker + +# Optional extended soak test flags (read by pytest from the process environment, NOT from .env). +# Note: `-m soak` alone does not enable the endurance test; export these flags in your shell first: +# export RUN_SOAK_TESTS=1 SOAK_DURATION_SECONDS=28800 && pytest -v -m soak test/chaos +# RUN_SOAK_TESTS=0 +# SOAK_DURATION_SECONDS=28800 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..8818a51 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,287 @@ +name: rag-tabular-analytics-worker CI + +on: + push: + branches: + - main + - master + - rag-tabular-analytics-worker + - "feat/**" + - "fix/**" + pull_request: + branches: + - main + - master + - rag-tabular-analytics-worker + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +env: + PYTHON_VERSION: "3.12" + SERVICE_NAME: "rag-tabular-analytics-worker" + +jobs: + lint-and-typecheck: + name: Lint & Typecheck + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Python ${{ env.PYTHON_VERSION }} + uses: actions/setup-python@v5 + with: + python-version: ${{ env.PYTHON_VERSION }} + + - name: Install lint and typecheck tools + run: | + python -m pip install --upgrade pip + pip install ruff==0.11.0 mypy==1.15.0 + if [ -f pyproject.toml ]; then + pip install -e ".[dev]" || pip install -e . || true + fi + + - name: Run Ruff lint & format checks + run: | + set -euo pipefail + if [ -d cmd ] || [ -d internal ] || [ -d api ] || [ -d test ]; then + ruff check . + ruff format --check . + else + echo "No Python source directories yet; skipping ruff." + fi + + - name: Run Mypy static type checking + run: | + set -euo pipefail + TARGETS=() + for dir in cmd internal api; do + if [ -d "$dir" ] && [ -n "$(find "$dir" -type f \( -name '*.py' -o -name '*.pyi' \) -print -quit)" ]; then + TARGETS+=("$dir") + fi + done + if [ ${#TARGETS[@]} -gt 0 ]; then + mypy "${TARGETS[@]}" + else + echo "No Python source directories yet; skipping mypy." + fi + + security-and-logging-audit: + name: Security & Secured Logging Audit + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Python ${{ env.PYTHON_VERSION }} + uses: actions/setup-python@v5 + with: + python-version: ${{ env.PYTHON_VERSION }} + + - name: Install security audit tools + run: | + python -m pip install --upgrade pip + pip install bandit==1.8.3 pip-audit==2.8.0 + + - name: Run Bandit SAST scan + run: | + set -euo pipefail + TARGETS=() + for dir in cmd internal api; do + if [ -d "$dir" ] && [ -n "$(find "$dir" -type f \( -name '*.py' -o -name '*.pyi' \) -print -quit)" ]; then + TARGETS+=("$dir") + fi + done + if [ ${#TARGETS[@]} -gt 0 ]; then + bandit -r "${TARGETS[@]}" -ll + else + echo "No Python source directories yet; skipping bandit." + fi + + - name: Run dependency vulnerability audit (pip-audit) + run: | + set -euo pipefail + if [ -f pyproject.toml ]; then + pip install -e . + pip-audit + elif [ -f requirements.txt ]; then + pip-audit -r requirements.txt + else + echo "No pyproject.toml or requirements.txt yet; skipping pip-audit." + fi + + - name: Audit Secured Logging & DuckDB Sandbox Invariants + run: | + set -euo pipefail + SRC_DIRS=() + for dir in cmd internal api; do + if [ -d "$dir" ]; then + SRC_DIRS+=("$dir") + fi + done + + if [ ${#SRC_DIRS[@]} -eq 0 ]; then + echo "No Python source directories yet; skipping source audit." + exit 0 + fi + + # 1. Disallow raw print() in production code (must use structured JSON logger) + if grep -rnE '^\s*print\(' "${SRC_DIRS[@]}"; then + echo "ERROR: Raw print() detected in production code. Use structured JSON logging." + exit 1 + fi + + # 2. Disallow logging sensitive credential/signature fields + if grep -rnE '(logger|logging)\.[a-z]+\(.*(X-Amz-Signature|X-Amz-Credential|X-Amz-Security-Token|AWS_SECRET_ACCESS_KEY|presigned_url)' "${SRC_DIRS[@]}"; then + echo "ERROR: Potential unredacted secret or presigned URL parameter in log statement." + exit 1 + fi + + # 3. Enforce DuckDB sandbox lock (enable_external_access must never be enabled) + if grep -rnE 'enable_external_access\s*=\s*(true|True|1)' "${SRC_DIRS[@]}"; then + echo "ERROR: DuckDB enable_external_access must remain disabled (false) in sandbox." + exit 1 + fi + + echo "Secured logging and DuckDB sandbox audit passed." + + openapi-and-doc-api-sync: + name: OpenAPI 3.1 & Doc API Contract Sync + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Set up Python ${{ env.PYTHON_VERSION }} + uses: actions/setup-python@v5 + with: + python-version: ${{ env.PYTHON_VERSION }} + + - name: Verify .ai/doc-api.md exists and is populated + run: | + set -euo pipefail + test -s .ai/doc-api.md + grep -q "rag-tabular-analytics-worker" .ai/doc-api.md + grep -q "OpenAPI 3.1" .ai/doc-api.md + grep -q "Arrow Flight" .ai/doc-api.md + + - name: Validate OpenAPI 3.1 Specification + run: | + set -euo pipefail + if [ -f api/openapi/openapi.yaml ]; then + pip install --quiet openapi-spec-validator==0.7.1 pyyaml==6.0.2 + python -c " + import yaml + from openapi_spec_validator import validate + with open('api/openapi/openapi.yaml', 'r', encoding='utf-8') as f: + spec = yaml.safe_load(f) + assert str(spec.get('openapi', '')).startswith('3.1'), f'Expected OpenAPI 3.1.x, got {spec.get(\"openapi\")}' + validate(spec) + print('OpenAPI 3.1 specification is valid.') + " + else + echo "api/openapi/openapi.yaml not created yet; skipping OpenAPI validation." + fi + + - name: Enforce Doc API & OpenAPI Sync on API/Contract PR Changes + if: github.event_name == 'pull_request' + run: | + set -euo pipefail + BASE_SHA="${{ github.event.pull_request.base.sha }}" + HEAD_SHA="${{ github.event.pull_request.head.sha }}" + MERGE_BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA") + CHANGED_FILES=$(git diff --name-only "$MERGE_BASE" "$HEAD_SHA") + echo "Changed files:" + echo "$CHANGED_FILES" + + if echo "$CHANGED_FILES" | grep -qE '^(api/|internal/flight/handlers\.py|internal/telemetry/metrics\.py)'; then + if ! echo "$CHANGED_FILES" | grep -qE '^(\.ai/doc-api\.md|api/openapi/openapi\.yaml)$'; then + echo "ERROR: API, Flight handler, or Prometheus metrics changed without updating .ai/doc-api.md or api/openapi/openapi.yaml." + exit 1 + fi + fi + echo "Doc API & OpenAPI contract sync check passed." + + test-suites: + name: Test Suites (Unit, Security, Integration, Chaos) + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Python ${{ env.PYTHON_VERSION }} + uses: actions/setup-python@v5 + with: + python-version: ${{ env.PYTHON_VERSION }} + + - name: Install test dependencies + run: | + python -m pip install --upgrade pip + pip install pytest==8.3.5 pytest-asyncio==0.25.3 pytest-cov==6.0.0 + if [ -f pyproject.toml ]; then + pip install -e ".[test]" || pip install -e . || true + fi + + - name: Run Unit Tests (AST Validator, Cache LRU) + run: | + set -euo pipefail + if [ -d test/unit ]; then + pytest -v test/unit + else + echo "No test/unit directory yet; skipping unit tests." + fi + + - name: Run Security & AST Fuzzing Tests + run: | + set -euo pipefail + if [ -d test/security ]; then + pytest -v test/security + else + echo "No test/security directory yet; skipping security tests." + fi + + - name: Run Integration Tests (Arrow Flight IPC & Mock S3) + run: | + set -euo pipefail + if [ -d test/integration ]; then + pytest -v test/integration + else + echo "No test/integration directory yet; skipping integration tests." + fi + + - name: Run Chaos & Subprocess Isolation Tests (SIGKILL, 5s Timeout) + run: | + set -euo pipefail + if [ -d test/chaos ]; then + pytest -v test/chaos + else + echo "No test/chaos directory yet; skipping chaos tests." + fi + + container-and-helm-lint: + name: Container & Helm Manifest Lint + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Lint Dockerfile (Hadolint) + if: hashFiles('deployments/docker/Dockerfile') != '' + uses: hadolint/hadolint-action@v3.1.0 + with: + dockerfile: deployments/docker/Dockerfile + + - name: Lint & Render Helm Chart + if: hashFiles('deployments/helm/Chart.yaml') != '' + run: | + set -euo pipefail + helm lint deployments/helm + helm template rag-tabular-analytics-worker deployments/helm > /dev/null + echo "Helm lint and template rendering passed." diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml new file mode 100644 index 0000000..bb8d20b --- /dev/null +++ b/.github/workflows/deploy.yml @@ -0,0 +1,109 @@ +name: Deploy rag-tabular-analytics-worker + +on: + workflow_dispatch: + inputs: + environment: + description: "Target deployment environment" + required: true + default: "staging" + type: choice + options: + - staging + - prod + image_tag: + description: "Container image tag to deploy" + required: false + default: "0.1.0" + type: string + +permissions: + contents: read + +concurrency: + group: deploy-${{ inputs.environment }} + cancel-in-progress: false + +jobs: + deploy: + name: Inject Secrets & Deploy (${{ inputs.environment }}) + runs-on: ubuntu-latest + environment: ${{ inputs.environment }} + env: + RELEASE_NAME: rag-tabular-analytics-worker + SECRET_NAME: rag-tabular-analytics-worker-secrets + KUBE_NAMESPACE: ${{ inputs.environment == 'prod' && 'rag-tabular-prod' || 'rag-tabular-staging' }} + VALUES_OVERRIDE: ${{ inputs.environment == 'prod' && 'deployments/helm/values-prod.yaml' || 'deployments/helm/values-staging.yaml' }} + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up kubectl + uses: azure/setup-kubectl@v4 + + - name: Set up Helm + uses: azure/setup-helm@v4 + + - name: Configure kubeconfig + env: + KUBE_CONFIG_DATA: ${{ secrets.KUBE_CONFIG_DATA }} + run: | + set -euo pipefail + if [ -z "${KUBE_CONFIG_DATA}" ]; then + echo "ERROR: KUBE_CONFIG_DATA secret is not configured for environment '${{ inputs.environment }}'." >&2 + exit 1 + fi + mkdir -p "${HOME}/.kube" + printf '%s' "${KUBE_CONFIG_DATA}" | base64 --decode > "${HOME}/.kube/config" + chmod 600 "${HOME}/.kube/config" + + - name: Ensure target namespace exists + run: | + set -euo pipefail + kubectl create namespace "${KUBE_NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f - + + - name: Inject Kubernetes Secret (Cloudflare R2 S3 credentials) + env: + R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }} + R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }} + OTEL_EXPORTER_OTLP_HEADERS: ${{ secrets.OTEL_EXPORTER_OTLP_HEADERS }} + run: | + set -euo pipefail + if [ -z "${R2_ACCESS_KEY_ID}" ] || [ -z "${R2_SECRET_ACCESS_KEY}" ]; then + echo "ERROR: R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY secrets must be set." >&2 + exit 1 + fi + + ENV_FILE="$(mktemp)" + trap 'rm -f "${ENV_FILE}"' EXIT + chmod 600 "${ENV_FILE}" + + { + printf 'AWS_ACCESS_KEY_ID=%s\n' "${R2_ACCESS_KEY_ID}" + printf 'AWS_SECRET_ACCESS_KEY=%s\n' "${R2_SECRET_ACCESS_KEY}" + if [ -n "${OTEL_EXPORTER_OTLP_HEADERS}" ]; then + printf 'OTEL_EXPORTER_OTLP_HEADERS=%s\n' "${OTEL_EXPORTER_OTLP_HEADERS}" + fi + } > "${ENV_FILE}" + + kubectl create secret generic "${SECRET_NAME}" \ + --namespace "${KUBE_NAMESPACE}" \ + --from-env-file="${ENV_FILE}" \ + --dry-run=client -o yaml | kubectl apply -f - + + - name: Deploy Helm chart + env: + IMAGE_TAG: ${{ inputs.image_tag }} + run: | + set -euo pipefail + if [[ ! "${IMAGE_TAG}" =~ ^[A-Za-z0-9_][A-Za-z0-9._-]{0,127}$ ]]; then + echo "ERROR: Invalid image_tag '${IMAGE_TAG}'." >&2 + exit 1 + fi + helm upgrade --install "${RELEASE_NAME}" deployments/helm \ + --namespace "${KUBE_NAMESPACE}" \ + --values deployments/helm/values.yaml \ + --values "${VALUES_OVERRIDE}" \ + --set-string "image.tag=${IMAGE_TAG}" \ + --wait \ + --timeout 5m diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b0e186d --- /dev/null +++ b/.gitignore @@ -0,0 +1,70 @@ +# Project planning & specification PDFs (local reference only) +tasks/ + +# Python bytecode & caches +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# Virtual environments +.venv/ +venv/ +ENV/ +env/ +.conda/ + +# Testing, coverage & type-checking caches +.pytest_cache/ +.coverage +.coverage.* +htmlcov/ +.tox/ +.nox/ +.mypy_cache/ +.ruff_cache/ +.hypothesis/ +coverage.xml +*.cover + +# DuckDB & local Parquet cache artifacts +*.duckdb +*.duckdb.wal +*.parquet +tmp_cache/ +nvme_cache/ + +# Environment & secrets +.env +.env.* +!.env.example +*.pem +*.key +*.crt + +# IDE & OS +.vscode/ +.idea/ +*.swp +*.swo +*~ +.DS_Store +Thumbs.db diff --git a/GEMINI.MD b/GEMINI.MD new file mode 100644 index 0000000..fa3f0f4 --- /dev/null +++ b/GEMINI.MD @@ -0,0 +1,47 @@ +# GEMINI.MD — Tabular Analytics Worker (`rag-tabular-analytics-worker`) + +This repository implements **`rag-tabular-analytics-worker`**, a stateless, horizontally autoscaling Python 3.12+ Apache Arrow Flight microservice executing sandboxed DuckDB analytical queries over Parquet artifacts. + +--- + +## Mandatory Context & Reading Order + +Before planning, modifying code, or making any architectural decision, **always read and follow**: + +1. [`.ai/agents.md`](.ai/agents.md) — **Primary AI Agent Persona, System Role, and Execution Protocol** +2. [`.ai.md`](.ai.md) — **Root Application Building Context & Non-Negotiable Invariants** +3. [`.ai/project-overview.md`](.ai/project-overview.md) — Product definition, strategic objectives, SLOs/SLIs, and scope boundaries +4. [`.ai/architecture.md`](.ai/architecture.md) — 5-layer backend architecture, memory partitioning (`6.0 GiB` pod / `2.5 GiB` DuckDB), and folder structure +5. [`.ai/doc-api.md`](.ai/doc-api.md) — Authoritative OpenAPI 3.1 HTTP ops endpoints, Arrow Flight IPC (`DoGet`/`DoAction`) contracts, and Prometheus metrics +6. [`.ai/code-standards.md`](.ai/code-standards.md) — Python 3.12+, `ruff`, `mypy`, `pebble.ProcessPool` / `multiprocessing` supervisor (`max_tasks=100`, keeping sibling queries serviceable after a worker timeout per `ADR-001`), `sqlglot` AST checks, and `jemalloc` hygiene +7. [`.ai/security.md`](.ai/security.md) — Threat model, DuckDB sandbox locks (`enable_external_access=false`, `temp_directory=''`), and Secured Logging invariants +8. [`.ai/ai-workflow-rules.md`](.ai/ai-workflow-rules.md) — Spec-driven workflow, human-in-the-loop rules, and pre-commit verification checklist +9. [`.ai/progress-tracker.md`](.ai/progress-tracker.md) — 4-phase roadmap (Weeks 1–8), `ADR-001`–`ADR-005`, and operational runbooks + +--- + +## Non-Negotiable Engineering Rules + +1. **Continuous Doc API & OpenAPI 3.1 Synchronization**: + - [`.ai/doc-api.md`](.ai/doc-api.md) and `api/openapi/openapi.yaml` **MUST** be kept updated on **every code change or addition**, no matter how small, whenever endpoints, Flight tickets/actions, error mappings, metrics, or schemas are touched. +2. **Secured Logging**: + - Emit structured JSON logs only (never raw `print()` in `cmd/`, `internal/`, or `api/`). + - Strictly redact authorization headers, bearer tokens, AWS credentials, presigned S3 query parameters (`X-Amz-Signature`, `X-Amz-Credential`, `X-Amz-Security-Token`), and sensitive tenant PII/PHI or raw SQL literals before emission. +3. **Process-Isolated DuckDB Sandboxing**: + - Never run DuckDB in the main Arrow Flight daemon process. Always execute inside an isolated worker pool (`pebble.ProcessPool` / `multiprocessing` supervisor, `max_tasks=100`) with `duckdb.connect(":memory:")`, `enable_external_access=false` (applied after binding the single validated `{hash}.parquet` view), `temp_directory=''`, aggregate `memory_limit <= 2.5GB` across `POOL_SIZE` workers, `threads=2`, and a `5s` per-worker `SIGKILL` circuit breaker. + - Validate all SQL queries with `sqlglot` AST whitelisting ($\le 2\text{ ms}$) prior to subprocess dispatch. +4. **CI & Quality Gates**: + - All changes must pass the GitHub Actions gates in [`.github/workflows/ci.yml`](.github/workflows/ci.yml): `lint-and-typecheck`, `security-and-logging-audit`, `OpenAPI 3.1 & Doc API Contract Sync`, `test-suites`, and `container-and-helm-lint`. + +--- + +## Git Commit Standards + +- **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**. +- **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes: + - `feat:` for new features + - `fix:` for bug fixes + - `refactor:` for code refactoring + - `docs:` for documentation updates + - `tests:` for tests addition and modification + - `chore:` for maintenance tasks and environment setup diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..66a60b5 --- /dev/null +++ b/Makefile @@ -0,0 +1,31 @@ +.PHONY: format lint typecheck security test-unit test-security test-chaos test check + +PYTHON ?= python + +format: + $(PYTHON) -m ruff format . + +lint: + $(PYTHON) -m ruff check . + $(PYTHON) -m ruff format --check . + +typecheck: + $(PYTHON) -m mypy cmd internal api + +security: + $(PYTHON) -m bandit -r cmd internal api -ll + $(PYTHON) -m pip_audit + +test-unit: + $(PYTHON) -m pytest -v test/unit + +test-security: + $(PYTHON) -m pytest -v test/security + +test-chaos: + $(PYTHON) -m pytest -v test/chaos + +test: + $(PYTHON) -m pytest -v test/unit test/security test/chaos test/integration + +check: lint typecheck security test diff --git a/README.md b/README.md new file mode 100644 index 0000000..8120371 --- /dev/null +++ b/README.md @@ -0,0 +1,64 @@ +# `rag-tabular-analytics-worker` + +Stateless, horizontally autoscaling Python 3.12+ Apache Arrow Flight microservice executing sandboxed DuckDB analytical queries over content-addressed Parquet artifacts. + +## Phase 1 Capabilities (Implemented) + +- **Deterministic SQL AST Security Validator (`internal/validator/`)**: + - `sqlglot` AST whitelisting (`<= 2 ms` warm steady-state) permitting only read-only `SELECT`, `WITH` (CTEs), `UNION` / `INTERSECT` / `EXCEPT`, and `Subquery` constructs. + - Strict allowlist for SQL functions (including `exp.Anonymous` filtering), column/table/CTE alias validation, join count limits (`MAX_JOINS`), and query length limits (`MAX_SQL_LENGTH`). + - Blocks DML/DDL, `PRAGMA`, `SET`, `RESET`, `SUMMARIZE`, `DESCRIBE`, `SHOW`, `PIVOT`, `COPY`, `ATTACH`, `DETACH`, `LOAD`, `INSTALL`, and DuckDB filesystem/network functions. +- **Ephemeral DuckDB Sandbox Runtime (`internal/sandbox/`)**: + - Materializes only the single validated `{hash}.parquet` artifact (`CREATE TEMP TABLE AS SELECT * FROM read_parquet(...)`) inside `duckdb.connect(":memory:")`. + - Immediately locks `enable_external_access = false`, `lock_configuration = true`, `temp_directory = ''`, `threads = 2`, and per-worker `memory_limit` (`floor(2.5 GiB / POOL_SIZE)` in `MiB`, e.g., `640MiB` at `POOL_SIZE=4`). + - Streams chunked PyArrow `RecordBatch` frames via `RecordBatchReader` without `fetch_arrow_table()` full materialization. +- **Disposable Subprocess Pool & Circuit Breaker (`internal/pool/`)**: + - `pebble.ProcessPool` supervisor (`max_workers=POOL_SIZE`, `max_tasks=100`) streaming Arrow IPC `RecordBatch` chunks over an OS pipe / `multiprocessing.connection` without full-result `bytes` buffering. + - Per-worker `5s` hard execution deadline with isolated child termination (`SIGKILL` on POSIX / `TerminateProcess` on Windows) and automatic worker replacement (`ADR-001`). + +## Local Development Setup + +```bash +python -m venv .venv +# Linux/macOS: source .venv/bin/activate +# Windows PowerShell: .venv\Scripts\Activate.ps1 +pip install -e ".[dev,test]" +``` + +## Verification Commands + +```bash +ruff check . +ruff format --check . +mypy cmd internal api +bandit -r cmd internal api -ll +pip-audit +pytest -v test/unit test/security test/chaos test/integration +``` + +### Extended Soak & SLO Endurance Tests + +The default `test/chaos` suite runs a fast 500-stream bounded concurrency soak and burst backpressure test. To run the opt-in 8-hour `jemalloc` and `P95 <= 45ms` endurance suite: + +```bash +RUN_SOAK_TESTS=1 pytest -v -m soak test/chaos +``` + +## Container & Kubernetes Deployment + +### Multi-Stage Production Docker Image (`jemalloc` Preloaded) + +```bash +docker build -f deployments/docker/Dockerfile -t rag-tabular-analytics-worker:0.1.0 . +docker run --rm -p 8815:8815 -p 8080:8080 --read-only -e TABULAR_WORKER_CACHE_MAX_BYTES=805306368 --tmpfs /var/cache/tabular-worker:rw,noexec,nosuid,uid=10001,gid=10001,size=1g --tmpfs /tmp:rw,size=256m rag-tabular-analytics-worker:0.1.0 +``` + +### Helm & ArgoCD Progressive Canary Rollout + +- **Staging**: `deployments/helm/values-staging.yaml` and `deployments/argocd/application-staging.yaml` +- **Production**: `deployments/helm/values-prod.yaml`, `deployments/argocd/application-prod.yaml`, and `deployments/argocd/rollout-canary.yaml` (`10% -> 25% -> 50% -> 100%` gated by `AnalysisTemplate` over Prometheus SLIs) + +```bash +helm lint deployments/helm +helm template rag-tabular-analytics-worker deployments/helm -f deployments/helm/values-prod.yaml > /dev/null +``` diff --git a/api/__init__.py b/api/__init__.py new file mode 100644 index 0000000..f65d98f --- /dev/null +++ b/api/__init__.py @@ -0,0 +1 @@ +"""Public API schemas and OpenAPI 3.1 contracts for rag-tabular-analytics-worker.""" diff --git a/api/flight/__init__.py b/api/flight/__init__.py new file mode 100644 index 0000000..da0e1c6 --- /dev/null +++ b/api/flight/__init__.py @@ -0,0 +1,21 @@ +"""Apache Arrow Flight ticket, action, and error contracts.""" + +from api.flight.actions import CacheWarmRequest, CacheWarmResponse, HealthCheckResponse +from api.flight.schema import ( + ConcurrencyLimitError, + FlightErrorCode, + FlightQueryTicket, + map_exception_to_flight_error, + sanitize_client_message, +) + +__all__ = [ + "CacheWarmRequest", + "CacheWarmResponse", + "ConcurrencyLimitError", + "FlightErrorCode", + "FlightQueryTicket", + "HealthCheckResponse", + "map_exception_to_flight_error", + "sanitize_client_message", +] diff --git a/api/flight/actions.py b/api/flight/actions.py new file mode 100644 index 0000000..7d480bb --- /dev/null +++ b/api/flight/actions.py @@ -0,0 +1,36 @@ +"""Apache Arrow Flight DoAction request and response schemas.""" + +from __future__ import annotations + +from typing import Literal + +from pydantic import BaseModel, ConfigDict, Field + + +class CacheWarmRequest(BaseModel): + """Request payload for Arrow Flight `DoAction("cache_warm")`.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + tenant_id: str = Field(..., min_length=1, max_length=128) + table_hash: str = Field(..., min_length=64, max_length=64) + s3_bucket: str = Field(..., min_length=3, max_length=63) + s3_key: str = Field(..., min_length=1, max_length=512) + + +class CacheWarmResponse(BaseModel): + """Response payload for Arrow Flight `DoAction("cache_warm")`.""" + + model_config = ConfigDict(extra="forbid") + + status: Literal["warmed"] = "warmed" + bytes: int = Field(..., ge=0) + + +class HealthCheckResponse(BaseModel): + """Response payload for Arrow Flight `DoAction("health_check")`.""" + + model_config = ConfigDict(extra="forbid") + + status: Literal["ok", "degraded"] + active_workers: int = Field(..., ge=0) diff --git a/api/flight/schema.py b/api/flight/schema.py new file mode 100644 index 0000000..ac061b1 --- /dev/null +++ b/api/flight/schema.py @@ -0,0 +1,112 @@ +"""Arrow Flight DoGet ticket schema and sanitized gRPC/Flight error mapping.""" + +from __future__ import annotations + +from enum import StrEnum + +import pyarrow.flight as fl +from pydantic import BaseModel, ConfigDict, Field +from pydantic import ValidationError as PydanticValidationError + +from internal.storage.cache_manager import CacheCapacityError, CorruptedArtifactError +from internal.storage.s3_streamer import ArtifactNotFoundError, S3AccessError +from internal.storage.table_resolver import TableValidationError, TenantIsolationError +from internal.validator.exceptions import ValidationError as SQLValidationError + + +class FlightErrorCode(StrEnum): + """Canonical gRPC/Flight error codes documented in .ai/doc-api.md.""" + + INVALID_ARGUMENT = "INVALID_ARGUMENT" + PERMISSION_DENIED = "PERMISSION_DENIED" + DEADLINE_EXCEEDED = "DEADLINE_EXCEEDED" + INTERNAL = "INTERNAL" + RESOURCE_EXHAUSTED = "RESOURCE_EXHAUSTED" + NOT_FOUND = "NOT_FOUND" + + +class FlightQueryTicket(BaseModel): + """Validated JSON payload for Arrow Flight `DoGet(Ticket)`.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + request_id: str = Field(..., min_length=1, max_length=128) + tenant_id: str = Field(..., min_length=1, max_length=128) + sql: str = Field(..., min_length=1, max_length=65536) + table_alias: str = Field(..., min_length=1, max_length=128) + table_hash: str = Field(..., min_length=64, max_length=64) + s3_bucket: str = Field(..., min_length=3, max_length=63) + s3_key: str = Field(..., min_length=1, max_length=512) + + +class ConcurrencyLimitError(RuntimeError): + """Raised when MAX_CONCURRENT_QUERIES is saturated for immediate backpressure.""" + + +def sanitize_client_message(code: FlightErrorCode, detail: str) -> str: + """Format a client-safe error message without internal paths, buckets, or S3 keys.""" + return f"{code.value}: {detail}" + + +def map_exception_to_flight_error(exc: BaseException) -> fl.FlightError: + """Map internal domain exceptions to sanitized PyArrow Flight exceptions.""" + if isinstance(exc, fl.FlightError): + return exc + + if isinstance(exc, TenantIsolationError): + return fl.FlightUnauthorizedError( + sanitize_client_message( + FlightErrorCode.PERMISSION_DENIED, + "Tenant or artifact access is not permitted.", + ) + ) + + if isinstance(exc, PydanticValidationError | TableValidationError | SQLValidationError): + if isinstance(exc, SQLValidationError): + detail = f"SQL validation failed ({exc.rule})." + elif isinstance(exc, TableValidationError): + detail = "Malformed table or tenant identifier." + else: + detail = "Malformed Flight ticket payload." + return fl.FlightServerError( + sanitize_client_message(FlightErrorCode.INVALID_ARGUMENT, detail) + ) + + if isinstance(exc, ArtifactNotFoundError): + return fl.FlightServerError( + sanitize_client_message( + FlightErrorCode.NOT_FOUND, + "Requested Parquet artifact was not found.", + ) + ) + + if isinstance(exc, ConcurrencyLimitError | CacheCapacityError): + return fl.FlightServerError( + sanitize_client_message( + FlightErrorCode.RESOURCE_EXHAUSTED, + "Worker concurrency or cache capacity is saturated; retry later.", + ) + ) + + if isinstance(exc, TimeoutError): + return fl.FlightTimedOutError( + sanitize_client_message( + FlightErrorCode.DEADLINE_EXCEEDED, + "Query execution exceeded the 5s deadline.", + ) + ) + + if isinstance(exc, CorruptedArtifactError | S3AccessError): + return fl.FlightInternalError( + sanitize_client_message( + FlightErrorCode.INTERNAL, + "Upstream artifact retrieval or verification failed.", + ) + ) + + return fl.FlightInternalError( + sanitize_client_message( + FlightErrorCode.INTERNAL, + "Internal query execution failure.", + ) + ) diff --git a/api/openapi/openapi.yaml b/api/openapi/openapi.yaml new file mode 100644 index 0000000..98a2293 --- /dev/null +++ b/api/openapi/openapi.yaml @@ -0,0 +1,135 @@ +openapi: 3.1.0 +info: + title: rag-tabular-analytics-worker Ops & Control API + version: 0.2.0 + description: > + Operational, health, Prometheus observability, and OpenAPI 3.1 documentation + endpoints for the stateless Apache Arrow Flight + DuckDB analytics worker. + license: + name: MIT +servers: + - url: http://localhost:8080 + description: Internal Service / Kubernetes Pod Ops Listener +paths: + /healthz: + get: + summary: Kubernetes Liveness Probe + description: Verifies the Python host daemon process and async event loop are alive. + operationId: getHealthz + responses: + "200": + description: Daemon is alive + content: + application/json: + schema: + $ref: "#/components/schemas/HealthzResponse" + /ready: + get: + summary: Kubernetes Readiness Probe + description: > + Verifies the ProcessManager worker pool is healthy, the Tier-1 NVMe/tmpfs + cache mount is writable, and the pod is not draining. + operationId: getReady + responses: + "200": + description: Pod is ready to serve Arrow Flight queries + content: + application/json: + schema: + $ref: "#/components/schemas/ReadyResponse" + "503": + description: Pod is draining or dependency checks failed + content: + application/json: + schema: + $ref: "#/components/schemas/NotReadyResponse" + /metrics: + get: + summary: Prometheus Metrics Scrape Endpoint + description: Exposes Prometheus text exposition metrics (text/plain; version=0.0.4). + operationId: getMetrics + responses: + "200": + description: Prometheus metrics payload + content: + text/plain: + schema: + type: string + /openapi.yaml: + get: + summary: Raw OpenAPI 3.1 Specification + description: Serves the authoritative OpenAPI 3.1 YAML specification. + operationId: getOpenApiSpec + responses: + "200": + description: OpenAPI 3.1 YAML document + content: + application/yaml: + schema: + type: string + /docs: + get: + summary: Swagger UI Documentation + description: Interactive Swagger UI HTML page backed by /openapi.yaml. + operationId: getSwaggerDocs + responses: + "200": + description: Swagger UI HTML page + content: + text/html: + schema: + type: string +components: + schemas: + HealthzResponse: + type: object + required: + - status + - service + properties: + status: + type: string + enum: + - ok + service: + type: string + example: rag-tabular-analytics-worker + ReadyResponse: + type: object + required: + - status + - checks + properties: + status: + type: string + enum: + - ready + checks: + type: object + required: + - process_pool + - cache_mount + properties: + process_pool: + type: string + enum: + - healthy + cache_mount: + type: string + enum: + - writable + NotReadyResponse: + type: object + required: + - status + - checks + properties: + status: + type: string + enum: + - draining + - not_ready + checks: + type: object + additionalProperties: + type: string diff --git a/cmd/worker/main.py b/cmd/worker/main.py new file mode 100644 index 0000000..05b77ba --- /dev/null +++ b/cmd/worker/main.py @@ -0,0 +1,15 @@ +"""CLI entrypoint for rag-tabular-analytics-worker.""" + +from __future__ import annotations + +from internal.flight.daemon import WorkerDaemon + + +def main() -> None: + """Start the rag-tabular-analytics-worker daemon and block until termination.""" + daemon = WorkerDaemon() + daemon.run_forever() + + +if __name__ == "__main__": + main() diff --git a/deployments/SECRET_MANAGEMENT.md b/deployments/SECRET_MANAGEMENT.md new file mode 100644 index 0000000..832ac06 --- /dev/null +++ b/deployments/SECRET_MANAGEMENT.md @@ -0,0 +1,108 @@ +# Secret Management (`rag-tabular-analytics-worker`) + +This service separates **non-sensitive runtime configuration** (stored in `.env.prod` and synced to the Helm `ConfigMap`) from **sensitive credentials** (injected into a Kubernetes `Secret` via CI/CD). + +--- + +## 1. Architecture Overview + +| Category | Examples | Storage / Source of Truth | Kubernetes Mechanism | +| :--- | :--- | :--- | :--- | +| **Non-sensitive config** | `TABULAR_WORKER_POOL_SIZE`, `TABULAR_WORKER_S3_ENDPOINT_URL`, `TABULAR_WORKER_S3_REGION`, `TABULAR_WORKER_S3_BUCKET_ALLOWLIST` | `.env.prod` -> `deployments/helm/values.yaml` (via `scripts/sync_env_to_deployments.py --sync`) | `ConfigMap` (`-config`) via `envFrom.configMapRef` | +| **Sensitive credentials** | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` (Cloudflare R2 S3 API tokens), `OTEL_EXPORTER_OTLP_HEADERS` | GitHub Actions Environment Secrets (`staging`, `prod`) | `Secret` (`rag-tabular-analytics-worker-secrets`) via `envFrom.secretRef` | + +> [!IMPORTANT] +> Never commit credentials to `.env.example`, `deployments/helm/values*.yaml`, or Git. +> `scripts/sync_env_to_deployments.py` automatically excludes any key containing `PASSWORD`, `SECRET`, `TOKEN`, or `KEY` (while preserving `env.secretName`). + +--- + +## 2. Cloudflare R2 S3 Configuration + +[`internal/storage/s3_streamer.py`](../internal/storage/s3_streamer.py) uses `aiobotocore` / `botocore`, which reads standard AWS S3 credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) from the pod environment. + +### Non-sensitive R2 settings (`.env.prod` / `ConfigMap`) + +Set these in `.env.prod` and sync to `deployments/helm/values.yaml`: + +```dotenv +TABULAR_WORKER_S3_ENDPOINT_URL=https://.r2.cloudflarestorage.com +TABULAR_WORKER_S3_REGION=auto +TABULAR_WORKER_S3_BUCKET_ALLOWLIST=["your-r2-bucket"] +``` + +### Sensitive R2 credentials (Kubernetes `Secret`) + +Create an R2 API token in the Cloudflare dashboard with **Object Read only** (or Read & Write if needed) scoped to the target bucket(s), and store its S3-compatible credentials in GitHub Environment Secrets: + +- `R2_ACCESS_KEY_ID` -> injected as `AWS_ACCESS_KEY_ID` +- `R2_SECRET_ACCESS_KEY` -> injected as `AWS_SECRET_ACCESS_KEY` + +--- + +## 3. Helm & Kubernetes Wiring + +1. **`deployments/helm/values.yaml`** sets: + ```yaml + env: + secretName: "rag-tabular-analytics-worker-secrets" + ``` +2. **`deployments/helm/templates/deployment.yaml`** mounts both the `ConfigMap` and `Secret` via `envFrom`: + ```yaml + envFrom: + - configMapRef: + name: {{ include "rag-tabular-analytics-worker.fullname" . }}-config + {{- if .Values.env.secretName }} + - secretRef: + name: {{ .Values.env.secretName }} + {{- end }} + ``` +3. **`deployments/helm/templates/configmap.yaml`** explicitly skips `secretName` and `nil` values so the `Secret` name is never emitted into the `ConfigMap`. +4. **`deployments/helm/templates/secret.yaml`** is disabled by default (`secrets.create: false`) so Helm never stores plaintext secrets in release values when CI/CD injects `rag-tabular-analytics-worker-secrets` directly. + +--- + +## 4. CI/CD Secret Injection (GitHub Actions) + +[`.github/workflows/deploy.yml`](../.github/workflows/deploy.yml) handles secret injection and Helm deployment on `workflow_dispatch`: + +1. **Configure GitHub Environments** (`staging` and `prod`) under **Settings -> Environments** in GitHub. +2. **Add the following Environment Secrets** to each environment: + | GitHub Secret | Required | Description | + | :--- | :--- | :--- | + | `KUBE_CONFIG_DATA` | Yes | Base64-encoded `kubeconfig` with deploy permissions for the target cluster/namespace. | + | `R2_ACCESS_KEY_ID` | Yes | Cloudflare R2 S3 Access Key ID (mapped to `AWS_ACCESS_KEY_ID`). | + | `R2_SECRET_ACCESS_KEY` | Yes | Cloudflare R2 S3 Secret Access Key (mapped to `AWS_SECRET_ACCESS_KEY`). | + | `OTEL_EXPORTER_OTLP_HEADERS` | No | Optional auth headers for OpenTelemetry OTLP exporter. | +3. **How injection works safely:** + - The workflow writes secrets to a temporary `chmod 600` file (`mktemp` with `trap 'rm -f ...' EXIT`) rather than passing secret values on the `kubectl` command line. + - It runs `kubectl create secret generic rag-tabular-analytics-worker-secrets --from-env-file=... --dry-run=client -o yaml | kubectl apply -f -` to idempotently create or update the `Secret` in `rag-tabular-staging` or `rag-tabular-prod`. + - It then runs `helm upgrade --install` with `values.yaml` and the environment overlay (`values-staging.yaml` or `values-prod.yaml`). + +--- + +## 5. Local Development & Manual Operations + +### Local Development +Put your `TABULAR_WORKER_*` S3 settings in `.env` (which is gitignored): + +```dotenv +TABULAR_WORKER_S3_BUCKET_ALLOWLIST=["your-dev-bucket"] +TABULAR_WORKER_S3_ENDPOINT_URL=https://.r2.cloudflarestorage.com +TABULAR_WORKER_S3_REGION=auto +``` + +> **Note:** `Settings` uses `env_prefix="TABULAR_WORKER_"` with `extra="ignore"` and does **not** load unprefixed `AWS_*` keys from `.env` into the process environment. Export your development R2 (or local MinIO) credentials in the shell environment that launches the worker so its S3 client (`aioboto3`/`botocore`) can read them: + +```bash +export AWS_ACCESS_KEY_ID=your-dev-r2-access-key-id +export AWS_SECRET_ACCESS_KEY=your-dev-r2-secret-access-key +``` + +### Secret Rotation +When rotating Cloudflare R2 credentials: +1. Update `R2_ACCESS_KEY_ID` and `R2_SECRET_ACCESS_KEY` in the GitHub Environment. +2. Re-run the **Deploy** workflow (or apply the updated `Secret` manually), then trigger a rolling restart so pods pick up the new environment variables: + ```bash + kubectl rollout restart deployment/rag-tabular-analytics-worker -n rag-tabular-prod + ``` diff --git a/deployments/argocd/application-prod.yaml b/deployments/argocd/application-prod.yaml new file mode 100644 index 0000000..65a3252 --- /dev/null +++ b/deployments/argocd/application-prod.yaml @@ -0,0 +1,25 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: rag-tabular-analytics-worker-prod + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + # Production deployments promote pinned release tags via progressive Argo Rollouts canary. + source: + repoURL: https://github.com/org/rag-tabular-analytics-worker.git + targetRevision: v0.1.0 + path: deployments/helm + helm: + releaseName: rag-tabular-analytics-worker + valueFiles: + - values.yaml + - values-prod.yaml + destination: + server: https://kubernetes.default.svc + namespace: rag-tabular-prod + syncPolicy: + syncOptions: + - CreateNamespace=true diff --git a/deployments/argocd/application-staging.yaml b/deployments/argocd/application-staging.yaml new file mode 100644 index 0000000..04b55ab --- /dev/null +++ b/deployments/argocd/application-staging.yaml @@ -0,0 +1,27 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: rag-tabular-analytics-worker-staging + namespace: argocd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + source: + repoURL: https://github.com/Gemejime/Tabular-SQL-Engine.git + targetRevision: HEAD + path: deployments/helm + helm: + releaseName: rag-tabular-analytics-worker + valueFiles: + - values.yaml + - values-staging.yaml + destination: + server: https://kubernetes.default.svc + namespace: rag-tabular-staging + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true diff --git a/deployments/argocd/rollout-canary.yaml b/deployments/argocd/rollout-canary.yaml new file mode 100644 index 0000000..12f45bd --- /dev/null +++ b/deployments/argocd/rollout-canary.yaml @@ -0,0 +1,85 @@ +apiVersion: argoproj.io/v1alpha1 +kind: AnalysisTemplate +metadata: + name: rag-tabular-analytics-worker-canary-analysis + namespace: rag-tabular-prod +spec: + args: + - name: prometheus-url + value: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090 + metrics: + - name: query-latency-p99 + interval: 30s + count: 4 + failureLimit: 1 + successCondition: "len(result) > 0 && result[0] <= 0.12" + provider: + prometheus: + address: "{{args.prometheus-url}}" + query: | + histogram_quantile(0.99, sum(rate(tabular_worker_query_duration_seconds_bucket{cache_state="warm_cache"}[2m])) by (le)) + - name: subprocess-abnormal-recycles + interval: 30s + count: 4 + failureLimit: 1 + successCondition: "len(result) > 0 && result[0] <= 0.05" + provider: + prometheus: + address: "{{args.prometheus-url}}" + query: | + sum(rate(tabular_worker_subprocess_recycles_total{reason=~"TIMEOUT|MEMORY_LIMIT|CRASH"}[2m])) + - name: cache-hit-ratio + interval: 30s + count: 4 + failureLimit: 2 + successCondition: "len(result) > 0 && result[0] >= 0.80" + provider: + prometheus: + address: "{{args.prometheus-url}}" + query: | + avg(tabular_worker_cache_hit_ratio) + - name: active-queries + interval: 30s + count: 4 + successCondition: "len(result) > 0 && result[0] >= 0" + provider: + prometheus: + address: "{{args.prometheus-url}}" + query: | + avg(tabular_worker_active_queries) +--- +apiVersion: argoproj.io/v1alpha1 +kind: Rollout +metadata: + name: rag-tabular-analytics-worker + namespace: rag-tabular-prod +spec: + replicas: 4 + workloadRef: + apiVersion: apps/v1 + kind: Deployment + name: rag-tabular-analytics-worker + strategy: + canary: + canaryService: rag-tabular-analytics-worker-canary + stableService: rag-tabular-analytics-worker + steps: + - setWeight: 10 + - pause: + duration: 2m + - analysis: + templates: + - templateName: rag-tabular-analytics-worker-canary-analysis + - setWeight: 25 + - pause: + duration: 5m + - analysis: + templates: + - templateName: rag-tabular-analytics-worker-canary-analysis + - setWeight: 50 + - pause: + duration: 10m + - analysis: + templates: + - templateName: rag-tabular-analytics-worker-canary-analysis + - setWeight: 100 diff --git a/deployments/docker/Dockerfile b/deployments/docker/Dockerfile new file mode 100644 index 0000000..c454157 --- /dev/null +++ b/deployments/docker/Dockerfile @@ -0,0 +1,62 @@ +# syntax=docker/dockerfile:1.7 + +FROM python:3.12-slim AS builder + +WORKDIR /build + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PIP_NO_CACHE_DIR=1 + +RUN python -m venv /opt/venv +ENV PATH="/opt/venv/bin:$PATH" + +COPY pyproject.toml README.md ./ +COPY cmd ./cmd +COPY internal ./internal +COPY api ./api + +# hadolint ignore=DL3013 +RUN pip install --no-cache-dir --upgrade pip setuptools wheel && \ + pip install --no-cache-dir . + +FROM python:3.12-slim AS runtime + +WORKDIR /app + +# hadolint ignore=DL3008 +RUN set -eux; \ + apt-get update; \ + apt-get install -y --no-install-recommends libjemalloc2; \ + rm -rf /var/lib/apt/lists/*; \ + jemalloc_so="$(find /usr/lib -name 'libjemalloc.so.2' -print -quit)"; \ + test -n "${jemalloc_so}"; \ + ln -sf "${jemalloc_so}" /usr/local/lib/libjemalloc.so; \ + groupadd --gid 10001 tabular; \ + useradd --uid 10001 --gid 10001 --no-create-home --shell /usr/sbin/nologin tabular; \ + mkdir -p /var/cache/tabular-worker; \ + chown -R 10001:10001 /var/cache/tabular-worker /app + +COPY --from=builder /opt/venv /opt/venv +COPY cmd ./cmd +COPY internal ./internal +COPY api ./api + +ENV PATH="/opt/venv/bin:$PATH" \ + PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + LD_PRELOAD=/usr/local/lib/libjemalloc.so \ + TABULAR_WORKER_CACHE_DIR=/var/cache/tabular-worker \ + TABULAR_WORKER_FLIGHT_HOST=0.0.0.0 \ + TABULAR_WORKER_FLIGHT_PORT=8815 \ + TABULAR_WORKER_HTTP_HOST=0.0.0.0 \ + TABULAR_WORKER_HTTP_PORT=8080 + +USER 10001:10001 + +EXPOSE 8815 8080 + +HEALTHCHECK --interval=15s --timeout=3s --start-period=10s --retries=3 \ + CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/healthz', timeout=2)"] + +CMD ["python", "-m", "cmd.worker.main"] diff --git a/deployments/helm/Chart.yaml b/deployments/helm/Chart.yaml new file mode 100644 index 0000000..6bbf0e3 --- /dev/null +++ b/deployments/helm/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: rag-tabular-analytics-worker +description: Stateless, horizontally autoscaling Apache Arrow Flight microservice executing sandboxed DuckDB queries over Parquet artifacts. +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/deployments/helm/templates/_helpers.tpl b/deployments/helm/templates/_helpers.tpl new file mode 100644 index 0000000..80f35a4 --- /dev/null +++ b/deployments/helm/templates/_helpers.tpl @@ -0,0 +1,28 @@ +{{- define "rag-tabular-analytics-worker.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "rag-tabular-analytics-worker.fullname" -}} +{{- if .Values.fullnameOverride -}} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- if contains $name .Release.Name -}} +{{- .Release.Name | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{- define "rag-tabular-analytics-worker.labels" -}} +app.kubernetes.io/name: {{ include "rag-tabular-analytics-worker.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end -}} + +{{- define "rag-tabular-analytics-worker.selectorLabels" -}} +app.kubernetes.io/name: {{ include "rag-tabular-analytics-worker.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} diff --git a/deployments/helm/templates/configmap.yaml b/deployments/helm/templates/configmap.yaml new file mode 100644 index 0000000..f3a086b --- /dev/null +++ b/deployments/helm/templates/configmap.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "rag-tabular-analytics-worker.fullname" . }}-config + labels: + {{- include "rag-tabular-analytics-worker.labels" . | nindent 4 }} +data: + PORT: {{ default "8080" (get .Values.env "PORT") | quote }} + GRPC_PORT: {{ default "50051" (get .Values.env "GRPC_PORT") | quote }} + {{- range $key, $val := .Values.env }} + {{- if and (ne $key "secretName") (ne $key "PORT") (ne $key "GRPC_PORT") (ne $val nil) }} + {{ $key }}: {{ $val | quote }} + {{- end }} + {{- end }} diff --git a/deployments/helm/templates/deployment.yaml b/deployments/helm/templates/deployment.yaml new file mode 100644 index 0000000..ea5042f --- /dev/null +++ b/deployments/helm/templates/deployment.yaml @@ -0,0 +1,94 @@ +{{- if and .Values.env.TABULAR_WORKER_CACHE_DIR (ne .Values.env.TABULAR_WORKER_CACHE_DIR .Values.cache.mountPath) -}} +{{- fail (printf "env.TABULAR_WORKER_CACHE_DIR (%s) must match cache.mountPath (%s)" .Values.env.TABULAR_WORKER_CACHE_DIR .Values.cache.mountPath) -}} +{{- end -}} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "rag-tabular-analytics-worker.fullname" . }} + labels: + {{- include "rag-tabular-analytics-worker.labels" . | nindent 4 }} +spec: + {{- if not .Values.keda.enabled }} + replicas: {{ .Values.replicaCount }} + {{- end }} + selector: + matchLabels: + {{- include "rag-tabular-analytics-worker.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "rag-tabular-analytics-worker.selectorLabels" . | nindent 8 }} + {{- $httpPort := default 8080 .Values.env.TABULAR_WORKER_HTTP_PORT | int | toString }} + {{- $annotations := merge (dict "prometheus.io/port" $httpPort) (default dict .Values.podAnnotations) }} + annotations: + checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} + {{- with $annotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }} + securityContext: + runAsNonRoot: {{ .Values.securityContext.runAsNonRoot }} + runAsUser: {{ .Values.securityContext.runAsUser }} + runAsGroup: {{ .Values.securityContext.runAsGroup }} + fsGroup: {{ .Values.securityContext.runAsGroup }} + containers: + - name: {{ .Chart.Name }} + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + securityContext: + allowPrivilegeEscalation: {{ .Values.securityContext.allowPrivilegeEscalation }} + readOnlyRootFilesystem: {{ .Values.securityContext.readOnlyRootFilesystem }} + capabilities: + {{- toYaml .Values.securityContext.capabilities | nindent 14 }} + ports: + - name: grpc + containerPort: {{ default 8815 .Values.env.TABULAR_WORKER_FLIGHT_PORT | int }} + protocol: TCP + - name: http + containerPort: {{ default 8080 .Values.env.TABULAR_WORKER_HTTP_PORT | int }} + protocol: TCP + envFrom: + - configMapRef: + name: {{ include "rag-tabular-analytics-worker.fullname" . }}-config + {{- if .Values.env.secretName }} + - secretRef: + name: {{ .Values.env.secretName }} + {{- end }} + env: + {{- if .Values.jemalloc.enabled }} + - name: LD_PRELOAD + value: {{ .Values.jemalloc.path | quote }} + {{- end }} + resources: + {{- toYaml .Values.resources | nindent 12 }} + livenessProbe: + httpGet: + path: /healthz + port: http + initialDelaySeconds: 10 + periodSeconds: 10 + timeoutSeconds: 2 + failureThreshold: 3 + readinessProbe: + httpGet: + path: /ready + port: http + initialDelaySeconds: 5 + periodSeconds: 5 + timeoutSeconds: 2 + failureThreshold: 3 + volumeMounts: + - name: nvme-cache + mountPath: {{ .Values.cache.mountPath }} + - name: tmp + mountPath: /tmp + volumes: + - name: nvme-cache + emptyDir: + medium: "" + sizeLimit: {{ .Values.cache.sizeLimit }} + - name: tmp + emptyDir: + medium: Memory + sizeLimit: 256Mi diff --git a/deployments/helm/templates/scaledobject.yaml b/deployments/helm/templates/scaledobject.yaml new file mode 100644 index 0000000..2fce90c --- /dev/null +++ b/deployments/helm/templates/scaledobject.yaml @@ -0,0 +1,22 @@ +{{- if .Values.keda.enabled }} +apiVersion: keda.sh/v1alpha1 +kind: ScaledObject +metadata: + name: {{ include "rag-tabular-analytics-worker.fullname" . }} + labels: + {{- include "rag-tabular-analytics-worker.labels" . | nindent 4 }} +spec: + scaleTargetRef: + name: {{ include "rag-tabular-analytics-worker.fullname" . }} + pollingInterval: {{ .Values.keda.pollingInterval }} + cooldownPeriod: {{ .Values.keda.cooldownPeriod }} + minReplicaCount: {{ .Values.keda.minReplicaCount }} + maxReplicaCount: {{ .Values.keda.maxReplicaCount }} + triggers: + - type: prometheus + metadata: + serverAddress: {{ .Values.keda.prometheus.serverAddress }} + metricName: tabular_worker_active_queries + threshold: {{ .Values.keda.prometheus.threshold | quote }} + query: sum(tabular_worker_active_queries{namespace="{{ .Release.Namespace }}",pod=~"{{ include "rag-tabular-analytics-worker.fullname" . }}-.*"}) +{{- end }} diff --git a/deployments/helm/templates/secret.yaml b/deployments/helm/templates/secret.yaml new file mode 100644 index 0000000..367f2fe --- /dev/null +++ b/deployments/helm/templates/secret.yaml @@ -0,0 +1,13 @@ +{{- if and .Values.secrets .Values.secrets.create }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ default "rag-tabular-analytics-worker-secrets" .Values.env.secretName }} + labels: + {{- include "rag-tabular-analytics-worker.labels" . | nindent 4 }} +type: Opaque +stringData: + {{- range $key, $val := .Values.secrets.data }} + {{ $key }}: {{ $val | quote }} + {{- end }} +{{- end }} diff --git a/deployments/helm/templates/service.yaml b/deployments/helm/templates/service.yaml new file mode 100644 index 0000000..54f0561 --- /dev/null +++ b/deployments/helm/templates/service.yaml @@ -0,0 +1,22 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "rag-tabular-analytics-worker.fullname" . }} + labels: + {{- include "rag-tabular-analytics-worker.labels" . | nindent 4 }} +spec: + type: {{ .Values.service.type }} + {{- if .Values.service.headless }} + clusterIP: None + {{- end }} + selector: + {{- include "rag-tabular-analytics-worker.selectorLabels" . | nindent 4 }} + ports: + - name: grpc + port: {{ .Values.service.grpcPort }} + targetPort: grpc + protocol: TCP + - name: http + port: {{ .Values.service.httpPort }} + targetPort: http + protocol: TCP diff --git a/deployments/helm/values-prod.yaml b/deployments/helm/values-prod.yaml new file mode 100644 index 0000000..12da561 --- /dev/null +++ b/deployments/helm/values-prod.yaml @@ -0,0 +1,32 @@ +replicaCount: 4 + +resources: + requests: + cpu: 2000m + memory: 4Gi + limits: + cpu: 4000m + memory: 6Gi + +cache: + mountPath: /var/cache/tabular-worker + sizeLimit: 10Gi + +jemalloc: + enabled: true + path: /usr/local/lib/libjemalloc.so + +env: + TABULAR_WORKER_OTEL_ENABLED: "true" + TABULAR_WORKER_OTEL_EXPORTER_ENDPOINT: "http://otel-collector.monitoring.svc.cluster.local:4317" + TABULAR_WORKER_OTEL_SERVICE_NAME: "rag-tabular-analytics-worker" + +keda: + enabled: true + minReplicaCount: 4 + maxReplicaCount: 24 + pollingInterval: 10 + cooldownPeriod: 60 + prometheus: + serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090 + threshold: "3" diff --git a/deployments/helm/values-staging.yaml b/deployments/helm/values-staging.yaml new file mode 100644 index 0000000..d711905 --- /dev/null +++ b/deployments/helm/values-staging.yaml @@ -0,0 +1,20 @@ +replicaCount: 1 + +jemalloc: + enabled: true + path: /usr/local/lib/libjemalloc.so + +env: + TABULAR_WORKER_OTEL_ENABLED: "true" + TABULAR_WORKER_OTEL_EXPORTER_ENDPOINT: "http://otel-collector.monitoring.svc.cluster.local:4317" + TABULAR_WORKER_OTEL_SERVICE_NAME: "rag-tabular-analytics-worker-staging" + +keda: + enabled: true + minReplicaCount: 1 + maxReplicaCount: 6 + pollingInterval: 10 + cooldownPeriod: 60 + prometheus: + serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090 + threshold: "3" diff --git a/deployments/helm/values.yaml b/deployments/helm/values.yaml new file mode 100644 index 0000000..29ba275 --- /dev/null +++ b/deployments/helm/values.yaml @@ -0,0 +1,84 @@ +replicaCount: 4 + +image: + repository: rag-tabular-analytics-worker + tag: "0.1.0" + pullPolicy: IfNotPresent + +service: + type: ClusterIP + headless: false + grpcPort: 8815 + httpPort: 8080 + +resources: + requests: + cpu: 2000m + memory: 4Gi + limits: + cpu: 4000m + memory: 6Gi + +terminationGracePeriodSeconds: 30 + +cache: + mountPath: /var/cache/tabular-worker + sizeLimit: 10Gi + +jemalloc: + enabled: true + path: /usr/local/lib/libjemalloc.so + +securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + +podAnnotations: + prometheus.io/scrape: "true" + prometheus.io/port: "8080" + prometheus.io/path: "/metrics" + +env: + secretName: "rag-tabular-analytics-worker-secrets" + TABULAR_WORKER_SERVICE_NAME: "rag-tabular-analytics-worker" + TABULAR_WORKER_LOG_LEVEL: "INFO" + TABULAR_WORKER_POOL_SIZE: "4" + TABULAR_WORKER_MAX_TASKS_PER_CHILD: "100" + TABULAR_WORKER_QUERY_TIMEOUT_SECONDS: "5.0" + TABULAR_WORKER_DUCKDB_TOTAL_MEMORY_BYTES: "2684354560" + TABULAR_WORKER_DUCKDB_THREADS: "2" + TABULAR_WORKER_ARROW_BATCH_SIZE: "8192" + TABULAR_WORKER_MAX_SQL_LENGTH: "8192" + TABULAR_WORKER_MAX_JOINS: "5" + TABULAR_WORKER_FLIGHT_HOST: "0.0.0.0" + TABULAR_WORKER_FLIGHT_PORT: "8815" + TABULAR_WORKER_HTTP_HOST: "0.0.0.0" + TABULAR_WORKER_HTTP_PORT: "8080" + TABULAR_WORKER_CACHE_DIR: "/var/cache/tabular-worker" + TABULAR_WORKER_CACHE_MAX_BYTES: "10737418240" + TABULAR_WORKER_S3_BUCKET_ALLOWLIST: "[]" + TABULAR_WORKER_S3_ENDPOINT_URL: "" + TABULAR_WORKER_S3_REGION: "us-east-1" + TABULAR_WORKER_S3_CHUNK_SIZE_BYTES: "8388608" + TABULAR_WORKER_S3_TIMEOUT_SECONDS: "4.0" + TABULAR_WORKER_MAX_CONCURRENT_QUERIES: "4" + TABULAR_WORKER_WORKER_PROCESS_RLIMIT_BYTES: "" + TABULAR_WORKER_OTEL_ENABLED: "false" + TABULAR_WORKER_OTEL_EXPORTER_ENDPOINT: "" + TABULAR_WORKER_OTEL_SERVICE_NAME: "rag-tabular-analytics-worker" + +keda: + enabled: true + minReplicaCount: 4 + maxReplicaCount: 24 + pollingInterval: 10 + cooldownPeriod: 60 + prometheus: + serverAddress: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090 + threshold: "3" diff --git a/internal/__init__.py b/internal/__init__.py new file mode 100644 index 0000000..659d4e4 --- /dev/null +++ b/internal/__init__.py @@ -0,0 +1 @@ +"""Internal packages for rag-tabular-analytics-worker.""" diff --git a/internal/config/__init__.py b/internal/config/__init__.py new file mode 100644 index 0000000..94ed34e --- /dev/null +++ b/internal/config/__init__.py @@ -0,0 +1,5 @@ +"""Configuration package for rag-tabular-analytics-worker.""" + +from internal.config.settings import Settings, get_settings + +__all__ = ["Settings", "get_settings"] diff --git a/internal/config/settings.py b/internal/config/settings.py new file mode 100644 index 0000000..e6fc465 --- /dev/null +++ b/internal/config/settings.py @@ -0,0 +1,137 @@ +"""12-factor typed configuration for rag-tabular-analytics-worker.""" + +from __future__ import annotations + +import json +import math +from functools import lru_cache +from pathlib import Path +from typing import Annotated, Any + +from pydantic import Field, field_validator, model_validator +from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict + +GIB_BYTES = 1024 * 1024 * 1024 +MIB_BYTES = 1024 * 1024 +DEFAULT_DUCKDB_TOTAL_MEMORY_BYTES = int(2.5 * GIB_BYTES) # 2,684,354,560 bytes (2.5 GiB) +DEFAULT_CACHE_MAX_BYTES = 10 * GIB_BYTES # 10 GiB NVMe emptyDir ceiling (ADR-002) +_PROJECT_ROOT = Path(__file__).resolve().parents[2] + + +class Settings(BaseSettings): + """Runtime settings for the tabular analytics worker.""" + + model_config = SettingsConfigDict( + env_prefix="TABULAR_WORKER_", + env_file=(_PROJECT_ROOT / ".env", ".env"), + env_file_encoding="utf-8", + case_sensitive=True, + extra="ignore", + ) + + SERVICE_NAME: str = Field(default="rag-tabular-analytics-worker") + LOG_LEVEL: str = Field(default="INFO") + + # Subprocess pool & circuit breaker (ADR-001) + POOL_SIZE: int = Field(default=4, ge=1, le=32) + MAX_TASKS_PER_CHILD: int = Field(default=100, ge=1, le=10000) + QUERY_TIMEOUT_SECONDS: float = Field(default=5.0, gt=0.0, le=30.0) + + # DuckDB sandbox memory & concurrency limits (ADR-003) + DUCKDB_TOTAL_MEMORY_BYTES: int = Field( + default=DEFAULT_DUCKDB_TOTAL_MEMORY_BYTES, + ge=64 * MIB_BYTES, + le=DEFAULT_DUCKDB_TOTAL_MEMORY_BYTES, + ) + DUCKDB_THREADS: int = Field(default=2, ge=1, le=8) + ARROW_BATCH_SIZE: int = Field(default=8192, ge=128, le=65536) + + # SQL AST validator limits (ADR-005) + MAX_SQL_LENGTH: int = Field(default=8192, ge=64, le=65536) + MAX_JOINS: int = Field(default=5, ge=0, le=20) + + # Arrow Flight & HTTP Ops Server endpoints + FLIGHT_HOST: str = Field(default="0.0.0.0") # noqa: S104 # nosec B104 + FLIGHT_PORT: int = Field(default=8815, ge=0, le=65535) + HTTP_HOST: str = Field(default="0.0.0.0") # noqa: S104 # nosec B104 + HTTP_PORT: int = Field(default=8080, ge=0, le=65535) + + # Tier-1 Disk-Backed NVMe Parquet Cache (ADR-002) + CACHE_DIR: Path = Field(default=Path("/var/cache/tabular-worker")) + CACHE_MAX_BYTES: int = Field(default=DEFAULT_CACHE_MAX_BYTES, ge=MIB_BYTES) + + # S3 VPC Endpoint Streaming & Tenant Allowlist + S3_BUCKET_ALLOWLIST: Annotated[list[str], NoDecode] = Field(default_factory=list) + S3_ENDPOINT_URL: str | None = Field(default=None) + S3_REGION: str = Field(default="us-east-1") + S3_CHUNK_SIZE_BYTES: int = Field(default=8 * MIB_BYTES, ge=64 * 1024, le=128 * MIB_BYTES) + S3_TIMEOUT_SECONDS: float = Field(default=4.0, gt=0.0, le=60.0) + + # Concurrency & KEDA Backpressure (ADR-004) + MAX_CONCURRENT_QUERIES: int = Field(default=4, ge=1, le=64) + + # Optional POSIX worker address-space guardrail (disabled by default; ADR-003 uses DuckDB limit) + WORKER_PROCESS_RLIMIT_BYTES: int | None = Field(default=None, ge=64 * MIB_BYTES) + + # OpenTelemetry Distributed Tracing + OTEL_ENABLED: bool = Field(default=False) + OTEL_EXPORTER_ENDPOINT: str | None = Field(default=None) + OTEL_SERVICE_NAME: str = Field(default="rag-tabular-analytics-worker") + + @field_validator( + "S3_ENDPOINT_URL", + "WORKER_PROCESS_RLIMIT_BYTES", + "OTEL_EXPORTER_ENDPOINT", + mode="before", + ) + @classmethod + def _empty_str_to_none(cls, value: Any) -> Any: + """Treat blank .env values for optional settings as None.""" + if isinstance(value, str) and not value.strip(): + return None + return value + + @field_validator("S3_BUCKET_ALLOWLIST", mode="before") + @classmethod + def _parse_bucket_allowlist(cls, value: Any) -> list[str]: + """Parse S3_BUCKET_ALLOWLIST from JSON array, comma-separated string, or list.""" + if value is None: + return [] + if isinstance(value, list): + return [str(item).strip() for item in value if str(item).strip()] + if isinstance(value, str): + stripped = value.strip() + if not stripped: + return [] + if stripped.startswith("["): + parsed = json.loads(stripped) + if not isinstance(parsed, list): + raise ValueError("S3_BUCKET_ALLOWLIST JSON value must be a list of strings") + return [str(item).strip() for item in parsed if str(item).strip()] + return [item.strip() for item in stripped.split(",") if item.strip()] + raise ValueError("S3_BUCKET_ALLOWLIST must be a list or string") + + @model_validator(mode="after") + def _validate_worker_memory_budget(self) -> Settings: + """Ensure each worker receives at least 1 MiB within the aggregate DuckDB budget.""" + if self.worker_memory_limit_mib < 1: + raise ValueError( + "DUCKDB_TOTAL_MEMORY_BYTES divided by POOL_SIZE must be at least 1 MiB" + ) + return self + + @property + def worker_memory_limit_mib(self) -> int: + """Return per-worker DuckDB memory limit floored to whole MiB.""" + return math.floor(self.DUCKDB_TOTAL_MEMORY_BYTES / self.POOL_SIZE / MIB_BYTES) + + @property + def worker_memory_limit_str(self) -> str: + """Return per-worker DuckDB memory_limit pragma string (e.g. '640MiB').""" + return f"{self.worker_memory_limit_mib}MiB" + + +@lru_cache(maxsize=1) +def get_settings() -> Settings: + """Return cached singleton settings instance.""" + return Settings() diff --git a/internal/flight/__init__.py b/internal/flight/__init__.py new file mode 100644 index 0000000..602954b --- /dev/null +++ b/internal/flight/__init__.py @@ -0,0 +1,12 @@ +"""Apache Arrow Flight daemon, handlers, and HTTP operational server.""" + +from internal.flight.handlers import FlightHandlers +from internal.flight.ops_server import OpsHTTPServer, OpsState +from internal.flight.server import TabularFlightServer + +__all__ = [ + "FlightHandlers", + "OpsHTTPServer", + "OpsState", + "TabularFlightServer", +] diff --git a/internal/flight/daemon.py b/internal/flight/daemon.py new file mode 100644 index 0000000..df9a747 --- /dev/null +++ b/internal/flight/daemon.py @@ -0,0 +1,294 @@ +"""Daemon lifecycle orchestrator for Arrow Flight server, Ops HTTP, S3 hydration, and pool.""" + +from __future__ import annotations + +import asyncio +import signal +import threading +from types import FrameType +from typing import Any + +from internal.config.settings import Settings, get_settings +from internal.flight.handlers import FlightHandlers +from internal.flight.ops_server import OpsHTTPServer, OpsState +from internal.flight.server import TabularFlightServer +from internal.pool.process_manager import ProcessManager +from internal.storage.cache_manager import CacheManager +from internal.storage.s3_streamer import S3Streamer +from internal.storage.table_resolver import TableResolver +from internal.telemetry.logging import configure_logging, get_logger +from internal.telemetry.metrics import MetricsRegistry +from internal.telemetry.tracing import ( + FlightTracingMiddlewareFactory, + init_tracing, + shutdown_tracing, +) +from internal.validator.ast_validator import ASTValidator + +_logger = get_logger("internal.flight.daemon") + + +class HydrationLoopThread: + """Dedicated background thread running the asyncio event loop for S3 hydration.""" + + def __init__(self) -> None: + self._loop = asyncio.new_event_loop() + self._ready = threading.Event() + self._thread = threading.Thread( + target=self._run, + name="s3-hydration-loop", + daemon=True, + ) + + @property + def loop(self) -> asyncio.AbstractEventLoop: + """Return the managed asyncio event loop.""" + return self._loop + + def _run(self) -> None: + asyncio.set_event_loop(self._loop) + self._loop.call_soon(self._ready.set) + self._loop.run_forever() + self._loop.close() + + def start(self) -> None: + """Start the background asyncio thread and wait until the loop is running.""" + if self._thread.is_alive(): + return + if self._loop.is_closed(): + self._loop = asyncio.new_event_loop() + self._ready = threading.Event() + self._thread = threading.Thread( + target=self._run, + name="s3-hydration-loop", + daemon=True, + ) + self._thread.start() + self._ready.wait(timeout=5.0) + + def stop(self) -> None: + """Stop the asyncio event loop and join its thread.""" + if self._loop.is_running(): + self._loop.call_soon_threadsafe(self._loop.stop) + if self._thread.is_alive(): + self._thread.join(timeout=5.0) + + +class WorkerDaemon: + """Lifecycle orchestrator for the Arrow Flight daemon and Ops HTTP server.""" + + def __init__( + self, + settings: Settings | None = None, + *, + s3_client: Any | None = None, + ) -> None: + self.settings = settings or get_settings() + configure_logging( + level=self.settings.LOG_LEVEL, + service_name=self.settings.SERVICE_NAME, + ) + self.metrics = MetricsRegistry() + self.hydration_thread = HydrationLoopThread() + self.ast_validator = ASTValidator( + max_sql_length=self.settings.MAX_SQL_LENGTH, + max_joins=self.settings.MAX_JOINS, + ) + self.table_resolver = TableResolver(self.settings.S3_BUCKET_ALLOWLIST) + self.cache_manager = CacheManager( + cache_dir=self.settings.CACHE_DIR, + max_bytes=self.settings.CACHE_MAX_BYTES, + metrics=self.metrics, + ) + self.s3_streamer = S3Streamer( + endpoint_url=self.settings.S3_ENDPOINT_URL, + region_name=self.settings.S3_REGION, + chunk_size_bytes=self.settings.S3_CHUNK_SIZE_BYTES, + timeout_seconds=self.settings.S3_TIMEOUT_SECONDS, + client=s3_client, + ) + self.process_manager = ProcessManager(self.settings, metrics=self.metrics) + self.handlers = FlightHandlers( + settings=self.settings, + ast_validator=self.ast_validator, + table_resolver=self.table_resolver, + cache_manager=self.cache_manager, + s3_streamer=self.s3_streamer, + process_manager=self.process_manager, + hydration_loop=self.hydration_thread.loop, + metrics=self.metrics, + ) + self.ops_state = OpsState( + service_name=self.settings.SERVICE_NAME, + process_manager=self.process_manager, + cache_manager=self.cache_manager, + metrics=self.metrics, + ) + self.flight_server: TabularFlightServer | None = None + self.ops_server: OpsHTTPServer | None = None + self._stop_event = threading.Event() + self._flight_thread: threading.Thread | None = None + self._is_shut_down = False + + def start(self) -> None: + """Start hydration loop, S3 client, cache, worker pool, Flight server, and Ops HTTP.""" + self._is_shut_down = False + self._stop_event.clear() + self.ops_state.clear_draining() + + try: + init_tracing(self.settings) + self.hydration_thread.start() + self.handlers.set_hydration_loop(self.hydration_thread.loop) + asyncio.run_coroutine_threadsafe( + self.s3_streamer.start(), + self.hydration_thread.loop, + ).result(timeout=5.0) + + self.cache_manager.initialize() + self.process_manager.start() + + location = f"grpc://{self.settings.FLIGHT_HOST}:{self.settings.FLIGHT_PORT}" + self.flight_server = TabularFlightServer( + location, + self.handlers, + middleware={"otel_tracing": FlightTracingMiddlewareFactory()}, + ) + self._flight_thread = threading.Thread( + target=self.flight_server.serve, + name="arrow-flight-server", + daemon=True, + ) + self._flight_thread.start() + + self.ops_server = OpsHTTPServer( + self.settings.HTTP_HOST, + self.settings.HTTP_PORT, + self.ops_state, + ) + self.ops_server.start() + + _logger.info( + "Started rag-tabular-analytics-worker daemon", + extra={ + "extra_fields": { + "flight_port": self.flight_server.port, + "http_port": self.ops_server.port, + } + }, + ) + except BaseException: + self.shutdown() + raise + + def _shutdown_flight_server(self, drain_timeout_seconds: float = 5.0) -> None: + """Bound Flight server drain and abort stalled GeneratorStream responses on timeout.""" + server = self.flight_server + if server is None: + return + self.flight_server = None + + def _drain() -> None: + server.shutdown() + server.wait() + + drain_thread = threading.Thread( + target=_drain, + name="arrow-flight-drain", + daemon=True, + ) + drain_thread.start() + drain_thread.join(timeout=drain_timeout_seconds) + if drain_thread.is_alive(): + closed_streams = self.handlers.close_active_streams() + _logger.warning( + "Flight server drain timed out; closed active streams", + extra={"extra_fields": {"closed_streams": closed_streams}}, + ) + drain_thread.join(timeout=1.0) + + def shutdown(self) -> None: + """Gracefully drain readiness, shut down Flight server, pool, S3 client, and loop.""" + if self._is_shut_down: + return + self._is_shut_down = True + self._stop_event.set() + self.ops_state.mark_draining() + + _logger.info("Initiating graceful shutdown drain") + + first_exc: BaseException | None = None + try: + self._shutdown_flight_server(drain_timeout_seconds=5.0) + except BaseException as exc: + first_exc = exc + + if self._flight_thread is not None: + if self._flight_thread.is_alive(): + self._flight_thread.join(timeout=5.0) + self._flight_thread = None + + try: + self.process_manager.close() + except BaseException as exc: + if first_exc is None: + first_exc = exc + + if self.ops_server is not None: + try: + self.ops_server.close() + except BaseException as exc: + if first_exc is None: + first_exc = exc + finally: + self.ops_server = None + + if self.hydration_thread.loop.is_running(): + close_future = asyncio.run_coroutine_threadsafe( + self.s3_streamer.close(), + self.hydration_thread.loop, + ) + try: + close_future.result(timeout=5.0) + except BaseException as exc: + close_future.cancel() + if first_exc is None: + first_exc = exc + + try: + self.hydration_thread.stop() + except BaseException as exc: + if first_exc is None: + first_exc = exc + finally: + self.s3_streamer.reset_client_state() + + try: + shutdown_tracing() + except BaseException as exc: + if first_exc is None: + first_exc = exc + + if first_exc is not None: + raise first_exc + _logger.info("Shutdown complete") + + def run_forever(self) -> None: + """Start daemon and block until SIGINT or SIGTERM triggers graceful shutdown.""" + + def _handle_signal(signum: int, _frame: FrameType | None) -> None: + _logger.info( + "Received termination signal", + extra={"extra_fields": {"signal": signum}}, + ) + self._stop_event.set() + + signal.signal(signal.SIGINT, _handle_signal) + signal.signal(signal.SIGTERM, _handle_signal) + + try: + self.start() + while not self._stop_event.is_set(): + self._stop_event.wait(timeout=0.5) + finally: + self.shutdown() diff --git a/internal/flight/handlers.py b/internal/flight/handlers.py new file mode 100644 index 0000000..befe421 --- /dev/null +++ b/internal/flight/handlers.py @@ -0,0 +1,484 @@ +"""Arrow Flight DoGet and DoAction handlers coordinating validation, cache, and DuckDB pool.""" + +from __future__ import annotations + +import asyncio +import json +import threading +import time +import weakref +from collections.abc import Callable, Iterator +from concurrent.futures import Future +from typing import NoReturn + +import pyarrow as pa +import pyarrow.flight as fl +from opentelemetry.trace import Span +from pydantic import ValidationError as PydanticValidationError + +from api.flight.actions import CacheWarmRequest, CacheWarmResponse, HealthCheckResponse +from api.flight.schema import ( + ConcurrencyLimitError, + FlightQueryTicket, + map_exception_to_flight_error, +) +from internal.config.settings import Settings +from internal.pool.circuit_breaker import QueryTimeoutError, RecycleReason, WorkerCrashError +from internal.pool.process_manager import ProcessManager, QueryStream +from internal.storage.cache_manager import CacheManager, PinnedArtifact +from internal.storage.s3_streamer import S3Streamer +from internal.storage.table_resolver import ResolvedTable, TableResolver +from internal.telemetry.logging import get_logger +from internal.telemetry.metrics import MetricsRegistry +from internal.telemetry.tracing import ( + get_current_trace_id, + get_tracer, + record_sanitized_exception, + sanitize_span_attributes, + start_span, +) +from internal.validator.ast_validator import ASTValidator +from internal.validator.exceptions import ValidationError as SQLValidationError + +_logger = get_logger("internal.flight.handlers") + + +class _OwnedFlightBatchIterator: + """Iterator owning Flight query stream cleanup across iteration, close, and GC.""" + + def __init__( + self, + *, + query_stream: QueryStream, + pinned: PinnedArtifact, + metrics: MetricsRegistry, + semaphore: threading.BoundedSemaphore, + start_time: float, + request_id: str, + tenant_id: str, + table_hash: str, + exec_span: Span | None = None, + trace_id: str | None = None, + on_cleanup: Callable[[_OwnedFlightBatchIterator], None] | None = None, + ) -> None: + self._query_stream = query_stream + self._batch_iter = iter(query_stream) + self._pinned = pinned + self._metrics = metrics + self._semaphore = semaphore + self._start_time = start_time + self._request_id = request_id + self._tenant_id = tenant_id + self._table_hash = table_hash + self._exec_span = exec_span + self._trace_id = trace_id + self._on_cleanup = on_cleanup + self._stream_status = "ok" + self._stream_bytes = 0 + self._cleanup_lock = threading.Lock() + self._cleaned_up = False + self._aborted = False + + def __iter__(self) -> Iterator[pa.RecordBatch]: + return self + + def _raise_terminal_exception(self) -> NoReturn: + if self._aborted: + raise fl.FlightCancelledError("Query stream aborted during shutdown.") from None + raise StopIteration from None + + def __next__(self) -> pa.RecordBatch: + with self._cleanup_lock: + if self._cleaned_up: + self._raise_terminal_exception() + try: + batch = next(self._batch_iter) + except StopIteration: + with self._cleanup_lock: + aborted = self._aborted + self._cleanup() + if aborted: + raise fl.FlightCancelledError("Query stream aborted during shutdown.") from None + raise + except QueryTimeoutError as exc: + with self._cleanup_lock: + if self._cleaned_up: + self._raise_terminal_exception() + self._stream_status = "timeout" + self._metrics.inc_subprocess_recycle(RecycleReason.TIMEOUT.value) + if self._exec_span is not None: + record_sanitized_exception(self._exec_span, exc) + self._cleanup() + raise map_exception_to_flight_error(exc) from exc + except WorkerCrashError as exc: + with self._cleanup_lock: + if self._cleaned_up: + self._raise_terminal_exception() + self._stream_status = "worker_crash" + self._metrics.inc_subprocess_recycle(exc.reason.value) + if self._exec_span is not None: + record_sanitized_exception(self._exec_span, exc) + self._cleanup() + raise map_exception_to_flight_error(exc) from exc + except Exception as exc: + with self._cleanup_lock: + if self._cleaned_up: + self._raise_terminal_exception() + self._stream_status = "error" + if self._exec_span is not None: + record_sanitized_exception(self._exec_span, exc) + self._cleanup() + raise map_exception_to_flight_error(exc) from exc + + self._stream_bytes += batch.nbytes + self._metrics.add_stream_bytes(batch.nbytes) + return batch + + def close(self) -> None: + """Close underlying stream and release owned resources idempotently.""" + with self._cleanup_lock: + if not self._cleaned_up: + self._aborted = True + if self._stream_status == "ok": + self._stream_status = "closed" + self._cleanup() + + def __del__(self) -> None: + self._cleanup() + + def _cleanup(self) -> None: + with self._cleanup_lock: + if self._cleaned_up: + return + self._cleaned_up = True + + self._query_stream.close() + duration_s = time.perf_counter() - self._start_time + self._metrics.observe_query_duration(self._pinned.cache_state, duration_s) + self._metrics.dec_active_queries() + self._pinned.release() + self._semaphore.release() + if self._on_cleanup is not None: + self._on_cleanup(self) + if self._exec_span is not None: + self._exec_span.set_attributes( + sanitize_span_attributes( + { + "cache_state": self._pinned.cache_state, + "stream_bytes": self._stream_bytes, + "status": self._stream_status, + } + ) + ) + self._exec_span.end() + log_extra: dict[str, object] = { + "request_id": self._request_id, + "tenant_id": self._tenant_id, + "table_hash": self._table_hash, + "cache_state": self._pinned.cache_state, + "duration_ms": round(duration_s * 1000.0, 2), + "status": self._stream_status, + "extra_fields": {"stream_bytes": self._stream_bytes}, + } + if self._trace_id is not None: + log_extra["trace_id"] = self._trace_id + _logger.info("Completed Flight DoGet query stream", extra=log_extra) + + +class FlightHandlers: + """Coordinates Arrow Flight RPCs with AST validation, NVMe cache, S3, and ProcessManager.""" + + def __init__( + self, + *, + settings: Settings, + ast_validator: ASTValidator, + table_resolver: TableResolver, + cache_manager: CacheManager, + s3_streamer: S3Streamer, + process_manager: ProcessManager, + hydration_loop: asyncio.AbstractEventLoop, + metrics: MetricsRegistry, + ) -> None: + self._settings = settings + self._ast_validator = ast_validator + self._table_resolver = table_resolver + self._cache_manager = cache_manager + self._s3_streamer = s3_streamer + self._process_manager = process_manager + self._hydration_loop = hydration_loop + self._metrics = metrics + self._concurrency_semaphore = threading.BoundedSemaphore(settings.MAX_CONCURRENT_QUERIES) + self._active_streams_lock = threading.RLock() + self._active_streams: weakref.WeakSet[_OwnedFlightBatchIterator] = weakref.WeakSet() + + def _unregister_stream(self, stream: _OwnedFlightBatchIterator) -> None: + with self._active_streams_lock: + self._active_streams.discard(stream) + + def close_active_streams(self) -> int: + """Force-close any active GeneratorStream iterators during shutdown drain timeout.""" + with self._active_streams_lock: + streams = list(self._active_streams) + for stream in streams: + stream.close() + return len(streams) + + def set_hydration_loop(self, loop: asyncio.AbstractEventLoop) -> None: + """Update the asyncio event loop used for S3 hydration.""" + self._hydration_loop = loop + + def _await_hydration(self, resolved: ResolvedTable) -> PinnedArtifact: + """Await hydration on the asyncio loop, releasing any artifact if the caller abandons.""" + future: Future[PinnedArtifact] = asyncio.run_coroutine_threadsafe( + self._cache_manager.get_or_hydrate(resolved, self._s3_streamer), + self._hydration_loop, + ) + lock = threading.Lock() + caller_took_ownership = False + caller_abandoned = False + released = False + + def _release_once(done_fut: Future[PinnedArtifact]) -> None: + nonlocal released + if done_fut.cancelled(): + return + try: + artifact = done_fut.result() + except BaseException: + return + with lock: + if caller_took_ownership or not caller_abandoned or released: + return + released = True + artifact.release() + + future.add_done_callback(_release_once) + try: + artifact = future.result(timeout=self._settings.S3_TIMEOUT_SECONDS + 2.0) + with lock: + caller_took_ownership = True + return artifact + except BaseException: + with lock: + caller_abandoned = True + if future.done(): + _release_once(future) + raise + + def _hydrate_sync( + self, + tenant_id: str, + table_hash: str, + s3_bucket: str, + s3_key: str, + ) -> PinnedArtifact: + """Resolve tenant/S3 invariants and hydrate/pin artifact on the asyncio loop.""" + resolved = self._table_resolver.resolve( + tenant_id=tenant_id, + table_hash=table_hash, + s3_bucket=s3_bucket, + s3_key=s3_key, + ) + return self._await_hydration(resolved) + + def handle_do_get(self, ticket_bytes: bytes) -> fl.FlightDataStream: + """Validate ticket and SQL AST, hydrate Parquet, and stream RecordBatch chunks.""" + start_time = time.perf_counter() + + with start_span("flight.do_get") as do_get_span: + try: + ticket = FlightQueryTicket.model_validate_json(ticket_bytes) + except (PydanticValidationError, ValueError) as exc: + _logger.warning("Rejected malformed Flight ticket payload") + raise map_exception_to_flight_error(exc) from exc + + do_get_span.set_attributes( + sanitize_span_attributes( + { + "request_id": ticket.request_id, + "tenant_id": ticket.tenant_id, + "table_hash": ticket.table_hash, + "table_alias": ticket.table_alias, + } + ) + ) + + try: + resolved = self._table_resolver.resolve( + tenant_id=ticket.tenant_id, + table_hash=ticket.table_hash, + s3_bucket=ticket.s3_bucket, + s3_key=ticket.s3_key, + ) + with start_span( + "sqlglot.validate", + attributes={ + "request_id": ticket.request_id, + "tenant_id": ticket.tenant_id, + "table_hash": ticket.table_hash, + }, + ): + self._ast_validator.validate(ticket.sql, ticket.table_alias) + except SQLValidationError as exc: + self._metrics.inc_ast_rejection(exc.rule) + _logger.warning( + "Rejected SQL query during AST validation", + extra={ + "request_id": ticket.request_id, + "tenant_id": ticket.tenant_id, + "table_hash": ticket.table_hash, + "rule": exc.rule, + }, + ) + raise map_exception_to_flight_error(exc) from exc + except Exception as exc: + _logger.warning( + "Rejected Flight ticket during table resolution", + extra={ + "request_id": ticket.request_id, + "tenant_id": ticket.tenant_id, + "table_hash": ticket.table_hash, + }, + ) + raise map_exception_to_flight_error(exc) from exc + + if not self._concurrency_semaphore.acquire(blocking=False): + _logger.warning( + "Pod concurrency limit saturated; returning RESOURCE_EXHAUSTED", + extra={ + "request_id": ticket.request_id, + "tenant_id": ticket.tenant_id, + "table_hash": ticket.table_hash, + }, + ) + raise map_exception_to_flight_error(ConcurrencyLimitError()) + + pinned: PinnedArtifact | None = None + active_incremented = False + handed_off = False + exec_span: Span | None = None + + try: + self._metrics.inc_active_queries() + active_incremented = True + + with start_span( + "storage.get_or_hydrate", + attributes={ + "request_id": ticket.request_id, + "tenant_id": ticket.tenant_id, + "table_hash": ticket.table_hash, + }, + ) as hydrate_span: + pinned = self._await_hydration(resolved) + hydrate_span.set_attribute("cache_state", pinned.cache_state) + + exec_span = get_tracer().start_span( + "duckdb.subprocess_execute", + attributes=sanitize_span_attributes( + { + "request_id": ticket.request_id, + "tenant_id": ticket.tenant_id, + "table_hash": ticket.table_hash, + "cache_state": pinned.cache_state, + } + ), + ) + query_stream = self._process_manager.stream_query( + ticket.sql, + pinned.path, + ticket.table_alias, + ) + owned_iter = _OwnedFlightBatchIterator( + query_stream=query_stream, + pinned=pinned, + metrics=self._metrics, + semaphore=self._concurrency_semaphore, + start_time=start_time, + request_id=ticket.request_id, + tenant_id=ticket.tenant_id, + table_hash=ticket.table_hash, + exec_span=exec_span, + trace_id=get_current_trace_id(), + on_cleanup=self._unregister_stream, + ) + with self._active_streams_lock: + self._active_streams.add(owned_iter) + handed_off = True + return fl.GeneratorStream(query_stream.schema, owned_iter) + except QueryTimeoutError as exc: + self._metrics.inc_subprocess_recycle(RecycleReason.TIMEOUT.value) + if exec_span is not None: + record_sanitized_exception(exec_span, exc) + raise map_exception_to_flight_error(exc) from exc + except WorkerCrashError as exc: + self._metrics.inc_subprocess_recycle(exc.reason.value) + if exec_span is not None: + record_sanitized_exception(exec_span, exc) + raise map_exception_to_flight_error(exc) from exc + except Exception as exc: + if exec_span is not None: + record_sanitized_exception(exec_span, exc) + raise map_exception_to_flight_error(exc) from exc + finally: + if not handed_off: + if exec_span is not None: + exec_span.end() + if active_incremented: + self._metrics.dec_active_queries() + if pinned is not None: + pinned.release() + self._concurrency_semaphore.release() + + def handle_cache_warm(self, body: bytes) -> bytes: + """Idempotently hydrate a tenant-validated Parquet artifact into the NVMe cache.""" + with start_span("flight.do_action.cache_warm") as span: + try: + req = CacheWarmRequest.model_validate_json(body) + except (PydanticValidationError, ValueError) as exc: + raise map_exception_to_flight_error(exc) from exc + + span.set_attributes( + sanitize_span_attributes( + { + "tenant_id": req.tenant_id, + "table_hash": req.table_hash, + } + ) + ) + pinned: PinnedArtifact | None = None + try: + pinned = self._hydrate_sync( + tenant_id=req.tenant_id, + table_hash=req.table_hash, + s3_bucket=req.s3_bucket, + s3_key=req.s3_key, + ) + span.set_attribute("cache_state", pinned.cache_state) + resp = CacheWarmResponse(status="warmed", bytes=pinned.size_bytes) + _logger.info( + "Warmed Parquet artifact in Tier-1 cache", + extra={ + "tenant_id": req.tenant_id, + "table_hash": req.table_hash, + "cache_state": pinned.cache_state, + "extra_fields": {"bytes": pinned.size_bytes}, + }, + ) + return json.dumps(resp.model_dump()).encode("utf-8") + except Exception as exc: + raise map_exception_to_flight_error(exc) from exc + finally: + if pinned is not None: + pinned.release() + + def handle_health_check(self) -> bytes: + """Return worker pool health status for Flight DoAction('health_check').""" + with start_span("flight.do_action.health_check"): + healthy = self._process_manager.is_healthy() + resp = HealthCheckResponse( + status="ok" if healthy else "degraded", + active_workers=self._settings.POOL_SIZE if healthy else 0, + ) + return json.dumps(resp.model_dump()).encode("utf-8") diff --git a/internal/flight/ops_server.py b/internal/flight/ops_server.py new file mode 100644 index 0000000..e82a726 --- /dev/null +++ b/internal/flight/ops_server.py @@ -0,0 +1,203 @@ +"""HTTP operational & OpenAPI 3.1 server (/healthz, /ready, /metrics, /openapi.yaml, /docs).""" + +from __future__ import annotations + +import json +import threading +from http import HTTPStatus +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Any + +from internal.pool.process_manager import ProcessManager +from internal.storage.cache_manager import CacheManager +from internal.telemetry.logging import get_logger +from internal.telemetry.metrics import MetricsRegistry + +_logger = get_logger("internal.flight.ops_server") + +_DEFAULT_OPENAPI_PATH = Path(__file__).resolve().parents[2] / "api" / "openapi" / "openapi.yaml" + +_SWAGGER_UI_HTML = """ + + + + + rag-tabular-analytics-worker API Docs + + + +
+ + + + +""" + + +class OpsState: + """Shared readiness, drain, metrics, and OpenAPI state for the HTTP ops server.""" + + def __init__( + self, + *, + service_name: str, + process_manager: ProcessManager, + cache_manager: CacheManager, + metrics: MetricsRegistry, + openapi_path: Path = _DEFAULT_OPENAPI_PATH, + ) -> None: + self.service_name = service_name + self.process_manager = process_manager + self.cache_manager = cache_manager + self.metrics = metrics + self.openapi_path = openapi_path + self._draining = threading.Event() + + def mark_draining(self) -> None: + """Mark the worker as draining so /ready returns 503 Service Unavailable.""" + self._draining.set() + + def clear_draining(self) -> None: + """Clear the draining flag when starting or restarting the daemon.""" + self._draining.clear() + + def is_draining(self) -> bool: + """Return True if graceful shutdown drain has been initiated.""" + return self._draining.is_set() + + def readiness_payload(self) -> tuple[int, dict[str, Any]]: + """Evaluate readiness checks and return (HTTP status code, JSON body).""" + if self.is_draining(): + return HTTPStatus.SERVICE_UNAVAILABLE, { + "status": "draining", + "checks": {"drain": "shutting_down"}, + } + + pool_ok = self.process_manager.is_healthy() + cache_ok = self.cache_manager.is_writable() + if pool_ok and cache_ok: + return HTTPStatus.OK, { + "status": "ready", + "checks": { + "process_pool": "healthy", + "cache_mount": "writable", + }, + } + + return HTTPStatus.SERVICE_UNAVAILABLE, { + "status": "not_ready", + "checks": { + "process_pool": "healthy" if pool_ok else "unhealthy", + "cache_mount": "writable" if cache_ok else "unwritable", + }, + } + + +class _OpsHTTPServer(ThreadingHTTPServer): + """ThreadingHTTPServer carrying typed OpsState.""" + + daemon_threads = True + + def __init__( + self, + server_address: tuple[str, int], + request_handler_class: type[BaseHTTPRequestHandler], + ops_state: OpsState, + ) -> None: + self.ops_state = ops_state + super().__init__(server_address, request_handler_class) + + +class _OpsRequestHandler(BaseHTTPRequestHandler): + """Handles Kubernetes probes, Prometheus scrape, OpenAPI spec, and Swagger UI.""" + + server: _OpsHTTPServer + + def log_message(self, format: str, *args: Any) -> None: # noqa: A002 + """Suppress default stderr logging in favor of structured JSON logging.""" + _logger.debug( + "Ops HTTP request", + extra={"extra_fields": {"path": self.path}}, + ) + + def _send_bytes(self, status_code: int, content_type: str, body: bytes) -> None: + self.send_response(status_code) + self.send_header("Content-Type", content_type) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self) -> None: + state = self.server.ops_state + path = self.path.split("?", 1)[0] + + if path == "/healthz": + body = json.dumps({"status": "ok", "service": state.service_name}).encode("utf-8") + self._send_bytes(HTTPStatus.OK, "application/json", body) + return + + if path == "/ready": + status_code, payload = state.readiness_payload() + body = json.dumps(payload).encode("utf-8") + self._send_bytes(status_code, "application/json", body) + return + + if path == "/metrics": + body = state.metrics.render_prometheus() + self._send_bytes(HTTPStatus.OK, "text/plain; version=0.0.4; charset=utf-8", body) + return + + if path == "/openapi.yaml": + try: + spec_bytes = state.openapi_path.read_bytes() + except OSError: + self._send_bytes( + HTTPStatus.NOT_FOUND, + "application/json", + b'{"error":"OpenAPI specification not found"}', + ) + return + self._send_bytes(HTTPStatus.OK, "application/yaml; charset=utf-8", spec_bytes) + return + + if path == "/docs": + self._send_bytes(HTTPStatus.OK, "text/html; charset=utf-8", _SWAGGER_UI_HTML.encode()) + return + + self._send_bytes(HTTPStatus.NOT_FOUND, "application/json", b'{"error":"Not found"}') + + +class OpsHTTPServer: + """Lifecycle wrapper around ThreadingHTTPServer running on a background daemon thread.""" + + def __init__(self, host: str, port: int, ops_state: OpsState) -> None: + self._server = _OpsHTTPServer((host, port), _OpsRequestHandler, ops_state) + self._thread: threading.Thread | None = None + + @property + def port(self) -> int: + """Return bound TCP port (useful when binding to ephemeral port 0 in tests).""" + return int(self._server.server_address[1]) + + def start(self) -> None: + """Start serving HTTP ops requests on a background thread.""" + if self._thread is None: + self._thread = threading.Thread( + target=self._server.serve_forever, + name="ops-http-server", + daemon=True, + ) + self._thread.start() + + def close(self) -> None: + """Shut down the HTTP ops server and join its thread.""" + self._server.shutdown() + self._server.server_close() + if self._thread is not None: + self._thread.join(timeout=5.0) + self._thread = None diff --git a/internal/flight/server.py b/internal/flight/server.py new file mode 100644 index 0000000..9abaa8a --- /dev/null +++ b/internal/flight/server.py @@ -0,0 +1,65 @@ +"""Apache Arrow FlightServerBase implementation for rag-tabular-analytics-worker.""" + +from __future__ import annotations + +from collections.abc import Iterator +from typing import Any + +import pyarrow.flight as fl + +from api.flight.schema import FlightErrorCode, sanitize_client_message +from internal.flight.handlers import FlightHandlers +from internal.telemetry.tracing import attach_flight_trace_context + + +class TabularFlightServer(fl.FlightServerBase): # type: ignore[misc] + """HTTP/2 gRPC Arrow Flight server handling DoGet and DoAction RPCs.""" + + def __init__( + self, + location: str, + handlers: FlightHandlers, + **kwargs: Any, + ) -> None: + super().__init__(location, **kwargs) + self._handlers = handlers + + def do_get( + self, + context: fl.ServerCallContext, + ticket: fl.Ticket, + ) -> fl.FlightDataStream: + """Execute a validated SQL query and stream zero-copy Arrow RecordBatch chunks.""" + with attach_flight_trace_context(context): + return self._handlers.handle_do_get(ticket.ticket) + + def list_actions(self, context: fl.ServerCallContext) -> list[tuple[str, str]]: + """Advertise supported Flight DoAction operations.""" + return [ + ("cache_warm", "Pre-hydrate a content-addressed Parquet artifact into Tier-1 cache."), + ("health_check", "Check worker pool health and active worker capacity."), + ] + + def do_action( + self, + context: fl.ServerCallContext, + action: fl.Action, + ) -> Iterator[fl.Result]: + """Handle `cache_warm` and `health_check` Flight actions.""" + action_type = action.type + with attach_flight_trace_context(context): + if action_type == "cache_warm": + payload = self._handlers.handle_cache_warm(action.body.to_pybytes()) + yield fl.Result(payload) + return + if action_type == "health_check": + payload = self._handlers.handle_health_check() + yield fl.Result(payload) + return + + raise fl.FlightServerError( + sanitize_client_message( + FlightErrorCode.INVALID_ARGUMENT, + f"Unsupported Flight action: {action_type}", + ) + ) diff --git a/internal/pool/__init__.py b/internal/pool/__init__.py new file mode 100644 index 0000000..19b3df7 --- /dev/null +++ b/internal/pool/__init__.py @@ -0,0 +1,17 @@ +"""Disposable subprocess worker pool and circuit breaker package.""" + +from internal.pool.circuit_breaker import ( + CircuitBreaker, + QueryTimeoutError, + RecycleReason, + WorkerCrashError, +) +from internal.pool.process_manager import ProcessManager + +__all__ = [ + "CircuitBreaker", + "ProcessManager", + "QueryTimeoutError", + "RecycleReason", + "WorkerCrashError", +] diff --git a/internal/pool/circuit_breaker.py b/internal/pool/circuit_breaker.py new file mode 100644 index 0000000..2b82718 --- /dev/null +++ b/internal/pool/circuit_breaker.py @@ -0,0 +1,93 @@ +"""Per-worker 5s execution circuit breaker and recycle reason tracking.""" + +from __future__ import annotations + +from concurrent.futures import Future +from enum import StrEnum +from threading import Lock +from typing import Any + +from pebble import ProcessExpired + +from internal.telemetry.logging import get_logger + +_logger = get_logger("internal.pool.circuit_breaker") + + +class RecycleReason(StrEnum): + """Canonical worker subprocess recycle reasons (matches Prometheus label values).""" + + TIMEOUT = "TIMEOUT" + MEMORY_LIMIT = "MEMORY_LIMIT" + TASK_LIMIT = "TASK_LIMIT" + CRASH = "CRASH" + + +class QueryTimeoutError(TimeoutError): + """Raised when a child worker exceeds the hard execution deadline.""" + + +class WorkerCrashError(RuntimeError): + """Raised when a child worker terminates unexpectedly (SIGKILL, OOM, or panic).""" + + def __init__( + self, + message: str, + *, + reason: RecycleReason = RecycleReason.CRASH, + ) -> None: + super().__init__(message) + self.reason = reason + + +class CircuitBreaker: + """Monitors subprocess futures against hard deadlines and tracks recycle reasons.""" + + def __init__(self, timeout_seconds: float = 5.0) -> None: + self._timeout_seconds = timeout_seconds + self._lock = Lock() + self._recycle_counts: dict[RecycleReason, int] = {reason: 0 for reason in RecycleReason} + + @property + def timeout_seconds(self) -> float: + """Return configured hard execution deadline in seconds.""" + return self._timeout_seconds + + def record_recycle(self, reason: RecycleReason) -> None: + """Increment counter for a worker recycle reason.""" + with self._lock: + self._recycle_counts[reason] += 1 + + def get_recycle_counts(self) -> dict[RecycleReason, int]: + """Return a snapshot of worker recycle counts by reason.""" + with self._lock: + return dict(self._recycle_counts) + + def await_result(self, future: Future[Any]) -> Any: + """Wait for a pebble subprocess future, translating timeouts and crashes.""" + try: + return future.result(timeout=self._timeout_seconds) + except TimeoutError as exc: + self.record_recycle(RecycleReason.TIMEOUT) + _logger.warning( + "Worker subprocess exceeded execution deadline and was terminated", + extra={"reason": RecycleReason.TIMEOUT.value}, + ) + raise QueryTimeoutError( + f"Query exceeded {self._timeout_seconds:.1f}s execution deadline." + ) from exc + except ProcessExpired as exc: + exit_code = getattr(exc, "exitcode", None) + reason = RecycleReason.MEMORY_LIMIT if exit_code in (-9, 137) else RecycleReason.CRASH + self.record_recycle(reason) + _logger.error( + "Worker subprocess terminated unexpectedly", + extra={ + "reason": reason.value, + "extra_fields": {"exit_code": exit_code}, + }, + ) + raise WorkerCrashError( + f"Worker subprocess exited unexpectedly (exit_code={exit_code}).", + reason=reason, + ) from exc diff --git a/internal/pool/process_manager.py b/internal/pool/process_manager.py new file mode 100644 index 0000000..fd09aee --- /dev/null +++ b/internal/pool/process_manager.py @@ -0,0 +1,377 @@ +"""Disposable pebble.ProcessPool supervisor with chunked Arrow IPC pipe streaming.""" + +from __future__ import annotations + +import contextlib +import io +import os +import threading +from collections.abc import Callable, Iterator +from concurrent.futures import Future +from multiprocessing.connection import Client, Listener +from pathlib import Path +from typing import Any, Protocol + +import pyarrow as pa +from pebble import ProcessPool + +from internal.config.settings import Settings +from internal.pool.circuit_breaker import ( + CircuitBreaker, + QueryTimeoutError, + RecycleReason, + WorkerCrashError, +) +from internal.pool.worker_task import ( + init_worker_process, + ping_worker, + run_sandboxed_query_task, +) +from internal.telemetry.logging import get_logger +from internal.telemetry.metrics import MetricsRegistry + +_logger = get_logger("internal.pool.process_manager") + + +class _IpcRecvConn(Protocol): + """Structural protocol for Connection / PipeConnection receive side.""" + + def recv_bytes(self, maxlength: int | None = None, /) -> bytes: ... + def close(self) -> None: ... + + +class _ConnectionRawReader(io.RawIOBase): + """Readable binary stream backed by a multiprocessing Connection.""" + + def __init__(self, conn: _IpcRecvConn) -> None: + super().__init__() + self._conn = conn + self._buffer = bytearray() + self._eof = False + + def readable(self) -> bool: + return True + + def readinto(self, b: bytearray | memoryview) -> int: # type: ignore[override] + while not self._buffer and not self._eof: + try: + chunk = self._conn.recv_bytes() + except EOFError: + self._eof = True + break + if not chunk: + self._eof = True + break + self._buffer.extend(chunk) + + if not self._buffer: + return 0 + + n = min(len(b), len(self._buffer)) + b[:n] = self._buffer[:n] + del self._buffer[:n] + return n + + +class QueryStream: + """Iterable Arrow RecordBatch stream carrying the IPC schema and owning IPC cleanup.""" + + def __init__( + self, + *, + schema: pa.Schema, + stream_reader: pa.ipc.RecordBatchStreamReader, + conn: _IpcRecvConn, + listener: Listener, + future: Future[tuple[int, int]], + circuit_breaker: CircuitBreaker, + on_task_complete: Callable[[int], None] | None = None, + ) -> None: + self.schema = schema + self._stream_reader = stream_reader + self._conn = conn + self._listener = listener + self._future = future + self._circuit_breaker = circuit_breaker + self._on_task_complete = on_task_complete + self._closed = False + self._iterated = False + + def close(self) -> None: + """Close the Arrow stream reader, IPC connection, and listener idempotently.""" + if self._closed: + return + self._closed = True + with contextlib.suppress(Exception): + self._stream_reader.close() + with contextlib.suppress(OSError): + self._conn.close() + with contextlib.suppress(OSError): + self._listener.close() + + def __del__(self) -> None: + self.close() + + def __iter__(self) -> Iterator[pa.RecordBatch]: + if self._iterated: + raise RuntimeError("QueryStream can only be iterated once.") + self._iterated = True + try: + try: + yield from self._stream_reader + except (EOFError, OSError, pa.ArrowInvalid) as exc: + self._circuit_breaker.await_result(self._future) + raise WorkerCrashError(f"Arrow IPC stream interrupted: {exc}") from exc + result = self._circuit_breaker.await_result(self._future) + if ( + self._on_task_complete is not None + and isinstance(result, tuple) + and len(result) == 2 + ): + self._on_task_complete(int(result[1])) + finally: + self.close() + + +class ProcessManager: + """Supervises disposable DuckDB worker subprocesses via pebble.ProcessPool.""" + + def __init__(self, settings: Settings, metrics: MetricsRegistry | None = None) -> None: + self._settings = settings + self._metrics = metrics + self._circuit_breaker = CircuitBreaker(timeout_seconds=settings.QUERY_TIMEOUT_SECONDS) + self._pool: ProcessPool | None = None + self._task_counts_lock = threading.Lock() + self._pid_task_counts: dict[int, int] = {} + + @property + def circuit_breaker(self) -> CircuitBreaker: + """Return the circuit breaker instance.""" + return self._circuit_breaker + + def _record_worker_task_completion(self, worker_pid: int) -> None: + """Track completed tasks per worker PID and emit TASK_LIMIT recycle events.""" + recycled = False + with self._task_counts_lock: + count = self._pid_task_counts.get(worker_pid, 0) + 1 + if count >= self._settings.MAX_TASKS_PER_CHILD: + self._pid_task_counts.pop(worker_pid, None) + recycled = True + else: + self._pid_task_counts[worker_pid] = count + if len(self._pid_task_counts) > self._settings.POOL_SIZE * 4: + oldest_pid = next(iter(self._pid_task_counts)) + self._pid_task_counts.pop(oldest_pid, None) + + if recycled: + self._circuit_breaker.record_recycle(RecycleReason.TASK_LIMIT) + if self._metrics is not None: + self._metrics.inc_subprocess_recycle(RecycleReason.TASK_LIMIT.value) + _logger.info( + "Worker subprocess reached MAX_TASKS_PER_CHILD and was recycled", + extra={ + "reason": RecycleReason.TASK_LIMIT.value, + "worker_pid": worker_pid, + "extra_fields": {"max_tasks_per_child": self._settings.MAX_TASKS_PER_CHILD}, + }, + ) + + def start(self) -> None: + """Initialize and pre-warm the disposable subprocess pool.""" + if self._pool is None: + self._pool = ProcessPool( + max_workers=self._settings.POOL_SIZE, + max_tasks=self._settings.MAX_TASKS_PER_CHILD, + initializer=init_worker_process, + initargs=(self._settings.WORKER_PROCESS_RLIMIT_BYTES,), + ) + if self._settings.MAX_TASKS_PER_CHILD > 1: + warmup_timeout = max(30.0, self._settings.QUERY_TIMEOUT_SECONDS * 4.0) + warmup_futures = [ + self._pool.schedule(ping_worker, timeout=warmup_timeout) + for _ in range(self._settings.POOL_SIZE) + ] + for fut in warmup_futures: + try: + worker_pid = int(fut.result(timeout=warmup_timeout)) + self._record_worker_task_completion(worker_pid) + except Exception as exc: + _logger.warning( + "Worker warm-up task did not complete cleanly", + extra={"extra_fields": {"error_type": type(exc).__name__}}, + ) + _logger.info( + "Started disposable DuckDB worker pool", + extra={ + "extra_fields": { + "pool_size": self._settings.POOL_SIZE, + "max_tasks_per_child": self._settings.MAX_TASKS_PER_CHILD, + "worker_memory_limit": self._settings.worker_memory_limit_str, + } + }, + ) + + def close(self) -> None: + """Shut down the subprocess pool and wait for workers to exit.""" + if self._pool is not None: + self._pool.close() # type: ignore[no-untyped-call] + self._pool.join() + self._pool = None + + def __enter__(self) -> ProcessManager: + self.start() + return self + + def __exit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None: + self.close() + + def is_healthy(self) -> bool: + """Return True if the worker pool is initialized and active.""" + return self._pool is not None and self._pool.active + + def stream_query( + self, + sql: str, + parquet_path: Path, + table_alias: str, + ) -> QueryStream: + """Dispatch a validated SQL query to a worker and return a schema-aware batch stream.""" + if self._pool is None: + raise RuntimeError("ProcessManager must be started before executing queries.") + + import threading + + authkey = os.urandom(32) + listener = Listener(authkey=authkey) + conn: _IpcRecvConn | None = None + future: Future[tuple[int, int]] = self._pool.schedule( + run_sandboxed_query_task, + args=( + sql, + str(parquet_path), + table_alias, + self._settings.worker_memory_limit_str, + self._settings.DUCKDB_THREADS, + "", + self._settings.ARROW_BATCH_SIZE, + str(listener.address), + authkey, + ), + timeout=self._settings.QUERY_TIMEOUT_SECONDS, + ) + + accept_done = threading.Event() + aborted = threading.Event() + wake_lock = threading.Lock() + accepted_conn: list[_IpcRecvConn | None] = [None] + accept_error: list[BaseException | None] = [None] + + def _accept_worker() -> None: + try: + incoming = listener.accept() + if aborted.is_set(): + with contextlib.suppress(OSError): + incoming.close() + return + accepted_conn[0] = incoming + except BaseException as exc: + if not aborted.is_set(): + accept_error[0] = exc + finally: + accept_done.set() + + accept_thread = threading.Thread( + target=_accept_worker, + name="ipc-listener-accept", + daemon=True, + ) + accept_thread.start() + + def _wake_listener() -> None: + with wake_lock: + aborted.set() + if accept_done.is_set(): + with contextlib.suppress(OSError): + listener.close() + return + + def _connect_wake() -> None: + with contextlib.suppress(OSError, EOFError, Exception): + wake_conn = Client(listener.address) + wake_conn.close() + + wake_thread = threading.Thread( + target=_connect_wake, + name="ipc-listener-wake", + daemon=True, + ) + wake_thread.start() + accept_done.wait(timeout=0.5) + wake_thread.join(timeout=0.2) + with contextlib.suppress(OSError): + listener.close() + + def _on_future_done(_done_future: Future[tuple[int, int]]) -> None: + if not accept_done.is_set() and ( + _done_future.cancelled() or _done_future.exception() is not None + ): + _wake_listener() + + future.add_done_callback(_on_future_done) + + handed_off = False + try: + if not accept_done.wait(timeout=self._settings.QUERY_TIMEOUT_SECONDS): + _wake_listener() + accept_thread.join(timeout=1.0) + try: + self._circuit_breaker.await_result(future) + except QueryTimeoutError: + raise + except (OSError, EOFError, WorkerCrashError) as exc: + self._circuit_breaker.record_recycle(RecycleReason.TIMEOUT) + raise QueryTimeoutError( + "Worker failed to connect within " + f"{self._settings.QUERY_TIMEOUT_SECONDS:.1f}s." + ) from exc + self._circuit_breaker.record_recycle(RecycleReason.TIMEOUT) + raise QueryTimeoutError( + f"Worker failed to connect within {self._settings.QUERY_TIMEOUT_SECONDS:.1f}s." + ) + + if accept_error[0] is not None: + self._circuit_breaker.await_result(future) + raise WorkerCrashError(f"IPC accept failed: {accept_error[0]}") from accept_error[0] + + conn = accepted_conn[0] + if conn is None: + self._circuit_breaker.await_result(future) + raise WorkerCrashError("IPC accept aborted before worker connected.") + + raw_reader = _ConnectionRawReader(conn) + buffered_reader = io.BufferedReader(raw_reader) + try: + stream_reader = pa.ipc.open_stream(buffered_reader) + except (EOFError, OSError, pa.ArrowInvalid) as exc: + self._circuit_breaker.await_result(future) + raise WorkerCrashError(f"Arrow IPC stream interrupted: {exc}") from exc + + handed_off = True + return QueryStream( + schema=stream_reader.schema, + stream_reader=stream_reader, + conn=conn, + listener=listener, + future=future, + circuit_breaker=self._circuit_breaker, + on_task_complete=self._record_worker_task_completion, + ) + except (QueryTimeoutError, WorkerCrashError): + raise + finally: + if not handed_off: + if conn is not None: + with contextlib.suppress(OSError): + conn.close() + with contextlib.suppress(OSError): + listener.close() diff --git a/internal/pool/worker_task.py b/internal/pool/worker_task.py new file mode 100644 index 0000000..5441c10 --- /dev/null +++ b/internal/pool/worker_task.py @@ -0,0 +1,101 @@ +"""Top-level picklable worker task executed inside disposable pebble subprocesses.""" + +from __future__ import annotations + +import io +import os +import sys +from multiprocessing.connection import Client +from pathlib import Path +from typing import Protocol + +from internal.sandbox.duckdb_runtime import stream_query_to_ipc +from internal.sandbox.security_profile import SandboxProfile +from internal.telemetry.logging import get_logger + +_logger = get_logger("internal.pool.worker_task") + + +def init_worker_process(rlimit_bytes: int | None = None) -> None: + """Initialize a newly spawned worker subprocess with optional POSIX address-space limit.""" + if rlimit_bytes is None or sys.platform == "win32": + return + try: + import resource + + _soft, hard = resource.getrlimit(resource.RLIMIT_AS) + target = rlimit_bytes if hard == resource.RLIM_INFINITY else min(rlimit_bytes, hard) + resource.setrlimit(resource.RLIMIT_AS, (target, hard)) + except (ImportError, OSError, ValueError) as exc: + _logger.warning( + "Unable to apply WORKER_PROCESS_RLIMIT_BYTES in worker subprocess", + extra={"extra_fields": {"error_type": type(exc).__name__}}, + ) + + +def ping_worker() -> int: + """Warm up worker subprocess imports (DuckDB, PyArrow) and return worker PID.""" + return os.getpid() + + +class _IpcSendConn(Protocol): + """Structural protocol for Connection / PipeConnection send side.""" + + def send_bytes(self, buf: bytes, /) -> None: ... + def close(self) -> None: ... + + +class _ConnectionBinarySink(io.RawIOBase): + """Streams Arrow IPC bytes chunk-by-chunk over a multiprocessing Connection.""" + + def __init__(self, conn: _IpcSendConn) -> None: + super().__init__() + self._conn = conn + + def writable(self) -> bool: + return True + + def write(self, b: bytes | bytearray | memoryview) -> int: # type: ignore[override] + data = bytes(b) + if data: + self._conn.send_bytes(data) + return len(data) + + +def run_sandboxed_query_task( + sql: str, + parquet_path_str: str, + table_alias: str, + memory_limit: str, + threads: int, + temp_directory: str, + batch_size: int, + ipc_address: str, + authkey: bytes, +) -> tuple[int, int]: + """Execute a validated SQL query in an ephemeral DuckDB sandbox and stream Arrow IPC. + + This function is top-level so it is safe under ``spawn`` multiprocessing. + Returns ``(total_rows, worker_pid)``. + """ + profile = SandboxProfile( + memory_limit=memory_limit, + threads=threads, + temp_directory=temp_directory, + ) + conn = Client(ipc_address, authkey=authkey) + try: + sink = _ConnectionBinarySink(conn) + total_rows = stream_query_to_ipc( + sql=sql, + parquet_path=Path(parquet_path_str), + table_alias=table_alias, + profile=profile, + sink=sink, + batch_size=batch_size, + ) + # Signal clean end-of-stream to parent reader. + conn.send_bytes(b"") + return total_rows, os.getpid() + finally: + conn.close() diff --git a/internal/sandbox/__init__.py b/internal/sandbox/__init__.py new file mode 100644 index 0000000..9a17797 --- /dev/null +++ b/internal/sandbox/__init__.py @@ -0,0 +1,11 @@ +"""Ephemeral DuckDB sandbox runtime package.""" + +from internal.sandbox.duckdb_runtime import execute_sandboxed_query, stream_query_to_ipc +from internal.sandbox.security_profile import SandboxProfile, apply_sandbox_lock + +__all__ = [ + "SandboxProfile", + "apply_sandbox_lock", + "execute_sandboxed_query", + "stream_query_to_ipc", +] diff --git a/internal/sandbox/duckdb_runtime.py b/internal/sandbox/duckdb_runtime.py new file mode 100644 index 0000000..d08d25c --- /dev/null +++ b/internal/sandbox/duckdb_runtime.py @@ -0,0 +1,71 @@ +"""Ephemeral DuckDB :memory: runtime with chunked PyArrow RecordBatch streaming.""" + +from __future__ import annotations + +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import duckdb +import pyarrow as pa + +from internal.sandbox.security_profile import SandboxProfile, apply_sandbox_lock + + +def _get_arrow_reader( + conn: duckdb.DuckDBPyConnection, + sql: str, + batch_size: int, +) -> pa.RecordBatchReader: + """Return a streaming PyArrow RecordBatchReader without full-table copies.""" + reader: pa.RecordBatchReader = conn.sql(sql).fetch_arrow_reader(batch_size=batch_size) + return reader + + +def execute_sandboxed_query( + sql: str, + parquet_path: Path, + table_alias: str, + profile: SandboxProfile, + *, + batch_size: int = 8192, +) -> Iterator[pa.RecordBatch]: + """Execute validated SQL inside an ephemeral locked DuckDB :memory: connection. + + Yields PyArrow ``RecordBatch`` chunks without full-table materialization. + """ + conn = duckdb.connect(":memory:") + try: + apply_sandbox_lock(conn, parquet_path, table_alias, profile) + reader = _get_arrow_reader(conn, sql, batch_size) + yield from reader + finally: + conn.close() + + +def stream_query_to_ipc( + sql: str, + parquet_path: Path, + table_alias: str, + profile: SandboxProfile, + sink: Any, + *, + batch_size: int = 8192, +) -> int: + """Execute validated SQL and write Arrow IPC stream frames directly to ``sink``. + + Returns: + Total number of rows written across all RecordBatches. + """ + conn = duckdb.connect(":memory:") + total_rows = 0 + try: + apply_sandbox_lock(conn, parquet_path, table_alias, profile) + reader = _get_arrow_reader(conn, sql, batch_size) + with pa.ipc.new_stream(sink, reader.schema) as writer: + for batch in reader: + writer.write_batch(batch) + total_rows += batch.num_rows + return total_rows + finally: + conn.close() diff --git a/internal/sandbox/security_profile.py b/internal/sandbox/security_profile.py new file mode 100644 index 0000000..94b92e2 --- /dev/null +++ b/internal/sandbox/security_profile.py @@ -0,0 +1,54 @@ +"""DuckDB sandbox security profile: single-artifact materialization and hard locks.""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path + +import duckdb + +from internal.validator.rules import SAFE_IDENTIFIER_RE + + +@dataclass(frozen=True, slots=True) +class SandboxProfile: + """Immutable DuckDB sandbox configuration applied to every child worker connection.""" + + memory_limit: str = "640MiB" + threads: int = 2 + temp_directory: str = "" + + +def apply_sandbox_lock( + conn: duckdb.DuckDBPyConnection, + parquet_path: Path, + table_alias: str, + profile: SandboxProfile, +) -> None: + """Materialize a single validated Parquet artifact and lock down DuckDB external access. + + The artifact is materialized via ``CREATE TEMP TABLE`` before disabling external access + so read-only queries execute entirely in memory without filesystem/network capabilities. + """ + if not SAFE_IDENTIFIER_RE.match(table_alias): + raise ValueError(f"Unsafe table alias: {table_alias!r}") + + resolved_path = parquet_path.resolve(strict=True) + if not resolved_path.is_file(): + raise FileNotFoundError(f"Parquet artifact is not a regular file: {resolved_path}") + + escaped_tmp = profile.temp_directory.replace("'", "''") + conn.execute(f"SET temp_directory = '{escaped_tmp}'") + escaped_mem = profile.memory_limit.replace("'", "''") + conn.execute(f"SET memory_limit = '{escaped_mem}'") + conn.execute(f"SET threads = {int(profile.threads)}") + + escaped_path = str(resolved_path).replace("'", "''") + conn.execute( + f'CREATE TEMP TABLE "{table_alias}" ' # noqa: S608 # nosec B608 + f"AS SELECT * FROM read_parquet('{escaped_path}')" + ) + + # Lock external access and configuration immediately after materializing the single artifact. + conn.execute("SET enable_external_access = false") + conn.execute("SET lock_configuration = true") diff --git a/internal/storage/__init__.py b/internal/storage/__init__.py new file mode 100644 index 0000000..7d945c1 --- /dev/null +++ b/internal/storage/__init__.py @@ -0,0 +1,29 @@ +"""Tier-1 NVMe Parquet cache, S3 streaming hydration, and tenant isolation.""" + +from internal.storage.cache_manager import ( + CacheCapacityError, + CacheManager, + CorruptedArtifactError, + PinnedArtifact, +) +from internal.storage.s3_streamer import ArtifactNotFoundError, S3AccessError, S3Streamer +from internal.storage.table_resolver import ( + ResolvedTable, + TableResolver, + TableValidationError, + TenantIsolationError, +) + +__all__ = [ + "ArtifactNotFoundError", + "CacheCapacityError", + "CacheManager", + "CorruptedArtifactError", + "PinnedArtifact", + "ResolvedTable", + "S3AccessError", + "S3Streamer", + "TableResolver", + "TableValidationError", + "TenantIsolationError", +] diff --git a/internal/storage/cache_manager.py b/internal/storage/cache_manager.py new file mode 100644 index 0000000..2d4792e --- /dev/null +++ b/internal/storage/cache_manager.py @@ -0,0 +1,424 @@ +"""Tier-1 disk-backed NVMe Parquet cache with LRU eviction, refcounts, and single-flight.""" + +from __future__ import annotations + +import asyncio +import contextlib +import os +import tempfile +import threading +from collections import OrderedDict +from dataclasses import dataclass +from pathlib import Path +from typing import Literal + +import pyarrow as pa +import pyarrow.parquet as pq + +from internal.storage.s3_streamer import S3Streamer +from internal.storage.table_resolver import ResolvedTable +from internal.telemetry.logging import get_logger +from internal.telemetry.metrics import MetricsRegistry + +_logger = get_logger("internal.storage.cache_manager") + +CacheKey = tuple[str, str, str] +CacheState = Literal["warm_cache", "cold_s3"] +_PARQUET_MAGIC = b"PAR1" + + +class CacheCapacityError(RuntimeError): + """Raised when CACHE_MAX_BYTES is exhausted and no unpinned entries can be evicted.""" + + +class CorruptedArtifactError(ValueError): + """Raised when a hydrated Parquet file fails PAR1 magic or metadata verification.""" + + +@dataclass(slots=True) +class _CacheEntry: + path: Path + size_bytes: int + refcount: int = 0 + pending_evict: bool = False + + +@dataclass(slots=True) +class PinnedArtifact: + """Reader-pinned cached Parquet artifact handle; must call release() when done.""" + + key: CacheKey + path: Path + size_bytes: int + cache_state: CacheState + _manager: CacheManager + _entry: _CacheEntry | None = None + _released: bool = False + + def release(self) -> None: + """Decrement the active reader refcount (idempotent).""" + if not self._released: + self._released = True + self._manager.release(self.key, entry=self._entry) + + +def _validate_parquet_file(path: Path) -> int: + """Verify PAR1 header/footer magic bytes and PyArrow ParquetFile metadata.""" + try: + size_bytes = path.stat().st_size + except OSError as exc: + raise CorruptedArtifactError(f"Cannot stat Parquet artifact: {exc}") from exc + + if size_bytes < 12: + raise CorruptedArtifactError("Parquet artifact is smaller than minimum 12-byte format.") + + try: + with path.open("rb") as fh: + header = fh.read(4) + fh.seek(-4, os.SEEK_END) + footer = fh.read(4) + except OSError as exc: + raise CorruptedArtifactError(f"Failed reading Parquet magic bytes: {exc}") from exc + + if header != _PARQUET_MAGIC or footer != _PARQUET_MAGIC: + raise CorruptedArtifactError("Artifact failed Parquet PAR1 header/footer validation.") + + try: + pf = pq.ParquetFile(path) + _ = pf.metadata + del pf + except (pa.ArrowException, OSError, ValueError) as exc: + raise CorruptedArtifactError(f"Artifact failed Parquet metadata validation: {exc}") from exc + + return size_bytes + + +class CacheManager: + """Thread-safe disk-backed LRU Parquet cache with single-flight S3 hydration.""" + + def __init__( + self, + cache_dir: Path, + max_bytes: int, + metrics: MetricsRegistry | None = None, + ) -> None: + self._cache_dir = cache_dir + self._max_bytes = max_bytes + self._metrics = metrics + self._lock = threading.Lock() + self._entries: OrderedDict[CacheKey, _CacheEntry] = OrderedDict() + self._total_bytes = 0 + self._reserved_bytes = 0 + self._inflight: dict[CacheKey, asyncio.Future[tuple[Path, int]]] = {} + + @property + def cache_dir(self) -> Path: + """Return root cache directory.""" + return self._cache_dir + + @property + def total_bytes(self) -> int: + """Return total tracked cached bytes.""" + with self._lock: + return self._total_bytes + + def initialize(self) -> None: + """Create cache_dir, sweep orphaned .tmp files, and reconcile existing .parquet files.""" + self._cache_dir.mkdir(parents=True, exist_ok=True) + entries: OrderedDict[CacheKey, _CacheEntry] = OrderedDict() + total_bytes = 0 + + for path in self._cache_dir.rglob("*"): + if not path.is_file(): + continue + if path.name.endswith(".tmp") or ".tmp." in path.name: + with contextlib.suppress(OSError): + path.unlink(missing_ok=True) + continue + if path.suffix != ".parquet": + continue + + try: + rel = path.relative_to(self._cache_dir) + except ValueError: + continue + if len(rel.parts) != 3: + continue + bucket, tenant_id, filename = rel.parts + table_hash = filename.removesuffix(".parquet") + key: CacheKey = (bucket, tenant_id, table_hash) + try: + size_bytes = path.stat().st_size + except OSError: + continue + entries[key] = _CacheEntry(path=path, size_bytes=size_bytes) + total_bytes += size_bytes + + with self._lock: + self._entries = entries + self._total_bytes = total_bytes + + if self._metrics is not None: + self._metrics.set_cache_bytes_used(total_bytes) + + _logger.info( + "Initialized Tier-1 Parquet cache", + extra={ + "extra_fields": { + "cached_artifacts": len(entries), + "cache_bytes_used": total_bytes, + "cache_max_bytes": self._max_bytes, + } + }, + ) + + def is_writable(self) -> bool: + """Return True if cache_dir exists and is writable.""" + return self._cache_dir.is_dir() and os.access(self._cache_dir, os.W_OK | os.R_OK) + + def _try_pin_warm(self, key: CacheKey) -> PinnedArtifact | None: + """Attempt to pin an already-cached artifact under lock.""" + with self._lock: + entry = self._entries.get(key) + if entry is None: + return None + if entry.pending_evict: + if entry.refcount == 0: + self._remove_entry_locked(key, unlink_file=True) + return None + raise CorruptedArtifactError( + "Artifact is pending eviction while active readers finish." + ) + if not entry.path.exists(): + if entry.refcount == 0: + self._remove_entry_locked(key, unlink_file=False) + return None + entry.pending_evict = True + raise CorruptedArtifactError("Cached artifact missing while active readers finish.") + self._entries.move_to_end(key) + entry.refcount += 1 + return PinnedArtifact( + key=key, + path=entry.path, + size_bytes=entry.size_bytes, + cache_state="warm_cache", + _manager=self, + _entry=entry, + ) + + def _remove_entry_locked(self, key: CacheKey, *, unlink_file: bool) -> None: + """Remove an entry from LRU tracking (caller must hold self._lock).""" + entry = self._entries.pop(key, None) + if entry is None: + return + self._total_bytes = max(0, self._total_bytes - entry.size_bytes) + if unlink_file: + with contextlib.suppress(OSError): + entry.path.unlink(missing_ok=True) + if self._metrics is not None: + self._metrics.set_cache_bytes_used(self._total_bytes) + + def release(self, key: CacheKey, *, entry: _CacheEntry | None = None) -> None: + """Decrement active reader refcount and unlink if pending eviction.""" + with self._lock: + tracked = self._entries.get(key) + target = entry if entry is not None else tracked + if target is None: + return + target.refcount = max(0, target.refcount - 1) + if target.refcount == 0 and target.pending_evict: + if tracked is target: + self._remove_entry_locked(key, unlink_file=True) + else: + with contextlib.suppress(OSError): + target.path.unlink(missing_ok=True) + + def evict_corrupted(self, key: CacheKey) -> None: + """Evict a corrupted artifact immediately or once active readers finish.""" + with self._lock: + entry = self._entries.get(key) + if entry is None: + return + if entry.refcount == 0: + self._remove_entry_locked(key, unlink_file=True) + else: + entry.pending_evict = True + + def _commit_tmp_file( + self, + key: CacheKey, + tmp_path: Path, + dest_path: Path, + size_bytes: int, + *, + reserved_bytes: int = 0, + ) -> None: + """Evict unpinned LRU entries if needed and atomically rename tmp_path to dest_path.""" + with self._lock: + other_reserved = max(0, self._reserved_bytes - reserved_bytes) + + if size_bytes > self._max_bytes: + with contextlib.suppress(OSError): + tmp_path.unlink(missing_ok=True) + raise CacheCapacityError( + "Artifact size exceeds configured CACHE_MAX_BYTES ceiling." + ) + + # Remove stale entry for same key if present and unpinned. + existing = self._entries.get(key) + if existing is not None: + if existing.refcount == 0: + self._remove_entry_locked(key, unlink_file=True) + else: + with contextlib.suppress(OSError): + tmp_path.unlink(missing_ok=True) + raise CorruptedArtifactError( + "Existing cache entry is still pinned; cannot replace." + ) + + for candidate_key, candidate_entry in list(self._entries.items()): + if self._total_bytes + other_reserved + size_bytes <= self._max_bytes: + break + if candidate_entry.refcount > 0 or candidate_entry.pending_evict: + continue + self._remove_entry_locked(candidate_key, unlink_file=True) + + if self._total_bytes + other_reserved + size_bytes > self._max_bytes: + with contextlib.suppress(OSError): + tmp_path.unlink(missing_ok=True) + raise CacheCapacityError( + "Cache capacity exhausted with all cached entries pinned by active readers." + ) + + dest_path.parent.mkdir(parents=True, exist_ok=True) + os.replace(tmp_path, dest_path) + self._entries[key] = _CacheEntry(path=dest_path, size_bytes=size_bytes, refcount=0) + self._entries.move_to_end(key) + self._total_bytes += size_bytes + if self._metrics is not None: + self._metrics.set_cache_bytes_used(self._total_bytes) + + def _pin_after_hydration(self, key: CacheKey) -> PinnedArtifact: + """Pin a newly hydrated entry for the caller.""" + pinned = self._try_pin_after_hydration(key) + if pinned is None: + raise RuntimeError("Hydrated artifact missing from cache immediately after commit.") + return pinned + + def _try_pin_after_hydration(self, key: CacheKey) -> PinnedArtifact | None: + """Pin a newly hydrated entry if it is still present in the cache.""" + with self._lock: + entry = self._entries.get(key) + if entry is None or entry.pending_evict or not entry.path.exists(): + return None + self._entries.move_to_end(key) + entry.refcount += 1 + return PinnedArtifact( + key=key, + path=entry.path, + size_bytes=entry.size_bytes, + cache_state="cold_s3", + _manager=self, + _entry=entry, + ) + + async def get_or_hydrate( + self, + resolved: ResolvedTable, + s3_streamer: S3Streamer, + ) -> PinnedArtifact: + """Return a reader-pinned Parquet artifact, hydrating from S3 on cache miss.""" + key = resolved.cache_key + recorded_miss = False + + while True: + warm = self._try_pin_warm(key) + if warm is not None: + if not recorded_miss and self._metrics is not None: + self._metrics.record_cache_hit() + return warm + + if not recorded_miss: + if self._metrics is not None: + self._metrics.record_cache_miss() + recorded_miss = True + + inflight_fut = self._inflight.get(key) + if inflight_fut is not None: + await asyncio.shield(inflight_fut) + pinned = self._try_pin_after_hydration(key) + if pinned is not None: + return pinned + continue + + loop = asyncio.get_running_loop() + leader_fut: asyncio.Future[tuple[Path, int]] = loop.create_future() + self._inflight[key] = leader_fut + + tmp_path: Path | None = None + reserved_bytes = 0 + + def _reserve_chunk(nbytes: int) -> None: + nonlocal reserved_bytes + with self._lock: + if reserved_bytes + nbytes > self._max_bytes: + raise CacheCapacityError( + "Artifact size exceeds configured CACHE_MAX_BYTES ceiling." + ) + for candidate_key, candidate_entry in list(self._entries.items()): + if self._total_bytes + self._reserved_bytes + nbytes <= self._max_bytes: + break + if candidate_entry.refcount > 0 or candidate_entry.pending_evict: + continue + self._remove_entry_locked(candidate_key, unlink_file=True) + if self._total_bytes + self._reserved_bytes + nbytes > self._max_bytes: + raise CacheCapacityError( + "Cache capacity exhausted with all cached entries pinned " + "by active readers." + ) + self._reserved_bytes += nbytes + reserved_bytes += nbytes + + try: + dest_path = self._cache_dir / resolved.relative_cache_path + dest_path.parent.mkdir(parents=True, exist_ok=True) + fd, tmp_name = tempfile.mkstemp( + prefix=f".{resolved.table_hash[:12]}.", + suffix=".tmp", + dir=dest_path.parent, + ) + os.close(fd) + tmp_path = Path(tmp_name) + + await s3_streamer.stream_to_file( + resolved.s3_bucket, + resolved.s3_key, + tmp_path, + on_chunk=_reserve_chunk, + ) + size_bytes = _validate_parquet_file(tmp_path) + self._commit_tmp_file( + key, + tmp_path, + dest_path, + size_bytes, + reserved_bytes=reserved_bytes, + ) + pinned = self._pin_after_hydration(key) + if not leader_fut.done(): + leader_fut.set_result((dest_path, size_bytes)) + return pinned + except BaseException as exc: + if tmp_path is not None: + with contextlib.suppress(OSError): + tmp_path.unlink(missing_ok=True) + if not leader_fut.done(): + leader_fut.set_exception(exc) + # Mark exception retrieved in case there are no concurrent waiters. + leader_fut.exception() + raise + finally: + if reserved_bytes > 0: + with self._lock: + self._reserved_bytes = max(0, self._reserved_bytes - reserved_bytes) + self._inflight.pop(key, None) diff --git a/internal/storage/s3_streamer.py b/internal/storage/s3_streamer.py new file mode 100644 index 0000000..bc2145a --- /dev/null +++ b/internal/storage/s3_streamer.py @@ -0,0 +1,191 @@ +"""Async S3 streaming hydration over internal VPC endpoints without heap buffering.""" + +from __future__ import annotations + +import asyncio +from collections.abc import Callable +from pathlib import Path +from typing import Any + +from aiobotocore.session import get_session +from botocore.config import Config +from botocore.exceptions import BotoCoreError, ClientError + +from internal.storage.table_resolver import TenantIsolationError +from internal.telemetry.logging import get_logger + +_logger = get_logger("internal.storage.s3_streamer") + + +class ArtifactNotFoundError(FileNotFoundError): + """Raised when the requested Parquet artifact does not exist in S3.""" + + +class S3AccessError(RuntimeError): + """Raised when S3 retrieval fails due to upstream transport or service errors.""" + + +class S3Streamer: + """Streams Parquet artifacts from S3 directly to disk inside the hydration asyncio loop.""" + + def __init__( + self, + *, + endpoint_url: str | None = None, + region_name: str = "us-east-1", + chunk_size_bytes: int = 8 * 1024 * 1024, + timeout_seconds: float = 4.0, + client: Any | None = None, + ) -> None: + self._endpoint_url = endpoint_url + self._region_name = region_name + self._chunk_size_bytes = chunk_size_bytes + self._timeout_seconds = timeout_seconds + self._injected_client = client + self._client_ctx: Any | None = None + self._client: Any | None = client + self._client_lock = asyncio.Lock() + + async def start(self) -> None: + """Initialize the aiobotocore S3 client inside the active asyncio event loop.""" + if self._client is not None: + return + async with self._client_lock: + if self._client is not None: + return + session = get_session() + boto_config = Config( + region_name=self._region_name, + connect_timeout=self._timeout_seconds, + read_timeout=self._timeout_seconds, + retries={"max_attempts": 2, "mode": "standard"}, + ) + ctx = session.create_client( + "s3", + endpoint_url=self._endpoint_url, + region_name=self._region_name, + config=boto_config, + ) + client = await ctx.__aenter__() + self._client_ctx = ctx + self._client = client + + async def close(self) -> None: + """Close the aiobotocore S3 client inside the active asyncio event loop.""" + async with self._client_lock: + try: + if self._client_ctx is not None: + await self._client_ctx.__aexit__(None, None, None) + finally: + self._client_ctx = None + if self._injected_client is None: + self._client = None + + def reset_client_state(self) -> None: + """Reset client handles after a timed-out or stopped hydration loop.""" + self._client_ctx = None + if self._injected_client is None: + self._client = None + self._client_lock = asyncio.Lock() + + async def stream_to_file( + self, + bucket: str, + key: str, + dest_tmp_path: Path, + *, + on_chunk: Callable[[int], None] | None = None, + ) -> int: + """Stream an S3 Parquet object chunk-by-chunk directly to a local temp file.""" + if self._client is None: + await self.start() + assert self._client is not None # noqa: S101 + + dest_tmp_path.parent.mkdir(parents=True, exist_ok=True) + total_bytes = 0 + + try: + async with asyncio.timeout(self._timeout_seconds): + response = await self._client.get_object(Bucket=bucket, Key=key) + body = response["Body"] + try: + with dest_tmp_path.open("wb") as fh: + if hasattr(body, "iter_chunks"): + async for chunk in body.iter_chunks(chunk_size=self._chunk_size_bytes): + if chunk: + if on_chunk is not None: + on_chunk(len(chunk)) + fh.write(chunk) + total_bytes += len(chunk) + else: + while True: + chunk = await body.read(self._chunk_size_bytes) + if not chunk: + break + if on_chunk is not None: + on_chunk(len(chunk)) + fh.write(chunk) + total_bytes += len(chunk) + fh.flush() + finally: + close_fn = getattr(body, "close", None) + if callable(close_fn): + res = close_fn() + if asyncio.iscoroutine(res): + await res + except TimeoutError as exc: + _logger.warning( + "S3 artifact stream timed out", + extra={"extra_fields": {"s3_bucket": bucket, "s3_key": key}}, + ) + raise S3AccessError("S3 artifact stream timed out.") from exc + except ClientError as exc: + error_info = exc.response.get("Error", {}) if hasattr(exc, "response") else {} + meta = exc.response.get("ResponseMetadata", {}) if hasattr(exc, "response") else {} + code = str(error_info.get("Code", "")) + status_code = int(meta.get("HTTPStatusCode", 0) or 0) + + if code in ("NoSuchKey", "404", "NotFound") or status_code == 404: + _logger.info( + "Requested Parquet artifact not found in S3", + extra={"extra_fields": {"s3_bucket": bucket, "s3_key": key}}, + ) + raise ArtifactNotFoundError("Target Parquet artifact not found.") from exc + + if code in ("AccessDenied", "403", "Forbidden") or status_code == 403: + _logger.warning( + "S3 access denied for artifact", + extra={"extra_fields": {"s3_bucket": bucket, "s3_key": key}}, + ) + raise TenantIsolationError("Access denied to requested S3 artifact.") from exc + + _logger.error( + "S3 client error during artifact hydration", + extra={ + "extra_fields": { + "s3_bucket": bucket, + "s3_key": key, + "s3_error_code": code, + "http_status": status_code, + } + }, + ) + raise S3AccessError(f"S3 retrieval failed with code {code or status_code}.") from exc + except BotoCoreError as exc: + _logger.error( + "S3 transport error during artifact hydration", + extra={"extra_fields": {"s3_bucket": bucket, "s3_key": key}}, + ) + raise S3AccessError("S3 transport failure during artifact hydration.") from exc + + _logger.info( + "Streamed S3 Parquet artifact to local temp file", + extra={ + "extra_fields": { + "s3_bucket": bucket, + "s3_key": key, + "bytes": total_bytes, + } + }, + ) + return total_bytes diff --git a/internal/storage/table_resolver.py b/internal/storage/table_resolver.py new file mode 100644 index 0000000..641a82a --- /dev/null +++ b/internal/storage/table_resolver.py @@ -0,0 +1,97 @@ +"""Tenant isolation, S3 bucket allowlist, and canonical artifact path resolution.""" + +from __future__ import annotations + +import re +from collections.abc import Iterable +from dataclasses import dataclass +from pathlib import Path + +from internal.validator.exceptions import ValidationError + +_TENANT_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$") +_TABLE_HASH_RE = re.compile(r"^[0-9a-f]{64}$") +_S3_BUCKET_RE = re.compile(r"^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$") + + +class TenantIsolationError(PermissionError): + """Raised when a request violates tenant isolation, bucket allowlist, or canonical S3 key.""" + + +class TableValidationError(ValidationError): + """Raised when tenant_id, table_hash, s3_bucket, or s3_key format is malformed.""" + + rule = "INVALID_TABLE_SPEC" + + +@dataclass(frozen=True, slots=True) +class ResolvedTable: + """Validated artifact coordinates for cache lookup and S3 hydration.""" + + tenant_id: str + table_hash: str + s3_bucket: str + s3_key: str + relative_cache_path: Path + + @property + def cache_key(self) -> tuple[str, str, str]: + """Return content-addressed cache key scoped by bucket and tenant.""" + return (self.s3_bucket, self.tenant_id, self.table_hash) + + +class TableResolver: + """Validates tenant, hash, bucket allowlist, and canonical S3 key invariants.""" + + def __init__(self, s3_bucket_allowlist: Iterable[str]) -> None: + self._allowlist: frozenset[str] = frozenset( + bucket.strip() for bucket in s3_bucket_allowlist if bucket.strip() + ) + + @property + def allowlist(self) -> frozenset[str]: + """Return configured S3 bucket allowlist.""" + return self._allowlist + + def resolve( + self, + *, + tenant_id: str, + table_hash: str, + s3_bucket: str, + s3_key: str, + ) -> ResolvedTable: + """Validate artifact coordinates and return a ResolvedTable. + + Raises: + TableValidationError: For malformed tenant_id, table_hash, s3_bucket, or s3_key. + TenantIsolationError: If allowlist is empty, bucket is unallowlisted, + or s3_key does not match canonical `artifacts/{tenant_id}/{table_hash}.parquet`. + """ + if not tenant_id or not _TENANT_ID_RE.match(tenant_id) or ".." in tenant_id: + raise TableValidationError("Invalid tenant_id format.") + + if not table_hash or not _TABLE_HASH_RE.match(table_hash): + raise TableValidationError("table_hash must be a 64-character lowercase SHA-256 hex.") + + if not s3_bucket or not _S3_BUCKET_RE.match(s3_bucket) or ".." in s3_bucket: + raise TableValidationError("Invalid s3_bucket format.") + + if not s3_key or not s3_key.strip(): + raise TableValidationError("s3_key must not be empty.") + + # Fail-closed when S3_BUCKET_ALLOWLIST is empty or does not contain the requested bucket. + if not self._allowlist or s3_bucket not in self._allowlist: + raise TenantIsolationError("Requested S3 bucket is not permitted.") + + expected_key = f"artifacts/{tenant_id}/{table_hash}.parquet" + if s3_key != expected_key: + raise TenantIsolationError("s3_key does not match canonical tenant artifact path.") + + return ResolvedTable( + tenant_id=tenant_id, + table_hash=table_hash, + s3_bucket=s3_bucket, + s3_key=s3_key, + relative_cache_path=Path(s3_bucket) / tenant_id / f"{table_hash}.parquet", + ) diff --git a/internal/telemetry/__init__.py b/internal/telemetry/__init__.py new file mode 100644 index 0000000..36037d0 --- /dev/null +++ b/internal/telemetry/__init__.py @@ -0,0 +1,10 @@ +"""Telemetry and structured JSON logging package.""" + +from internal.telemetry.logging import ( + configure_logging, + get_logger, + redact_mapping, + redact_text, +) + +__all__ = ["configure_logging", "get_logger", "redact_mapping", "redact_text"] diff --git a/internal/telemetry/logging.py b/internal/telemetry/logging.py new file mode 100644 index 0000000..1faa4cf --- /dev/null +++ b/internal/telemetry/logging.py @@ -0,0 +1,168 @@ +"""Structured JSON logging with strict secret, credential, and SQL literal redaction.""" + +from __future__ import annotations + +import json +import logging +import re +import sys +from datetime import UTC, datetime +from typing import Any + +from opentelemetry import trace as otel_trace + +REDACTED = "[REDACTED]" + +_BEARER_RE = re.compile(r"(?i)(Bearer\s+)[A-Za-z0-9\-._~+/]+=*") +_AMZ_QUERY_RE = re.compile( + r"(?i)([?&](?:X-Amz-Signature|X-Amz-Credential|X-Amz-Security-Token" + r"|X-Amz-Algorithm|X-Amz-Date|X-Amz-Expires|X-Amz-SignedHeaders)=)[^&\s\"']+" +) +_AWS_KEY_RE = re.compile(r"\bAKIA[0-9A-Z]{16}\b") + +_SENSITIVE_KEYS: frozenset[str] = frozenset( + { + "authorization", + "bearer", + "token", + "access_token", + "secret", + "password", + "aws_access_key_id", + "aws_secret_access_key", + "aws_session_token", + "presigned_url", + "raw_sql", + "sql", + "sql_literals", + } +) + +_CORRELATION_FIELDS: tuple[str, ...] = ( + "trace_id", + "request_id", + "tenant_id", + "table_hash", + "cache_state", + "duration_ms", + "status", + "rule", + "reason", + "worker_pid", +) + + +def redact_text(value: str) -> str: + """Redact bearer tokens, presigned S3 query params, and AWS access key IDs from text.""" + sanitized = _BEARER_RE.sub(rf"\1{REDACTED}", value) + sanitized = _AMZ_QUERY_RE.sub(rf"\1{REDACTED}", sanitized) + sanitized = _AWS_KEY_RE.sub(REDACTED, sanitized) + return sanitized + + +def _redact_value(val: Any) -> Any: + """Recursively sanitize nested mappings, sequences, and strings.""" + if isinstance(val, dict): + return redact_mapping(val) + if isinstance(val, str): + return redact_text(val) + if isinstance(val, list | tuple): + return [_redact_value(item) for item in val] + return val + + +def redact_mapping(data: dict[str, Any]) -> dict[str, Any]: + """Recursively redact sensitive keys and string values in a structured log mapping.""" + result: dict[str, Any] = {} + for key, val in data.items(): + if key.lower() in _SENSITIVE_KEYS: + result[key] = REDACTED + else: + result[key] = _redact_value(val) + return result + + +class JsonLogFormatter(logging.Formatter): + """Formats log records as single-line structured JSON with redaction.""" + + def __init__(self, service_name: str = "rag-tabular-analytics-worker") -> None: + super().__init__() + self._service_name = service_name + + def format(self, record: logging.LogRecord) -> str: + payload: dict[str, Any] = { + "timestamp": datetime.now(UTC).isoformat(), + "level": record.levelname, + "service": self._service_name, + "logger": record.name, + "message": redact_text(record.getMessage()), + } + + for field_name in _CORRELATION_FIELDS: + field_val = getattr(record, field_name, None) + if field_val is not None: + payload[field_name] = _redact_value(field_val) + + if "trace_id" not in payload: + span_ctx = otel_trace.get_current_span().get_span_context() + if span_ctx.is_valid: + payload["trace_id"] = format(span_ctx.trace_id, "032x") + + extra_fields = getattr(record, "extra_fields", None) + if isinstance(extra_fields, dict): + payload.update(redact_mapping(extra_fields)) + + if record.exc_info and record.exc_info[1] is not None: + payload["error_type"] = type(record.exc_info[1]).__name__ + payload["error_message"] = redact_text(str(record.exc_info[1])) + + return json.dumps(payload, default=str, separators=(",", ":")) + + +_DEFAULT_LOG_LEVEL: str = "INFO" +_DEFAULT_SERVICE_NAME: str = "rag-tabular-analytics-worker" + + +def _resolve_log_level(level: str) -> int: + """Resolve a case-insensitive log level string to a stdlib logging constant.""" + resolved = getattr(logging, level.upper(), None) + return resolved if isinstance(resolved, int) else logging.INFO + + +def configure_logging(level: str = "INFO", service_name: str | None = None) -> None: + """Apply runtime log level and service name to all worker loggers.""" + global _DEFAULT_LOG_LEVEL, _DEFAULT_SERVICE_NAME + _DEFAULT_LOG_LEVEL = level.upper() + if service_name: + _DEFAULT_SERVICE_NAME = service_name + + numeric_level = _resolve_log_level(level) + for name, logger_obj in list(logging.Logger.manager.loggerDict.items()): + if not isinstance(logger_obj, logging.Logger): + continue + if ( + name == "internal" + or name.startswith("internal.") + or name == "cmd" + or name.startswith("cmd.") + ): + logger_obj.setLevel(numeric_level) + if service_name: + for handler in logger_obj.handlers: + if isinstance(handler.formatter, JsonLogFormatter): + handler.formatter._service_name = service_name + + +def get_logger(name: str, level: str | None = None) -> logging.Logger: + """Return a configured structured JSON logger without duplicate stream handlers.""" + logger = logging.getLogger(name) + numeric_level = _resolve_log_level(level or _DEFAULT_LOG_LEVEL) + logger.setLevel(numeric_level) + logger.propagate = False + + if not logger.handlers: + handler = logging.StreamHandler(sys.stdout) + handler.setFormatter(JsonLogFormatter(service_name=_DEFAULT_SERVICE_NAME)) + logger.addHandler(handler) + + return logger diff --git a/internal/telemetry/metrics.py b/internal/telemetry/metrics.py new file mode 100644 index 0000000..d7915f7 --- /dev/null +++ b/internal/telemetry/metrics.py @@ -0,0 +1,128 @@ +"""Thread-safe Prometheus metrics registry and text exposition helper.""" + +from __future__ import annotations + +import threading + +from prometheus_client import CollectorRegistry, Counter, Gauge, Histogram, generate_latest + + +class MetricsRegistry: + """Isolated Prometheus metrics registry for rag-tabular-analytics-worker.""" + + def __init__(self, registry: CollectorRegistry | None = None) -> None: + self._registry = registry or CollectorRegistry(auto_describe=True) + self._ratio_lock = threading.Lock() + self._hits_count = 0 + self._misses_count = 0 + + self.active_queries = Gauge( + "tabular_worker_active_queries", + "Concurrent in-flight queries (primary KEDA autoscaling metric)", + registry=self._registry, + ) + self.query_duration_seconds = Histogram( + "tabular_worker_query_duration_seconds", + "End-to-end query execution latency in seconds", + labelnames=("cache_state",), + buckets=(0.005, 0.01, 0.025, 0.045, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0), + registry=self._registry, + ) + self.cache_hit_ratio = Gauge( + "tabular_worker_cache_hit_ratio", + "Local NVMe Parquet cache hit ratio", + registry=self._registry, + ) + self.cache_hits_total = Counter( + "tabular_worker_cache_hits_total", + "Total local Parquet cache hits", + registry=self._registry, + ) + self.cache_misses_total = Counter( + "tabular_worker_cache_misses_total", + "Total local Parquet cache misses", + registry=self._registry, + ) + self.cache_bytes_used = Gauge( + "tabular_worker_cache_bytes_used", + "Current local NVMe cache disk utilization in bytes", + registry=self._registry, + ) + self.subprocess_recycles_total = Counter( + "tabular_worker_subprocess_recycles_total", + "Child worker subprocess recycling events", + labelnames=("reason",), + registry=self._registry, + ) + # Initialize abnormal recycle series to 0 so Prometheus rate() queries + # return a zero rate rather than an empty vector on healthy workers. + for reason in ("TIMEOUT", "MEMORY_LIMIT", "CRASH"): + self.subprocess_recycles_total.labels(reason=reason) + self.flight_stream_bytes_total = Counter( + "tabular_worker_flight_stream_bytes_total", + "Total zero-copy Arrow Flight bytes streamed", + registry=self._registry, + ) + self.ast_rejections_total = Counter( + "tabular_worker_ast_rejections_total", + "Rejected SQL statements by violated AST rule", + labelnames=("rule",), + registry=self._registry, + ) + + @property + def registry(self) -> CollectorRegistry: + """Return underlying Prometheus CollectorRegistry.""" + return self._registry + + def inc_active_queries(self) -> None: + """Increment active in-flight queries gauge.""" + self.active_queries.inc() + + def dec_active_queries(self) -> None: + """Decrement active in-flight queries gauge.""" + self.active_queries.dec() + + def observe_query_duration(self, cache_state: str, duration_seconds: float) -> None: + """Record query execution duration partitioned by cache_state.""" + self.query_duration_seconds.labels(cache_state=cache_state).observe( + max(0.0, duration_seconds) + ) + + def record_cache_hit(self) -> None: + """Increment cache hit counter and update hit ratio gauge.""" + self.cache_hits_total.inc() + with self._ratio_lock: + self._hits_count += 1 + total = self._hits_count + self._misses_count + self.cache_hit_ratio.set(self._hits_count / total if total > 0 else 0.0) + + def record_cache_miss(self) -> None: + """Increment cache miss counter and update hit ratio gauge.""" + self.cache_misses_total.inc() + with self._ratio_lock: + self._misses_count += 1 + total = self._hits_count + self._misses_count + self.cache_hit_ratio.set(self._hits_count / total if total > 0 else 0.0) + + def set_cache_bytes_used(self, num_bytes: int) -> None: + """Update current local cache disk utilization gauge.""" + self.cache_bytes_used.set(max(0, num_bytes)) + + def inc_subprocess_recycle(self, reason: str) -> None: + """Increment subprocess recycle counter for a given reason.""" + self.subprocess_recycles_total.labels(reason=reason).inc() + + def add_stream_bytes(self, num_bytes: int) -> None: + """Add streamed Arrow IPC bytes to the Flight egress counter.""" + if num_bytes > 0: + self.flight_stream_bytes_total.inc(num_bytes) + + def inc_ast_rejection(self, rule: str) -> None: + """Increment AST rejection counter for a violated rule.""" + self.ast_rejections_total.labels(rule=rule).inc() + + def render_prometheus(self) -> bytes: + """Render all registered metrics in Prometheus text exposition format 0.0.4.""" + result: bytes = generate_latest(self._registry) + return result diff --git a/internal/telemetry/tracing.py b/internal/telemetry/tracing.py new file mode 100644 index 0000000..f90e3fc --- /dev/null +++ b/internal/telemetry/tracing.py @@ -0,0 +1,214 @@ +"""OpenTelemetry distributed tracing, W3C traceparent extraction, and safe span helpers.""" + +from __future__ import annotations + +from collections.abc import Iterator, Mapping +from contextlib import contextmanager +from typing import Any + +import pyarrow.flight as fl +from opentelemetry import trace +from opentelemetry.context import Context, attach, detach +from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter +from opentelemetry.sdk.resources import Resource +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import BatchSpanProcessor +from opentelemetry.trace import Span, Status, StatusCode +from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator + +from internal.config.settings import Settings +from internal.telemetry.logging import REDACTED, get_logger, redact_text + +_logger = get_logger("internal.telemetry.tracing") +_PROPAGATOR = TraceContextTextMapPropagator() + +_SENSITIVE_SPAN_KEYS: frozenset[str] = frozenset( + { + "authorization", + "bearer", + "token", + "access_token", + "secret", + "password", + "aws_access_key_id", + "aws_secret_access_key", + "aws_session_token", + "presigned_url", + "raw_sql", + "sql", + "sql_literals", + } +) + +_ACTIVE_PROVIDER: TracerProvider | None = None + + +def init_tracing(settings: Settings) -> TracerProvider | None: + """Initialize OpenTelemetry TracerProvider when OTEL_ENABLED is True.""" + global _ACTIVE_PROVIDER + if not settings.OTEL_ENABLED: + return None + + if _ACTIVE_PROVIDER is not None: + _ACTIVE_PROVIDER.shutdown() + _ACTIVE_PROVIDER = None + + resource = Resource.create({"service.name": settings.OTEL_SERVICE_NAME}) + provider = TracerProvider(resource=resource) + if settings.OTEL_EXPORTER_ENDPOINT: + exporter = OTLPSpanExporter(endpoint=settings.OTEL_EXPORTER_ENDPOINT) + provider.add_span_processor(BatchSpanProcessor(exporter)) + if isinstance(trace.get_tracer_provider(), trace.ProxyTracerProvider): + trace.set_tracer_provider(provider) + _ACTIVE_PROVIDER = provider + _logger.info( + "Initialized OpenTelemetry tracing", + extra={ + "extra_fields": { + "otel_service_name": settings.OTEL_SERVICE_NAME, + "exporter_configured": bool(settings.OTEL_EXPORTER_ENDPOINT), + } + }, + ) + return provider + + +def shutdown_tracing() -> None: + """Flush and shut down the active TracerProvider if initialized.""" + global _ACTIVE_PROVIDER + if _ACTIVE_PROVIDER is not None: + _ACTIVE_PROVIDER.shutdown() + _ACTIVE_PROVIDER = None + + +def get_tracer(name: str = "rag-tabular-analytics-worker") -> trace.Tracer: + """Return an OpenTelemetry tracer from the currently owned provider or global fallback.""" + if _ACTIVE_PROVIDER is not None: + return _ACTIVE_PROVIDER.get_tracer(name) + return trace.get_tracer(name) + + +def get_current_trace_id() -> str | None: + """Return the active 32-char hex trace_id if a valid span is active.""" + span = trace.get_current_span() + span_ctx = span.get_span_context() + if span_ctx.is_valid: + return format(span_ctx.trace_id, "032x") + return None + + +def sanitize_span_attributes( + attributes: Mapping[str, Any] | None, +) -> dict[str, str | int | float | bool]: + """Filter and redact span attributes so raw SQL, credentials, and presigned URLs never leak.""" + if not attributes: + return {} + sanitized: dict[str, str | int | float | bool] = {} + for key, val in attributes.items(): + if key.lower() in _SENSITIVE_SPAN_KEYS: + sanitized[key] = REDACTED + elif isinstance(val, str): + sanitized[key] = redact_text(val) + elif isinstance(val, bool | int | float): + sanitized[key] = val + elif val is not None: + sanitized[key] = redact_text(str(val)) + return sanitized + + +def record_sanitized_exception(span: Span, exc: BaseException) -> None: + """Record allowlisted error-type and rule metadata without leaking raw SQL/secrets.""" + exc_type = type(exc).__name__ + span.set_status(Status(StatusCode.ERROR, exc_type)) + span.set_attribute("exception.type", exc_type) + rule = getattr(exc, "rule", None) + if isinstance(rule, str) and rule: + span.set_attribute("error.rule", rule) + span.set_attribute("exception.message", f"{exc_type}:{rule}") + else: + span.set_attribute("exception.message", exc_type) + + +def extract_trace_context(headers: Mapping[str, Any] | None) -> Context: + """Extract W3C traceparent/tracestate context from gRPC or HTTP metadata headers.""" + if not headers: + return Context() + carrier: dict[str, str] = {} + for key, val in headers.items(): + lower_key = str(key).lower() + if lower_key in ("traceparent", "tracestate"): + if isinstance(val, list | tuple) and val: + carrier[lower_key] = str(val[0]) + elif isinstance(val, bytes): + carrier[lower_key] = val.decode("utf-8", errors="ignore") + elif isinstance(val, str): + carrier[lower_key] = val + return _PROPAGATOR.extract(carrier=carrier) + + +@contextmanager +def start_span( + name: str, + *, + tracer_name: str = "rag-tabular-analytics-worker", + context: Context | None = None, + attributes: Mapping[str, Any] | None = None, +) -> Iterator[Span]: + """Start an OpenTelemetry span with sanitized attributes and safe error recording.""" + tracer = get_tracer(tracer_name) + safe_attrs = sanitize_span_attributes(attributes) + with tracer.start_as_current_span( + name, + context=context, + attributes=safe_attrs, + record_exception=False, + set_status_on_exception=False, + ) as span: + try: + yield span + except BaseException as exc: + record_sanitized_exception(span, exc) + raise + + +class FlightTracingMiddleware(fl.ServerMiddleware): # type: ignore[misc] + """Per-RPC PyArrow Flight middleware carrying extracted W3C trace context.""" + + def __init__(self, trace_context: Context) -> None: + super().__init__() + self.trace_context = trace_context + + def sending_headers(self) -> dict[str, str]: + return {} + + def call_completed(self, exception: Exception | None) -> None: + return None + + +class FlightTracingMiddlewareFactory(fl.ServerMiddlewareFactory): # type: ignore[misc] + """PyArrow Flight ServerMiddlewareFactory extracting W3C traceparent headers.""" + + def start_call( + self, + info: fl.CallInfo, + headers: dict[str, list[str]], + ) -> FlightTracingMiddleware: + ctx = extract_trace_context(headers) + return FlightTracingMiddleware(ctx) + + +@contextmanager +def attach_flight_trace_context(context: fl.ServerCallContext | None) -> Iterator[None]: + """Attach the extracted W3C trace context from Flight ServerCallContext if present.""" + if context is None: + yield + return + middleware = context.get_middleware("otel_tracing") + if isinstance(middleware, FlightTracingMiddleware): + token = attach(middleware.trace_context) + try: + yield + finally: + detach(token) + return + yield diff --git a/internal/validator/__init__.py b/internal/validator/__init__.py new file mode 100644 index 0000000..6cb7dc3 --- /dev/null +++ b/internal/validator/__init__.py @@ -0,0 +1,25 @@ +"""SQL AST security validator package.""" + +from internal.validator.ast_validator import ASTValidator +from internal.validator.exceptions import ( + ASTSyntaxError, + DisallowedAliasError, + DisallowedFunctionError, + DisallowedStatementError, + DisallowedTableReferenceError, + JoinLimitExceededError, + QueryLengthExceededError, + ValidationError, +) + +__all__ = [ + "ASTSyntaxError", + "ASTValidator", + "DisallowedAliasError", + "DisallowedFunctionError", + "DisallowedStatementError", + "DisallowedTableReferenceError", + "JoinLimitExceededError", + "QueryLengthExceededError", + "ValidationError", +] diff --git a/internal/validator/ast_validator.py b/internal/validator/ast_validator.py new file mode 100644 index 0000000..1987bc0 --- /dev/null +++ b/internal/validator/ast_validator.py @@ -0,0 +1,235 @@ +"""Deterministic sqlglot AST security validator for read-only DuckDB queries.""" + +from __future__ import annotations + +import sqlglot +from sqlglot import Dialect, exp +from sqlglot.errors import ParseError, TokenError + +from internal.validator.exceptions import ( + ASTSyntaxError, + DisallowedAliasError, + DisallowedFunctionError, + DisallowedStatementError, + DisallowedTableReferenceError, + JoinLimitExceededError, + QueryLengthExceededError, +) +from internal.validator.rules import ( + ALLOWED_ANONYMOUS_FUNCTIONS, + ALLOWED_ROOT_EXPRESSIONS, + ALLOWED_STRUCTURAL_NODES, + APPROVED_DATE_UNITS, + BANNED_AST_NODES, + BANNED_FUNCTION_NAMES, + BANNED_FUNCTION_PREFIXES, + SAFE_IDENTIFIER_RE, + ASTNode, +) + + +class ASTValidator: + """Validates incoming SQL queries against a strict read-only DuckDB AST allowlist.""" + + def __init__(self, *, max_sql_length: int = 8192, max_joins: int = 5) -> None: + self._max_sql_length = max_sql_length + self._max_joins = max_joins + self._banned_type_cache: dict[type[ASTNode], bool] = {} + self._allowed_type_cache: dict[type[ASTNode], bool] = {} + self._dialect = Dialect.get_or_raise("duckdb") + # Pre-warm sqlglot DuckDB dialect parser so steady-state validation stays <= 2 ms. + sqlglot.parse_one("SELECT 1", read=self._dialect) + + def _is_banned_node(self, node: ASTNode) -> bool: + node_type = type(node) + cached = self._banned_type_cache.get(node_type) + if cached is None: + cached = isinstance(node, BANNED_AST_NODES) + self._banned_type_cache[node_type] = cached + return cached + + def _is_allowed_node(self, node: ASTNode) -> bool: + node_type = type(node) + cached = self._allowed_type_cache.get(node_type) + if cached is None: + cached = isinstance(node, ALLOWED_STRUCTURAL_NODES) + self._allowed_type_cache[node_type] = cached + return cached + + def validate(self, sql: str, allowed_table_alias: str) -> ASTNode: + """Validate SQL against length, syntax, statement, function, table, alias, and join rules. + + Args: + sql: Raw SQL string from the incoming Flight ticket. + allowed_table_alias: The single table alias bound in the DuckDB sandbox. + + Returns: + The validated root sqlglot AST expression. + + Raises: + ValidationError: If any security or structural rule is violated. + """ + if not sql or not sql.strip(): + raise ASTSyntaxError("SQL query must not be empty.") + + if len(sql) > self._max_sql_length: + raise QueryLengthExceededError( + f"SQL query length {len(sql)} exceeds maximum of {self._max_sql_length} characters." + ) + + if not SAFE_IDENTIFIER_RE.match(allowed_table_alias): + raise DisallowedAliasError(f"Invalid allowed table alias: {allowed_table_alias!r}") + + try: + statements = sqlglot.parse(sql, read=self._dialect) + except (ParseError, TokenError) as exc: + raise ASTSyntaxError(f"Failed to parse SQL query: {exc}") from exc + + non_empty = [stmt for stmt in statements if stmt is not None] + if len(non_empty) != 1: + raise ASTSyntaxError(f"Expected exactly 1 SQL statement, got {len(non_empty)}.") + + root = non_empty[0] + if self._is_banned_node(root) or not isinstance(root, ALLOWED_ROOT_EXPRESSIONS): + raise DisallowedStatementError( + f"Statement type {type(root).__name__} is not permitted; " + "only read-only SELECT is allowed." + ) + + self._validate_cte_aliases(root) + join_count = 0 + + for node in root.walk(): + if self._is_banned_node(node): + raise DisallowedStatementError( + f"Disallowed AST construct {type(node).__name__} detected." + ) + + if not self._is_allowed_node(node) and not self._is_approved_date_unit_var(node): + raise DisallowedStatementError( + f"Unallowlisted AST node {type(node).__name__} detected." + ) + + if isinstance(node, exp.Join): + join_count += 1 + if join_count > self._max_joins: + raise JoinLimitExceededError( + f"Query contains {join_count} joins, exceeding MAX_JOINS={self._max_joins}." + ) + + if isinstance(node, exp.Func): + self._validate_function(node) + + if isinstance(node, exp.Table): + self._validate_table(node, allowed_table_alias) + + if isinstance(node, exp.Alias | exp.TableAlias): + self._validate_alias(node) + + return root + + @staticmethod + def _is_approved_date_unit_var(node: ASTNode) -> bool: + """Allow exp.Var only when used as an approved unit argument on date functions.""" + if not isinstance(node, exp.Var) or node.arg_key != "unit": + return False + if not isinstance( + node.parent, + exp.DateDiff | exp.DateTrunc | exp.DateAdd | exp.DateSub | exp.Extract, + ): + return False + return node.name.lower() in APPROVED_DATE_UNITS + + @staticmethod + def _validate_cte_aliases(root: ASTNode) -> None: + """Validate all CTE alias identifiers defined in WITH clauses.""" + for cte in root.find_all(exp.CTE): + alias_name = cte.alias_or_name + if not alias_name or not SAFE_IDENTIFIER_RE.match(alias_name): + raise DisallowedAliasError(f"Invalid CTE alias: {alias_name!r}") + + @staticmethod + def _is_descendant_or_self(node: ASTNode, ancestor: ASTNode) -> bool: + curr: ASTNode | None = node + while curr is not None: + if curr is ancestor: + return True + curr = curr.parent + return False + + @classmethod + def _visible_cte_names(cls, table: exp.Table) -> set[str]: + """Collect CTE names visible from ``table``'s lexical SQL scope.""" + visible: set[str] = set() + child: ASTNode = table + curr = table.parent + while curr is not None: + with_clause = curr.args.get("with_") or curr.args.get("with") + if isinstance(with_clause, exp.With): + is_recursive = bool(with_clause.args.get("recursive")) + for cte in with_clause.expressions: + if not isinstance(cte, exp.CTE): + continue + alias_name = cte.alias_or_name + if child is with_clause and cls._is_descendant_or_self(table, cte): + if is_recursive and alias_name: + visible.add(alias_name.lower()) + break + if alias_name: + visible.add(alias_name.lower()) + child = curr + curr = curr.parent + return visible + + @staticmethod + def _validate_function(node: exp.Func) -> None: + """Ensure function calls are not banned and anonymous functions are allowlisted.""" + if isinstance(node, exp.Anonymous): + func_name = node.name.lower() + if func_name in BANNED_FUNCTION_NAMES or func_name.startswith(BANNED_FUNCTION_PREFIXES): + raise DisallowedFunctionError(f"Function '{func_name}' is explicitly banned.") + if func_name not in ALLOWED_ANONYMOUS_FUNCTIONS: + raise DisallowedFunctionError( + f"Function '{func_name}' is not in the approved function allowlist." + ) + return + + sql_name = (node.key or type(node).__name__).lower() + if sql_name in BANNED_FUNCTION_NAMES or sql_name.startswith(BANNED_FUNCTION_PREFIXES): + raise DisallowedFunctionError(f"Function '{sql_name}' is explicitly banned.") + + @classmethod + def _validate_table( + cls, + table: exp.Table, + allowed_table_alias: str, + ) -> None: + """Ensure table references point only to the bound alias or a visible local CTE.""" + if table.catalog or table.db: + raise DisallowedTableReferenceError( + "Schema-qualified or catalog-qualified table references are not permitted." + ) + + if not isinstance(table.this, exp.Identifier): + raise DisallowedTableReferenceError( + "Table reference must be a simple identifier " + "(file literals and sub-expressions are banned)." + ) + + table_name = table.name + if not SAFE_IDENTIFIER_RE.match(table_name): + raise DisallowedTableReferenceError(f"Invalid table identifier: {table_name!r}") + + lowered = table_name.lower() + if lowered != allowed_table_alias.lower() and lowered not in cls._visible_cte_names(table): + raise DisallowedTableReferenceError( + f"Table '{table_name}' is not permitted; " + f"only '{allowed_table_alias}' or local CTEs are allowed." + ) + + @staticmethod + def _validate_alias(node: exp.Alias | exp.TableAlias) -> None: + """Validate column and table aliases against the safe identifier pattern.""" + alias_name = node.alias_or_name + if alias_name and not SAFE_IDENTIFIER_RE.match(alias_name): + raise DisallowedAliasError(f"Disallowed alias identifier: {alias_name!r}") diff --git a/internal/validator/exceptions.py b/internal/validator/exceptions.py new file mode 100644 index 0000000..96e2d71 --- /dev/null +++ b/internal/validator/exceptions.py @@ -0,0 +1,57 @@ +"""Validation exception hierarchy for SQL AST security checks.""" + +from __future__ import annotations + + +class ValidationError(Exception): + """Base class for deterministic SQL AST validation rejections.""" + + rule: str = "VALIDATION_ERROR" + + def __init__(self, message: str, *, rule: str | None = None) -> None: + if rule is not None: + self.rule = rule + super().__init__(message) + self.message = message + + +class QueryLengthExceededError(ValidationError): + """Raised when incoming SQL exceeds MAX_SQL_LENGTH.""" + + rule = "QUERY_LENGTH_EXCEEDED" + + +class ASTSyntaxError(ValidationError): + """Raised when SQL cannot be parsed or contains multiple statements.""" + + rule = "SYNTAX_ERROR" + + +class DisallowedStatementError(ValidationError): + """Raised when SQL contains non-SELECT statements or disallowed AST nodes.""" + + rule = "DISALLOWED_STATEMENT" + + +class DisallowedFunctionError(ValidationError): + """Raised when SQL invokes a banned or non-allowlisted function.""" + + rule = "DISALLOWED_FUNCTION" + + +class DisallowedTableReferenceError(ValidationError): + """Raised when SQL references an unallowlisted table, file literal, or schema.""" + + rule = "DISALLOWED_TABLE_REFERENCE" + + +class DisallowedAliasError(ValidationError): + """Raised when a column, table, or CTE alias fails identifier validation.""" + + rule = "DISALLOWED_ALIAS" + + +class JoinLimitExceededError(ValidationError): + """Raised when SQL exceeds MAX_JOINS.""" + + rule = "JOIN_LIMIT_EXCEEDED" diff --git a/internal/validator/rules.py b/internal/validator/rules.py new file mode 100644 index 0000000..f8e717a --- /dev/null +++ b/internal/validator/rules.py @@ -0,0 +1,443 @@ +"""SQL AST allowlist rules, banned constructs, and identifier constraints.""" + +from __future__ import annotations + +import re +from typing import Any + +from sqlglot import exp + +# sqlglot <29 uses exp.Expression as the AST base class, while sqlglot >=29 uses exp.Expr. +type ASTNode = Any +_AST_NODE_BASE: type[Any] = getattr(exp, "Expr", exp.Expression) + +SAFE_IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,127}$") + +# Top-level root statements permitted by the sandbox. +ALLOWED_ROOT_EXPRESSIONS: tuple[type[ASTNode], ...] = ( + exp.Select, + exp.Union, + exp.Intersect, + exp.Except, + exp.Subquery, +) + + +def _optional_exp_nodes(*names: str) -> tuple[type[ASTNode], ...]: + """Return sqlglot AST node subclasses that exist in the installed sqlglot version.""" + nodes: list[type[ASTNode]] = [] + for name in names: + cls = getattr(exp, name, None) + if isinstance(cls, type) and issubclass(cls, _AST_NODE_BASE): + nodes.append(cls) + return tuple(nodes) + + +# Explicitly banned statement/command AST node types. +BANNED_AST_NODES: tuple[type[ASTNode], ...] = ( + exp.Command, + exp.Insert, + exp.Update, + exp.Delete, + exp.Merge, + exp.Drop, + exp.Alter, + exp.Create, + exp.Pragma, + exp.Set, + exp.Copy, + exp.Attach, + exp.Detach, + exp.LoadData, + exp.Describe, + exp.Show, + exp.Summarize, + exp.Pivot, + exp.Use, + exp.Transaction, + exp.Commit, + exp.Rollback, + exp.Grant, + *_optional_exp_nodes( + "Install", + "Revoke", + "ReadParquet", + "ReadCSV", + "ReadJSON", + "ScanParquet", + ), +) + +# Approved date/time unit identifiers when sqlglot parses them as exp.Var(unit=...). +APPROVED_DATE_UNITS: frozenset[str] = frozenset( + { + "year", + "years", + "quarter", + "quarters", + "month", + "months", + "week", + "weeks", + "day", + "days", + "hour", + "hours", + "minute", + "minutes", + "second", + "seconds", + "millisecond", + "milliseconds", + "microsecond", + "microseconds", + "nanosecond", + "nanoseconds", + } +) + +# Banned function names (lowercased) that access files, network, or DuckDB internals. +BANNED_FUNCTION_NAMES: frozenset[str] = frozenset( + { + "read_csv", + "read_csv_auto", + "read_parquet", + "scan_parquet", + "parquet_scan", + "parquet_metadata", + "parquet_schema", + "parquet_file_metadata", + "read_json", + "read_json_auto", + "read_ndjson", + "read_ndjson_auto", + "read_text", + "read_blob", + "read_xlsx", + "st_read", + "write_csv", + "write_parquet", + "copy_to", + "glob", + "list_files", + "current_setting", + "set_config", + "getenv", + "system", + "exec", + "eval", + "query", + "query_table", + "sql_query", + "pragma_table_info", + "pragma_database_list", + "pragma_version", + "duckdb_tables", + "duckdb_columns", + "duckdb_databases", + "duckdb_extensions", + "duckdb_functions", + "duckdb_settings", + "duckdb_secrets", + "duckdb_views", + "duckdb_schemas", + "duckdb_memory", + "duckdb_temporary_files", + "version", + } +) + +# Banned function name prefixes (lowercased). +BANNED_FUNCTION_PREFIXES: tuple[str, ...] = ( + "read_", + "write_", + "scan_", + "duckdb_", + "pragma_", + "httpfs_", + "s3_", + "azure_", + "gcs_", + "iceberg_", + "delta_", + "sqlite_", + "postgres_", + "mysql_", +) + +# Pre-approved function names when sqlglot parses a call as exp.Anonymous. +ALLOWED_ANONYMOUS_FUNCTIONS: frozenset[str] = frozenset( + { + # Analytical / statistical / aggregate + "approx_count_distinct", + "arg_max", + "arg_min", + "array_agg", + "corr", + "covar_pop", + "covar_samp", + "first", + "last", + "list", + "list_agg", + "median", + "mode", + "percentile_cont", + "percentile_disc", + "quantile", + "quantile_cont", + "quantile_disc", + "stddev", + "stddev_pop", + "stddev_samp", + "string_agg", + "var_pop", + "var_samp", + "variance", + # Math + "abs", + "acos", + "asin", + "atan", + "atan2", + "cbrt", + "ceil", + "ceiling", + "cos", + "degrees", + "exp", + "floor", + "greatest", + "isfinite", + "isinf", + "isnan", + "least", + "ln", + "log", + "log10", + "log2", + "mod", + "pi", + "pow", + "power", + "radians", + "round", + "sign", + "sin", + "sqrt", + "tan", + "trunc", + # String + "ascii", + "bit_length", + "char_length", + "character_length", + "chr", + "concat", + "concat_ws", + "contains", + "ends_with", + "format", + "ilike", + "instr", + "lcase", + "left", + "len", + "length", + "lower", + "lpad", + "ltrim", + "octet_length", + "position", + "prefix", + "printf", + "regexp_extract", + "regexp_matches", + "regexp_replace", + "repeat", + "replace", + "reverse", + "right", + "rpad", + "rtrim", + "split_part", + "starts_with", + "strlen", + "strpos", + "substr", + "substring", + "suffix", + "trim", + "ucase", + "upper", + # Date / time + "age", + "current_date", + "current_time", + "current_timestamp", + "date_diff", + "date_part", + "date_sub", + "date_trunc", + "datediff", + "datepart", + "day", + "dayname", + "dayofmonth", + "dayofweek", + "dayofyear", + "epoch", + "epoch_ms", + "epoch_ns", + "epoch_us", + "extract", + "hour", + "last_day", + "make_date", + "make_timestamp", + "minute", + "month", + "monthname", + "now", + "quarter", + "second", + "strftime", + "strptime", + "time_bucket", + "today", + "week", + "year", + # Conditional / type + "coalesce", + "if", + "ifnull", + "nullif", + "typeof", + } +) + +# Complete allowlist of non-function AST node types permitted in read-only SELECT queries. +ALLOWED_STRUCTURAL_NODES: tuple[type[ASTNode], ...] = ( + # Query & set operations + exp.Select, + exp.With, + exp.CTE, + exp.Union, + exp.Intersect, + exp.Except, + exp.Subquery, + exp.From, + exp.Join, + exp.Where, + exp.Group, + exp.Having, + exp.Qualify, + exp.Order, + exp.Ordered, + exp.Limit, + exp.Offset, + exp.Distinct, + exp.Window, + exp.WindowSpec, + exp.Partition, + exp.Filter, + exp.WithinGroup, + # Aliases, identifiers, columns, literals + exp.Alias, + exp.TableAlias, + exp.Table, + exp.Column, + exp.Identifier, + exp.Star, + exp.Literal, + exp.Boolean, + exp.Null, + exp.DataType, + exp.DataTypeParam, + exp.Interval, + exp.Paren, + exp.Tuple, + exp.Array, + exp.Struct, + exp.Bracket, + exp.Dot, + exp.Lambda, + # Predicates & operators + exp.Condition, + exp.Binary, + exp.Unary, + exp.Connector, + exp.And, + exp.Or, + exp.Not, + exp.In, + exp.Between, + exp.Is, + exp.Like, + exp.ILike, + exp.SimilarTo, + exp.Exists, + exp.Case, + exp.If, + exp.Cast, + exp.TryCast, + exp.Extract, + exp.Substring, + exp.Trim, + exp.Count, + exp.Sum, + exp.Avg, + exp.Min, + exp.Max, + exp.Coalesce, + exp.Nullif, + exp.RowNumber, + exp.Rank, + exp.DenseRank, + exp.Lag, + exp.Lead, + exp.FirstValue, + exp.LastValue, + exp.NthValue, + exp.Ntile, + exp.Stddev, + exp.StddevPop, + exp.StddevSamp, + exp.Variance, + exp.VariancePop, + exp.Quantile, + exp.ApproxDistinct, + exp.ArrayAgg, + exp.GroupConcat, + exp.Lower, + exp.Upper, + exp.Length, + exp.Concat, + exp.ConcatWs, + exp.Replace, + exp.SplitPart, + exp.RegexpLike, + exp.RegexpReplace, + exp.RegexpExtract, + exp.Abs, + exp.Round, + exp.Ceil, + exp.Floor, + exp.Pow, + exp.Sqrt, + exp.Log, + exp.Exp, + exp.Greatest, + exp.Least, + exp.DateTrunc, + exp.DateDiff, + exp.DateAdd, + exp.DateSub, + exp.StrToTime, + exp.TimeToStr, + exp.CurrentDate, + exp.CurrentTimestamp, + exp.CurrentTime, + exp.Anonymous, + exp.Func, +) diff --git a/prompts/phase-1-subprocess-sandboxing-ast-validation.md b/prompts/phase-1-subprocess-sandboxing-ast-validation.md new file mode 100644 index 0000000..0c650af --- /dev/null +++ b/prompts/phase-1-subprocess-sandboxing-ast-validation.md @@ -0,0 +1,65 @@ +# Implementation Plan: Phase 1 — Subprocess Sandboxing & AST Validation + +## 1. Goal & Scope +Implement **Phase 1: Subprocess Sandboxing & AST Validation** for `rag-tabular-analytics-worker` (`GEMINI.MD`, `.ai/progress-tracker.md`, `tasks/Tabular Analytics Worker Implementation Plan.pdf`): +- Deterministic `sqlglot` AST security validator ($\le 2\text{ ms}$ warm steady-state) enforcing strict AST node and pre-approved function allowlisting (`SELECT` / `WITH` / `UNION` / `INTERSECT` / `EXCEPT`), column alias checks, join limits, and blocking DML/DDL, `PRAGMA`/`SET`/`RESET`/`SUMMARIZE`/`DESCRIBE`/`SHOW`/`PIVOT`, `COPY`/`ATTACH`/`DETACH`/`LOAD`/`INSTALL`, and DuckDB filesystem/network functions. +- Ephemeral DuckDB `:memory:` sandbox (`duckdb >= 1.2.0`) materializing only the single validated `{hash}.parquet` artifact (`CREATE TEMP TABLE AS SELECT * FROM read_parquet(...)`, since lazy `TEMP VIEW`s fail at query time once external access is disabled) before locking `enable_external_access=false`, `lock_configuration=true`, `temp_directory=''`, `threads=2`, and `memory_limit=floor(2.5GiB / POOL_SIZE)` (e.g., `640MiB` at `POOL_SIZE=4`). +- Disposable worker subprocess pool (`pebble.ProcessPool` with `max_tasks=100`) streaming chunked Arrow IPC `RecordBatch` frames over an OS pipe / `multiprocessing.connection` or Unix socket (never pickling full-result `bytes` or calling `fetch_arrow_table()`), paired with a `5s` per-worker circuit breaker (`SIGKILL` on POSIX / `TerminateProcess` on Windows + respawn) keeping sibling workers serviceable (`ADR-001`). +- Project scaffolding (`pyproject.toml`, `Makefile`, `internal/config/settings.py`, structured JSON logging) and Phase 1 test suites (`test/unit/`, `test/security/`, `test/chaos/`, `test/fixtures/`). + +--- + +## 2. Files to Create / Modify + +### Project Scaffolding & Config +- `pyproject.toml` — Python `>=3.12` package metadata, runtime deps (`duckdb>=1.2.0`, `pyarrow`, `sqlglot`, `pebble`, `pydantic`, `pydantic-settings`), dev/test deps (`ruff`, `mypy`, `bandit`, `pip-audit`, `pytest`, `pytest-asyncio`, `pytest-cov`), and `mypy` overrides for untyped third-party packages (e.g., `pebble`) if needed. +- `Makefile` — Local targets for `format`, `lint`, `typecheck`, `security`, `test-unit`, `test-security`, `test-chaos`, `test`, and `check`. +- `README.md` — Concise service overview, local dev setup, and verification commands. +- `internal/__init__.py`, `internal/config/__init__.py`, `internal/config/settings.py` — Typed `Settings` (`POOL_SIZE=4`, `MAX_TASKS_PER_CHILD=100`, `DUCKDB_TOTAL_MEMORY_BYTES=2.5 GiB`, per-worker `memory_limit` formatted explicitly in `MiB`, `QUERY_TIMEOUT_SECONDS=5.0`, `MAX_SQL_LENGTH`, `MAX_JOINS`, `DUCKDB_THREADS=2`). +- `internal/telemetry/__init__.py`, `internal/telemetry/logging.py` — Structured JSON logger with strict redaction of credentials, bearer tokens, presigned S3 params, and raw SQL literals (no `print()`, and keep redaction key literals outside `logger.(...)` calls to satisfy CI static regex audits). + +### SQL AST Security Validator (`internal/validator/`) +- `internal/validator/__init__.py` +- `internal/validator/exceptions.py` — `ValidationError` hierarchy (`ASTSyntaxError`, `DisallowedStatementError`, `DisallowedFunctionError`, `DisallowedTableReferenceError`, `DisallowedAliasError`, `JoinLimitExceededError`, `QueryLengthExceededError`) with `rule` codes for metrics/logging. +- `internal/validator/rules.py` — Strict read-only AST node allowlist (`Select`, `With`, `Union`, `Intersect`, `Except`, `Subquery`), explicit rejection of `exp.Command` and banned statements (`INSERT`, `UPDATE`, `DELETE`, `DROP`, `ALTER`, `CREATE`, `PRAGMA`, `SET`, `RESET`, `SUMMARIZE`, `DESCRIBE`, `SHOW`, `PIVOT`, `COPY`, `ATTACH`, `DETACH`, `LOAD`, `INSTALL`), banned DuckDB I/O & system functions (`read_csv`, `read_parquet`, `scan_parquet`, `write_csv`, `glob`, `httpfs_*`, `duckdb_*`, `getenv`, etc.), pre-approved function allowlist (including strict filtering of `exp.Anonymous`), column alias checks, join count limit (`MAX_JOINS`), and table/CTE alias validation. +- `internal/validator/ast_validator.py` — `ASTValidator.validate(sql, allowed_table_alias)` using `sqlglot.parse(..., read="duckdb")`, walking the full AST against the allowlist, rejecting multi-statement queries, and enforcing $\le 2\text{ ms}$ warm steady-state validation. + +### DuckDB Sandbox Runtime (`internal/sandbox/`) +- `internal/sandbox/__init__.py` +- `internal/sandbox/security_profile.py` — Binds the single validated `{hash}.parquet` artifact via `CREATE TEMP TABLE AS SELECT * FROM read_parquet(...)` (avoiding lazy `TEMP VIEW` failures after lockdown and never setting `enable_external_access=true`), then immediately locks `enable_external_access = false`, `lock_configuration = true`, `temp_directory = ''`, `memory_limit` (`MiB`), and `threads = 2`. +- `internal/sandbox/duckdb_runtime.py` — Ephemeral `duckdb.connect(":memory:")` lifecycle and chunked Arrow `RecordBatchReader` IPC streaming (no `fetch_arrow_table()` full materialization). + +### Disposable Subprocess Pool & Circuit Breaker (`internal/pool/`) +- `internal/pool/__init__.py` +- `internal/pool/worker_task.py` — Top-level picklable (`spawn`-safe) child subprocess entrypoint running `duckdb_runtime` + `security_profile` and streaming Arrow IPC `RecordBatch` chunks over a dedicated OS pipe / `multiprocessing.connection` (or Unix socket) without buffering the full result set on the heap. +- `internal/pool/circuit_breaker.py` — `5s` execution deadline enforcement with cross-platform isolated child termination (`SIGKILL` on POSIX / `TerminateProcess` via `pebble` on Windows) and recycle reason tracking (`TIMEOUT`, `MEMORY_LIMIT`, `TASK_LIMIT`). +- `internal/pool/process_manager.py` — `pebble.ProcessPool` supervisor (`max_workers=POOL_SIZE`, `max_tasks=100`), pre-allocated worker pool, and automatic worker replacement without `BrokenProcessPool`. + +### Tests & Fixtures (`test/`) +- `test/fixtures/sample_data.py` (or generated `.parquet` fixture) — Normalized test Parquet dataset. +- `test/unit/test_settings.py` — Memory budget (`floor(2.5GiB / POOL_SIZE)` in `MiB`) and config validation tests. +- `test/unit/test_ast_validator.py` — Valid `SELECT`/`WITH`/`UNION`, column/table/CTE alias checks, join limit checks, and $\le 2\text{ ms}$ warm steady-state latency tests (post parser warmup). +- `test/unit/test_duckdb_sandbox.py` — Ephemeral `:memory:` execution, `TEMP TABLE` binding + `enable_external_access=false` verification, and chunked Arrow IPC batch streaming tests. +- `test/security/test_sql_injection_fuzz.py` — DML/DDL, `PRAGMA`, `SET`, `RESET`, `SUMMARIZE`, `DESCRIBE`, `SHOW`, `PIVOT`, `COPY`, `ATTACH`, `LOAD`/`INSTALL`, DuckDB file/system functions (`exp.Anonymous` bypass attempts), multi-statement injection, and sandbox escape tests (`enable_external_access=false`). +- `test/chaos/test_subprocess_isolation.py` — Cross-platform forced child termination/panic isolation (without hardcoding POSIX-only `signal.SIGKILL` on Windows), `5s` timeout circuit breaker (cross-join), and `max_tasks=100` recycling tests. + +### Documentation Sync +- `.ai/progress-tracker.md` — Update Phase 1 status and session notes upon completion. +- `.ai/doc-api.md` — Verify all AST rules, error mappings, and config invariants remain synchronized. + +--- + +## 3. Verification Plan +1. `ruff format --check .` and `ruff check .` +2. `mypy cmd internal api` (targeting `internal` in Phase 1) +3. `bandit -r internal -ll` and `pip-audit` +4. Secured logging & DuckDB sandbox audit (no `print()`, no unredacted secrets in logger calls, no `enable_external_access=true`) +5. `pytest -v test/unit test/security test/chaos` +6. At least 2 conventional git commits (`chore:`, `feat:`, `tests:`, `docs:`) as Phase 1 is built. + +--- + +## 4. Open Questions for Human Confirmation +1. **Function Allowlist & Join Limit Defaults**: Confirm strict `exp.Anonymous` allowlisting (standard analytical, math, string, date/time, window, cast/conditional functions) and a default `MAX_JOINS` ceiling (e.g., `5`) in `internal/validator/rules.py`. +2. **Phase 1 Cross-Process IPC Transport**: Confirm using a dedicated OS pipe / `multiprocessing.connection` (or Unix socket on POSIX) to stream Arrow IPC `RecordBatch` chunks from `worker_task.py` to the parent process without full-result `bytes` buffering. +3. **Proceed**: Please confirm if we should proceed with implementing Phase 1 according to this updated plan. diff --git a/prompts/phase-2-nvme-cache-arrow-flight-daemon.md b/prompts/phase-2-nvme-cache-arrow-flight-daemon.md new file mode 100644 index 0000000..5535c14 --- /dev/null +++ b/prompts/phase-2-nvme-cache-arrow-flight-daemon.md @@ -0,0 +1,72 @@ +# Implementation Plan: Phase 2 — Disk-Backed NVMe Cache & Arrow Flight Daemon + +## 1. Goal & Scope +Implement **Phase 2: Disk-Backed NVMe Cache & Arrow Flight Daemon** for `rag-tabular-analytics-worker` (`GEMINI.MD`, `.ai/progress-tracker.md`, `.ai/doc-api.md`, `tasks/Tabular Analytics Worker Implementation Plan.pdf`): +- **Tier-1 Disk-Backed NVMe Parquet Cache (`internal/storage/cache_manager.py`)**: Content-addressed `/{s3_bucket}/{tenant_id}/{table_hash}.parquet` local cache (`10 GiB` default ceiling) with startup disk-state reconciliation + `.tmp` sweep, `threading.Lock`-protected `OrderedDict` LRU eviction + active-reader refcounts (independent of `noatime` mounts), per-artifact single-flight download coalescing, atomic `os.replace` writes, pre-commit Parquet `PAR1`/metadata verification, and safe eviction of corrupted artifacts. +- **S3 Streaming & Tenant Isolation (`internal/storage/s3_streamer.py`, `internal/storage/table_resolver.py`)**: Async S3 range/chunked streaming (`aiobotocore`) over internal VPC endpoints on a dedicated `asyncio` event-loop thread, streaming chunks directly to `.tmp` files on disk (zero monolithic Python heap buffering) with bounded connect/read timeouts, plus strict `s3_bucket` allowlist and canonical `s3_key == f"artifacts/{tenant_id}/{table_hash}.parquet"` tenant isolation (`PERMISSION_DENIED`). +- **Apache Arrow Flight Daemon (`api/flight/`, `internal/flight/`)**: `FlightServerBase` handling HTTP/2 gRPC `DoGet` (streaming zero-copy `RecordBatch` chunks from `ProcessManager.stream_query()` via `GeneratorStream` / `RecordBatchReader` without `fetch_arrow_table()` or JSON row conversion) and `DoAction` (`cache_warm`, `health_check`), bounded by `MAX_CONCURRENT_QUERIES` (`4`, matching `POOL_SIZE` for immediate `RESOURCE_EXHAUSTED` backpressure without hidden pool queuing). +- **Telemetry, OpenAPI 3.1 Ops HTTP Server & Entrypoint (`internal/telemetry/metrics.py`, `api/openapi/openapi.yaml`, `cmd/worker/main.py`)**: Prometheus metrics registry + HTTP ops listener on port `8080` serving `/healthz`, `/ready` (verifying `ProcessManager.is_healthy()` + writable cache mount + drain state), `/metrics`, `/openapi.yaml`, and `/docs` (Swagger UI), with `SIGINT`/`SIGTERM` graceful drain. + +--- + +## 2. Files to Create / Modify + +### Configuration (`internal/config/settings.py`, `pyproject.toml`) +- `pyproject.toml` — Add Phase 2 runtime dependencies (`aiobotocore`, `pyyaml`) and dev/test dependency (`openapi-spec-validator`), plus `mypy` `ignore_missing_imports` overrides for `aiobotocore` / `aiobotocore.*`. +- `internal/config/settings.py` — Add `FLIGHT_HOST`, `FLIGHT_PORT` (`8815`), `HTTP_HOST`, `HTTP_PORT` (`8080`), `CACHE_DIR`, `CACHE_MAX_BYTES` (`10 GiB`), `S3_BUCKET_ALLOWLIST`, `S3_ENDPOINT_URL`, `S3_REGION`, `S3_CHUNK_SIZE_BYTES` (`8 MiB`), `S3_TIMEOUT_SECONDS` (`4.0`), and `MAX_CONCURRENT_QUERIES` (default `4`, aligned with `POOL_SIZE=4` and KEDA `tabular_worker_active_queries >= 4`). + +### Tier-1 Storage & Tenant Isolation (`internal/storage/`) +- `internal/storage/__init__.py` +- `internal/storage/table_resolver.py` — `TableResolver` validating `tenant_id`, 64-hex SHA-256 `table_hash`, `s3_bucket` allowlist (fail-closed if empty), and canonical `s3_key`; raises `TenantIsolationError` (`PERMISSION_DENIED`) or `ValidationError` (`INVALID_ARGUMENT`). +- `internal/storage/s3_streamer.py` — `S3Streamer` whose `aiobotocore` `AioSession`/client lifecycle is created and closed inside the dedicated `asyncio` hydration loop thread; streams `GetObject` / bounded range reads (`Range: bytes=...` / `iter_chunks`) directly to a temp file in `CACHE_DIR` with explicit connect/read timeouts; maps `NoSuchKey`/`404` to `ArtifactNotFoundError` (`NOT_FOUND`), `403` to `TenantIsolationError`/`S3AccessError`, and never logs presigned URLs, `X-Amz-*` parameters, or AWS credentials. +- `internal/storage/cache_manager.py` — `CacheManager` managing `/{s3_bucket}/{tenant_id}/{table_hash}.parquet`: + - Scans `CACHE_DIR` on startup to seed LRU size tracking from existing `.parquet` files and remove orphaned `.tmp` files. + - Uses a `threading.Lock` for `OrderedDict` LRU state + per-artifact reader `refcount`s (safe across gRPC threads and the `asyncio` loop thread) and per-key single-flight coalescing on the `asyncio` loop thread. + - Verifies `PAR1` header/footer magic bytes and `pyarrow.parquet.ParquetFile` metadata before `os.replace`, and evicts corrupted entries on read failure. + - If `CACHE_MAX_BYTES` is exhausted and all cached entries are pinned (`refcount > 0`), fails new cold hydrations with `CacheCapacityError` (`RESOURCE_EXHAUSTED`) rather than exceeding the `10 GiB` volume limit. + +### Arrow Flight Contracts, Telemetry & Server (`api/flight/`, `internal/flight/`, `internal/telemetry/`) +- `api/flight/__init__.py` +- `api/flight/schema.py` — Pydantic `FlightQueryTicket` model (`request_id`, `tenant_id`, `sql`, `table_alias`, `table_hash`, `s3_bucket`, `s3_key`) and sanitized gRPC/Flight error mapping (`INVALID_ARGUMENT`, `PERMISSION_DENIED`, `DEADLINE_EXCEEDED`, `INTERNAL`, `RESOURCE_EXHAUSTED`, `NOT_FOUND`) that never leaks stack traces, bucket names, or S3 keys to clients. +- `api/flight/actions.py` — `CacheWarmRequest`/`CacheWarmResponse` and `HealthCheckResponse` schemas for `DoAction`. +- `internal/telemetry/metrics.py` — Thread-safe Prometheus metrics registry & text exposition helper (`tabular_worker_active_queries`, `tabular_worker_cache_hits_total`, `tabular_worker_cache_misses_total`, `tabular_worker_cache_hit_ratio`, `tabular_worker_cache_bytes_used`). +- `internal/flight/__init__.py` +- `internal/flight/handlers.py` — `FlightHandlers` coordinating ticket validation, `request_id` structured log correlation, `ASTValidator`, `TableResolver` (enforced on both `DoGet` and `DoAction("cache_warm")`), `CacheManager`/`S3Streamer` via `asyncio.run_coroutine_threadsafe`, `MAX_CONCURRENT_QUERIES` semaphore, and `ProcessManager.stream_query(...)`: + - Adapts `ProcessManager.stream_query()` (`Iterator[pa.RecordBatch]`) to `GeneratorStream` / `RecordBatchReader` without materializing a `pa.Table`. + - Releases the `MAX_CONCURRENT_QUERIES` semaphore, decrements `tabular_worker_active_queries`, and unpins the `CacheManager` reader refcount inside the **stream generator's `finally:` block** (not when `do_get` returns). + - Catches `QueryTimeoutError` → `FlightTimedOutError` (`DEADLINE_EXCEEDED`) and `WorkerCrashError` → `FlightInternalError` (`INTERNAL`) inside the stream wrapper. +- `internal/flight/server.py` — `TabularFlightServer(pyarrow.flight.FlightServerBase)` implementing `do_get`, `do_action` (`cache_warm` idempotent, `health_check`), and `list_actions`. + +### OpenAPI 3.1 & Ops HTTP Server (`api/openapi/`, `internal/flight/ops_server.py`, `cmd/worker/main.py`) +- `api/__init__.py`, `api/openapi/openapi.yaml` — OpenAPI 3.1.0 specification for `/healthz`, `/ready`, `/metrics`, `/openapi.yaml`, and `/docs` (validated by `openapi-spec-validator`). +- `internal/flight/ops_server.py` — `http.server.ThreadingHTTPServer` serving `/healthz`, `/ready` (returning `503` if draining, `!ProcessManager.is_healthy()`, or `CACHE_DIR` not writable), `/metrics` (from `internal/telemetry/metrics.py`), `/openapi.yaml`, and `/docs`. +- `cmd/__init__.py`, `cmd/worker/__init__.py`, `cmd/worker/main.py` — Daemon entrypoint starting the `asyncio` hydration loop thread (with `S3Streamer` client lifecycle), `ProcessManager`, `TabularFlightServer`, and Ops HTTP server, and handling `SIGINT`/`SIGTERM` graceful drain (`503` on `/ready` → `TabularFlightServer.shutdown()` → `ProcessManager.close()` → close S3 client & stop `asyncio` loop). + +### Tests (`test/unit/`, `test/integration/`) +- `test/unit/test_table_resolver.py` — Tenant isolation, `s3_bucket` allowlist, canonical `s3_key`, and `table_hash` validation tests. +- `test/unit/test_cache_manager.py` — Startup `.parquet` reconciliation & `.tmp` sweep, single-flight coalescing, LRU eviction under capacity pressure, active-reader pinning + all-pinned `RESOURCE_EXHAUSTED`, atomic `os.replace`, and corrupted Parquet handling tests. +- `test/unit/test_flight_schemas.py` — Ticket/action Pydantic validation, sanitized gRPC error mapping, and `api/openapi/openapi.yaml` OpenAPI 3.1 validation tests. +- `test/integration/test_flight_server.py` — End-to-end `FlightClient` `DoGet` & `DoAction` (`cache_warm`, `health_check`) against `TabularFlightServer` with mocked S3 streaming, subprocess DuckDB execution, mid-stream timeout/crash error mapping, HTTP ops endpoints (`/healthz`, `/ready` healthy/unhealthy/draining `503`, `/metrics`, `/openapi.yaml`, `/docs`), and `RESOURCE_EXHAUSTED` backpressure. + +### Documentation Sync +- `.ai/doc-api.md` — Keep HTTP ops endpoints, Arrow Flight `DoGet`/`DoAction` schemas, error mappings, Prometheus metrics, and cache conventions synchronized. +- `.ai/progress-tracker.md` — Update Phase 2 status and session notes upon completion. + +--- + +## 3. Verification Plan +1. `ruff format --check .` and `ruff check .` +2. `mypy cmd internal api` +3. `bandit -r cmd internal api -ll` and `pip-audit` +4. OpenAPI 3.1 validation (`openapi-spec-validator`) on `api/openapi/openapi.yaml` +5. Secured logging & DuckDB sandbox audit (no `print()`, no unredacted secrets/presigned URLs in logger calls, no `enable_external_access=true`) +6. `pytest -v test/unit test/security test/integration test/chaos` +7. At least 2 conventional git commits (`feat:`, `tests:`, `docs:`) as Phase 2 is built. + +--- + +## 4. Confirmed Decisions & Proceed Check +1. **S3 Bucket Allowlist & Endpoint Defaults (Confirmed)**: `S3_BUCKET_ALLOWLIST` defaults to an empty list (`[]`, fail-closed unless explicitly configured via `TABULAR_WORKER_S3_BUCKET_ALLOWLIST`). +2. **Flight Authentication Claims (Confirmed)**: For Phase 2, `DoGet`/`DoAction` accept local/VPC gRPC tickets carrying `tenant_id` per `.ai/doc-api.md`, with mTLS/signed-token middleware deferred to Phase 3/4. +3. **HTTP Ops Server Implementation (Confirmed)**: Use Python's built-in `http.server.ThreadingHTTPServer` for `/healthz`, `/ready`, `/metrics`, `/openapi.yaml`, and `/docs` (zero extra runtime dependencies, clean `503` drain support). +4. **Proceed**: Awaiting explicit confirmation to begin implementing Phase 2. diff --git a/prompts/phase-3-subprocess-telemetry-keda-integration.md b/prompts/phase-3-subprocess-telemetry-keda-integration.md new file mode 100644 index 0000000..f0ddb0e --- /dev/null +++ b/prompts/phase-3-subprocess-telemetry-keda-integration.md @@ -0,0 +1,63 @@ +# Implementation Plan: Phase 3 — Subprocess Telemetry & KEDA Concurrency Integration + +## 1. Goal & Scope +Implement **Phase 3: Subprocess Telemetry & KEDA Concurrency Integration** for `rag-tabular-analytics-worker` (per `GEMINI.MD`, `.ai/progress-tracker.md`, `.ai/architecture.md`, `.ai/doc-api.md`, `tasks/Tabular Analytics Worker Implementation Plan.pdf`, and `tasks/Production Grade Backend Concepts.pdf`): + +1. **Subprocess Recycle Telemetry & Worker Memory Guardrails (`internal/pool/`, `internal/flight/handlers.py`)**: + - Carry `RecycleReason` on `WorkerCrashError` (e.g., `reason: RecycleReason`) so `FlightHandlers` increments `tabular_worker_subprocess_recycles_total` with the exact reason (`MEMORY_LIMIT` for exit codes `-9` / `137`, `CRASH` for other unexpected exits, `TIMEOUT` for `5s` circuit-breaker kills) instead of masking `MEMORY_LIMIT` as `CRASH`. + - Track and emit `TASK_LIMIT` when `pebble.ProcessPool` recycles a child worker after `MAX_TASKS_PER_CHILD` (e.g., via per-worker PID/generation tracking in `ProcessManager` / `worker_task.py`). Standardize the canonical label set (`TIMEOUT`, `MEMORY_LIMIT`, `TASK_LIMIT`, `CRASH`) and sync `.ai/doc-api.md`. + - Enforce DuckDB's `SET memory_limit` (`ADR-003`) + `pebble` `MAX_TASKS_PER_CHILD=100` recycling rather than tight per-task `RLIMIT_AS` / `RLIMIT_DATA` (which breaks DuckDB, PyArrow, and `jemalloc` virtual address reservations). If an OS-level process guard is enabled, apply `WORKER_PROCESS_RLIMIT_BYTES` (optional, disabled by default) once at POSIX worker startup via a pool `initializer`. + - Investigate and fix the Windows named-pipe startup race in `test/chaos/test_subprocess_isolation.py` (`test_timeout_circuit_breaker_kills_slow_query_and_keeps_siblings_serviceable`) if caused by cold-spawn warm-up timeout or listener cleanup. + +2. **OpenTelemetry Distributed Tracing & Log Correlation (`internal/telemetry/tracing.py`, `internal/telemetry/logging.py`, `internal/flight/server.py`, `internal/config/settings.py`)**: + - Add `opentelemetry-api`, `opentelemetry-sdk`, and `opentelemetry-exporter-otlp-proto-grpc` (or `http`) to `pyproject.toml`. + - Create `internal/telemetry/tracing.py` providing tracer initialization (enabled/disabled via `TABULAR_WORKER_OTEL_ENABLED`, `TABULAR_WORKER_OTEL_EXPORTER_ENDPOINT`, `TABULAR_WORKER_OTEL_SERVICE_NAME`), W3C `traceparent` context extraction, and span helpers/decorators. + - Wire a PyArrow Flight `fl.ServerMiddlewareFactory` into `TabularFlightServer` (`internal/flight/server.py`) to extract incoming gRPC `traceparent` metadata and propagate context into `FlightHandlers` (`DoGet`/`DoAction`). + - Instrument **Flight ingress (`DoGet`/`DoAction`)**, **AST validation (`sqlglot`)**, **Tier-1 cache / S3 hydration**, and **parent-side subprocess DuckDB execution** (around `ProcessManager.stream_query` + `QueryStream` iteration, avoiding OTel exporters inside `pebble` child processes so `SIGKILL` timeouts never drop spans). + - Ensure active `trace_id` is automatically injected into structured JSON logs (`internal/telemetry/logging.py`) and strictly redact/exclude raw SQL literals, credentials, and presigned S3 URLs from all OTel span attributes and `record_exception` messages. + +3. **KEDA `ScaledObject` & Helm Deployment Blueprint (`deployments/helm/`)**: + - Create `deployments/helm/Chart.yaml`, `deployments/helm/values.yaml`, and templates: + - `deployments/helm/templates/deployment.yaml` — `6.0 GiB` memory limit / `4.0 GiB` request, `4000m` / `2000m` CPU, `10 GiB` disk-backed `emptyDir` (`medium: ""`, `ADR-002`), non-root `securityContext` (`capabilities: drop: ["ALL"]`), configurable `LD_PRELOAD` for `jemalloc` (disabled or arch-aware via `.Values.jemalloc.enabled` / `.Values.jemalloc.path` so ARM64 `c7g.xlarge` and x86_64 `c6i.xlarge` nodes remain compatible until Phase 4's `Dockerfile`), `terminationGracePeriodSeconds` (`>= 20`s, exceeding `QUERY_TIMEOUT_SECONDS=5.0`) for graceful `SIGTERM` stream draining, `/healthz` & `/ready` probes, and Prometheus scrape annotations. + - `deployments/helm/templates/service.yaml` — Exposes named ports for Arrow Flight gRPC (`8815`, `grpc`) and Ops HTTP (`8080`, `http`), with configurable `headless` (`clusterIP: None`) support for HTTP/2 gRPC client-side load balancing from `rag-search-orchestrator`. + - `deployments/helm/templates/scaledobject.yaml` — KEDA `ScaledObject` (`minReplicaCount: 4`, `maxReplicaCount: 24`) with a per-pod Prometheus PromQL trigger (`avg(tabular_worker_active_queries{...})`) and scale-out threshold aligned with `MAX_CONCURRENT_QUERIES` headroom (e.g., `threshold: "3"` when `MAX_CONCURRENT_QUERIES=4`, or raising `MAX_CONCURRENT_QUERIES` above `4`, so pods scale out before shedding `RESOURCE_EXHAUSTED`). + +4. **Tests & Documentation Sync**: + - `test/unit/test_tracing.py` — Span creation, `ServerMiddlewareFactory` `traceparent` extraction, `trace_id` log correlation, SQL/secret redaction on spans, and OTel-disabled no-op behavior. + - `test/unit/test_helm_manifests.py` — Validate Helm YAML structure, KEDA `ScaledObject` bounds (`4..24`, per-pod `avg(...)` PromQL query, `tabular_worker_active_queries`), memory limits (`6Gi`/`4Gi`), `terminationGracePeriodSeconds`, configurable `jemalloc` `LD_PRELOAD`, and disk-backed `emptyDir` (`medium` not `Memory`). + - Expand `test/integration/test_flight_server.py` and `test/chaos/test_subprocess_isolation.py` to verify `tabular_worker_subprocess_recycles_total` labels (`TIMEOUT`, `MEMORY_LIMIT`, `TASK_LIMIT`, `CRASH`), `tabular_worker_active_queries` gauge behavior, and OpenTelemetry spans on `DoGet`. + - Sync `.ai/doc-api.md` (add `CRASH` label to `tabular_worker_subprocess_recycles_total`, document `traceparent` header), `api/openapi/openapi.yaml` (if any config/metric descriptions are updated), and `.ai/progress-tracker.md`. + +--- + +## 2. Files to Create / Modify +- `pyproject.toml` — Add `opentelemetry-api`, `opentelemetry-sdk`, `opentelemetry-exporter-otlp-proto-grpc` (or `http`). +- `internal/config/settings.py` — Add `OTEL_ENABLED`, `OTEL_EXPORTER_ENDPOINT`, `OTEL_SERVICE_NAME`, and optional `WORKER_PROCESS_RLIMIT_BYTES`. +- `internal/telemetry/tracing.py` — New module for OpenTelemetry tracer provider, `traceparent` propagation/middleware helpers, span decorators/context managers, and `trace_id` helper. +- `internal/telemetry/logging.py` — Inject active `trace_id` from current OTel span when not explicitly passed in `extra`. +- `internal/pool/circuit_breaker.py`, `internal/pool/process_manager.py`, & `internal/pool/worker_task.py` — Carry `RecycleReason` on `WorkerCrashError`, detect and emit `TASK_LIMIT` recycles, and optionally apply `WORKER_PROCESS_RLIMIT_BYTES` once at POSIX worker startup if configured. +- `internal/flight/server.py` & `internal/flight/handlers.py` — Wire Flight `ServerMiddlewareFactory` for `traceparent` extraction, instrument `DoGet` and `DoAction` with OpenTelemetry spans, and report exact `RecycleReason` (`MEMORY_LIMIT` vs `CRASH` vs `TIMEOUT`) to `MetricsRegistry`. +- `cmd/worker/main.py` / `internal/flight/daemon.py` — Initialize and gracefully flush/shut down OTel tracer provider. +- `deployments/helm/Chart.yaml`, `deployments/helm/values.yaml`, `deployments/helm/templates/{deployment,service,scaledobject}.yaml`. +- `test/unit/test_tracing.py`, `test/unit/test_helm_manifests.py`, plus updates to `test/integration/test_flight_server.py` and `test/chaos/test_subprocess_isolation.py`. +- `.ai/doc-api.md`, `.ai/progress-tracker.md`. + +--- + +## 3. Verification Plan +1. `ruff format --check .` and `ruff check .` +2. `mypy cmd internal api` +3. `bandit -r cmd internal api -ll` and `pip-audit` +4. OpenAPI 3.1 validation (`openapi-spec-validator`) on `api/openapi/openapi.yaml` +5. Secured logging & DuckDB sandbox audit (no `print()`, no unredacted secrets/presigned URLs/raw SQL in logs or OTel spans, no `enable_external_access=true`) +6. `pytest -v test/` (all unit, security, integration, and chaos suites) +7. `helm lint deployments/helm` and `helm template` YAML validation +8. At least 2 conventional git commits (`feat:`, `tests:`, `docs:`) as Phase 3 is built. + +--- + +## 4. Open Questions for Confirmation +1. **OpenTelemetry Exporter Protocol**: Should `OTEL_EXPORTER_ENDPOINT` use OTLP/gRPC (`opentelemetry-exporter-otlp-proto-grpc`) with `OTEL_ENABLED=false` by default when no collector endpoint is configured? +2. **KEDA Headroom vs. `MAX_CONCURRENT_QUERIES`**: Should we set the KEDA `ScaledObject` threshold to `3` per pod (keeping `MAX_CONCURRENT_QUERIES=4` so pods scale out before returning `RESOURCE_EXHAUSTED`), or keep the KEDA threshold at `4` and raise `MAX_CONCURRENT_QUERIES` (e.g., `6`)? +3. **KEDA Prometheus Server Address**: In `deployments/helm/values.yaml` & `scaledobject.yaml`, is `http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090` (configurable via `.Values.keda.prometheus.serverAddress`) acceptable as the default placeholder? +4. **Proceed**: Please confirm if you would like me to proceed with implementing Phase 3 according to this updated plan. diff --git a/prompts/phase-4-chaos-hardening-fuzzing-canary-rollout.md b/prompts/phase-4-chaos-hardening-fuzzing-canary-rollout.md new file mode 100644 index 0000000..2920aa1 --- /dev/null +++ b/prompts/phase-4-chaos-hardening-fuzzing-canary-rollout.md @@ -0,0 +1,72 @@ +# Implementation Plan: Phase 4 — Chaos Hardening, Fuzzing & Canary Rollout + +## 1. Goal & Scope +Implement **Phase 4: Chaos Hardening, Fuzzing & Canary Rollout** for `rag-tabular-analytics-worker` (per `GEMINI.MD`, `.ai/progress-tracker.md`, `.ai/architecture.md`, `.ai/security.md`, `.ai/doc-api.md`, and `tasks/Tabular Analytics Worker Implementation Plan.pdf`): + +1. **Multi-Stage Production `Dockerfile` with `jemalloc` (`deployments/docker/Dockerfile`, `.dockerignore`, `deployments/helm/values.yaml`)**: + - Build a minimal, non-root multi-stage Python `3.12-slim` image (`builder` stage installing the package and runtime dependencies into `/opt/venv` + `runtime` stage copying `/opt/venv` and app code). + - Install `libjemalloc2` (`--no-install-recommends`) and symlink `/usr/lib/*-linux-gnu/libjemalloc.so.2` to `/usr/local/lib/libjemalloc.so` so `LD_PRELOAD=/usr/local/lib/libjemalloc.so` works identically on both `amd64` (`c6i.xlarge`) and `arm64` (`c7g.xlarge`) nodes. + - Create `/var/cache/tabular-worker` owned by `10001:10001`, run as non-root user `10001:10001` (compatible with `readOnlyRootFilesystem: true` + mounted `/tmp` and `/var/cache/tabular-worker`), expose Flight gRPC (`8815`) and Ops HTTP (`8080`), set `HEALTHCHECK` against `/healthz`, and use JSON `CMD ["python", "-m", "cmd.worker.main"]`. + - Ensure the `Dockerfile` passes `hadolint/hadolint-action@v3.1.0` in `.github/workflows/ci.yml` (e.g., `--no-install-recommends`, pinned or explicitly documented `hadolint ignore=DL3008`, consolidated `RUN` layers). + - Update `deployments/helm/values.yaml` to set `jemalloc.enabled: true` and `jemalloc.path: /usr/local/lib/libjemalloc.so`, and update the corresponding assertion in `test/unit/test_helm_manifests.py`. + +2. **Expanded SQL Injection & AST Bypass Fuzzing (`test/security/test_sql_injection_fuzz.py`)**: + - Add deterministic randomized/combinatorial SQL fuzzing covering: + - Nested CTEs (`WITH ...`), `UNION`/`INTERSECT`/`EXCEPT` branches, and scalar/table subqueries smuggling banned functions (`read_parquet`, `read_csv`, `read_csv_auto`, `scan_parquet`, `glob`, `getenv`, `current_setting`, `duckdb_tables`, `duckdb_settings`, `httpfs_get`, `write_csv`, `copy`). + - Comment obfuscation (`/* ... */`, `-- ...`), whitespace/newline/case variations, stacked statements (`;`), deeply nested expressions, and disallowed table literals/paths. + - Warm up `ASTValidator` once in a fixture before timing, then verify every malicious/malformed payload raises `ValidationError` within `<= 2ms` per query and increments `tabular_worker_ast_rejections_total` when routed through `FlightHandlers`. + - Verify `apply_sandbox_lock` in DuckDB (`enable_external_access = false`, `lock_configuration = true`) directly blocks `read_parquet`, `read_csv`, `PRAGMA`, `SET`, `ATTACH`, `DETACH`, `COPY ... TO`, `INSTALL`, `LOAD`, and `getenv` even if a hypothetical query bypassed the validator. + +3. **500-Stream Concurrent Chaos & Memory-Boundary Soak Tests (`test/chaos/test_soak_and_memory.py`)**: + - **Sustained 500-Query Concurrent Stream Test** (fast ~2–5s default CI run): Execute 500 mixed queries (valid aggregations, AST rejections, and periodic `MAX_TASKS_PER_CHILD` recycles) through a bounded client concurrency pool (`<= MAX_CONCURRENT_QUERIES`) against `FlightServerBase` / `ProcessManager`, asserting: + - `0` host daemon crashes or unhandled exceptions. + - `0` leaked child processes after pool shutdown. + - Memory RSS growth remains bounded across `max_tasks` recycling cycles (measured using stdlib `resource` / `/proc/self/statm` / `tracemalloc` with cross-platform fallback—do **not** require `psutil`, which is not in `pyproject.toml`). + - **Burst Backpressure Test**: Send an over-limit concurrent burst (`> MAX_CONCURRENT_QUERIES`) to `FlightHandlers` / `FlightServerBase` and verify `RESOURCE_EXHAUSTED` (`ConcurrencyLimitError`) is returned cleanly without leaking semaphore permits or leaving `tabular_worker_active_queries > 0`. + - **Extended Endurance & SLO Marker (`@pytest.mark.soak`)**: Gate the 8-hour `jemalloc` continuous soak run and large-dataset `250 MB/sec` / `P95 <= 45ms` stress checks behind `@pytest.mark.soak` (skipped unless explicitly enabled, e.g., via `RUN_SOAK_TESTS=1` or `-m soak`) so `pytest -v test/chaos` remains fast in CI. + +4. **ArgoCD Canary Rollout & Staging/Production Overlays (`deployments/argocd/`, `deployments/helm/`)**: + - Add `deployments/helm/values-staging.yaml` (e.g., `minReplicaCount: 1`, `maxReplicaCount: 6`, `OTEL_ENABLED: "true"`) and `deployments/helm/values-prod.yaml` (`minReplicaCount: 4`, `maxReplicaCount: 24`, `4Gi/6Gi` RAM, `2000m/4000m` CPU, `10Gi` NVMe cache, `jemalloc.enabled: true`). + - Add ArgoCD manifests under `deployments/argocd/`: + - `application-staging.yaml` and `application-prod.yaml` (ArgoCD `Application` resources pointing to `deployments/helm` with their respective values files). + - `rollout-canary.yaml` containing an Argo Rollouts `Rollout` (using `spec.workloadRef` to the Helm `Deployment` so existing `deployments/helm/templates/deployment.yaml` and KEDA `ScaledObject` remain intact and pass `helm lint`) with progressive canary steps (`10% -> 25% -> 50% -> 100%`) and an `AnalysisTemplate` gated only on real metrics defined in `internal/telemetry/metrics.py`: + - `tabular_worker_query_duration_seconds` P99 (`<= 0.12s` warm-cache SLO) + - `tabular_worker_subprocess_recycles_total{reason=~"TIMEOUT|MEMORY_LIMIT|CRASH"}` rate + - `tabular_worker_cache_hit_ratio` (`>= 0.80` per Runbook 1) + - `tabular_worker_active_queries` visibility + +5. **Documentation & Tracker Sync (`.ai/progress-tracker.md`, `.ai/doc-api.md`, `README.md`)**: + - Update `.ai/progress-tracker.md` to mark Phase 4 as `Completed` with session verification notes. + - Verify `.ai/doc-api.md` and `api/openapi/openapi.yaml` remain in sync with any configuration, container, or metric updates. + - Update `README.md` with Docker build/run, fuzzing/soak test (`@pytest.mark.soak`), and ArgoCD canary deployment instructions. + +--- + +## 2. Files to Create / Modify +- `deployments/docker/Dockerfile` (new) & `.dockerignore` (new) +- `deployments/helm/values.yaml` (enable `jemalloc` & set `/usr/local/lib/libjemalloc.so`) +- `deployments/helm/values-staging.yaml` & `deployments/helm/values-prod.yaml` (new) +- `deployments/argocd/application-staging.yaml`, `deployments/argocd/application-prod.yaml`, & `deployments/argocd/rollout-canary.yaml` (new) +- `test/security/test_sql_injection_fuzz.py` (expand AST bypass & DuckDB sandbox fuzzing with warm `<= 2ms` budget) +- `test/chaos/test_soak_and_memory.py` (new — 500-stream bounded concurrency soak, burst backpressure, & `@pytest.mark.soak` endurance suite) +- `test/unit/test_helm_manifests.py` (update `jemalloc.enabled` assertion; validate `Dockerfile`, staging/prod Helm values, and ArgoCD manifests) +- `README.md`, `.ai/progress-tracker.md` (and `.ai/doc-api.md` / `api/openapi/openapi.yaml` if any contracts change) + +--- + +## 3. Verification Plan +1. `ruff format --check .` and `ruff check .` +2. `mypy cmd internal api` +3. `bandit -r cmd internal api -ll` and `pip-audit` +4. OpenAPI 3.1 validation (`openapi-spec-validator`) on `api/openapi/openapi.yaml` +5. Secured logging & DuckDB sandbox audit (no `print()`, no unredacted secrets/presigned URLs, no `enable_external_access=true`) +6. `pytest -v test/` (unit, security/fuzz, integration, and chaos/500-stream soak tests) +7. `helm lint` / `helm template` (with default, `values-staging.yaml`, and `values-prod.yaml`) and `Dockerfile` lint checks (`hadolint` where available, plus unit tests covering both) +8. At least 2 conventional git commits (`feat:`, `tests:`, `docs:`) as Phase 4 is built. + +--- + +## 4. Confirmed Architectural Decisions +1. **Canary Rollout Format**: Argo Rollouts `Rollout` (with `spec.workloadRef` referencing the Helm `Deployment`) + `AnalysisTemplate` (progressive steps `10% -> 25% -> 50% -> 100%`) alongside ArgoCD `Application` manifests (`application-staging.yaml`, `application-prod.yaml`). +2. **`jemalloc` Path**: `Dockerfile` symlinks `/usr/lib/*-linux-gnu/libjemalloc.so.2` to `/usr/local/lib/libjemalloc.so` for both `amd64` (`c6i.xlarge`) and `arm64` (`c7g.xlarge`), with `jemalloc.enabled: true` in `deployments/helm/values.yaml`. +3. **500-Stream Soak Test in CI**: Fast (~2–5s) 500-query bounded stream + burst backpressure test runs by default in `test/chaos/`, while the 8-hour `jemalloc` endurance run is gated behind `@pytest.mark.soak`. diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..5ab62ba --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,101 @@ +[build-system] +requires = ["setuptools>=75.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "rag-tabular-analytics-worker" +version = "0.1.0" +description = "Stateless, horizontally autoscaling Apache Arrow Flight microservice executing sandboxed DuckDB analytical queries over Parquet artifacts." +readme = "README.md" +requires-python = ">=3.12" +license = { text = "MIT" } +dependencies = [ + "aiobotocore>=2.15.0", + "duckdb>=1.2.0", + "opentelemetry-api>=1.30.0", + "opentelemetry-exporter-otlp-proto-grpc>=1.30.0", + "opentelemetry-sdk>=1.30.0", + "pebble>=5.1.0", + "prometheus-client>=0.21.0", + "pyarrow>=18.0.0", + "pydantic>=2.10.0", + "pydantic-settings>=2.7.0", + "pyyaml>=6.0.2", + "sqlglot>=26.0.0", +] + +[project.optional-dependencies] +dev = [ + "bandit>=1.8.0", + "mypy>=1.15.0", + "openapi-spec-validator>=0.7.1", + "pip-audit>=2.8.0", + "ruff>=0.11.0", +] +test = [ + "openapi-spec-validator>=0.7.1", + "pytest>=8.3.0", + "pytest-asyncio>=0.25.0", + "pytest-cov>=6.0.0", +] + +[tool.setuptools.packages.find] +include = ["cmd*", "internal*", "api*"] + +[tool.ruff] +target-version = "py312" +line-length = 100 + +[tool.ruff.lint] +select = [ + "E", + "W", + "F", + "I", + "B", + "UP", + "SIM", + "S", +] +ignore = [ + "S101", # Allow assert in tests (handled per-file below too) +] + +[tool.ruff.lint.per-file-ignores] +"test/**/*.py" = ["S101", "S105", "S106", "S108", "S608"] + +[tool.mypy] +python_version = "3.12" +strict = true +warn_return_any = true +warn_unused_configs = true +disallow_untyped_defs = true +disallow_incomplete_defs = true +check_untyped_defs = true +no_implicit_optional = true + +[[tool.mypy.overrides]] +module = [ + "aiobotocore", + "aiobotocore.*", + "botocore", + "botocore.*", + "duckdb", + "duckdb.*", + "pebble", + "pebble.*", + "pyarrow", + "pyarrow.*", + "prometheus_client", + "prometheus_client.*", +] +ignore_missing_imports = true + +[tool.pytest.ini_options] +testpaths = ["test"] +pythonpath = ["."] +asyncio_mode = "auto" +addopts = "-ra" +markers = [ + "soak: Extended endurance and SLO soak tests (opt-in via RUN_SOAK_TESTS=1 or -m soak)", +] diff --git a/scripts/sync_env_to_deployments.py b/scripts/sync_env_to_deployments.py new file mode 100644 index 0000000..8031891 --- /dev/null +++ b/scripts/sync_env_to_deployments.py @@ -0,0 +1,293 @@ +"""Synchronize `.env.prod` into `deployments/helm/values.yaml` and detect drift.""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path +from typing import Any + +import yaml + +SENSITIVE_KEY_MARKERS: tuple[str, ...] = ( + "PASSWORD", + "SECRET", + "TOKEN", + "KEY", + "OTEL_EXPORTER_OTLP_HEADERS", +) +TOP_LEVEL_KEY_RE = re.compile(r"^[A-Za-z0-9_-]+:") + + +def configure_utf8_streams() -> None: + """Ensure stdout/stderr emit UTF-8 cleanly on Windows consoles.""" + if hasattr(sys.stdout, "reconfigure"): + sys.stdout.reconfigure(encoding="utf-8", errors="replace") + if hasattr(sys.stderr, "reconfigure"): + sys.stderr.reconfigure(encoding="utf-8", errors="replace") + + +def is_sensitive_key(key: str) -> bool: + """Return True if *key* looks like a credential that must stay out of values.yaml.""" + if key == "secretName": + return False + upper = key.upper() + return any(marker in upper for marker in SENSITIVE_KEY_MARKERS) + + +def _unquote_env_value(raw: str) -> str: + value = raw.strip() + if not value: + return "" + if value[0] in ('"', "'"): + quote = value[0] + escaped = False + for idx in range(1, len(value)): + ch = value[idx] + if quote == '"' and escaped: + escaped = False + continue + if quote == '"' and ch == "\\": + escaped = True + continue + if ch == quote: + trailing = value[idx + 1 :].strip() + if not trailing or trailing.startswith("#"): + inner = value[1:idx] + if quote == '"': + inner = inner.replace('\\"', '"') + return inner + break + return value + if value.startswith("#"): + return "" + return re.split(r"\s+#", value, maxsplit=1)[0].rstrip() + + +def parse_env_file(env_path: Path) -> tuple[dict[str, str], list[str]]: + """Parse a dotenv file, returning (non_sensitive_env, excluded_sensitive_keys).""" + if not env_path.is_file(): + raise FileNotFoundError(f"Environment file not found: {env_path}") + + env_vars: dict[str, str] = {} + excluded: list[str] = [] + + for raw_line in env_path.read_text(encoding="utf-8").splitlines(): + line = raw_line.strip() + if not line or line.startswith("#"): + continue + if line.startswith("export "): + line = line[len("export ") :].lstrip() + if "=" not in line: + continue + + key, _, raw_value = line.partition("=") + key = key.strip() + if not key: + continue + + value = _unquote_env_value(raw_value) + if is_sensitive_key(key): + excluded.append(key) + continue + + env_vars[key] = value + + return env_vars, excluded + + +def _normalize_yaml_env_value(value: Any) -> str: + if value is None: + return "" + if isinstance(value, bool): + return "true" if value else "false" + return str(value) + + +def load_helm_env(values_path: Path) -> tuple[dict[str, str], Any, bool]: + """Load `env` from Helm values.yaml. + + Returns (env_without_secret_name, secret_name_value, has_secret_name). + """ + if not values_path.is_file(): + raise FileNotFoundError(f"Helm values file not found: {values_path}") + + data = yaml.safe_load(values_path.read_text(encoding="utf-8")) or {} + if not isinstance(data, dict): + raise ValueError(f"Expected mapping at root of {values_path}") + + raw_env = data.get("env") or {} + if not isinstance(raw_env, dict): + raise ValueError(f"Expected mapping for 'env' in {values_path}") + + has_secret_name = "secretName" in raw_env + secret_name_val = raw_env.get("secretName") + + env_map: dict[str, str] = {} + for key, val in raw_env.items(): + if key == "secretName": + continue + env_map[str(key)] = _normalize_yaml_env_value(val) + + return env_map, secret_name_val, has_secret_name + + +def compute_drift( + expected: dict[str, str], + actual: dict[str, str], +) -> tuple[list[str], list[str], list[str], list[str]]: + """Return (missing, extra, changed, forbidden_sensitive) key lists.""" + forbidden_sensitive = sorted(k for k in actual if is_sensitive_key(k)) + missing = sorted(k for k in expected if k not in actual) + extra = sorted(k for k in actual if k not in expected and not is_sensitive_key(k)) + changed = sorted(k for k in expected if k in actual and actual[k] != expected[k]) + return missing, extra, changed, forbidden_sensitive + + +def _render_env_block( + env_vars: dict[str, str], + secret_name_val: Any, + has_secret_name: bool, +) -> list[str]: + lines = ["env:"] + if has_secret_name: + if secret_name_val is None: + lines.append(" secretName: null") + else: + lines.append(f" secretName: {json.dumps(str(secret_name_val))}") + for key, value in env_vars.items(): + lines.append(f" {key}: {json.dumps(value)}") + return lines + + +def sync_values_file( + env_vars: dict[str, str], + values_path: Path, + secret_name_val: Any, + has_secret_name: bool, +) -> None: + """Replace only the top-level `env:` block in values.yaml, preserving all other blocks.""" + original_text = values_path.read_text(encoding="utf-8") + newline = "\r\n" if "\r\n" in original_text else "\n" + trailing_newline = original_text.endswith(("\n", "\r\n")) + lines = original_text.splitlines() + + new_env_lines = _render_env_block(env_vars, secret_name_val, has_secret_name) + + start_idx: int | None = None + end_idx: int | None = None + for idx, line in enumerate(lines): + if start_idx is None: + if re.match(r"^env:\s*(#.*)?$", line): + start_idx = idx + elif TOP_LEVEL_KEY_RE.match(line): + end_idx = idx + break + + if start_idx is None: + if lines and lines[-1].strip(): + lines.append("") + updated_lines = lines + new_env_lines + else: + if end_idx is None: + end_idx = len(lines) + # Preserve a single blank line before the next top-level block if one existed. + had_separator = ( + end_idx < len(lines) and end_idx > start_idx + 1 and not lines[end_idx - 1].strip() + ) + if had_separator: + new_env_lines.append("") + updated_lines = lines[:start_idx] + new_env_lines + lines[end_idx:] + + output = newline.join(updated_lines) + if trailing_newline: + output += newline + values_path.write_text(output, encoding="utf-8") + + +def build_parser() -> argparse.ArgumentParser: + repo_root = Path(__file__).resolve().parents[1] + parser = argparse.ArgumentParser( + description="Sync .env.prod into deployments/helm/values.yaml or check for drift." + ) + mode = parser.add_mutually_exclusive_group(required=True) + mode.add_argument( + "--sync", + action="store_true", + help="Update deployments/helm/values.yaml from .env.prod.", + ) + mode.add_argument( + "--check-only", + action="store_true", + help="Verify zero drift between .env.prod and deployments/helm/values.yaml.", + ) + parser.add_argument( + "--env-file", + type=Path, + default=repo_root / ".env.prod", + help="Path to source .env.prod file.", + ) + parser.add_argument( + "--values-file", + type=Path, + default=repo_root / "deployments" / "helm" / "values.yaml", + help="Path to target Helm values.yaml file.", + ) + return parser + + +def main(argv: list[str] | None = None) -> int: + configure_utf8_streams() + args = build_parser().parse_args(argv) + + try: + expected_env, excluded_keys = parse_env_file(args.env_file) + actual_env, secret_name_val, has_secret_name = load_helm_env(args.values_file) + except (FileNotFoundError, ValueError, yaml.YAMLError) as exc: + print(f"Error: {exc}", file=sys.stderr) + return 1 + + if excluded_keys: + print(f"Excluded sensitive keys from sync: {', '.join(excluded_keys)}") + + missing, extra, changed, forbidden = compute_drift(expected_env, actual_env) + + if args.check_only: + if not (missing or extra or changed or forbidden): + print("OK: Zero drift between .env.prod and deployments/helm/values.yaml.") + return 0 + + print("DRIFT DETECTED between .env.prod and deployments/helm/values.yaml:", file=sys.stderr) + if missing: + print(f" Missing in values.yaml: {', '.join(missing)}", file=sys.stderr) + if extra: + print(f" Extra in values.yaml: {', '.join(extra)}", file=sys.stderr) + if changed: + print(f" Changed values: {', '.join(changed)}", file=sys.stderr) + if forbidden: + print( + f" Forbidden sensitive keys in values.yaml: {', '.join(forbidden)}", + file=sys.stderr, + ) + print( + "Run `python scripts/sync_env_to_deployments.py --sync` to update values.yaml.", + file=sys.stderr, + ) + return 1 + + if not (missing or extra or changed or forbidden): + print("Already in sync: deployments/helm/values.yaml matches .env.prod.") + return 0 + + sync_values_file(expected_env, args.values_file, secret_name_val, has_secret_name) + print( + f"Synced {len(expected_env)} environment variables from " + f"{args.env_file} to {args.values_file}." + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test/__init__.py b/test/__init__.py new file mode 100644 index 0000000..ea4ba6c --- /dev/null +++ b/test/__init__.py @@ -0,0 +1 @@ +"""Test suite package for rag-tabular-analytics-worker.""" diff --git a/test/chaos/test_soak_and_memory.py b/test/chaos/test_soak_and_memory.py new file mode 100644 index 0000000..a02affc --- /dev/null +++ b/test/chaos/test_soak_and_memory.py @@ -0,0 +1,377 @@ +"""Chaos soak, bounded 500-stream concurrency, and backpressure tests.""" + +from __future__ import annotations + +import hashlib +import json +import multiprocessing as mp +import os +import sys +import threading +import time +import tracemalloc +from concurrent.futures import ThreadPoolExecutor, as_completed +from pathlib import Path +from typing import Any + +import pyarrow.flight as fl +import pytest + +from internal.config.settings import Settings +from internal.flight.daemon import WorkerDaemon +from internal.pool.circuit_breaker import RecycleReason +from test.fixtures.sample_data import write_sample_parquet + + +class _FakeBody: + def __init__(self, data: bytes) -> None: + self._data = data + + async def iter_chunks(self, chunk_size: int = 8192) -> Any: + for idx in range(0, len(self._data), chunk_size): + yield self._data[idx : idx + chunk_size] + + def close(self) -> None: + return None + + +class _FakeS3Client: + def __init__(self, objects: dict[tuple[str, str], bytes]) -> None: + self._objects = objects + + async def get_object(self, *, Bucket: str, Key: str) -> dict[str, Any]: + return {"Body": _FakeBody(self._objects[(Bucket, Key)])} + + +def _get_rss_bytes() -> int: + """Return current process RSS in bytes using stdlib-only cross-platform fallbacks.""" + statm = Path("/proc/self/statm") + if statm.exists(): + pages = int(statm.read_text(encoding="utf-8").split()[1]) + return pages * os.sysconf("SC_PAGE_SIZE") + + if sys.platform != "win32": + import resource # pragma: no cover + + usage = resource.getrusage(resource.RUSAGE_SELF) + scale = 1 if sys.platform == "darwin" else 1024 + return int(usage.ru_maxrss) * scale + + current, _peak = tracemalloc.get_traced_memory() + return int(current) + + +def _make_ticket( + *, + request_id: str, + tenant_id: str, + sql: str, + table_hash: str, + bucket: str, + s3_key: str, +) -> fl.Ticket: + payload = json.dumps( + { + "request_id": request_id, + "tenant_id": tenant_id, + "sql": sql, + "table_alias": "tbl", + "table_hash": table_hash, + "s3_bucket": bucket, + "s3_key": s3_key, + } + ).encode("utf-8") + return fl.Ticket(payload) + + +def test_500_stream_bounded_concurrency_soak_and_memory_stability(tmp_path: Path) -> None: + """Execute 500 mixed streams across worker recycles with 0 leaks or daemon crashes.""" + sample_path = write_sample_parquet(tmp_path / "soak.parquet", num_rows=200) + parquet_bytes = sample_path.read_bytes() + table_hash = hashlib.sha256(parquet_bytes).hexdigest() + bucket = "rag-tabular-artifacts" + tenant_id = "tenant-soak-01" + s3_key = f"artifacts/{tenant_id}/{table_hash}.parquet" + + fake_s3 = _FakeS3Client({(bucket, s3_key): parquet_bytes}) + settings = Settings( + FLIGHT_HOST="127.0.0.1", + FLIGHT_PORT=0, + HTTP_HOST="127.0.0.1", + HTTP_PORT=0, + CACHE_DIR=tmp_path / "nvme_cache", + CACHE_MAX_BYTES=64 * 1024 * 1024, + S3_BUCKET_ALLOWLIST=[bucket], + POOL_SIZE=4, + MAX_TASKS_PER_CHILD=50, + MAX_CONCURRENT_QUERIES=4, + QUERY_TIMEOUT_SECONDS=15.0, + ) + + tracemalloc.start() + daemon = WorkerDaemon(settings=settings, s3_client=fake_s3) + daemon.start() + try: + assert daemon.flight_server is not None + port = daemon.flight_server.port + + # Warm the cache once so the 500-stream soak exercises warm-cache Flight + DuckDB. + warm_client = fl.FlightClient(f"grpc://127.0.0.1:{port}") + warm_ticket = _make_ticket( + request_id="req-warm", + tenant_id=tenant_id, + sql="SELECT COUNT(*) AS cnt FROM tbl", + table_hash=table_hash, + bucket=bucket, + s3_key=s3_key, + ) + assert warm_client.do_get(warm_ticket).read_all().column("cnt")[0].as_py() == 200 + baseline_rss = _get_rss_bytes() + + thread_local = threading.local() + + def _run_one(idx: int) -> str: + client: fl.FlightClient | None = getattr(thread_local, "client", None) + if client is None: + client = fl.FlightClient(f"grpc://127.0.0.1:{port}") + thread_local.client = client + + # Mix valid analytical aggregations (90%) with AST-rejected SQL (10%). + if idx % 10 == 9: + bad_ticket = _make_ticket( + request_id=f"req-soak-{idx}", + tenant_id=tenant_id, + sql="SELECT * FROM read_parquet('/etc/passwd')", + table_hash=table_hash, + bucket=bucket, + s3_key=s3_key, + ) + with pytest.raises(fl.FlightError): + client.do_get(bad_ticket).read_all() + return "rejected" + + sql = ( + "SELECT region, COUNT(*) AS cnt, SUM(revenue) AS rev FROM tbl GROUP BY region" + if idx % 2 == 0 + else "SELECT COUNT(*) AS cnt FROM tbl" + ) + ticket = _make_ticket( + request_id=f"req-soak-{idx}", + tenant_id=tenant_id, + sql=sql, + table_hash=table_hash, + bucket=bucket, + s3_key=s3_key, + ) + table = client.do_get(ticket).read_all() + assert table.num_rows >= 1 + return "ok" + + total_streams = 500 + results: list[str] = [] + with ThreadPoolExecutor(max_workers=settings.MAX_CONCURRENT_QUERIES) as executor: + futures = [executor.submit(_run_one, i) for i in range(total_streams)] + for fut in as_completed(futures): + results.append(fut.result()) + + assert results.count("ok") == 450 + assert results.count("rejected") == 50 + + # Verify task-limit recycling occurred across 450 valid queries with MAX_TASKS_PER_CHILD=50. + assert daemon.process_manager is not None + recycles = daemon.process_manager.circuit_breaker.get_recycle_counts() + assert recycles[RecycleReason.TASK_LIMIT] >= 1 + + # Verify active queries returned to 0 and RSS growth stayed bounded. + active_val = daemon.metrics.active_queries._value.get() # noqa: SLF001 + assert active_val == 0.0 + final_rss = _get_rss_bytes() + assert (final_rss - baseline_rss) < 128 * 1024 * 1024 + finally: + daemon.shutdown() + tracemalloc.stop() + + # Ensure zero child processes leaked after daemon shutdown. + deadline = time.monotonic() + 5.0 + while time.monotonic() < deadline: + alive = [p for p in mp.active_children() if p.is_alive()] + if not alive: + break + time.sleep(0.05) + assert not [p for p in mp.active_children() if p.is_alive()] + + +def test_burst_backpressure_returns_resource_exhausted_without_permit_leak( + tmp_path: Path, +) -> None: + """Over-limit concurrent burst returns RESOURCE_EXHAUSTED and leaks 0 permits.""" + sample_path = write_sample_parquet(tmp_path / "burst.parquet", num_rows=500) + parquet_bytes = sample_path.read_bytes() + table_hash = hashlib.sha256(parquet_bytes).hexdigest() + bucket = "rag-tabular-artifacts" + tenant_id = "tenant-burst-01" + s3_key = f"artifacts/{tenant_id}/{table_hash}.parquet" + + fake_s3 = _FakeS3Client({(bucket, s3_key): parquet_bytes}) + settings = Settings( + FLIGHT_HOST="127.0.0.1", + FLIGHT_PORT=0, + HTTP_HOST="127.0.0.1", + HTTP_PORT=0, + CACHE_DIR=tmp_path / "nvme_cache", + CACHE_MAX_BYTES=64 * 1024 * 1024, + S3_BUCKET_ALLOWLIST=[bucket], + POOL_SIZE=1, + MAX_CONCURRENT_QUERIES=1, + QUERY_TIMEOUT_SECONDS=15.0, + ) + daemon = WorkerDaemon(settings=settings, s3_client=fake_s3) + daemon.start() + try: + assert daemon.flight_server is not None + port = daemon.flight_server.port + + # Warm cache first. + warm_client = fl.FlightClient(f"grpc://127.0.0.1:{port}") + warm_client.do_get( + _make_ticket( + request_id="req-burst-warm", + tenant_id=tenant_id, + sql="SELECT COUNT(*) AS cnt FROM tbl", + table_hash=table_hash, + bucket=bucket, + s3_key=s3_key, + ) + ).read_all() + + # Hold the single concurrency permit on the server until the burst finishes. + permit_acquired = threading.Event() + release_holder = threading.Event() + holder_done = threading.Event() + orig_stream_query = daemon.process_manager.stream_query + + def _synchronized_stream_query(sql: str, parquet_path: Path, table_alias: str): # type: ignore[no-untyped-def] + if "CROSS JOIN" in sql: + permit_acquired.set() + release_holder.wait(timeout=10.0) + return orig_stream_query(sql, parquet_path, table_alias) + + daemon.process_manager.stream_query = _synchronized_stream_query # type: ignore[method-assign] + + def _hold_slot() -> None: + client = fl.FlightClient(f"grpc://127.0.0.1:{port}") + client.do_get( + _make_ticket( + request_id="req-burst-holder", + tenant_id=tenant_id, + sql="SELECT COUNT(*) AS cnt FROM tbl a CROSS JOIN tbl b", + table_hash=table_hash, + bucket=bucket, + s3_key=s3_key, + ) + ).read_all() + holder_done.set() + + holder = threading.Thread(target=_hold_slot, daemon=True) + burst_client = fl.FlightClient(f"grpc://127.0.0.1:{port}") + rejected_exhausted = 0 + try: + holder.start() + assert permit_acquired.wait(timeout=5.0) + assert daemon.metrics.active_queries._value.get() == 1.0 # noqa: SLF001 + + for i in range(5): + try: + burst_client.do_get( + _make_ticket( + request_id=f"req-burst-{i}", + tenant_id=tenant_id, + sql="SELECT COUNT(*) AS cnt FROM tbl", + table_hash=table_hash, + bucket=bucket, + s3_key=s3_key, + ) + ).read_all() + except fl.FlightError as exc: + if "MAX_CONCURRENT_QUERIES" in str(exc) or "RESOURCE_EXHAUSTED" in str(exc): + rejected_exhausted += 1 + + assert rejected_exhausted >= 1 + finally: + release_holder.set() + holder.join(timeout=10.0) + daemon.process_manager.stream_query = orig_stream_query # type: ignore[method-assign] + + assert holder_done.is_set() + + # Verify permit was cleanly released and subsequent query succeeds. + assert daemon.metrics.active_queries._value.get() == 0.0 # noqa: SLF001 + follow_up = burst_client.do_get( + _make_ticket( + request_id="req-burst-followup", + tenant_id=tenant_id, + sql="SELECT COUNT(*) AS cnt FROM tbl", + table_hash=table_hash, + bucket=bucket, + s3_key=s3_key, + ) + ).read_all() + assert follow_up.column("cnt")[0].as_py() == 500 + assert daemon.metrics.active_queries._value.get() == 0.0 # noqa: SLF001 + finally: + daemon.shutdown() + + +@pytest.mark.soak +@pytest.mark.skipif( + os.environ.get("RUN_SOAK_TESTS") != "1", + reason="Extended 8-hour jemalloc/SLO endurance test; set RUN_SOAK_TESTS=1 to execute.", +) +def test_extended_8h_jemalloc_endurance_and_slo(tmp_path: Path) -> None: # pragma: no cover + """8-hour continuous jemalloc endurance and P95 <= 45ms warm-cache SLO check.""" + duration_seconds = float(os.environ.get("SOAK_DURATION_SECONDS", "28800")) + sample_path = write_sample_parquet(tmp_path / "endurance.parquet", num_rows=50_000) + parquet_bytes = sample_path.read_bytes() + table_hash = hashlib.sha256(parquet_bytes).hexdigest() + bucket = "rag-tabular-artifacts" + tenant_id = "tenant-endurance" + s3_key = f"artifacts/{tenant_id}/{table_hash}.parquet" + + fake_s3 = _FakeS3Client({(bucket, s3_key): parquet_bytes}) + settings = Settings( + FLIGHT_HOST="127.0.0.1", + FLIGHT_PORT=0, + HTTP_HOST="127.0.0.1", + HTTP_PORT=0, + CACHE_DIR=tmp_path / "nvme_cache", + S3_BUCKET_ALLOWLIST=[bucket], + POOL_SIZE=4, + MAX_TASKS_PER_CHILD=100, + MAX_CONCURRENT_QUERIES=4, + ) + daemon = WorkerDaemon(settings=settings, s3_client=fake_s3) + daemon.start() + try: + assert daemon.flight_server is not None + client = fl.FlightClient(f"grpc://127.0.0.1:{daemon.flight_server.port}") + ticket = _make_ticket( + request_id="req-endurance", + tenant_id=tenant_id, + sql="SELECT region, SUM(revenue) AS rev FROM tbl GROUP BY region", + table_hash=table_hash, + bucket=bucket, + s3_key=s3_key, + ) + latencies_ms: list[float] = [] + deadline = time.monotonic() + duration_seconds + while time.monotonic() < deadline: + t0 = time.perf_counter() + table = client.do_get(ticket).read_all() + assert table.num_rows == 4 + latencies_ms.append((time.perf_counter() - t0) * 1000.0) + + latencies_ms.sort() + p95_idx = int(0.95 * (len(latencies_ms) - 1)) + assert latencies_ms[p95_idx] <= 45.0 + finally: + daemon.shutdown() diff --git a/test/chaos/test_subprocess_isolation.py b/test/chaos/test_subprocess_isolation.py new file mode 100644 index 0000000..9dd5758 --- /dev/null +++ b/test/chaos/test_subprocess_isolation.py @@ -0,0 +1,170 @@ +"""Chaos tests for subprocess isolation, 5s timeout circuit breaking, and max_tasks recycling.""" + +from __future__ import annotations + +import multiprocessing as mp +import os +import signal +import threading +import time +from pathlib import Path + +import pytest + +from internal.config.settings import Settings +from internal.pool import ProcessManager, QueryTimeoutError, RecycleReason, WorkerCrashError +from test.fixtures.sample_data import write_sample_parquet + + +@pytest.fixture +def chaos_parquet(tmp_path: Path) -> Path: + return write_sample_parquet(tmp_path / "chaos.parquet", num_rows=1000) + + +def test_timeout_circuit_breaker_kills_slow_query_and_keeps_siblings_serviceable( + chaos_parquet: Path, +) -> None: + # Exercise the 5.0s production circuit breaker with real cross-process termination + # (SIGKILL on POSIX / TerminateProcess on Windows). + settings = Settings(POOL_SIZE=1, MAX_TASKS_PER_CHILD=100, QUERY_TIMEOUT_SECONDS=5.0) + with ProcessManager(settings) as manager: + # Warm the single worker so we can identify its PID before the runaway query. + list(manager.stream_query("SELECT COUNT(*) AS cnt FROM tbl", chaos_parquet, "tbl")) + initial_pids = {proc.pid for proc in mp.active_children() if proc.pid is not None} + assert len(initial_pids) == 1 + timed_out_pid = next(iter(initial_pids)) + + # 1000^4 = 1,000,000,000,000 row cross-join will exceed 3.0s in DuckDB. + runaway_sql = ( + "SELECT COUNT(*) AS cnt FROM tbl a CROSS JOIN tbl b CROSS JOIN tbl c CROSS JOIN tbl d" + ) + with pytest.raises(QueryTimeoutError): + list(manager.stream_query(runaway_sql, chaos_parquet, "tbl")) + + counts = manager.circuit_breaker.get_recycle_counts() + assert counts[RecycleReason.TIMEOUT] >= 1 + + # Bounded check that the timed-out child process has exited before pool teardown. + deadline = time.monotonic() + 5.0 + while time.monotonic() < deadline: + live_pids = { + proc.pid + for proc in mp.active_children() + if proc.pid is not None and proc.is_alive() + } + if timed_out_pid not in live_pids: + break + time.sleep(0.05) + else: + pytest.fail(f"Timed-out worker PID {timed_out_pid} did not exit within 5.0s") + + # Sibling workers and replacement worker must remain healthy and serviceable. + assert manager.is_healthy() + for _ in range(4): + batches = list( + manager.stream_query( + "SELECT region, COUNT(*) AS cnt FROM tbl GROUP BY region", + chaos_parquet, + "tbl", + ) + ) + assert sum(b.num_rows for b in batches) == 4 + + +def test_max_tasks_recycling_executes_cleanly_across_generations(chaos_parquet: Path) -> None: + settings = Settings(POOL_SIZE=2, MAX_TASKS_PER_CHILD=2, QUERY_TIMEOUT_SECONDS=15.0) + with ProcessManager(settings) as manager: + # Startup warm-up before capturing recycle-count baseline and worker PIDs. + warmup = list( + manager.stream_query("SELECT COUNT(*) AS total_rows FROM tbl", chaos_parquet, "tbl") + ) + assert warmup[0].column("total_rows")[0].as_py() == 1000 + baseline_recycles = manager.circuit_breaker.get_recycle_counts()[RecycleReason.TASK_LIMIT] + initial_pids = { + proc.pid for proc in mp.active_children() if proc.pid is not None and proc.is_alive() + } + observed_pids = set(initial_pids) + + for _ in range(6): + batches = list( + manager.stream_query( + "SELECT COUNT(*) AS total_rows FROM tbl", + chaos_parquet, + "tbl", + ) + ) + assert len(batches) == 1 + assert batches[0].column("total_rows")[0].as_py() == 1000 + observed_pids.update( + proc.pid + for proc in mp.active_children() + if proc.pid is not None and proc.is_alive() + ) + counts = manager.circuit_breaker.get_recycle_counts() + assert counts[RecycleReason.TASK_LIMIT] > baseline_recycles + assert observed_pids - initial_pids + + +def test_worker_crash_or_forced_termination_does_not_break_pool(chaos_parquet: Path) -> None: + settings = Settings(POOL_SIZE=1, MAX_TASKS_PER_CHILD=100, QUERY_TIMEOUT_SECONDS=15.0) + with ProcessManager(settings) as manager: + parent_pid = os.getpid() + assert manager.is_healthy() + batches = list(manager.stream_query("SELECT COUNT(*) AS c FROM tbl", chaos_parquet, "tbl")) + assert batches[0].column("c")[0].as_py() == 1000 + + worker_pids = [ + proc.pid for proc in mp.active_children() if proc.pid is not None and proc.is_alive() + ] + assert len(worker_pids) == 1 + target_pid = worker_pids[0] + + query_scheduled = threading.Event() + scheduled_futures: list[object] = [] + assert manager._pool is not None # noqa: SLF001 + orig_schedule = manager._pool.schedule # noqa: SLF001 + + def _signaling_schedule(*args: object, **kwargs: object) -> object: + future = orig_schedule(*args, **kwargs) + scheduled_futures.append(future) + query_scheduled.set() + return future + + manager._pool.schedule = _signaling_schedule # type: ignore[method-assign] # noqa: SLF001 + + def _terminate_worker_soon() -> None: + if not query_scheduled.wait(timeout=5.0): + return + future = scheduled_futures[0] + deadline = time.monotonic() + 5.0 + while ( + not future.running() # type: ignore[attr-defined] + and not future.done() # type: ignore[attr-defined] + and time.monotonic() < deadline + ): + time.sleep(0.01) + if future.running(): # type: ignore[attr-defined] + os.kill(target_pid, signal.SIGTERM) + + killer = threading.Thread(target=_terminate_worker_soon, daemon=True) + killer.start() + try: + with pytest.raises(WorkerCrashError): + list( + manager.stream_query( + "SELECT COUNT(*) AS cnt FROM tbl a CROSS JOIN tbl b CROSS JOIN tbl c", + chaos_parquet, + "tbl", + ) + ) + finally: + manager._pool.schedule = orig_schedule # type: ignore[method-assign] # noqa: SLF001 + killer.join(timeout=2.0) + + counts = manager.circuit_breaker.get_recycle_counts() + assert counts[RecycleReason.CRASH] + counts[RecycleReason.MEMORY_LIMIT] >= 1 + + # Subsequent query must succeed on a replacement worker without affecting parent. + batches = list(manager.stream_query("SELECT COUNT(*) AS c FROM tbl", chaos_parquet, "tbl")) + assert batches[0].column("c")[0].as_py() == 1000 + assert os.getpid() == parent_pid diff --git a/test/fixtures/__init__.py b/test/fixtures/__init__.py new file mode 100644 index 0000000..d1b3395 --- /dev/null +++ b/test/fixtures/__init__.py @@ -0,0 +1 @@ +"""Test fixtures package.""" diff --git a/test/fixtures/sample_data.py b/test/fixtures/sample_data.py new file mode 100644 index 0000000..b738260 --- /dev/null +++ b/test/fixtures/sample_data.py @@ -0,0 +1,26 @@ +"""Test fixtures for rag-tabular-analytics-worker.""" + +from __future__ import annotations + +from pathlib import Path + +import pyarrow as pa +import pyarrow.parquet as pq + + +def write_sample_parquet(path: Path, num_rows: int = 1000) -> Path: + """Write a deterministic normalized Parquet dataset for unit, security, and chaos tests.""" + regions = ["north", "south", "east", "west"] + categories = ["electronics", "apparel", "grocery", "home"] + + table = pa.table( + { + "id": pa.array(range(1, num_rows + 1), type=pa.int64()), + "region": pa.array([regions[i % len(regions)] for i in range(num_rows)]), + "category": pa.array([categories[i % len(categories)] for i in range(num_rows)]), + "revenue": pa.array([float((i % 100) + 1) * 10.5 for i in range(num_rows)]), + "quantity": pa.array([(i % 10) + 1 for i in range(num_rows)], type=pa.int32()), + } + ) + pq.write_table(table, path) + return path diff --git a/test/integration/test_flight_server.py b/test/integration/test_flight_server.py new file mode 100644 index 0000000..772d249 --- /dev/null +++ b/test/integration/test_flight_server.py @@ -0,0 +1,300 @@ +"""Integration tests for Arrow Flight server, S3 hydration, DuckDB pool, and HTTP ops.""" + +from __future__ import annotations + +import hashlib +import json +import urllib.error +import urllib.request +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import pyarrow.flight as fl +import pytest + +from internal.config.settings import Settings +from internal.flight.daemon import WorkerDaemon +from test.fixtures.sample_data import write_sample_parquet + + +class _FakeBody: + def __init__(self, data: bytes) -> None: + self._data = data + + async def iter_chunks(self, chunk_size: int = 8192) -> Any: + for idx in range(0, len(self._data), chunk_size): + yield self._data[idx : idx + chunk_size] + + def close(self) -> None: + return None + + +class _FakeS3Client: + def __init__(self, objects: dict[tuple[str, str], bytes]) -> None: + self._objects = objects + self.get_calls = 0 + + async def get_object(self, *, Bucket: str, Key: str) -> dict[str, Any]: + self.get_calls += 1 + key = (Bucket, Key) + if key not in self._objects: + from botocore.exceptions import ClientError + + raise ClientError( + {"Error": {"Code": "NoSuchKey"}, "ResponseMetadata": {"HTTPStatusCode": 404}}, + "GetObject", + ) + return {"Body": _FakeBody(self._objects[key])} + + +@pytest.fixture +def running_daemon(tmp_path: Path) -> Iterator[tuple[WorkerDaemon, _FakeS3Client, str, str]]: + sample_path = write_sample_parquet(tmp_path / "source.parquet", num_rows=200) + parquet_bytes = sample_path.read_bytes() + table_hash = hashlib.sha256(parquet_bytes).hexdigest() + bucket = "rag-tabular-artifacts" + s3_key = f"artifacts/tenant-corp-01/{table_hash}.parquet" + + fake_s3 = _FakeS3Client({(bucket, s3_key): parquet_bytes}) + settings = Settings( + FLIGHT_HOST="127.0.0.1", + FLIGHT_PORT=0, + HTTP_HOST="127.0.0.1", + HTTP_PORT=0, + CACHE_DIR=tmp_path / "nvme_cache", + CACHE_MAX_BYTES=64 * 1024 * 1024, + S3_BUCKET_ALLOWLIST=[bucket], + POOL_SIZE=2, + MAX_CONCURRENT_QUERIES=2, + QUERY_TIMEOUT_SECONDS=20.0, + ) + daemon = WorkerDaemon(settings=settings, s3_client=fake_s3) + daemon.start() + try: + yield daemon, fake_s3, table_hash, s3_key + finally: + daemon.shutdown() + + +def _http_get(port: int, path: str) -> tuple[int, str]: + url = f"http://127.0.0.1:{port}{path}" + try: + with urllib.request.urlopen(url, timeout=5.0) as resp: # noqa: S310 + return resp.status, resp.read().decode("utf-8") + except urllib.error.HTTPError as exc: + return exc.code, exc.read().decode("utf-8") + + +def test_flight_do_get_do_action_and_ops_endpoints( + running_daemon: tuple[WorkerDaemon, _FakeS3Client, str, str], +) -> None: + daemon, fake_s3, table_hash, s3_key = running_daemon + assert daemon.flight_server is not None + assert daemon.ops_server is not None + + client = fl.FlightClient(f"grpc://127.0.0.1:{daemon.flight_server.port}") + + # 1. DoAction health_check + health_results = list(client.do_action(fl.Action("health_check", b"{}"))) + health_payload = json.loads(health_results[0].body.to_pybytes().decode("utf-8")) + assert health_payload == {"status": "ok", "active_workers": 2} + + # 2. DoAction cache_warm (cold S3 hydration) + warm_req = json.dumps( + { + "tenant_id": "tenant-corp-01", + "table_hash": table_hash, + "s3_bucket": "rag-tabular-artifacts", + "s3_key": s3_key, + } + ).encode("utf-8") + warm_results = list(client.do_action(fl.Action("cache_warm", warm_req))) + warm_payload = json.loads(warm_results[0].body.to_pybytes().decode("utf-8")) + assert warm_payload["status"] == "warmed" + assert warm_payload["bytes"] > 0 + assert fake_s3.get_calls == 1 + + # 3. DoGet (warm cache hit -> zero-copy RecordBatch stream) + ticket_bytes = json.dumps( + { + "request_id": "req_01", + "tenant_id": "tenant-corp-01", + "sql": ( + "SELECT region, SUM(revenue) AS total_rev FROM tbl GROUP BY region ORDER BY region" + ), + "table_alias": "tbl", + "table_hash": table_hash, + "s3_bucket": "rag-tabular-artifacts", + "s3_key": s3_key, + } + ).encode("utf-8") + reader = client.do_get(fl.Ticket(ticket_bytes)) + table = reader.read_all() + assert table.num_rows == 4 + assert fake_s3.get_calls == 1 # Still 1: served from warm NVMe cache + + # 3b. DoGet empty result must preserve typed schema columns + empty_ticket = json.dumps( + { + "request_id": "req_01_empty", + "tenant_id": "tenant-corp-01", + "sql": "SELECT id, region, revenue FROM tbl WHERE 1 = 0", + "table_alias": "tbl", + "table_hash": table_hash, + "s3_bucket": "rag-tabular-artifacts", + "s3_key": s3_key, + } + ).encode("utf-8") + empty_table = client.do_get(fl.Ticket(empty_ticket)).read_all() + assert empty_table.num_rows == 0 + assert empty_table.schema.names == ["id", "region", "revenue"] + + # 4. Error mapping checks: AST rejection, cross-tenant isolation, missing S3 object + bad_sql_ticket = json.dumps( + { + "request_id": "req_02", + "tenant_id": "tenant-corp-01", + "sql": "PRAGMA show_tables", + "table_alias": "tbl", + "table_hash": table_hash, + "s3_bucket": "rag-tabular-artifacts", + "s3_key": s3_key, + } + ).encode("utf-8") + with pytest.raises(fl.FlightServerError, match="INVALID_ARGUMENT"): + client.do_get(fl.Ticket(bad_sql_ticket)).read_all() + + cross_tenant_ticket = json.dumps( + { + "request_id": "req_03", + "tenant_id": "tenant-corp-01", + "sql": "SELECT COUNT(*) FROM tbl", + "table_alias": "tbl", + "table_hash": table_hash, + "s3_bucket": "rag-tabular-artifacts", + "s3_key": f"artifacts/tenant-other/{table_hash}.parquet", + } + ).encode("utf-8") + with pytest.raises(fl.FlightUnauthorizedError, match="PERMISSION_DENIED"): + client.do_get(fl.Ticket(cross_tenant_ticket)).read_all() + + missing_hash = hashlib.sha256(b"does-not-exist").hexdigest() + missing_ticket = json.dumps( + { + "request_id": "req_04", + "tenant_id": "tenant-corp-01", + "sql": "SELECT COUNT(*) FROM tbl", + "table_alias": "tbl", + "table_hash": missing_hash, + "s3_bucket": "rag-tabular-artifacts", + "s3_key": f"artifacts/tenant-corp-01/{missing_hash}.parquet", + } + ).encode("utf-8") + with pytest.raises(fl.FlightServerError, match="NOT_FOUND"): + client.do_get(fl.Ticket(missing_ticket)).read_all() + + # 5. Backpressure check: saturate concurrency semaphore + acquired_1 = daemon.handlers._concurrency_semaphore.acquire(blocking=False) + acquired_2 = daemon.handlers._concurrency_semaphore.acquire(blocking=False) + assert acquired_1 and acquired_2 + try: + with pytest.raises(fl.FlightServerError, match="RESOURCE_EXHAUSTED"): + client.do_get(fl.Ticket(ticket_bytes)) + finally: + daemon.handlers._concurrency_semaphore.release() + daemon.handlers._concurrency_semaphore.release() + + # 6. HTTP ops endpoints (/healthz, /ready, /metrics, /openapi.yaml, /docs) + http_port = daemon.ops_server.port + status, body = _http_get(http_port, "/healthz") + assert status == 200 + assert json.loads(body)["status"] == "ok" + + status, body = _http_get(http_port, "/ready") + assert status == 200 + assert json.loads(body)["status"] == "ready" + + status, body = _http_get(http_port, "/metrics") + assert status == 200 + assert "tabular_worker_active_queries" in body + assert "tabular_worker_cache_hits_total" in body + assert "tabular_worker_flight_stream_bytes_total" in body + + status, body = _http_get(http_port, "/openapi.yaml") + assert status == 200 + assert "openapi: 3.1.0" in body + + status, body = _http_get(http_port, "/docs") + assert status == 200 + assert "SwaggerUIBundle" in body + + # Drain -> /ready must return 503 + daemon.ops_state.mark_draining() + status, body = _http_get(http_port, "/ready") + assert status == 503 + assert json.loads(body)["status"] == "draining" + + +def test_flight_do_get_emits_otel_spans_and_updates_active_queries( + running_daemon: tuple[WorkerDaemon, _FakeS3Client, str, str], +) -> None: + from opentelemetry import trace + from opentelemetry.sdk.trace import TracerProvider + from opentelemetry.sdk.trace.export import SimpleSpanProcessor + from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter + + daemon, _fake_s3, table_hash, s3_key = running_daemon + assert daemon.flight_server is not None + assert daemon.ops_server is not None + + exporter = InMemorySpanExporter() + current_provider = trace.get_tracer_provider() + if isinstance(current_provider, TracerProvider): + provider = current_provider + owns_provider = False + else: + provider = TracerProvider() + trace.set_tracer_provider(provider) + owns_provider = True + processor = SimpleSpanProcessor(exporter) + provider.add_span_processor(processor) + try: + client = fl.FlightClient(f"grpc://127.0.0.1:{daemon.flight_server.port}") + trace_id = "abcdef0123456789abcdef0123456789" + call_opts = fl.FlightCallOptions( + headers=[(b"traceparent", f"00-{trace_id}-00f067aa0ba902b7-01".encode())] + ) + ticket_bytes = json.dumps( + { + "request_id": "req_otel_01", + "tenant_id": "tenant-corp-01", + "sql": "SELECT COUNT(*) AS cnt FROM tbl", + "table_alias": "tbl", + "table_hash": table_hash, + "s3_bucket": "rag-tabular-artifacts", + "s3_key": s3_key, + } + ).encode("utf-8") + + table = client.do_get(fl.Ticket(ticket_bytes), options=call_opts).read_all() + assert table.num_rows == 1 + + spans = exporter.get_finished_spans() + span_names = {s.name for s in spans} + assert "flight.do_get" in span_names + assert "sqlglot.validate" in span_names + assert "storage.get_or_hydrate" in span_names + assert "duckdb.subprocess_execute" in span_names + for span in spans: + assert format(span.context.trace_id, "032x") == trace_id + if span.attributes: + assert "sql" not in span.attributes + + status, body = _http_get(daemon.ops_server.port, "/metrics") + assert status == 200 + assert "tabular_worker_active_queries 0.0" in body + finally: + provider.force_flush() + if owns_provider: + provider.shutdown() diff --git a/test/security/test_sql_injection_fuzz.py b/test/security/test_sql_injection_fuzz.py new file mode 100644 index 0000000..d02ec09 --- /dev/null +++ b/test/security/test_sql_injection_fuzz.py @@ -0,0 +1,292 @@ +"""Security and fuzzing tests for SQL AST validation, DuckDB sandbox escapes, and logging.""" + +from __future__ import annotations + +import asyncio +import json +import logging +import random +import time +from pathlib import Path + +import duckdb +import pyarrow.flight as fl +import pytest + +from internal.config.settings import Settings +from internal.flight.handlers import FlightHandlers +from internal.pool.process_manager import ProcessManager +from internal.sandbox.security_profile import SandboxProfile, apply_sandbox_lock +from internal.storage.cache_manager import CacheManager +from internal.storage.s3_streamer import S3Streamer +from internal.storage.table_resolver import TableResolver +from internal.telemetry.logging import JsonLogFormatter, redact_mapping, redact_text +from internal.telemetry.metrics import MetricsRegistry +from internal.validator import ( + ASTSyntaxError, + ASTValidator, + DisallowedFunctionError, + DisallowedStatementError, + DisallowedTableReferenceError, + ValidationError, +) +from test.fixtures.sample_data import write_sample_parquet + + +@pytest.fixture(scope="module") +def validator() -> ASTValidator: + instance = ASTValidator(max_sql_length=8192, max_joins=5) + # Warm up sqlglot DuckDB parser/caches before steady-state latency checks. + instance.validate("SELECT COUNT(*) FROM tbl", allowed_table_alias="tbl") + return instance + + +@pytest.mark.parametrize( + "malicious_sql", + [ + "INSERT INTO tbl VALUES (1, 'a', 'b', 1.0, 1)", + "UPDATE tbl SET revenue = 0", + "DELETE FROM tbl WHERE id = 1", + "DROP TABLE tbl", + "ALTER TABLE tbl ADD COLUMN x INT", + "CREATE TABLE evil (id INT)", + "PRAGMA table_info('tbl')", + "PRAGMA version", + "SET memory_limit = '10GB'", + "RESET memory_limit", + "SUMMARIZE tbl", + "DESCRIBE tbl", + "SHOW TABLES", + "PIVOT tbl ON region USING SUM(revenue)", + "COPY tbl TO '/tmp/exfil.parquet'", + "ATTACH '/tmp/other.db' AS other", + "DETACH other", + "LOAD httpfs", + "INSTALL httpfs", + ], +) +def test_blocks_banned_dml_ddl_and_admin_statements( + validator: ASTValidator, + malicious_sql: str, +) -> None: + with pytest.raises((DisallowedStatementError, ValidationError)): + validator.validate(malicious_sql, allowed_table_alias="tbl") + + +@pytest.mark.parametrize( + "malicious_sql", + [ + "SELECT * FROM read_parquet('/etc/passwd')", + "SELECT * FROM read_csv('/etc/passwd')", + "SELECT * FROM read_csv_auto('/tmp/secret.csv')", + "SELECT * FROM scan_parquet('s3://bucket/key.parquet')", + "SELECT glob('*') FROM tbl", + "SELECT getenv('AWS_SECRET_ACCESS_KEY') FROM tbl", + "SELECT current_setting('memory_limit') FROM tbl", + "SELECT duckdb_tables() FROM tbl", + "SELECT duckdb_settings() FROM tbl", + "SELECT httpfs_get('https://evil.example.com') FROM tbl", + "SELECT write_csv(tbl, '/tmp/out.csv') FROM tbl", + "SELECT custom_unapproved_udf(revenue) FROM tbl", + ], +) +def test_blocks_duckdb_io_system_and_unapproved_anonymous_functions( + validator: ASTValidator, + malicious_sql: str, +) -> None: + with pytest.raises((DisallowedFunctionError, DisallowedTableReferenceError, ValidationError)): + validator.validate(malicious_sql, allowed_table_alias="tbl") + + +def test_blocks_file_literal_table_and_multi_statement_injection( + validator: ASTValidator, + tmp_path: Path, +) -> None: + parquet = write_sample_parquet(tmp_path / "secret.parquet", num_rows=10) + with pytest.raises(DisallowedTableReferenceError): + validator.validate(f"SELECT * FROM '{parquet}'", allowed_table_alias="tbl") + + with pytest.raises(ASTSyntaxError): + validator.validate("SELECT COUNT(*) FROM tbl; DROP TABLE tbl;", allowed_table_alias="tbl") + + +def test_structured_logging_redacts_secrets_and_presigned_urls() -> None: + raw = ( + "Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.secret.sig " + "url=https://s3.amazonaws.com/b/k?X-Amz-Signature=deadbeef&X-Amz-Credential=abc " + "key=AKIAIOSFODNN7EXAMPLE" + ) + sanitized = redact_text(raw) + assert "eyJhbGciOiJIUzI1NiJ9" not in sanitized + assert "deadbeef" not in sanitized + assert "AKIAIOSFODNN7EXAMPLE" not in sanitized + assert "[REDACTED]" in sanitized + + redacted_map = redact_mapping( + { + "tenant_id": "t-1", + "aws_secret_access_key": "supersecret", + "sql": "SELECT * FROM tbl WHERE ssn = '123-45-6789'", + } + ) + assert redacted_map["tenant_id"] == "t-1" + assert redacted_map["aws_secret_access_key"] == "[REDACTED]" + assert redacted_map["sql"] == "[REDACTED]" + + formatter = JsonLogFormatter() + record = logging.LogRecord( + name="test", + level=logging.INFO, + pathname=__file__, + lineno=1, + msg="request token Bearer abc123xyz", + args=(), + exc_info=None, + ) + parsed = json.loads(formatter.format(record)) + assert parsed["level"] == "INFO" + assert "abc123xyz" not in parsed["message"] + + +def _generate_fuzz_payloads(seed: int = 42, count: int = 120) -> list[str]: + """Generate deterministic obfuscated SQL injection and AST bypass payloads.""" + rng = random.Random(seed) # noqa: S311 + banned_calls = [ + "read_parquet('/etc/passwd')", + "read_csv('/etc/passwd')", + "read_csv_auto('/tmp/secret.csv')", + "scan_parquet('s3://evil-bucket/key.parquet')", + "glob('/var/run/*')", + "getenv('AWS_SECRET_ACCESS_KEY')", + "current_setting('memory_limit')", + "duckdb_tables()", + "duckdb_settings()", + "httpfs_get('https://evil.example.com')", + "write_csv(tbl, '/tmp/out.csv')", + "copy(tbl)", + ] + wrappers = [ + lambda call: f"WITH cte AS (SELECT {call} AS x FROM tbl) SELECT x FROM cte", + lambda call: f"SELECT COUNT(*) FROM tbl UNION ALL SELECT 1 FROM {call}", + lambda call: f"SELECT COUNT(*) FROM tbl INTERSECT SELECT 1 WHERE {call} IS NOT NULL", + lambda call: f"SELECT COUNT(*) FROM tbl EXCEPT SELECT 1 FROM {call}", + lambda call: f"SELECT (SELECT {call} FROM tbl LIMIT 1) AS leaked FROM tbl", + lambda call: f"SELECT * FROM (SELECT * FROM {call}) sub", + lambda call: f"SELECT /* bypass */ COUNT(*) FROM tbl; {call}", + lambda call: f"SELECT * FROM '/tmp/{rng.randint(1, 999)}.parquet'", + lambda call: f"SELECT * FROM tbl WHERE id = {'(' * 12}{call}{')' * 12}", + ] + comments = ["/*+ hint */ ", "/* fuzz */\n", "-- line comment\n", " \t\n "] + + payloads: list[str] = [] + for i in range(count): + call = banned_calls[i % len(banned_calls)] + if rng.random() < 0.5: + call = "".join(c.upper() if rng.random() < 0.5 else c.lower() for c in call) + prefix = rng.choice(comments) + wrapper = wrappers[i % len(wrappers)] + payloads.append(f"{prefix}{wrapper(call)}") + return payloads + + +def test_deterministic_sql_fuzzing_blocks_all_bypasses_within_2ms( + validator: ASTValidator, +) -> None: + payloads = _generate_fuzz_payloads(seed=42, count=120) + total_ms = 0.0 + for payload in payloads: + start = time.perf_counter() + with pytest.raises(ValidationError): + validator.validate(payload, allowed_table_alias="tbl") + elapsed_ms = (time.perf_counter() - start) * 1000.0 + total_ms += elapsed_ms + + avg_ms = total_ms / len(payloads) + assert avg_ms <= 2.0, f"Average AST validation latency {avg_ms:.3f}ms exceeded 2.0ms budget" + + +def test_flight_handlers_increment_ast_rejection_metric_on_fuzz_payloads( + tmp_path: Path, +) -> None: + settings = Settings( + CACHE_DIR=tmp_path / "cache", + S3_BUCKET_ALLOWLIST=["rag-tabular-artifacts"], + POOL_SIZE=1, + MAX_CONCURRENT_QUERIES=2, + ) + metrics = MetricsRegistry() + ast_validator = ASTValidator(max_sql_length=8192, max_joins=5) + table_resolver = TableResolver(settings.S3_BUCKET_ALLOWLIST) + cache_manager = CacheManager(tmp_path / "cache", max_bytes=1024 * 1024, metrics=metrics) + s3_streamer = S3Streamer(client=object()) + process_manager = ProcessManager(settings, metrics=metrics) + loop = asyncio.new_event_loop() + try: + handlers = FlightHandlers( + settings=settings, + ast_validator=ast_validator, + table_resolver=table_resolver, + cache_manager=cache_manager, + s3_streamer=s3_streamer, + process_manager=process_manager, + hydration_loop=loop, + metrics=metrics, + ) + payloads = _generate_fuzz_payloads(seed=7, count=25) + table_hash = "a" * 64 + for idx, sql in enumerate(payloads): + ticket_bytes = json.dumps( + { + "request_id": f"req-fuzz-{idx}", + "tenant_id": "tenant-fuzz", + "sql": sql, + "table_alias": "tbl", + "table_hash": table_hash, + "s3_bucket": "rag-tabular-artifacts", + "s3_key": f"artifacts/tenant-fuzz/{table_hash}.parquet", + } + ).encode("utf-8") + with pytest.raises(fl.FlightError): + handlers.handle_do_get(ticket_bytes) + + rendered = metrics.render_prometheus().decode("utf-8") + assert "tabular_worker_ast_rejections_total{" in rendered + total_rejections = sum( + sample.value + for metric in metrics.registry.collect() + if metric.name == "tabular_worker_ast_rejections" + for sample in metric.samples + if sample.name == "tabular_worker_ast_rejections_total" + ) + assert total_rejections == len(payloads) + finally: + loop.close() + + +@pytest.mark.parametrize( + "escape_sql", + [ + "SELECT * FROM read_parquet('/etc/passwd')", + "SELECT * FROM read_csv('/etc/passwd')", + "PRAGMA memory_limit='10GB'", + "SET memory_limit = '10GB'", + "ATTACH '/tmp/other.db' AS extra_db", + "DETACH memory", + "COPY tbl TO '/tmp/exfil.parquet'", + "INSTALL httpfs", + "LOAD httpfs", + "SELECT getenv('PATH')", + ], +) +def test_duckdb_sandbox_lock_blocks_direct_engine_escapes( + tmp_path: Path, + escape_sql: str, +) -> None: + parquet_path = write_sample_parquet(tmp_path / "sandbox.parquet", num_rows=10) + conn = duckdb.connect(":memory:") + try: + apply_sandbox_lock(conn, parquet_path, "tbl", SandboxProfile()) + with pytest.raises(duckdb.Error): + conn.execute(escape_sql).fetchall() + finally: + conn.close() diff --git a/test/unit/test_ast_validator.py b/test/unit/test_ast_validator.py new file mode 100644 index 0000000..443ac34 --- /dev/null +++ b/test/unit/test_ast_validator.py @@ -0,0 +1,164 @@ +"""Unit tests for deterministic sqlglot AST security validator.""" + +from __future__ import annotations + +import time + +import pytest + +from internal.validator import ( + ASTSyntaxError, + ASTValidator, + DisallowedAliasError, + DisallowedStatementError, + DisallowedTableReferenceError, + JoinLimitExceededError, + QueryLengthExceededError, +) + + +@pytest.fixture(scope="module") +def validator() -> ASTValidator: + return ASTValidator(max_sql_length=4096, max_joins=3) + + +def test_allows_valid_select_with_aggregations(validator: ASTValidator) -> None: + sql = ( + "SELECT region, SUM(revenue) AS total_rev, COUNT(*) AS row_cnt " + "FROM tbl WHERE quantity >= 2 GROUP BY region ORDER BY total_rev DESC LIMIT 10" + ) + ast = validator.validate(sql, allowed_table_alias="tbl") + assert ast is not None + + +def test_allows_with_cte_and_union_intersect_except(validator: ASTValidator) -> None: + sql = """ + WITH regional AS ( + SELECT region, SUM(revenue) AS total_rev + FROM tbl + GROUP BY region + ) + SELECT region, total_rev FROM regional WHERE total_rev > 100 + UNION ALL + SELECT region, total_rev FROM regional WHERE total_rev <= 100 + """ + assert validator.validate(sql, allowed_table_alias="tbl") is not None + + intersect_sql = ( + "SELECT region FROM tbl WHERE quantity > 1 " + "INTERSECT SELECT region FROM tbl WHERE revenue > 50" + ) + assert validator.validate(intersect_sql, allowed_table_alias="tbl") is not None + + except_sql = ( + "SELECT region FROM tbl WHERE quantity > 1 EXCEPT SELECT region FROM tbl WHERE revenue < 20" + ) + assert validator.validate(except_sql, allowed_table_alias="tbl") is not None + + +def test_allows_safe_anonymous_analytical_functions(validator: ASTValidator) -> None: + sql = ( + "SELECT region, MEDIAN(revenue) AS med_rev, " + "ROUND(AVG(revenue), 2) AS avg_rev, " + "APPROX_COUNT_DISTINCT(category) AS distinct_cats " + "FROM tbl GROUP BY region" + ) + assert validator.validate(sql, allowed_table_alias="tbl") is not None + + +def test_rejects_unallowlisted_table_and_schema_references(validator: ASTValidator) -> None: + with pytest.raises(DisallowedTableReferenceError): + validator.validate("SELECT * FROM other_table", allowed_table_alias="tbl") + + with pytest.raises(DisallowedTableReferenceError): + validator.validate("SELECT * FROM main.tbl", allowed_table_alias="tbl") + + with pytest.raises(DisallowedTableReferenceError): + validator.validate("SELECT * FROM information_schema.tables", allowed_table_alias="tbl") + + +def test_rejects_unsafe_aliases(validator: ASTValidator) -> None: + with pytest.raises(DisallowedAliasError): + validator.validate('SELECT revenue AS "bad;alias" FROM tbl', allowed_table_alias="tbl") + + with pytest.raises(DisallowedAliasError): + validator.validate( + 'WITH "bad-cte" AS (SELECT 1) SELECT * FROM tbl', + allowed_table_alias="tbl", + ) + + +def test_enforces_join_limit(validator: ASTValidator) -> None: + ok_sql = ( + "SELECT a.id FROM tbl a " + "JOIN tbl b ON a.id = b.id " + "JOIN tbl c ON a.id = c.id " + "JOIN tbl d ON a.id = d.id" + ) + assert validator.validate(ok_sql, allowed_table_alias="tbl") is not None + + too_many_joins_sql = ( + "SELECT a.id FROM tbl a " + "JOIN tbl b ON a.id = b.id " + "JOIN tbl c ON a.id = c.id " + "JOIN tbl d ON a.id = d.id " + "JOIN tbl e ON a.id = e.id" + ) + with pytest.raises(JoinLimitExceededError): + validator.validate(too_many_joins_sql, allowed_table_alias="tbl") + + +def test_enforces_query_length_and_empty_syntax(validator: ASTValidator) -> None: + with pytest.raises(ASTSyntaxError): + validator.validate(" ", allowed_table_alias="tbl") + + with pytest.raises(QueryLengthExceededError): + validator.validate("SELECT " + ("1," * 3000) + "1 FROM tbl", allowed_table_alias="tbl") + + +def test_warm_steady_state_validation_latency(validator: ASTValidator) -> None: + sql = ( + "SELECT region, category, SUM(revenue) AS total_rev, COUNT(*) AS cnt " + "FROM tbl WHERE quantity > 1 GROUP BY region, category ORDER BY total_rev DESC" + ) + # Warm up once before timing steady-state calls. + validator.validate(sql, allowed_table_alias="tbl") + + iterations = 50 + start = time.perf_counter() + for _ in range(iterations): + validator.validate(sql, allowed_table_alias="tbl") + avg_ms = ((time.perf_counter() - start) / iterations) * 1000.0 + # Pure-Python sqlglot.parse takes ~2-4 ms on shared CI runners; guard against regressions. + assert avg_ms <= 15.0, f"Expected warm AST validation <= 15.0 ms, got {avg_ms:.3f} ms" + + +def test_rejects_unterminated_sql_string_with_syntax_error(validator: ASTValidator) -> None: + with pytest.raises(ASTSyntaxError): + validator.validate("SELECT 'unterminated", allowed_table_alias="tbl") + + +def test_enforces_lexical_cte_scope(validator: ASTValidator) -> None: + nested_leak_sql = ( + "SELECT * FROM (WITH inner_cte AS (SELECT region FROM tbl) SELECT * FROM inner_cte) sub " + "UNION ALL SELECT * FROM inner_cte" + ) + with pytest.raises(DisallowedTableReferenceError): + validator.validate(nested_leak_sql, allowed_table_alias="tbl") + + +def test_allows_date_diff_with_approved_unit_and_rejects_unrelated_var( + validator: ASTValidator, +) -> None: + assert ( + validator.validate( + "SELECT DATE_DIFF('day', start_dt, end_dt) FROM tbl", + allowed_table_alias="tbl", + ) + is not None + ) + with pytest.raises(DisallowedStatementError): + validator.validate( + "SELECT DATE_DIFF('bogus_unit', start_dt, end_dt) FROM tbl", + allowed_table_alias="tbl", + ) diff --git a/test/unit/test_cache_manager.py b/test/unit/test_cache_manager.py new file mode 100644 index 0000000..a1bbe8e --- /dev/null +++ b/test/unit/test_cache_manager.py @@ -0,0 +1,203 @@ +"""Unit tests for Tier-1 NVMe CacheManager LRU, refcounting, single-flight, and Parquet checks.""" + +from __future__ import annotations + +import asyncio +import hashlib +from pathlib import Path +from typing import Any + +import pytest + +from internal.storage import ( + ArtifactNotFoundError, + CacheCapacityError, + CacheManager, + CorruptedArtifactError, + S3Streamer, + TableResolver, +) +from internal.telemetry.metrics import MetricsRegistry +from test.fixtures.sample_data import write_sample_parquet + + +class _FakeBody: + def __init__(self, data: bytes, delay: float = 0.0) -> None: + self._data = data + self._delay = delay + + async def iter_chunks(self, chunk_size: int = 1024) -> Any: + if self._delay > 0: + await asyncio.sleep(self._delay) + for idx in range(0, len(self._data), chunk_size): + yield self._data[idx : idx + chunk_size] + + def close(self) -> None: + return None + + +class _FakeS3Client: + def __init__(self, objects: dict[tuple[str, str], bytes], delay: float = 0.0) -> None: + self._objects = objects + self._delay = delay + self.get_calls = 0 + + async def get_object(self, *, Bucket: str, Key: str) -> dict[str, Any]: + self.get_calls += 1 + key = (Bucket, Key) + if key not in self._objects: + from botocore.exceptions import ClientError + + raise ClientError( + {"Error": {"Code": "NoSuchKey"}, "ResponseMetadata": {"HTTPStatusCode": 404}}, + "GetObject", + ) + return {"Body": _FakeBody(self._objects[key], delay=self._delay)} + + +def _make_hash(label: str) -> str: + return hashlib.sha256(label.encode("utf-8")).hexdigest() + + +def test_startup_reconciliation_and_tmp_sweep(tmp_path: Path) -> None: + cache_dir = tmp_path / "cache" + h1 = _make_hash("table-1") + valid_path = cache_dir / "bucket-a" / "tenant-1" / f"{h1}.parquet" + valid_path.parent.mkdir(parents=True, exist_ok=True) + write_sample_parquet(valid_path, num_rows=20) + expected_size = valid_path.stat().st_size + + orphan_tmp = valid_path.parent / ".orphan.tmp" + orphan_tmp.write_bytes(b"partial") + + metrics = MetricsRegistry() + manager = CacheManager(cache_dir=cache_dir, max_bytes=1024 * 1024, metrics=metrics) + manager.initialize() + + assert not orphan_tmp.exists() + assert manager.total_bytes == expected_size + assert metrics.cache_bytes_used._value.get() == expected_size + + +@pytest.mark.asyncio +async def test_single_flight_coalescing_hydrates_once(tmp_path: Path) -> None: + sample_file = write_sample_parquet(tmp_path / "src.parquet", num_rows=50) + parquet_bytes = sample_file.read_bytes() + h1 = _make_hash("singleflight") + s3_key = f"artifacts/tenant-1/{h1}.parquet" + + fake_client = _FakeS3Client({("bucket-a", s3_key): parquet_bytes}, delay=0.05) + streamer = S3Streamer(chunk_size_bytes=4096, timeout_seconds=4.0, client=fake_client) + metrics = MetricsRegistry() + manager = CacheManager( + cache_dir=tmp_path / "cache", max_bytes=10 * 1024 * 1024, metrics=metrics + ) + manager.initialize() + + resolver = TableResolver(["bucket-a"]) + resolved = resolver.resolve( + tenant_id="tenant-1", + table_hash=h1, + s3_bucket="bucket-a", + s3_key=s3_key, + ) + + pinned_list = await asyncio.gather( + manager.get_or_hydrate(resolved, streamer), + manager.get_or_hydrate(resolved, streamer), + manager.get_or_hydrate(resolved, streamer), + ) + try: + assert fake_client.get_calls == 1 + assert all(p.path.exists() for p in pinned_list) + assert manager.total_bytes == len(parquet_bytes) + finally: + for p in pinned_list: + p.release() + + +@pytest.mark.asyncio +async def test_lru_eviction_and_all_pinned_capacity_error(tmp_path: Path) -> None: + src = write_sample_parquet(tmp_path / "src.parquet", num_rows=40) + data = src.read_bytes() + size = len(data) + + h1 = _make_hash("t1") + h2 = _make_hash("t2") + h3 = _make_hash("t3") + k1 = f"artifacts/tenant-1/{h1}.parquet" + k2 = f"artifacts/tenant-1/{h2}.parquet" + k3 = f"artifacts/tenant-1/{h3}.parquet" + + fake_client = _FakeS3Client( + {("bucket-a", k1): data, ("bucket-a", k2): data, ("bucket-a", k3): data} + ) + streamer = S3Streamer(client=fake_client) + # Capacity fits exactly 2 artifacts, not 3. + manager = CacheManager(cache_dir=tmp_path / "cache", max_bytes=size * 2 + 10) + manager.initialize() + resolver = TableResolver(["bucket-a"]) + + r1 = resolver.resolve(tenant_id="tenant-1", table_hash=h1, s3_bucket="bucket-a", s3_key=k1) + r2 = resolver.resolve(tenant_id="tenant-1", table_hash=h2, s3_bucket="bucket-a", s3_key=k2) + r3 = resolver.resolve(tenant_id="tenant-1", table_hash=h3, s3_bucket="bucket-a", s3_key=k3) + + p1 = await manager.get_or_hydrate(r1, streamer) + p2 = await manager.get_or_hydrate(r2, streamer) + + # Both entries are pinned -> 3rd hydration must fail with CacheCapacityError. + with pytest.raises(CacheCapacityError): + await manager.get_or_hydrate(r3, streamer) + + # Release p1 (oldest), keep p2 pinned -> 3rd hydration evicts r1 and succeeds. + p1.release() + p3 = await manager.get_or_hydrate(r3, streamer) + try: + assert not (tmp_path / "cache" / r1.relative_cache_path).exists() + assert (tmp_path / "cache" / r2.relative_cache_path).exists() + assert (tmp_path / "cache" / r3.relative_cache_path).exists() + finally: + p2.release() + p3.release() + + +@pytest.mark.asyncio +async def test_corrupted_parquet_rejected_and_not_committed(tmp_path: Path) -> None: + h1 = _make_hash("corrupt") + s3_key = f"artifacts/tenant-1/{h1}.parquet" + fake_client = _FakeS3Client({("bucket-a", s3_key): b"NOT_A_PARQUET_FILE_AT_ALL"}) + streamer = S3Streamer(client=fake_client) + manager = CacheManager(cache_dir=tmp_path / "cache", max_bytes=1024 * 1024) + manager.initialize() + resolver = TableResolver(["bucket-a"]) + resolved = resolver.resolve( + tenant_id="tenant-1", + table_hash=h1, + s3_bucket="bucket-a", + s3_key=s3_key, + ) + + with pytest.raises(CorruptedArtifactError): + await manager.get_or_hydrate(resolved, streamer) + assert manager.total_bytes == 0 + assert not (tmp_path / "cache" / resolved.relative_cache_path).exists() + + +@pytest.mark.asyncio +async def test_missing_s3_object_raises_artifact_not_found(tmp_path: Path) -> None: + h1 = _make_hash("missing") + s3_key = f"artifacts/tenant-1/{h1}.parquet" + fake_client = _FakeS3Client({}) + streamer = S3Streamer(client=fake_client) + manager = CacheManager(cache_dir=tmp_path / "cache", max_bytes=1024 * 1024) + manager.initialize() + resolver = TableResolver(["bucket-a"]) + resolved = resolver.resolve( + tenant_id="tenant-1", + table_hash=h1, + s3_bucket="bucket-a", + s3_key=s3_key, + ) + + with pytest.raises(ArtifactNotFoundError): + await manager.get_or_hydrate(resolved, streamer) diff --git a/test/unit/test_daemon_and_metrics.py b/test/unit/test_daemon_and_metrics.py new file mode 100644 index 0000000..0637547 --- /dev/null +++ b/test/unit/test_daemon_and_metrics.py @@ -0,0 +1,196 @@ +"""Unit tests for MetricsRegistry recycle initialization, S3Streamer reset, and daemon shutdown.""" + +from __future__ import annotations + +import asyncio +import gc +import threading +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pyarrow as pa +import pyarrow.flight as fl +import pytest +from prometheus_client import generate_latest + +from internal.config.settings import Settings +from internal.flight.daemon import WorkerDaemon +from internal.flight.handlers import FlightHandlers, _OwnedFlightBatchIterator +from internal.storage.s3_streamer import S3Streamer +from internal.telemetry.metrics import MetricsRegistry + + +def test_metrics_registry_initializes_abnormal_recycle_series_to_zero() -> None: + metrics = MetricsRegistry() + exposition = generate_latest(metrics.registry).decode("utf-8") + + for reason in ("TIMEOUT", "MEMORY_LIMIT", "CRASH"): + assert f'tabular_worker_subprocess_recycles_total{{reason="{reason}"}} 0.0' in exposition + + +def test_s3_streamer_close_failure_and_reset_allow_fresh_restart() -> None: + streamer = S3Streamer() + ctx = MagicMock() + + async def _failing_aexit(*_args: object) -> None: + raise RuntimeError("close failed") + + ctx.__aexit__ = _failing_aexit + streamer._client_ctx = ctx + streamer._client = object() + + with pytest.raises(RuntimeError, match="close failed"): + asyncio.run(streamer.close()) + + assert streamer._client_ctx is None + assert streamer._client is None + + streamer._client_ctx = object() + streamer._client = object() + streamer.reset_client_state() + assert streamer._client_ctx is None + assert streamer._client is None + + +def test_daemon_shutdown_bounds_flight_drain_and_completes_cleanup( + monkeypatch: pytest.MonkeyPatch, +) -> None: + daemon = WorkerDaemon(Settings(_env_file=None)) + daemon.hydration_thread.start() + + unblock_flight = threading.Event() + stalled_server = MagicMock() + stalled_server.shutdown.side_effect = lambda: unblock_flight.wait(timeout=2.0) + daemon.flight_server = stalled_server + + closed_streams_called = threading.Event() + + def _close_active_streams() -> int: + closed_streams_called.set() + unblock_flight.set() + return 1 + + daemon.handlers.close_active_streams = _close_active_streams # type: ignore[method-assign] + daemon.process_manager.close = MagicMock() # type: ignore[method-assign] + daemon.ops_server = SimpleNamespace(close=MagicMock()) # type: ignore[assignment] + + async def _slow_s3_close() -> None: + await asyncio.sleep(2.0) + + daemon.s3_streamer.close = _slow_s3_close # type: ignore[method-assign] + daemon.s3_streamer._client_ctx = object() + daemon.s3_streamer._client = object() + + tracing_stopped = threading.Event() + monkeypatch.setattr("internal.flight.daemon.shutdown_tracing", tracing_stopped.set) + + # Speed up the test by using a short Flight drain timeout via _shutdown_flight_server + orig_shutdown_flight = daemon._shutdown_flight_server + monkeypatch.setattr( + daemon, + "_shutdown_flight_server", + lambda drain_timeout_seconds=0.05: orig_shutdown_flight(0.05), + ) + + daemon.shutdown() + + assert closed_streams_called.is_set() + daemon.process_manager.close.assert_called_once() + assert daemon.ops_server is None + assert not daemon.hydration_thread.loop.is_running() + assert daemon.s3_streamer._client_ctx is None + assert daemon.s3_streamer._client is None + assert tracing_stopped.is_set() + + +def test_owned_flight_batch_iterator_distinguishes_exhaustion_from_abort() -> None: + batch = pa.RecordBatch.from_pydict({"x": [1, 2]}) + + # Normal exhaustion raises StopIteration. + normal_stream = MagicMock() + normal_stream.__iter__.return_value = iter([batch]) + normal_pinned = MagicMock(cache_state="hit") + normal_sem = threading.BoundedSemaphore(1) + normal_sem.acquire() + normal_metrics = MetricsRegistry() + normal_metrics.inc_active_queries() + normal_iter = _OwnedFlightBatchIterator( + query_stream=normal_stream, + pinned=normal_pinned, + metrics=normal_metrics, + semaphore=normal_sem, + start_time=0.0, + request_id="req-ok", + tenant_id="tenant-1", + table_hash="a" * 64, + ) + assert next(normal_iter) == batch + with pytest.raises(StopIteration): + next(normal_iter) + assert normal_iter._stream_status == "ok" + normal_pinned.release.assert_called_once() + + # Forced close aborts the stream and raises FlightCancelledError on next(). + aborted_stream = MagicMock() + aborted_stream.__iter__.return_value = iter([batch]) + aborted_pinned = MagicMock(cache_state="hit") + aborted_sem = threading.BoundedSemaphore(1) + aborted_sem.acquire() + aborted_metrics = MetricsRegistry() + aborted_metrics.inc_active_queries() + aborted_iter = _OwnedFlightBatchIterator( + query_stream=aborted_stream, + pinned=aborted_pinned, + metrics=aborted_metrics, + semaphore=aborted_sem, + start_time=0.0, + request_id="req-abort", + tenant_id="tenant-1", + table_hash="a" * 64, + ) + aborted_iter.close() + assert aborted_iter._stream_status == "closed" + with pytest.raises(fl.FlightCancelledError): + next(aborted_iter) + aborted_pinned.release.assert_called_once() + + +def test_active_streams_registry_does_not_pin_abandoned_iterators() -> None: + handlers = FlightHandlers( + settings=Settings(_env_file=None), + ast_validator=MagicMock(), + table_resolver=MagicMock(), + cache_manager=MagicMock(), + s3_streamer=MagicMock(), + process_manager=MagicMock(), + hydration_loop=MagicMock(), + metrics=MetricsRegistry(), + ) + stream = MagicMock() + stream.__iter__.return_value = iter([]) + pinned = MagicMock(cache_state="hit") + sem = threading.BoundedSemaphore(1) + sem.acquire() + handlers._metrics.inc_active_queries() + + owned_iter: _OwnedFlightBatchIterator | None = _OwnedFlightBatchIterator( + query_stream=stream, + pinned=pinned, + metrics=handlers._metrics, + semaphore=sem, + start_time=0.0, + request_id="req-gc", + tenant_id="tenant-1", + table_hash="a" * 64, + on_cleanup=handlers._unregister_stream, + ) + with handlers._active_streams_lock: + handlers._active_streams.add(owned_iter) + + assert len(handlers._active_streams) == 1 + owned_iter = None + gc.collect() + + assert len(handlers._active_streams) == 0 + pinned.release.assert_called_once() + assert handlers.close_active_streams() == 0 diff --git a/test/unit/test_duckdb_sandbox.py b/test/unit/test_duckdb_sandbox.py new file mode 100644 index 0000000..1619147 --- /dev/null +++ b/test/unit/test_duckdb_sandbox.py @@ -0,0 +1,81 @@ +"""Unit tests for ephemeral DuckDB sandbox execution and chunked Arrow IPC streaming.""" + +from __future__ import annotations + +import io +from pathlib import Path + +import duckdb +import pyarrow as pa +import pytest + +from internal.sandbox import ( + SandboxProfile, + apply_sandbox_lock, + execute_sandboxed_query, + stream_query_to_ipc, +) +from test.fixtures.sample_data import write_sample_parquet + + +@pytest.fixture +def sample_parquet(tmp_path: Path) -> Path: + return write_sample_parquet(tmp_path / "sample.parquet", num_rows=500) + + +def test_execute_sandboxed_query_streams_record_batches(sample_parquet: Path) -> None: + profile = SandboxProfile(memory_limit="640MiB", threads=2, temp_directory="") + batches = list( + execute_sandboxed_query( + sql="SELECT region, COUNT(*) AS cnt, SUM(revenue) AS total FROM tbl GROUP BY region", + parquet_path=sample_parquet, + table_alias="tbl", + profile=profile, + batch_size=2, + ) + ) + assert len(batches) >= 2 + assert all(isinstance(batch, pa.RecordBatch) for batch in batches) + total_rows = sum(batch.num_rows for batch in batches) + assert total_rows == 4 + + +def test_stream_query_to_ipc_writes_valid_arrow_stream(sample_parquet: Path) -> None: + profile = SandboxProfile(memory_limit="640MiB", threads=2, temp_directory="") + buf = io.BytesIO() + rows = stream_query_to_ipc( + sql="SELECT id, region, revenue FROM tbl ORDER BY id", + parquet_path=sample_parquet, + table_alias="tbl", + profile=profile, + sink=buf, + batch_size=128, + ) + assert rows == 500 + buf.seek(0) + with pa.ipc.open_stream(buf) as reader: + streamed_batches = list(reader) + assert len(streamed_batches) == 4 # 128 * 3 + 116 + assert sum(b.num_rows for b in streamed_batches) == 500 + + +def test_sandbox_locks_external_access_and_configuration(sample_parquet: Path) -> None: + profile = SandboxProfile(memory_limit="640MiB", threads=2, temp_directory="") + conn = duckdb.connect(":memory:") + try: + apply_sandbox_lock(conn, sample_parquet, "tbl", profile) + + # Bound TEMP TABLE must remain readable after external access is disabled. + count_row = conn.execute("SELECT COUNT(*) FROM tbl").fetchone() + assert count_row == (500,) + + # Direct read_parquet must fail once enable_external_access=false. + escaped = str(sample_parquet).replace("'", "''") + with pytest.raises(duckdb.PermissionException): + conn.execute(f"SELECT * FROM read_parquet('{escaped}')").fetchall() + + # Re-enabling external access must fail once lock_configuration=true. + with pytest.raises(duckdb.InvalidInputException): + conn.execute("SET enable_external_access = true") + finally: + conn.close() diff --git a/test/unit/test_flight_schemas.py b/test/unit/test_flight_schemas.py new file mode 100644 index 0000000..64927f5 --- /dev/null +++ b/test/unit/test_flight_schemas.py @@ -0,0 +1,111 @@ +"""Unit tests for Arrow Flight schemas, sanitized error mappings, and OpenAPI 3.1 spec.""" + +from __future__ import annotations + +import hashlib +from pathlib import Path + +import pyarrow.flight as fl +import pytest +import yaml +from pydantic import ValidationError as PydanticValidationError + +from api.flight import ( + CacheWarmRequest, + CacheWarmResponse, + ConcurrencyLimitError, + FlightQueryTicket, + HealthCheckResponse, + map_exception_to_flight_error, +) +from internal.storage import ( + ArtifactNotFoundError, + CacheCapacityError, + S3AccessError, + TenantIsolationError, +) +from internal.validator.exceptions import DisallowedStatementError + +VALID_HASH = hashlib.sha256(b"flight-schema-test").hexdigest() + + +def test_flight_query_ticket_valid_and_rejects_extra_fields() -> None: + ticket = FlightQueryTicket( + request_id="req_01", + tenant_id="tenant-corp-01", + sql="SELECT COUNT(*) FROM tbl", + table_alias="tbl", + table_hash=VALID_HASH, + s3_bucket="rag-tabular-artifacts", + s3_key=f"artifacts/tenant-corp-01/{VALID_HASH}.parquet", + ) + assert ticket.tenant_id == "tenant-corp-01" + + with pytest.raises(PydanticValidationError): + FlightQueryTicket( + request_id="req_01", + tenant_id="tenant-corp-01", + sql="SELECT 1", + table_alias="tbl", + table_hash=VALID_HASH, + s3_bucket="rag-tabular-artifacts", + s3_key=f"artifacts/tenant-corp-01/{VALID_HASH}.parquet", + unexpected="boom", # type: ignore[call-arg] + ) + + +def test_action_models() -> None: + req = CacheWarmRequest( + tenant_id="tenant-1", + table_hash=VALID_HASH, + s3_bucket="rag-tabular-artifacts", + s3_key=f"artifacts/tenant-1/{VALID_HASH}.parquet", + ) + resp = CacheWarmResponse(bytes=2048) + health = HealthCheckResponse(status="ok", active_workers=4) + assert req.tenant_id == "tenant-1" + assert resp.status == "warmed" + assert health.active_workers == 4 + + +def test_sanitized_flight_error_mappings() -> None: + err_perm = map_exception_to_flight_error( + TenantIsolationError("secret-bucket/artifacts/tenant-x/secret.parquet") + ) + assert isinstance(err_perm, fl.FlightUnauthorizedError) + assert "PERMISSION_DENIED" in str(err_perm) + assert "secret-bucket" not in str(err_perm) + + err_ast = map_exception_to_flight_error(DisallowedStatementError("PRAGMA show_tables")) + assert isinstance(err_ast, fl.FlightServerError) + assert "INVALID_ARGUMENT" in str(err_ast) + + err_timeout = map_exception_to_flight_error(TimeoutError("deadline")) + assert isinstance(err_timeout, fl.FlightTimedOutError) + assert "DEADLINE_EXCEEDED" in str(err_timeout) + + err_not_found = map_exception_to_flight_error(ArtifactNotFoundError("s3://bucket/key")) + assert "NOT_FOUND" in str(err_not_found) + assert "s3://" not in str(err_not_found) + + err_capacity = map_exception_to_flight_error(CacheCapacityError("full")) + assert "RESOURCE_EXHAUSTED" in str(err_capacity) + + err_concurrency = map_exception_to_flight_error(ConcurrencyLimitError("busy")) + assert "RESOURCE_EXHAUSTED" in str(err_concurrency) + + err_internal = map_exception_to_flight_error(S3AccessError("internal trace /var/cache/secret")) + assert isinstance(err_internal, fl.FlightInternalError) + assert "INTERNAL" in str(err_internal) + assert "/var/cache/secret" not in str(err_internal) + + +def test_openapi_31_spec_is_valid() -> None: + from openapi_spec_validator import validate + + spec_path = Path(__file__).resolve().parents[2] / "api" / "openapi" / "openapi.yaml" + spec = yaml.safe_load(spec_path.read_text(encoding="utf-8")) + assert str(spec.get("openapi", "")).startswith("3.1") + validate(spec) + for endpoint in ("/healthz", "/ready", "/metrics", "/openapi.yaml", "/docs"): + assert endpoint in spec["paths"] diff --git a/test/unit/test_helm_manifests.py b/test/unit/test_helm_manifests.py new file mode 100644 index 0000000..7d8c300 --- /dev/null +++ b/test/unit/test_helm_manifests.py @@ -0,0 +1,126 @@ +"""Unit tests validating Helm chart structure, memory limits, and KEDA ScaledObject.""" + +from __future__ import annotations + +from pathlib import Path + +import yaml + +HELM_DIR = Path(__file__).resolve().parents[2] / "deployments" / "helm" + + +def test_helm_chart_and_values_conform_to_architecture_budgets() -> None: + chart = yaml.safe_load((HELM_DIR / "Chart.yaml").read_text(encoding="utf-8")) + assert chart["apiVersion"] == "v2" + assert chart["name"] == "rag-tabular-analytics-worker" + + values = yaml.safe_load((HELM_DIR / "values.yaml").read_text(encoding="utf-8")) + assert values["resources"]["requests"]["memory"] == "4Gi" + assert values["resources"]["limits"]["memory"] == "6Gi" + assert values["resources"]["requests"]["cpu"] == "2000m" + assert values["resources"]["limits"]["cpu"] == "4000m" + assert values["terminationGracePeriodSeconds"] >= 20 + assert values["cache"]["sizeLimit"] == "10Gi" + assert values["jemalloc"]["enabled"] is True + assert values["jemalloc"]["path"] == "/usr/local/lib/libjemalloc.so" + assert values["securityContext"]["runAsNonRoot"] is True + assert "ALL" in values["securityContext"]["capabilities"]["drop"] + + keda = values["keda"] + assert keda["enabled"] is True + assert keda["minReplicaCount"] == 4 + assert keda["maxReplicaCount"] == 24 + assert int(keda["prometheus"]["threshold"]) < int( + values["env"]["TABULAR_WORKER_MAX_CONCURRENT_QUERIES"] + ) + + +def test_helm_templates_contain_required_k8s_and_keda_directives() -> None: + deployment_tpl = (HELM_DIR / "templates" / "deployment.yaml").read_text(encoding="utf-8") + assert 'medium: ""' in deployment_tpl + assert "medium: Memory\n sizeLimit: 10Gi" not in deployment_tpl + assert "terminationGracePeriodSeconds" in deployment_tpl + assert "path: /healthz" in deployment_tpl + assert "path: /ready" in deployment_tpl + assert "LD_PRELOAD" in deployment_tpl + assert ".Values.jemalloc.enabled" in deployment_tpl + + service_tpl = (HELM_DIR / "templates" / "service.yaml").read_text(encoding="utf-8") + assert "name: grpc" in service_tpl + assert "name: http" in service_tpl + assert "clusterIP: None" in service_tpl + + scaledobject_tpl = (HELM_DIR / "templates" / "scaledobject.yaml").read_text(encoding="utf-8") + assert "apiVersion: keda.sh/v1alpha1" in scaledobject_tpl + assert "kind: ScaledObject" in scaledobject_tpl + assert "minReplicaCount:" in scaledobject_tpl + assert "maxReplicaCount:" in scaledobject_tpl + assert "sum(tabular_worker_active_queries{" in scaledobject_tpl + + +def test_dockerfile_and_dockerignore_harden_nonroot_jemalloc_runtime() -> None: + repo_root = Path(__file__).resolve().parents[2] + dockerfile = (repo_root / "deployments" / "docker" / "Dockerfile").read_text(encoding="utf-8") + assert "FROM python:3.12-slim AS builder" in dockerfile + assert "FROM python:3.12-slim AS runtime" in dockerfile + assert "libjemalloc2" in dockerfile + assert "/usr/local/lib/libjemalloc.so" in dockerfile + assert "LD_PRELOAD=/usr/local/lib/libjemalloc.so" in dockerfile + assert "USER 10001:10001" in dockerfile + assert "EXPOSE 8815 8080" in dockerfile + assert "HEALTHCHECK" in dockerfile + assert 'CMD ["python", "-m", "cmd.worker.main"]' in dockerfile + + dockerignore = (repo_root / ".dockerignore").read_text(encoding="utf-8") + assert ".venv" in dockerignore + assert ".git" in dockerignore + + +def test_helm_staging_and_prod_overlays_and_argocd_canary_manifests() -> None: + repo_root = Path(__file__).resolve().parents[2] + staging = yaml.safe_load((HELM_DIR / "values-staging.yaml").read_text(encoding="utf-8")) + assert staging["keda"]["minReplicaCount"] == 1 + assert staging["keda"]["maxReplicaCount"] == 6 + assert staging["env"]["TABULAR_WORKER_OTEL_ENABLED"] == "true" + assert staging["jemalloc"]["enabled"] is True + + prod = yaml.safe_load((HELM_DIR / "values-prod.yaml").read_text(encoding="utf-8")) + assert prod["keda"]["minReplicaCount"] == 4 + assert prod["keda"]["maxReplicaCount"] == 24 + assert prod["resources"]["requests"]["memory"] == "4Gi" + assert prod["resources"]["limits"]["memory"] == "6Gi" + assert prod["cache"]["sizeLimit"] == "10Gi" + assert prod["jemalloc"]["enabled"] is True + assert prod["jemalloc"]["path"] == "/usr/local/lib/libjemalloc.so" + + argocd_dir = repo_root / "deployments" / "argocd" + app_staging = yaml.safe_load((argocd_dir / "application-staging.yaml").read_text("utf-8")) + assert app_staging["kind"] == "Application" + assert "values-staging.yaml" in app_staging["spec"]["source"]["helm"]["valueFiles"] + + app_prod = yaml.safe_load((argocd_dir / "application-prod.yaml").read_text("utf-8")) + assert app_prod["kind"] == "Application" + assert "values-prod.yaml" in app_prod["spec"]["source"]["helm"]["valueFiles"] + + docs = list(yaml.safe_load_all((argocd_dir / "rollout-canary.yaml").read_text("utf-8"))) + kinds = {doc["kind"]: doc for doc in docs} + assert "AnalysisTemplate" in kinds + assert "Rollout" in kinds + + rollout = kinds["Rollout"] + assert rollout["spec"]["workloadRef"]["kind"] == "Deployment" + weights = [ + step["setWeight"] + for step in rollout["spec"]["strategy"]["canary"]["steps"] + if "setWeight" in step + ] + assert weights == [10, 25, 50, 100] + + analysis_text = (argocd_dir / "rollout-canary.yaml").read_text("utf-8") + for metric_name in ( + "tabular_worker_query_duration_seconds", + "tabular_worker_subprocess_recycles_total", + "tabular_worker_cache_hit_ratio", + "tabular_worker_active_queries", + ): + assert metric_name in analysis_text diff --git a/test/unit/test_settings.py b/test/unit/test_settings.py new file mode 100644 index 0000000..cbab83a --- /dev/null +++ b/test/unit/test_settings.py @@ -0,0 +1,148 @@ +"""Unit tests for typed 12-factor Settings, .env loading, and DuckDB memory budgets.""" + +from __future__ import annotations + +import logging +from pathlib import Path + +import pytest +from pydantic import ValidationError + +from internal.config.settings import ( + DEFAULT_DUCKDB_TOTAL_MEMORY_BYTES, + MIB_BYTES, + Settings, + get_settings, +) +from internal.telemetry.logging import JsonLogFormatter, configure_logging, get_logger + +_REPO_ROOT = Path(__file__).resolve().parents[2] + + +@pytest.fixture(autouse=True) +def _clear_tabular_worker_env( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + import os + + for key in list(os.environ): + if key.startswith("TABULAR_WORKER_"): + monkeypatch.delenv(key, raising=False) + monkeypatch.setitem(Settings.model_config, "env_file", tmp_path / ".env") + monkeypatch.chdir(tmp_path) + get_settings.cache_clear() + + +def test_default_memory_budget_per_worker() -> None: + settings = Settings(_env_file=None) + assert settings.POOL_SIZE == 4 + assert settings.MAX_TASKS_PER_CHILD == 100 + assert settings.DUCKDB_TOTAL_MEMORY_BYTES == DEFAULT_DUCKDB_TOTAL_MEMORY_BYTES + assert settings.worker_memory_limit_mib == 640 + assert settings.worker_memory_limit_str == "640MiB" + + +def test_floored_mib_budget_for_non_power_of_two_pool_size() -> None: + settings = Settings(_env_file=None, POOL_SIZE=3) + expected_mib = int(DEFAULT_DUCKDB_TOTAL_MEMORY_BYTES / 3 // MIB_BYTES) + assert settings.worker_memory_limit_mib == expected_mib + assert settings.worker_memory_limit_str == f"{expected_mib}MiB" + assert settings.worker_memory_limit_mib * settings.POOL_SIZE * MIB_BYTES <= ( + DEFAULT_DUCKDB_TOTAL_MEMORY_BYTES + ) + + +def test_rejects_excessive_aggregate_duckdb_memory() -> None: + with pytest.raises(ValidationError): + Settings(_env_file=None, DUCKDB_TOTAL_MEMORY_BYTES=DEFAULT_DUCKDB_TOTAL_MEMORY_BYTES + 1) + + +def test_dotenv_values_affect_settings(tmp_path: Path) -> None: + env_path = tmp_path / ".env" + env_path.write_text( + "\n".join( + [ + "TABULAR_WORKER_SERVICE_NAME=custom-analytics-worker", + "TABULAR_WORKER_LOG_LEVEL=DEBUG", + "TABULAR_WORKER_POOL_SIZE=2", + "TABULAR_WORKER_FLIGHT_PORT=9915", + "TABULAR_WORKER_S3_BUCKET_ALLOWLIST=bucket-a,bucket-b", + "TABULAR_WORKER_S3_ENDPOINT_URL=", + "TABULAR_WORKER_WORKER_PROCESS_RLIMIT_BYTES=", + "TABULAR_WORKER_OTEL_ENABLED=true", + ] + ), + encoding="utf-8", + ) + + first = get_settings() + assert first.SERVICE_NAME == "custom-analytics-worker" + assert first.LOG_LEVEL == "DEBUG" + assert first.POOL_SIZE == 2 + assert first.FLIGHT_PORT == 9915 + assert first.S3_BUCKET_ALLOWLIST == ["bucket-a", "bucket-b"] + assert first.S3_ENDPOINT_URL is None + assert first.WORKER_PROCESS_RLIMIT_BYTES is None + assert first.OTEL_ENABLED is True + assert first.worker_memory_limit_mib == 1280 + + # Updating .env changes subsequent Settings loads + env_path.write_text( + "\n".join( + [ + "TABULAR_WORKER_POOL_SIZE=8", + 'TABULAR_WORKER_S3_BUCKET_ALLOWLIST=["bucket-c"]', + ] + ), + encoding="utf-8", + ) + get_settings.cache_clear() + updated = get_settings() + assert updated.POOL_SIZE == 8 + assert updated.S3_BUCKET_ALLOWLIST == ["bucket-c"] + assert updated.worker_memory_limit_mib == 320 + + +def test_os_env_overrides_dotenv(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + (tmp_path / ".env").write_text("TABULAR_WORKER_POOL_SIZE=2\n", encoding="utf-8") + monkeypatch.setenv("TABULAR_WORKER_POOL_SIZE", "6") + + settings = Settings() + assert settings.POOL_SIZE == 6 + + +def test_env_example_covers_all_settings_fields_and_loads_cleanly() -> None: + env_example_path = _REPO_ROOT / ".env.example" + assert env_example_path.is_file() + + defined_keys: set[str] = set() + for raw_line in env_example_path.read_text(encoding="utf-8").splitlines(): + line = raw_line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key = line.split("=", 1)[0].strip() + defined_keys.add(key) + + expected_keys = {f"TABULAR_WORKER_{field}" for field in Settings.model_fields} + missing = expected_keys - defined_keys + assert not missing, f"Missing variables in .env.example: {sorted(missing)}" + + loaded = Settings(_env_file=env_example_path) + assert loaded.POOL_SIZE == 4 + assert loaded.S3_BUCKET_ALLOWLIST == [] + assert loaded.S3_ENDPOINT_URL is None + + +def test_configure_logging_applies_level_and_service_name() -> None: + logger = get_logger("internal.test_env_logging") + try: + configure_logging(level="DEBUG", service_name="custom-worker") + assert logger.level == logging.DEBUG + assert any( + isinstance(handler.formatter, JsonLogFormatter) + and handler.formatter._service_name == "custom-worker" + for handler in logger.handlers + ) + finally: + configure_logging(level="INFO", service_name="rag-tabular-analytics-worker") diff --git a/test/unit/test_sync_env_to_deployments.py b/test/unit/test_sync_env_to_deployments.py new file mode 100644 index 0000000..e1c54f4 --- /dev/null +++ b/test/unit/test_sync_env_to_deployments.py @@ -0,0 +1,114 @@ +"""Unit tests for scripts/sync_env_to_deployments.py and Helm ConfigMap/checksum wiring.""" + +from __future__ import annotations + +from pathlib import Path + +import yaml + +from scripts.sync_env_to_deployments import main as sync_main + +REPO_ROOT = Path(__file__).resolve().parents[2] +HELM_DIR = REPO_ROOT / "deployments" / "helm" + + +def test_sync_and_check_only_preserve_blocks_and_secret_name(tmp_path: Path) -> None: + env_file = tmp_path / ".env.prod" + env_file.write_text( + "\n".join( + [ + "# comment", + "TABULAR_WORKER_LOG_LEVEL=INFO # inline comment", + "TABULAR_WORKER_POOL_SIZE=8", + 'TABULAR_WORKER_QUOTED_HASH="literal#hash" # trailing comment', + "TABULAR_WORKER_SINGLE_QUOTED_HASH='single#hash'", + r'TABULAR_WORKER_S3_BUCKET_ALLOWLIST="[\"bucket-a\",\"bucket-b\"]"', + "TABULAR_WORKER_OTEL_ENABLED=true", + "TABULAR_WORKER_OTEL_EXPORTER_ENDPOINT=http://otel:4317", + "AWS_SECRET_ACCESS_KEY=super-secret", + "DB_PASSWORD=hunter2", + "API_TOKEN=tok_123", + "SIGNING_KEY=key_abc", + "OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer secret", + ] + ) + + "\n", + encoding="utf-8", + ) + + values_file = tmp_path / "values.yaml" + values_file.write_text( + "\n".join( + [ + "replicaCount: 4", + "image:", + " repository: example/worker", + ' tag: "1.0.0"', + "env:", + ' secretName: "my-k8s-secret"', + ' TABULAR_WORKER_LOG_LEVEL: "DEBUG"', + ' OTEL_EXPORTER_OTLP_HEADERS: "stale-secret"', + "keda:", + " enabled: true", + ] + ) + + "\n", + encoding="utf-8", + ) + + # Drift detected before sync + assert ( + sync_main(["--check-only", "--env-file", str(env_file), "--values-file", str(values_file)]) + == 1 + ) + + # Sync updates values.yaml + assert ( + sync_main(["--sync", "--env-file", str(env_file), "--values-file", str(values_file)]) == 0 + ) + + # Zero drift after sync + assert ( + sync_main(["--check-only", "--env-file", str(env_file), "--values-file", str(values_file)]) + == 0 + ) + + updated = yaml.safe_load(values_file.read_text(encoding="utf-8")) + assert updated["replicaCount"] == 4 + assert updated["image"] == {"repository": "example/worker", "tag": "1.0.0"} + assert updated["keda"] == {"enabled": True} + assert updated["env"] == { + "secretName": "my-k8s-secret", + "TABULAR_WORKER_LOG_LEVEL": "INFO", + "TABULAR_WORKER_POOL_SIZE": "8", + "TABULAR_WORKER_QUOTED_HASH": "literal#hash", + "TABULAR_WORKER_SINGLE_QUOTED_HASH": "single#hash", + "TABULAR_WORKER_S3_BUCKET_ALLOWLIST": '["bucket-a","bucket-b"]', + "TABULAR_WORKER_OTEL_ENABLED": "true", + "TABULAR_WORKER_OTEL_EXPORTER_ENDPOINT": "http://otel:4317", + } + + +def test_helm_configmap_and_deployment_checksum_templates() -> None: + configmap_tpl = (HELM_DIR / "templates" / "configmap.yaml").read_text(encoding="utf-8") + assert "kind: ConfigMap" in configmap_tpl + assert 'default "8080" (get .Values.env "PORT")' in configmap_tpl + assert 'default "50051" (get .Values.env "GRPC_PORT")' in configmap_tpl + assert 'ne $key "secretName"' in configmap_tpl + assert "ne $val nil" in configmap_tpl + assert "range $key, $val := .Values.env" in configmap_tpl + + deployment_tpl = (HELM_DIR / "templates" / "deployment.yaml").read_text(encoding="utf-8") + expected_checksum = ( + 'checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}' + ) + assert expected_checksum in deployment_tpl + assert "configMapRef:" in deployment_tpl + assert "secretRef:" in deployment_tpl + + secret_tpl = (HELM_DIR / "templates" / "secret.yaml").read_text(encoding="utf-8") + assert "kind: Secret" in secret_tpl + assert ".Values.secrets.create" in secret_tpl + + values = yaml.safe_load((HELM_DIR / "values.yaml").read_text(encoding="utf-8")) + assert values["env"]["secretName"] == "rag-tabular-analytics-worker-secrets" diff --git a/test/unit/test_table_resolver.py b/test/unit/test_table_resolver.py new file mode 100644 index 0000000..ee8d832 --- /dev/null +++ b/test/unit/test_table_resolver.py @@ -0,0 +1,86 @@ +"""Unit tests for TableResolver tenant isolation, bucket allowlist, and path validation.""" + +from __future__ import annotations + +import hashlib + +import pytest + +from internal.storage import TableResolver, TableValidationError, TenantIsolationError + +VALID_HASH = hashlib.sha256(b"sample-parquet").hexdigest() + + +def test_resolve_valid_canonical_artifact() -> None: + resolver = TableResolver(["rag-tabular-artifacts"]) + resolved = resolver.resolve( + tenant_id="tenant-corp-01", + table_hash=VALID_HASH, + s3_bucket="rag-tabular-artifacts", + s3_key=f"artifacts/tenant-corp-01/{VALID_HASH}.parquet", + ) + assert resolved.tenant_id == "tenant-corp-01" + assert resolved.table_hash == VALID_HASH + assert resolved.s3_bucket == "rag-tabular-artifacts" + assert resolved.relative_cache_path.parts == ( + "rag-tabular-artifacts", + "tenant-corp-01", + f"{VALID_HASH}.parquet", + ) + + +def test_empty_allowlist_fails_closed() -> None: + resolver = TableResolver([]) + with pytest.raises(TenantIsolationError): + resolver.resolve( + tenant_id="tenant-corp-01", + table_hash=VALID_HASH, + s3_bucket="rag-tabular-artifacts", + s3_key=f"artifacts/tenant-corp-01/{VALID_HASH}.parquet", + ) + + +def test_unallowlisted_bucket_raises_tenant_isolation_error() -> None: + resolver = TableResolver(["allowed-bucket"]) + with pytest.raises(TenantIsolationError): + resolver.resolve( + tenant_id="tenant-corp-01", + table_hash=VALID_HASH, + s3_bucket="other-bucket", + s3_key=f"artifacts/tenant-corp-01/{VALID_HASH}.parquet", + ) + + +def test_cross_tenant_s3_key_raises_tenant_isolation_error() -> None: + resolver = TableResolver(["rag-tabular-artifacts"]) + with pytest.raises(TenantIsolationError): + resolver.resolve( + tenant_id="tenant-a", + table_hash=VALID_HASH, + s3_bucket="rag-tabular-artifacts", + s3_key=f"artifacts/tenant-b/{VALID_HASH}.parquet", + ) + + +@pytest.mark.parametrize( + ("tenant_id", "table_hash"), + [ + ("../escape", VALID_HASH), + ("tenant/sub", VALID_HASH), + ("", VALID_HASH), + ("tenant-a", "not-a-sha256"), + ("tenant-a", VALID_HASH.upper()), + ], +) +def test_malformed_identifiers_raise_table_validation_error( + tenant_id: str, + table_hash: str, +) -> None: + resolver = TableResolver(["rag-tabular-artifacts"]) + with pytest.raises(TableValidationError): + resolver.resolve( + tenant_id=tenant_id, + table_hash=table_hash, + s3_bucket="rag-tabular-artifacts", + s3_key=f"artifacts/{tenant_id}/{table_hash}.parquet", + ) diff --git a/test/unit/test_tracing.py b/test/unit/test_tracing.py new file mode 100644 index 0000000..994269e --- /dev/null +++ b/test/unit/test_tracing.py @@ -0,0 +1,193 @@ +"""Unit tests for OpenTelemetry tracing, W3C traceparent extraction, and log correlation.""" + +from __future__ import annotations + +import json +import logging + +import pytest +from opentelemetry.sdk.trace import TracerProvider +from opentelemetry.sdk.trace.export import SimpleSpanProcessor +from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter + +from internal.config.settings import Settings +from internal.telemetry.logging import REDACTED, JsonLogFormatter +from internal.telemetry.tracing import ( + extract_trace_context, + get_current_trace_id, + get_tracer, + init_tracing, + record_sanitized_exception, + sanitize_span_attributes, + shutdown_tracing, + start_span, +) +from internal.validator.exceptions import DisallowedFunctionError + + +def test_init_tracing_disabled_is_noop() -> None: + provider = init_tracing(Settings(OTEL_ENABLED=False)) + assert provider is None + with start_span("noop.span", attributes={"request_id": "req_noop"}): + assert get_current_trace_id() is None + + +def test_traceparent_extraction_and_log_correlation(monkeypatch: pytest.MonkeyPatch) -> None: + exporter = InMemorySpanExporter() + provider = TracerProvider() + provider.add_span_processor(SimpleSpanProcessor(exporter)) + monkeypatch.setattr("internal.telemetry.tracing.get_tracer", provider.get_tracer) + try: + expected_trace_id = "4bf92f3577b34da6a3ce929d0e0e4736" + traceparent = f"00-{expected_trace_id}-00f067aa0ba902b7-01" + ctx = extract_trace_context({"traceparent": [traceparent]}) + + formatter = JsonLogFormatter() + with start_span("flight.do_get", context=ctx, attributes={"request_id": "req_01"}): + assert get_current_trace_id() == expected_trace_id + record = logging.LogRecord( + name="test.logger", + level=logging.INFO, + pathname=__file__, + lineno=1, + msg="Query executed", + args=(), + exc_info=None, + ) + payload = json.loads(formatter.format(record)) + assert payload["trace_id"] == expected_trace_id + + explicit_record = logging.LogRecord( + name="test.logger", + level=logging.INFO, + pathname=__file__, + lineno=1, + msg="Explicit trace", + args=(), + exc_info=None, + ) + explicit_record.trace_id = "custom-trace-id" + explicit_payload = json.loads(formatter.format(explicit_record)) + assert explicit_payload["trace_id"] == "custom-trace-id" + + spans = exporter.get_finished_spans() + assert len(spans) == 1 + assert format(spans[0].context.trace_id, "032x") == expected_trace_id + assert spans[0].attributes is not None + assert spans[0].attributes["request_id"] == "req_01" + finally: + provider.shutdown() + shutdown_tracing() + + +def test_span_attributes_and_exceptions_redact_sql_and_secrets() -> None: + attrs = sanitize_span_attributes( + { + "request_id": "req_02", + "sql": "SELECT * FROM tbl WHERE ssn = '123-45-6789'", + "raw_sql": "SELECT secret FROM tbl", + "authorization": "Bearer supersecrettoken123", + "url": "https://s3.amazonaws.com/b/k?X-Amz-Signature=deadbeef&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE", + } + ) + assert attrs["request_id"] == "req_02" + assert attrs["sql"] == REDACTED + assert attrs["raw_sql"] == REDACTED + assert attrs["authorization"] == REDACTED + assert "deadbeef" not in str(attrs["url"]) + assert "AKIAIOSFODNN7EXAMPLE" not in str(attrs["url"]) + assert REDACTED in str(attrs["url"]) + + exporter = InMemorySpanExporter() + provider = TracerProvider() + provider.add_span_processor(SimpleSpanProcessor(exporter)) + tracer = provider.get_tracer("test") + try: + with tracer.start_as_current_span("err.span") as span: + record_sanitized_exception( + span, + RuntimeError("Failed with Bearer mytoken123 and ?X-Amz-Signature=abc123"), + ) + with tracer.start_as_current_span("rule.span") as rule_span: + record_sanitized_exception( + rule_span, + DisallowedFunctionError("read_csv_auto('secret') is banned"), + ) + spans = exporter.get_finished_spans() + assert len(spans) == 2 + msg = str(spans[0].attributes["exception.message"]) # type: ignore[index] + assert msg == "RuntimeError" + assert "mytoken123" not in msg + assert "abc123" not in msg + + rule_attrs = spans[1].attributes + assert rule_attrs is not None + assert rule_attrs["exception.type"] == "DisallowedFunctionError" + assert rule_attrs["error.rule"] == "DISALLOWED_FUNCTION" + assert rule_attrs["exception.message"] == "DisallowedFunctionError:DISALLOWED_FUNCTION" + assert "read_csv_auto" not in str(rule_attrs["exception.message"]) + finally: + provider.shutdown() + + +def test_start_span_disables_automatic_exception_events_with_credentials( + monkeypatch: pytest.MonkeyPatch, +) -> None: + exporter = InMemorySpanExporter() + provider = TracerProvider() + provider.add_span_processor(SimpleSpanProcessor(exporter)) + monkeypatch.setattr("internal.telemetry.tracing.get_tracer", provider.get_tracer) + try: + secret_msg = ( + "Auth failed: Bearer supersecrettoken999 " + "https://s3.amazonaws.com/b/k?X-Amz-Signature=deadbeef&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE" + ) + with pytest.raises(RuntimeError), start_span("flight.do_get"): + raise RuntimeError(secret_msg) + + spans = exporter.get_finished_spans() + assert len(spans) == 1 + assert spans[0].events == () + attrs = spans[0].attributes + assert attrs is not None + assert attrs["exception.type"] == "RuntimeError" + assert attrs["exception.message"] == "RuntimeError" + assert "supersecrettoken999" not in str(attrs) + assert "deadbeef" not in str(attrs) + finally: + provider.shutdown() + + +def test_init_tracing_enabled_start_stop_start_lifecycle() -> None: + settings = Settings( + OTEL_ENABLED=True, + OTEL_EXPORTER_ENDPOINT=None, + OTEL_SERVICE_NAME="tabular-worker-test", + ) + try: + provider_1 = init_tracing(settings) + assert provider_1 is not None + exporter_1 = InMemorySpanExporter() + provider_1.add_span_processor(SimpleSpanProcessor(exporter_1)) + + with start_span("lifecycle.first", attributes={"request_id": "req_1"}): + assert get_current_trace_id() is not None + assert len(exporter_1.get_finished_spans()) == 1 + + shutdown_tracing() + + provider_2 = init_tracing(settings) + assert provider_2 is not None + assert provider_2 is not provider_1 + exporter_2 = InMemorySpanExporter() + provider_2.add_span_processor(SimpleSpanProcessor(exporter_2)) + + tracer_2 = get_tracer("test.lifecycle") + assert tracer_2 is not None + with start_span("lifecycle.second", attributes={"request_id": "req_2"}): + assert get_current_trace_id() is not None + + assert len(exporter_2.get_finished_spans()) == 1 + assert len(exporter_1.get_finished_spans()) == 1 + finally: + shutdown_tracing()