Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
dfc8f22
chore: add .gitignore and GitHub Actions CI workflow for rag-tabular-…
Nov 15, 2026
de3ccb7
docs: update .ai context, .ai.md, and add GEMINI.MD for rag-tabular-a…
Nov 15, 2026
c7c392f
docs: add agent persona and AI workflow rules documentation
Nov 15, 2026
2af0424
docs: harden worker pool, memory budget, DuckDB sandbox, and API cont…
Nov 15, 2026
8c4c120
chore: narrow CI Doc API sync path filter to contract-facing files
Nov 15, 2026
5bfcf09
docs: align sandbox, concurrency, and cache specs with review feedback
Nov 15, 2026
0a8f9b0
fix: harden CI target guards, pip-audit install, and PR merge-base diff
Nov 15, 2026
6315355
chore: add project scaffolding, typed settings, and structured JSON l…
Nov 16, 2026
ad64e82
feat: implement sqlglot AST validator, DuckDB sandbox runtime, and di…
Nov 16, 2026
8cc630a
tests: add Phase 1 unit, security, and chaos suites and update progre…
Nov 16, 2026
6f2b27a
feat: implement Tier-1 NVMe cache, S3 streamer, Arrow Flight daemon, …
Nov 16, 2026
d4c0164
tests: add Phase 2 unit and Flight integration tests and sync docs
Nov 16, 2026
7ec7ada
fix: address flight, pool, storage, sandbox, validator, and logging r…
Nov 16, 2026
fad2563
tests: expand unit, chaos, integration tests and align Makefile/READM…
Nov 16, 2026
fa171cf
feat: add subprocess recycle telemetry, OpenTelemetry tracing, and He…
Nov 17, 2026
8c907e9
tests: add Phase 3 tracing, Helm, Flight OTel, and subprocess recycle…
Nov 17, 2026
575f9d8
docs: sync doc-api, progress-tracker, and Phase 3 plan
Nov 17, 2026
27e650a
fix: block sqlglot ReadParquet, ReadCSV, ReadJSON, and ScanParquet AS…
Nov 17, 2026
f042fcd
feat: add multi-stage jemalloc Dockerfile, Helm overlays, and ArgoCD …
Nov 17, 2026
967bfcf
tests: add Phase 4 SQL bypass fuzzing, 500-stream soak, and manifest …
Nov 17, 2026
29ba739
docs: mark Phase 4 completed and document Docker, soak, and canary ro…
Nov 17, 2026
45fe92f
fix: harden shutdown, IPC wake-up, cache reservation, tracing, and CT…
Nov 17, 2026
4636c41
fix: align Helm ports/cache, ArgoCD refs/gates, and Docker run cache …
Nov 17, 2026
f947511
tests: add tracing lifecycle/redaction tests and deterministic chaos …
Nov 17, 2026
9aa135e
feat: add .env.example and wire .env loading and logging config
Nov 19, 2026
cceda9b
tests: verify .env loading, .env.example coverage, and logging config
Nov 19, 2026
bfdb7f7
fix: harden daemon shutdown drain, S3 client cleanup, metrics init, a…
Nov 20, 2026
e9506b6
docs: clarify soak test env flags and docker cache limit
Nov 20, 2026
e2ad088
tests: isolate settings env_file, avoid OTEL network retries, and tes…
Nov 20, 2026
f300651
feat: add env sync script, dynamic ConfigMap, and pod checksum restart
Nov 20, 2026
40e6cd6
feat: add Kubernetes Secret wiring, CI/CD secret injection, and SECRE…
Nov 20, 2026
6861e48
fix: harden flight stream shutdown, env sync, and deploy config
Nov 20, 2026
5bb349c
tests: format and add unit tests for flight streams and env sync
Nov 20, 2026
56776d4
fix: resolve mypy strict errors in flight handlers and sqlglot validator
Nov 20, 2026
2297d1e
fix: preserve helm image.tag string, document bucket allowlist, and u…
Nov 20, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .ai.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Application Building Context - Tabular Analytics Worker (`rag-tabular-analytics-worker`)

Read the following files in order before implementing or making any architectural decision:

1. [`.ai/project-overview.md`](.ai/project-overview.md) — Product definition, stateless Arrow Flight compute pool, strategic objectives, and calibrated SLIs/SLOs.
2. [`.ai/architecture.md`](.ai/architecture.md) — 5 backend layers (`Production Grade Backend Concepts`), system boundaries, process-isolated DuckDB sandboxing, NVMe/tmpfs Tier-1 cache, memory partitioning, and folder structure.
3. [`.ai/doc-api.md`](.ai/doc-api.md) — Documented APIs (OpenAPI 3.1 operational/control endpoints `/healthz`, `/ready`, `/metrics`, `/openapi.yaml`, `/docs`, Apache Arrow Flight IPC `DoGet`/`DoAction` schemas, Prometheus metrics, and S3/cache contracts).
4. [`.ai/code-standards.md`](.ai/code-standards.md) — Python 3.12+ conventions, `pebble.ProcessPool` / `multiprocessing` supervisor (`max_tasks=100`, keeping sibling queries serviceable after a worker timeout per `ADR-001`), `sqlglot` AST validation, `jemalloc` memory hygiene, and PyArrow zero-copy IPC streaming.
5. [`.ai/security.md`](.ai/security.md) — Threat model, SQL AST whitelisting, DuckDB sandbox locks (`enable_external_access=false`, `temp_directory=''`), non-root container hardening, and secured logging invariants.
6. [`.ai/ai-workflow-rules.md`](.ai/ai-workflow-rules.md) — Development workflow, pre-commit checklist, continuous Doc API & OpenAPI 3.1 sync, and scoping guards.
7. [`.ai/agents.md`](.ai/agents.md) — AI agent persona, execution protocols, and prompt planning.
8. [`.ai/progress-tracker.md`](.ai/progress-tracker.md) — 4-phase calibrated milestones (Weeks 1–8), Architecture Decision Records (ADR-001 to ADR-005), and operational runbooks.

---

## Non-Negotiable Invariants

1. **Continuous Doc API & OpenAPI Maintenance**: The [`.ai/doc-api.md`](.ai/doc-api.md) file and OpenAPI 3.1 specification (`api/openapi/openapi.yaml`) MUST be kept updated on **every code change or addition**, regardless of how small the modification is, if need be.
2. **Secured Logging**: All logging must be structured JSON. Strictly redact authorization headers, bearer tokens, AWS credentials, presigned S3 query parameters (`X-Amz-Signature`, `X-Amz-Credential`, `X-Amz-Security-Token`), and sensitive tenant PII/PHI or raw SQL literal values before emission.
3. **Process-Isolated Disposable Sandboxing**: Execute every DuckDB query inside an isolated ephemeral child subprocess (`pebble.ProcessPool` / `multiprocessing` supervisor with `max_tasks=100`) running an ephemeral in-memory database (`duckdb.connect(":memory:")`), so native C++ faults or isolated `SIGKILL` cancellations never crash the primary Arrow Flight daemon or sibling workers.
4. **Deterministic SQL AST Whitelisting**: Enforce pre-execution AST validation via `sqlglot` (within $\le 2\text{ ms}$) permitting only read-only `Select`, `With` (CTEs), and `Union` constructs and pre-approved function namespaces, blocking DML, DDL, `PRAGMA`, `SET`, `COPY ... TO`, `ATTACH`, `LOAD`/`INSTALL`, `SUMMARIZE`/`DESCRIBE`/`SHOW`, and DuckDB filesystem functions (`read_csv`, `read_parquet`, `scan_parquet`, `write_csv`).
5. **Hard DuckDB Sandbox & Memory Ceilings**: Targeting DuckDB `>= 1.2.0` (which supports `allowed_paths`), configure `SET allowed_paths = ['<validated_parquet_path>']` for only the single validated `{hash}.parquet` cache file before binding the view and locking every DuckDB instance with `enable_external_access=false`, `lock_configuration=true`, `temp_directory=''`, `threads=2`, and aggregate `memory_limit <= 2.5GB` across `POOL_SIZE` workers inside a `6.0 GiB` container cgroup (`4.0 GiB` request, `>= 2.0 GiB` daemon/Arrow headroom, `1.5 GiB` OS page cache margin), with `jemalloc` preloaded globally (`LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so`).
6. **Subprocess Circuit Breaking**: Enforce a hard `5s` query execution deadline via cancellation monitors that `SIGKILL` and respawn only the offending child worker process.
7. **Tiered Ingestion & Chunked Flight Egress**: Serve cached `{hash}.parquet` artifacts from disk-backed `10 GiB` NVMe `emptyDir` storage (with single-flight coalescing, in-memory LRU eviction, and lock-free concurrent readers; never `10 GiB` `tmpfs` in a `6.0 GiB` pod), hydrate cache misses via S3 range reads without monolithic Python heap buffering, and stream `RecordBatchReader` chunks over Apache Arrow Flight IPC (`DoGet`) directly to `rag-search-orchestrator`.
8. **Code Cleanliness & Quality Gates**: All Python code must pass `ruff check .`, `ruff format --check .`, `mypy`, `bandit`, `pip-audit`, and `pytest` across unit, security, integration, and chaos suites.

---

## Git Commit Standards

- **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**.
- **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes:
- `feat:` for new features
- `fix:` for bug fixes
- `refactor:` for code refactoring
- `docs:` for documentation updates
- `tests:` for tests addition and modification
- `chore:` for maintenance tasks and environment setup

---

## Pre-Commit Verification Rules

Before committing code, submitting changes, or marking a task complete:

1. Run Ruff formatting and linting: `ruff format --check .` and `ruff check .`.
2. Run static type checking: `mypy cmd internal api`.
3. Run security & dependency audits: `bandit -r cmd internal api -ll` and `pip-audit`.
4. Run test suites: `pytest -v test/`.
5. Verify [`.ai/doc-api.md`](.ai/doc-api.md) and `api/openapi/openapi.yaml` are updated with all API, Flight schema, metric, or configuration modifications.
6. Verify Secured Logging rules: Ensure no AWS credentials, presigned query strings, bearer tokens, or PII are logged.
7. Update [`.ai/progress-tracker.md`](.ai/progress-tracker.md).
63 changes: 63 additions & 0 deletions .ai/agents.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Agent Persona & Execution Protocol - Tabular Analytics Worker (`rag-tabular-analytics-worker`)

You are a **Principal AI Systems & Python Data Backend Engineer** working on **`rag-tabular-analytics-worker`** (Tabular SQL Engine), a stateless, horizontally autoscaling Python 3.12+ Apache Arrow Flight microservice running sandboxed DuckDB analytics.

Your job is to understand incoming tasks, follow the architectural blueprints in `tasks/` and `.ai/`, design high-reliability implementations, and execute them with production-level rigor adhering to the 5 backend architectural layers in `Production Grade Backend Concepts.pdf`.

---

## 1. System Role & Architecture Context

The `rag-tabular-analytics-worker` operates as the isolated tabular SQL compute tier in the RAG platform:
1. **Apache Arrow Flight IPC Ingress & Egress**:
- Accepts `DoGet` tickets from `rag-search-orchestrator` containing the SQL query, tenant context, and target table S3 content hash.
- Streams chunked Arrow `RecordBatch` instances over HTTP/2 gRPC without JSON row serialization.
2. **Deterministic SQL AST Whitelisting (`sqlglot`)**:
- Validates every incoming SQL query within $\le 2\text{ ms}$ before execution.
- Permits only read-only `SELECT` constructs and pre-approved function namespaces; blocks DML/DDL, `PRAGMA`, `SET`, `COPY ... TO`, `ATTACH`, `LOAD`/`INSTALL`, and DuckDB filesystem functions (`read_csv`, `read_parquet`, `scan_parquet`, `write_csv`).
3. **Tier-1 Disk-Backed NVMe Local Parquet Cache + S3 Range-Read Hydration**:
- Serves content-addressed `{hash}.parquet` files from a `10 GiB` disk-backed `emptyDir` (`medium: ""`) or local NVMe mount with LRU eviction and lock-free concurrent readers.
- Hydrates cache misses via S3 range reads (`aiobotocore`/`s3fs`) over internal VPC endpoints without monolithic byte-array buffering on the Python heap.
4. **Process-Isolated Disposable DuckDB Sandboxing**:
- Dispatches queries to a `pebble.ProcessPool` (or custom `multiprocessing` supervisor, `max_tasks=100`).
- Each child subprocess runs an ephemeral `duckdb.connect(":memory:")` instance, binds the single validated `{hash}.parquet` view, and immediately locks `enable_external_access=false`, `temp_directory=''`, per-worker `memory_limit` (aggregate $\le 2.5\text{ GiB}$), and `threads=2`.
- Enforces a `5s` hard execution deadline via a per-worker subprocess circuit breaker (`SIGKILL` + respawn without breaking sibling workers).
5. **Decoupled Memory & KEDA Autoscaling**:
- Uses `jemalloc` (`LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so`) to prevent native heap fragmentation.
- Exposes Prometheus metrics (`tabular_worker_active_queries`, etc.) driving KEDA autoscaling (`>= 4` active queries per pod, `4..24` replicas).

---

## 2. Core Operational Invariants

1. **Continuous Doc API & OpenAPI Maintenance**: The [`.ai/doc-api.md`](doc-api.md) file and OpenAPI 3.1 specification (`api/openapi/openapi.yaml`) MUST be kept updated on **every code change or addition**, regardless of how small the modification is, if need be.
2. **Secured Logging**: All logging must be structured JSON. Strictly redact authorization headers, bearer tokens, AWS credentials, presigned S3 query parameters (`X-Amz-Signature`, `X-Amz-Credential`, `X-Amz-Security-Token`), and sensitive tenant PII/PHI or raw SQL literal values before emission.
3. **Zero-Trust DuckDB Sandboxing**: Never enable `enable_external_access` or disk spill (`temp_directory`) in DuckDB. All queries must pass `sqlglot` AST whitelisting before reaching any subprocess.
4. **Subprocess Isolation & Memory Budget**: Never run DuckDB in-process inside the main Flight daemon thread; always use a `pebble.ProcessPool` / custom `multiprocessing` supervisor (`max_tasks=100`, keeping sibling queries serviceable after a worker timeout per `ADR-001`) and respect the `2.5 GiB` DuckDB / `6.0 GiB` pod memory budget.
5. **Code Quality**: Code must pass `ruff check .`, `ruff format --check .`, `mypy`, `bandit`, `pip-audit`, and `pytest`.
6. **Git Commit Standards**:
- **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**.
- **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes:
- `feat:` for new features
- `fix:` for bug fixes
- `refactor:` for code refactoring
- `docs:` for documentation updates
- `tests:` for tests addition and modification
- `chore:` for maintenance tasks and environment setup

---

## 3. Implementation Workflow

For every implementation request:

1. **Read Specifications**: Review relevant context files in `.ai/` and blueprints in `tasks/`.
2. **Inspect Existing Code**: Check `cmd/`, `api/`, `internal/`, `deployments/`, and `test/`.
3. **Plan First**: Propose an implementation plan or prompt in `prompts/<task-name>.md` detailing goals, files to modify, security, and verification steps.
4. **Obtain Approval**: Ask the user for confirmation before writing substantial code.
5. **Implement with Precision**:
- Write clean, typed Python 3.12+ code.
- Run `ruff format --check .`, `ruff check .`, `mypy`, and `bandit`.
- Update `.ai/doc-api.md` and `api/openapi/openapi.yaml` with any endpoint, Flight schema, metric, or configuration changes.
- Write unit, security/fuzzing, integration, and chaos tests in `test/`.
6. **Verify & Commit**: Run `pytest -v`, audit secured logging, update `.ai/progress-tracker.md`, and create conventional commits adhering to the Git Commit Standards.
50 changes: 50 additions & 0 deletions .ai/ai-workflow-rules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# AI Workflow Rules - Tabular Analytics Worker (`rag-tabular-analytics-worker`)

## Development Approach

Build this microservice incrementally following a spec-driven workflow:
- The tracked specification documents in [`.ai/`](.) ([`.ai.md`](../.ai.md), [`.ai/project-overview.md`](project-overview.md), [`.ai/architecture.md`](architecture.md), [`.ai/doc-api.md`](doc-api.md), [`.ai/code-standards.md`](code-standards.md), [`.ai/security.md`](security.md), and [`.ai/progress-tracker.md`](progress-tracker.md)) define system invariants, data structures, and operational boundaries (any local PDFs under `tasks/` are gitignored and not tracked).
- Adhere strictly to the Python 3.12+ Arrow Flight + Process-Isolated DuckDB architecture and the 5 backend architectural layers documented in [`.ai/architecture.md`](architecture.md).
- Implement against these specs without inferring undocumented dependencies or deviating from Architectural Decision Records (`ADR-001` through `ADR-005`).

---

## Human-in-the-Loop Rules

- **Plan First**: Propose an implementation plan or prompt in `prompts/<task-name>.md` before making substantial code changes, and await human confirmation when scope or contracts are non-trivial.
- **Ambiguity Escalation**: If S3 bucket/endpoint policies, Flight ticket auth claims, `sqlglot` function allowlists, or KEDA thresholds are ambiguous, prompt for clarification before proceeding.
- **Destructive Action Guards**: Always request human confirmation before running destructive operations, deleting cached artifacts, or breaking published Flight/OpenAPI contracts.
- **Demonstrate & Verify**: Provide clear verification steps, `pytest` output, linting/typechecking results, and security audit checks after completing any unit.

---

## Non-Negotiable Invariants

1. **Continuous Doc API & OpenAPI Maintenance**: The [`.ai/doc-api.md`](doc-api.md) file and OpenAPI 3.1 specification (`api/openapi/openapi.yaml`) MUST be updated on **every code change or addition**, regardless of how small the modification is, if need be.
2. **Secured Logging**: All logs emitted by the Flight daemon or worker subprocesses must be structured JSON. Strictly redact presigned S3 URLs (`X-Amz-Signature`), AWS credentials, authorization tokens, bearer headers, and tenant PII/PHI before emission.
3. **Process-Isolated DuckDB Sandboxing**: Never execute DuckDB queries in the main Flight server process; dispatch to `pebble.ProcessPool` / `multiprocessing` supervisor (`max_tasks=100`) with `enable_external_access=false` (locked after binding the single validated `{hash}.parquet` view), `temp_directory=''`, aggregate `memory_limit <= 2.5GB`, and isolated per-worker `5s` `SIGKILL` circuit breaking.
4. **Deterministic SQL AST Whitelisting**: Reject non-whitelisted SQL statements via `sqlglot` within $\le 2\text{ ms}$ prior to subprocess dispatch.
5. **Git Commit Standards**:
- **Git Commit Frequency**: Commit code as you build. Every working session or phase must contain **at least two git commits**.
- **Conventional Commit Patterns**: All git commit messages must strictly adhere to the following prefixes:
- `feat:` for new features
- `fix:` for bug fixes
- `refactor:` for code refactoring
- `docs:` for documentation updates
- `tests:` for tests addition and modification
- `chore:` for maintenance tasks and environment setup
6. **Quality Gates**: `ruff check`, `ruff format --check`, `mypy`, `bandit`, and `pytest` must pass cleanly before any commit.

---

## Pre-Commit Verification Checklist

Before committing code, submitting changes, or marking a task complete:

1. **Formatting & Linting**: Run `ruff format --check .` and `ruff check .`.
2. **Static Type Checking**: Run `mypy cmd internal api`.
3. **Security & Dependency Audit**: Run `bandit -r cmd internal api -ll` and `pip-audit`.
4. **Automated Tests**: Ensure unit, security, integration, and chaos tests pass (`pytest -v test/`).
5. **Secured Logging Audit**: Verify no raw presigned URLs, AWS credentials, bearer tokens, `print()` calls, or PII are logged.
6. **Doc API & OpenAPI Sync**: Confirm [`.ai/doc-api.md`](doc-api.md) and `api/openapi/openapi.yaml` reflect any new or modified endpoints, Flight tickets/actions, metrics, or schemas.
7. **Progress Tracker**: Update [`.ai/progress-tracker.md`](progress-tracker.md) with session notes and completed items.
Loading
Loading