Stateless, horizontally autoscaling Python 3.12+ Apache Arrow Flight microservice executing sandboxed DuckDB analytical queries over content-addressed Parquet artifacts.
- Deterministic SQL AST Security Validator (
internal/validator/):sqlglotAST whitelisting (<= 2 mswarm steady-state) permitting only read-onlySELECT,WITH(CTEs),UNION/INTERSECT/EXCEPT, andSubqueryconstructs.- Strict allowlist for SQL functions (including
exp.Anonymousfiltering), column/table/CTE alias validation, join count limits (MAX_JOINS), and query length limits (MAX_SQL_LENGTH). - Blocks DML/DDL,
PRAGMA,SET,RESET,SUMMARIZE,DESCRIBE,SHOW,PIVOT,COPY,ATTACH,DETACH,LOAD,INSTALL, and DuckDB filesystem/network functions.
- Ephemeral DuckDB Sandbox Runtime (
internal/sandbox/):- Materializes only the single validated
{hash}.parquetartifact (CREATE TEMP TABLE <alias> AS SELECT * FROM read_parquet(...)) insideduckdb.connect(":memory:"). - Immediately locks
enable_external_access = false,lock_configuration = true,temp_directory = '',threads = 2, and per-workermemory_limit(floor(2.5 GiB / POOL_SIZE)inMiB, e.g.,640MiBatPOOL_SIZE=4). - Streams chunked PyArrow
RecordBatchframes viaRecordBatchReaderwithoutfetch_arrow_table()full materialization.
- Materializes only the single validated
- Disposable Subprocess Pool & Circuit Breaker (
internal/pool/):pebble.ProcessPoolsupervisor (max_workers=POOL_SIZE,max_tasks=100) streaming Arrow IPCRecordBatchchunks over an OS pipe /multiprocessing.connectionwithout full-resultbytesbuffering.- Per-worker
5shard execution deadline with isolated child termination (SIGKILLon POSIX /TerminateProcesson Windows) and automatic worker replacement (ADR-001).
python -m venv .venv
# Linux/macOS: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
pip install -e ".[dev,test]"ruff check .
ruff format --check .
mypy cmd internal api
bandit -r cmd internal api -ll
pip-audit
pytest -v test/unit test/security test/chaos test/integrationThe default test/chaos suite runs a fast 500-stream bounded concurrency soak and burst backpressure test. To run the opt-in 8-hour jemalloc and P95 <= 45ms endurance suite:
RUN_SOAK_TESTS=1 pytest -v -m soak test/chaosdocker build -f deployments/docker/Dockerfile -t rag-tabular-analytics-worker:0.1.0 .
docker run --rm -p 8815:8815 -p 8080:8080 --read-only -e TABULAR_WORKER_CACHE_MAX_BYTES=805306368 --tmpfs /var/cache/tabular-worker:rw,noexec,nosuid,uid=10001,gid=10001,size=1g --tmpfs /tmp:rw,size=256m rag-tabular-analytics-worker:0.1.0- Staging:
deployments/helm/values-staging.yamlanddeployments/argocd/application-staging.yaml - Production:
deployments/helm/values-prod.yaml,deployments/argocd/application-prod.yaml, anddeployments/argocd/rollout-canary.yaml(10% -> 25% -> 50% -> 100%gated byAnalysisTemplateover Prometheus SLIs)
helm lint deployments/helm
helm template rag-tabular-analytics-worker deployments/helm -f deployments/helm/values-prod.yaml > /dev/null