Skip to content

Commit a6d4f9f

Browse files
kvlongekvlonge
authored
chore: add a Python boundary ratchet for the Rust-hosted migration (#1089)
* chore: add a Python boundary ratchet for the Rust-hosted migration * test: guard project-local adapter rendering overrides on build * fix: catch wrapper, conditional and suppressed native fallbacks in the boundary check * fix: require boundary-check handlers to raise on every path
1 parent 535f029 commit a6d4f9f

21 files changed

Lines changed: 864 additions & 3 deletions

File tree

‎AGENTS.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,13 @@
201201
- Members marked `[forced]` are adapter contract overrides required by `test_strict_adapter.py`. Record other intentional mirrors in `[dupes]` in `fensu.toml` with a reason.
202202
- Public adapter contract methods are duplicated per adapter on purpose: `test_strict_adapter.py` requires every first-class adapter to define them, and `[dupes]` exempts them. The private helpers behind them must not be copied. Keep them in shared builders, such as the snapshot SQL builders in `src/sqlbuild/adapter/contract/classes/`, and let each contract method delegate. Express real dialect differences as explicit parameters, such as a `SnapshotSqlDialect` field, not as a forked copy of the helper.
203203

204+
## Rust-Hosted Migration
205+
206+
- Rust is moving to host compile, planning and non-warehouse CLI work. Python stays only for the modules listed in `python-boundary.toml`: user Python code, adapters, the executor and public Python APIs.
207+
- A migration change deletes the Python it replaces in the same pull request and lowers `remaining_lines`. Never keep a parallel Python implementation, and never fall back to Python after a native failure; surface the failure as an error.
208+
- Adding a `stays` or `handled_native_errors` entry, or raising `remaining_lines`, requires a stated reason in the pull request and explicit user approval. Difficulty or a small measured speed gain is not a reason.
209+
- Moving code into a path listed in `stays`, or behind a reviewed handler, is not migration progress; only code that now runs in Rust lowers `remaining_lines`.
210+
204211
## Bounded Review Process
205212

206213
Orchestrating agents must run reviews as a single bounded cycle, not an open-ended loop:

‎Makefile‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
SHELL := /bin/bash
22

3-
.PHONY: verify verify-quick verify-pg coverage cli-preview rust-check check-native-layering \
3+
.PHONY: verify verify-quick verify-pg coverage cli-preview rust-check check-native-layering check-python-boundary \
44
check-e2e-shards test-e2e-duckdb test-e2e-duckdb-build-core \
55
test-e2e-duckdb-build-incremental \
66
test-e2e-duckdb-cli-data test-e2e-duckdb-cli test-e2e-duckdb-inspection \
@@ -604,7 +604,7 @@ verify-quick:
604604
exit $$status
605605

606606

607-
check-ci: check-e2e-shards check-native-layering
607+
check-ci: check-e2e-shards check-native-layering check-python-boundary
608608
uv run ruff format --check .
609609
uv run ruff check .
610610
uv run ty check src tests
@@ -616,6 +616,10 @@ check-native-layering:
616616
uv run python -m scripts.check_native_layering
617617

618618

619+
check-python-boundary:
620+
uv run python -m scripts.check_python_boundary
621+
622+
619623
check-pr-metadata:
620624
@test -n "$(PR_TITLE)" || { echo "PR_TITLE is required" >&2; exit 2; }
621625
@test -n "$(PR_BODY_FILE)" || { echo "PR_BODY_FILE is required" >&2; exit 2; }

‎python-boundary.toml‎

Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
# Python that stays once Rust hosts SQLBuild. Every other Python file under src/sqlbuild is
2+
# migration work that moves to Rust. `remaining_lines` must equal the current count of that work:
3+
# a PR that moves code lowers it, and adding movable Python fails the check.
4+
# Native failures must surface as errors; `handled_native_errors` lists the reviewed exceptions.
5+
remaining_lines = 146314
6+
7+
[[stays]]
8+
path = "executor"
9+
reason = "executor"
10+
note = "Runs plans against warehouses; a separate later decision."
11+
12+
[[stays]]
13+
path = "adapters"
14+
reason = "adapter"
15+
note = "Built-in warehouse adapters that project adapters subclass and override."
16+
17+
[[stays]]
18+
path = "adapter/contract"
19+
reason = "adapter"
20+
note = "BaseAdapter and StrictAdapter, the documented classes project adapters subclass."
21+
22+
[[stays]]
23+
path = "adapter/discovery"
24+
reason = "user-code"
25+
note = "Imports project adapter modules."
26+
27+
[[stays]]
28+
path = "adapter/relations"
29+
reason = "adapter"
30+
note = "Relation inspection wrappers adapters call, with per-connection caches."
31+
32+
[[stays]]
33+
path = "adapter/state_sql"
34+
reason = "adapter"
35+
note = "State-table SQL rendered through adapter callbacks."
36+
37+
[[stays]]
38+
path = "adapter/type_system"
39+
reason = "adapter"
40+
note = "Type normalisation adapters import; already a thin native wrapper."
41+
42+
[[stays]]
43+
path = "provider"
44+
reason = "user-code"
45+
note = "Provider base class users subclass, and injection into user functions."
46+
47+
[[stays]]
48+
path = "python_nodes"
49+
reason = "user-code"
50+
note = "Decorators that run when user modules are imported."
51+
52+
[[stays]]
53+
path = "microbatches"
54+
reason = "executor"
55+
note = "Microbatch state store the executor and adapters use."
56+
57+
[[stays]]
58+
path = "rules"
59+
reason = "public-api"
60+
note = "Public custom-rule API and test helper."
61+
62+
[[stays]]
63+
path = "rule_engine/_helpers/host"
64+
reason = "user-code"
65+
note = "Subprocess host that loads and runs user rule modules."
66+
67+
[[stays]]
68+
path = "rule_engine/classes/rule_context.py"
69+
reason = "user-code"
70+
note = "The context object user rules receive."
71+
72+
[[stays]]
73+
path = "runtime/observability"
74+
reason = "public-api"
75+
note = "Documented observability API used throughout the executor."
76+
77+
[[stays]]
78+
path = "runtime/execution_history"
79+
reason = "public-api"
80+
note = "Documented SQLite and Postgres execution-history backends."
81+
82+
[[stays]]
83+
path = "runtime/execution_limits"
84+
reason = "executor"
85+
note = "Deadline the executor scheduler and warehouse cursors enforce."
86+
87+
[[stays]]
88+
path = "integrations/dagster"
89+
reason = "public-api"
90+
note = "Documented orchestrator integration."
91+
92+
[[stays]]
93+
path = "integrations/rivers"
94+
reason = "public-api"
95+
note = "Documented orchestrator integration."
96+
97+
[[stays]]
98+
path = "integrations/dlt"
99+
reason = "user-code"
100+
note = "Loader runtime that runs inside user loaders."
101+
102+
[[stays]]
103+
path = "integrations/ingestr"
104+
reason = "user-code"
105+
note = "Loader runtime that runs inside the executor load phase."
106+
107+
[[stays]]
108+
path = "compiler/python_nodes/_helpers/identity.py"
109+
reason = "user-code"
110+
note = "Identity hashing over live user functions (source and globals)."
111+
112+
[[stays]]
113+
path = "compiler/discovery/_helpers/filesystem/command_output_sinks.py"
114+
reason = "user-code"
115+
note = "Inspects loaded user sink functions."
116+
117+
[[stays]]
118+
path = "compiler/pipeline/_helpers/materializations.py"
119+
reason = "user-code"
120+
note = "Imports user materialization modules."
121+
122+
[[stays]]
123+
path = "assets.py"
124+
reason = "public-api"
125+
note = "Public facade."
126+
127+
[[stays]]
128+
path = "audits.py"
129+
reason = "public-api"
130+
note = "Public facade."
131+
132+
[[stays]]
133+
path = "checks.py"
134+
reason = "public-api"
135+
note = "Public facade."
136+
137+
[[stays]]
138+
path = "compute_logs.py"
139+
reason = "public-api"
140+
note = "Public facade."
141+
142+
[[stays]]
143+
path = "execution_history.py"
144+
reason = "public-api"
145+
note = "Public facade."
146+
147+
[[stays]]
148+
path = "factories.py"
149+
reason = "public-api"
150+
note = "Public facade."
151+
152+
[[stays]]
153+
path = "functions.py"
154+
reason = "public-api"
155+
note = "Public facade."
156+
157+
[[stays]]
158+
path = "hooks.py"
159+
reason = "public-api"
160+
note = "Public facade."
161+
162+
[[stays]]
163+
path = "loaders.py"
164+
reason = "public-api"
165+
note = "Public facade."
166+
167+
[[stays]]
168+
path = "observability.py"
169+
reason = "public-api"
170+
note = "Public facade."
171+
172+
[[stays]]
173+
path = "postgres_history.py"
174+
reason = "public-api"
175+
note = "Public facade."
176+
177+
[[stays]]
178+
path = "providers.py"
179+
reason = "public-api"
180+
note = "Public facade."
181+
182+
[[stays]]
183+
path = "refs.py"
184+
reason = "public-api"
185+
note = "Public facade."
186+
187+
[[stays]]
188+
path = "retries.py"
189+
reason = "public-api"
190+
note = "Public facade."
191+
192+
[[stays]]
193+
path = "sinks.py"
194+
reason = "public-api"
195+
note = "Public facade."
196+
197+
[[stays]]
198+
path = "sqlite_history.py"
199+
reason = "public-api"
200+
note = "Public facade."
201+
202+
[[stays]]
203+
path = "tasks.py"
204+
reason = "public-api"
205+
note = "Public facade."
206+
207+
[[handled_native_errors]]
208+
location = "compiler/macro_bridge/_helpers/store_keys.py::macro_store_token"
209+
note = "A macro without a stable key is not stored; no Python implementation runs."
210+
211+
[[handled_native_errors]]
212+
location = "compiler/macro_bridge/_helpers/store_keys.py::context_store_token"
213+
note = "A context without a stable key is not stored; no Python implementation runs."
214+
215+
[[handled_native_errors]]
216+
location = "lint/_helpers/native_sql.py::_native_responses"
217+
note = "A native lint failure is reported as a lint error result, not recomputed in Python."
218+
219+
[[handled_native_errors]]
220+
location = "rule_engine/_helpers/run/rules.py::_cached_sql_findings"
221+
note = "An undecodable cached finding is a cache miss; the rules run again natively."

‎scripts/check_python_boundary.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
"""Direct wrapper for the Python boundary ratchet."""
2+
3+
from __future__ import annotations
4+
5+
from scripts.python_boundary.main.check import run_python_boundary_check
6+
7+
8+
def main(argv: list[str] | None = None) -> int:
9+
"""Check the Python boundary ratchet."""
10+
return run_python_boundary_check(argv)
11+
12+
13+
if __name__ == "__main__":
14+
raise SystemExit(main())

‎scripts/python_boundary/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""Python boundary ratchet for the Rust-hosted migration."""

‎scripts/python_boundary/_helpers/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""Python boundary ratchet helpers."""

‎scripts/python_boundary/_helpers/lines.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
"""Count the Python lines that remain to move to Rust."""
2+
3+
from __future__ import annotations
4+
5+
from pathlib import Path
6+
7+
from scripts.python_boundary.constants import SOURCE_ROOT
8+
from scripts.python_boundary.models import StayEntry
9+
10+
11+
def tracked_python_files(*, root: Path, stays: tuple[StayEntry, ...]) -> tuple[Path, ...]:
12+
"""Every Python file under the source root outside the modules that stay Python."""
13+
source: Path = root / SOURCE_ROOT
14+
staying: tuple[Path, ...] = tuple(source / entry.path for entry in stays)
15+
return tuple(
16+
path for path in sorted(source.rglob("*.py")) if not _stays(path=path, staying=staying)
17+
)
18+
19+
20+
def _stays(*, path: Path, staying: tuple[Path, ...]) -> bool:
21+
return any(path == stay or stay in path.parents for stay in staying)
22+
23+
24+
def count_lines(paths: tuple[Path, ...]) -> int:
25+
"""The total physical line count of `paths`."""
26+
return sum(len(path.read_text(encoding="utf-8").splitlines()) for path in paths)

0 commit comments

Comments
 (0)