Repository navigation
feat!: merge Python node folders into python/ - #850
Merged
Merged
Conversation
Discover loaders, tasks, assets, checks, factories, and audit factories anywhere under a single python/ root. The decorator defines each node kind, so folder-based kind validation is gone and undecorated helper modules under python/ are importable without becoming nodes. Remove the tasks/, assets/, checks/, loaders/, factories/, libs/, dagster/, and rivers_pipeline/ Python roots and the root definitions.py allowance. The Dagster and Rivers playgrounds now generate an orchestrator directory with definitions.py next to a nested waffle_shop/ SQLBuild project. Path selectors for Python nodes use the python/ root. BREAKING CHANGE: Python nodes and their helper modules must live under python/ (for example python/tasks/orders.py, imported as python.tasks.orders). Python under tasks/, assets/, checks/, loaders/, factories/, libs/, dagster/, rivers_pipeline/, or a root definitions.py now fails compilation with D016. Python path selectors use python/... instead of tasks/, assets/, checks/, or loaders/.
Update the Python nodes, project Python, selector, init, playground, quickstart, dlt, Dagster, and Rivers pages for the single python/ root and for orchestrator definitions that live outside the SQLBuild project, then regenerate the bundled skill references.
Discovery imports every python/ module as python.<dotted.path> through the normal import system, so helper packages can use relative imports and each module initializes once per discovery pass. python.* entries are evicted at the start of each pass so sequential projects and edited files are imported afresh.
A factory may return a decorated node that direct discovery already registers from its own python/ module. Skip that identical callable in factory output while keeping the duplicate-name error for distinct nodes.
Dagster and Rivers playgrounds nest the SQLBuild project under waffle_shop/. Install the scaffolded agent skills there, where sqb skills and stale-skill maintenance resolve them, and document the location.
Check node decorators directly when de-duplicating factory output, split the factory E2E into success and conflict tests, and use expected_ fields and the shared repo-file fixture in the new regressions.
…entity Discovery installs an explicit python package whose __path__ is the project's python/ folder, so project imports never resolve to an unrelated python package elsewhere on sys.path. Unrelated python.* modules displaced during discovery are restored afterwards. Discovery now imports every python/ module and registers direct nodes before running factories, then skips factory results that are the same callable direct discovery registered. A factory that binds a generated node to a module global registers it once instead of dropping it.
kvlonge
enabled auto-merge (squash)
September 27, 2026 04:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Python nodes were split across
tasks/,assets/,checks/,loaders/,factories/andlibs/, with folder-kind rules that added errors without adding meaning. Orchestrator code (dagster/,rivers_pipeline/, rootdefinitions.py) also lived inside the SQLBuild project, although nothing in SQLBuild reads it.Changes
python/root, discovered recursively. The decorator decides the node kind, and helper modules without decorators are imported but never become nodes.python.*package names, bound to the project's ownpython/directory, so relative imports work and helpers initialise once per discovery pass. An unrelated installedpythonpackage is restored afterwards.adapters,materializations,providers,python,rules,sinks, rootadapter.py. The removed roots fail with D016. D010 is removed; loader import failures report D011.sqb init, playground templates, examples and docs usepython/. The Dagster and Rivers playgrounds generate a wrapper directory withdefinitions.py, and the SQLBuild project nested inwaffle_shop/along with its skill files.BREAKING CHANGE: move Python node files into
python/, and move orchestrator code out of the SQLBuild project.Verification
make test: 7,848 passed.make check-cipassed.python/layouts, relative-import helpers, single initialisation, helper edits changing node identity, factory de-duplication, an unrelatedpythonpackage onPYTHONPATH, D016 for each removed root,sqb init, and the Dagster and Rivers wrapper playgrounds.