diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md index 11886b7e12..e925792cf9 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/build.md @@ -72,7 +72,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `--selection-diagnostics` | Warn when a selected model will build on changed upstream models that are not selected; see [Selection and staleness](../concepts/planning/selection-and-staleness.md) | | `--manifest` | Generate `target/manifest.json` with plan-aware project metadata | | `SELECTOR ...` | Positional selectors, the same as `--select` | -| `--select`, `-s` | Select specific models | +| `--select`, `-s` | Select specific models, or unit tests by name (`test:`); selecting only unit tests runs them without building models | | `--exclude` | Exclude specific models | | `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md index 4c3984075a..ff2664ca2c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/cli/test.md @@ -23,12 +23,19 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb |------|-------------| | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | | `SELECTOR ...` | Positional selectors, the same as `--select` | -| `--select`, `-s` | Select tests targeting specific models | -| `--exclude` | Exclude tests targeting specific models | +| `--select`, `-s` | Select tests by name (`order_status_rules` or `test:order_status_rules`), or the tests targeting specific models | +| `--exclude` | Exclude tests by name, or the tests targeting specific models | | `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](../concepts/project-configuration.md#command-group-warehouses) and the connection warehouse | -Parameterized cases use parent-level selection. Selecting a target model includes every case in -each matching `TEST` template; case names are not model selectors. +Test names are globally unique, so a bare test name selects exactly that test. `test:` is the +explicit form and accepts the same glob patterns as names, such as `test:orders_*`. A bare pattern +selects tests only when it matches no other resource. Test selectors are unioned with model +selectors, so `sqb test test:order_status_rules stg_orders` runs that test and every test targeting +`stg_orders`. Test selectors cannot use `+` graph expansion or `,` intersection. Only `sqb test` and +`sqb build` accept test selectors; other commands reject them with `S013`. + +Parameterized cases use parent-level selection. Selecting a test or a target model includes every +case of each matching `TEST` template, and `--case` narrows the run to one case name. Text output identifies each case as ` []` and includes its source path and safe typed parameters. Structured JSON emits one check per case with stable source/block/case identity, @@ -44,4 +51,10 @@ sqb test # Run tests for a specific model sqb test stg_orders + +# Run one test by name +sqb test order_status_rules + +# Run one case of a parameterized test +sqb test test:order_status_rules --case open_case ``` diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/overview.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/overview.md index 38cb8e3262..d2343d6641 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/overview.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/python-nodes/overview.md @@ -162,7 +162,7 @@ sqb build --select tag:exports # by tag sqb build --select +orders_export # with upstreams ``` -Names are globally unique across models, sources, seeds, functions, loaders, tasks, assets, checks, providers, and hooks. +Names are globally unique across models, sources, seeds, functions, loaders, tasks, assets, checks, providers, hooks, unit tests, and scenarios. ## Lifecycle: run, build, check diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md index fdde2184a0..cd6e10a5db 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/scenarios.md @@ -44,7 +44,7 @@ tests/ fulfillment__late_shipment.sql ``` -Folders are organizational only. The scenario name is the filename stem and must be globally unique across all discovered scenario files. +Folders are organizational only. The scenario name is the filename stem. Like model, source, seed, and unit-test names, it must be globally unique across the project. ### Scenario format diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/selectors.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/selectors.md index 0b1373b684..853d2f3c0f 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/selectors.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/selectors.md @@ -80,6 +80,19 @@ sqb build --select seed:waffle_types sqb build --select source:raw__orders ``` +### Unit test + +`sqb test` and `sqb build` select SQL unit tests by name. Test names are globally unique, so a bare +name works; `test:` is the explicit form and accepts glob patterns: + +```bash +sqb test order_status_rules +sqb test "test:orders_*" stg_customers +``` + +Test selectors are unioned with other selectors and cannot use `+` or `,`. Other commands reject +them with `S013`. + ## Graph expansion ### Upstream @@ -163,6 +176,14 @@ error[S007]: unknown selector name 'missing_*' error[S009]: no models found under path 'models/nonexistent'. error[S000]: no SQL resources or Python nodes found with tag 'nonexistent_tag' error[S003]: path selector 'fact_orders~' requires names on both sides of '~' +error[S013]: selector 'order_status_rules' selects a unit test; only `sqb test` and `sqb build` accept unit-test selectors +``` + +Unknown names suggest the closest selectable names, including unit tests: + +``` +error[S007]: unknown selector name 'order_status_rule' + = help: did you mean 'order_status_rules'? ``` If a path selector omits the root directory, SQLBuild asks for the explicit form: diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md index 9a4cb99578..584a200d7c 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/docs/concepts/testing.md @@ -548,7 +548,11 @@ CTE prefixes from other modes are not allowed. For example, `__source__` in a ma ## Multiple tests per file -A single test file can contain multiple `TEST()` blocks. Each block must have a unique `name`: +A single test file can contain multiple `TEST()` blocks. A test's name is its `name`, or the file +stem when `name` is omitted. Test names are globally unique: a test cannot share a name with another +test, a scenario, a model, a source, a seed, a function, or a Python node. Name the file or the block +after the behavior it checks, such as `orders_status_rules`, rather than after the model. The cases +of one parameterized test share its name. Each block in a file must have its own `name`: ```sql TEST (name "completed_orders_only"); diff --git a/src/sqlbuild/.agents/skills/sqlbuild/references/testing.md b/src/sqlbuild/.agents/skills/sqlbuild/references/testing.md index af77e18f97..f853cc1578 100644 --- a/src/sqlbuild/.agents/skills/sqlbuild/references/testing.md +++ b/src/sqlbuild/.agents/skills/sqlbuild/references/testing.md @@ -23,6 +23,10 @@ Files live under `tests/unit/` (any subfolder). Each file is a `TEST();` header, CTEs, and a required closing `SELECT 1`. +A test's name is its `name` or the file stem. Test names are globally unique across tests, +scenarios, models, sources, seeds, functions and Python nodes, so never name a test file after its +model (`test_stg_orders.sql`, not `stg_orders.sql`). Run one test with `sqb test ` or +`sqb test test:`; only `sqb test` and `sqb build` accept test selectors. ```sql -- tests/unit/test_stg_orders.sql diff --git a/src/sqlbuild/cli/commands/_helpers/build_planning/defer_clone.py b/src/sqlbuild/cli/commands/_helpers/build_planning/defer_clone.py index eb14232679..7a54795a27 100644 --- a/src/sqlbuild/cli/commands/_helpers/build_planning/defer_clone.py +++ b/src/sqlbuild/cli/commands/_helpers/build_planning/defer_clone.py @@ -32,6 +32,7 @@ from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.compiler.pipeline.main.clone_with_options import run_clone_pipeline_with_options from sqlbuild.compiler.pipeline.main.compiled_project import build_compiled_project +from sqlbuild.compiler.pipeline.main.unit_test_selection import split_unit_test_selectors from sqlbuild.compiler.pipeline.models import ( ClonePipelineConnection, ClonePipelineOptions, @@ -44,7 +45,7 @@ resolve_skipped_view_chain, ) from sqlbuild.compiler.planner.main.selection.scope import build_planner_scope -from sqlbuild.compiler.planner.models import PlannerScope +from sqlbuild.compiler.planner.models import PlannerScope, UnitTestSelectorSplit from sqlbuild.compiler.planner.types import MaterializationType from sqlbuild.executor.clone.main.build_destination_transient_models import ( build_destination_transient_models, @@ -173,6 +174,17 @@ def run_defer_clone_boundary_prephase( ) -> DeferClonePrephaseOutcome: """Resolve defer-clone boundaries and clone them before build planning.""" + split: UnitTestSelectorSplit = split_unit_test_selectors( + select=request.select, + exclude=request.exclude, + discovered_inputs=invocation.discovered_inputs, + ) + if not split.sql_test_selection.models_selected: + return DeferClonePrephaseOutcome( + destination_target_name=invocation.effective_target_name, + boundary_selectors=(), + view_chain_selectors=(), + ) cloned_project: CompiledProject boundary_selectors: tuple[str, ...] view_chain_selectors: tuple[str, ...] @@ -187,8 +199,8 @@ def run_defer_clone_boundary_prephase( selected_target=request.selected_target, no_sql_validation=request.no_sql_validation, no_cache=request.no_cache, - select=request.select, - exclude=request.exclude, + select=split.select, + exclude=split.exclude, cli_vars=request.cli_vars, project_dir=invocation.effective_project_dir, auto_load_sources=invocation.should_load_sources, @@ -208,7 +220,7 @@ def run_defer_clone_boundary_prephase( no_sql_validation=request.no_sql_validation, no_cache=request.no_cache, select=(*boundary_selectors, *view_chain_selectors), - caused_by_names=request.select, + caused_by_names=split.select, cli_vars=request.cli_vars, connection_config=invocation.connection_config, project_dir=invocation.effective_project_dir, diff --git a/src/sqlbuild/cli/commands/_helpers/build_planning/planning.py b/src/sqlbuild/cli/commands/_helpers/build_planning/planning.py index 49e05cb74d..bedc7c7dd5 100644 --- a/src/sqlbuild/cli/commands/_helpers/build_planning/planning.py +++ b/src/sqlbuild/cli/commands/_helpers/build_planning/planning.py @@ -59,6 +59,7 @@ def _run_build_compile_pipeline( max_microbatches=request.max_microbatches, selection_diagnostics=request.selection_diagnostics, plan_sql_tests=request.run_tests, + accepts_unit_test_selectors=True, ), hooks=ConnectionHooks( on_progress=invocation.planning_progress.on_progress, diff --git a/src/sqlbuild/cli/commands/_helpers/check/execution.py b/src/sqlbuild/cli/commands/_helpers/check/execution.py index 4011a33ad8..94689f7487 100644 --- a/src/sqlbuild/cli/commands/_helpers/check/execution.py +++ b/src/sqlbuild/cli/commands/_helpers/check/execution.py @@ -24,6 +24,7 @@ ) from sqlbuild.compiler.discovery.models import DiscoveredCheckFunction from sqlbuild.compiler.pipeline.main.project_relation_targets import build_project_relation_targets +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.models import CompilePipelineResult from sqlbuild.compiler.python_nodes.main.graph import build_discovered_python_node_graph from sqlbuild.compiler.python_nodes.main.run_lifecycle import build_python_sql_run_lifecycle @@ -57,6 +58,11 @@ def prepare_check_execution( python_graph: PythonNodeGraph = build_discovered_python_node_graph( discovered_inputs=invocation.discovered_inputs ) + reject_unit_test_selectors( + select=request.select, + exclude=request.exclude, + discovered_inputs=invocation.discovered_inputs, + ) check_names: frozenset[str] = resolve_selected_check_names( graph=python_graph, select=request.select, diff --git a/src/sqlbuild/cli/commands/_helpers/check/planning.py b/src/sqlbuild/cli/commands/_helpers/check/planning.py index 2689787c33..b0a712f1db 100644 --- a/src/sqlbuild/cli/commands/_helpers/check/planning.py +++ b/src/sqlbuild/cli/commands/_helpers/check/planning.py @@ -13,6 +13,7 @@ from sqlbuild.compiler.compile.models import CompiledObjectKey from sqlbuild.compiler.compile.types import CompiledResourceType from sqlbuild.compiler.graph.main.sql_ref_key import sql_ref_key +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.main.static_command import compile_static_command_context from sqlbuild.compiler.pipeline.main.static_result import build_static_pipeline_result from sqlbuild.compiler.pipeline.models import ( @@ -35,6 +36,11 @@ def compile_check_plan( python_graph: PythonNodeGraph = build_discovered_python_node_graph( discovered_inputs=invocation.discovered_inputs ) + reject_unit_test_selectors( + select=request.select, + exclude=request.exclude, + discovered_inputs=invocation.discovered_inputs, + ) check_names: frozenset[str] = resolve_selected_check_names( graph=python_graph, select=request.select, diff --git a/src/sqlbuild/cli/commands/_helpers/clone/invocation.py b/src/sqlbuild/cli/commands/_helpers/clone/invocation.py index 677b6c542b..599639cc66 100644 --- a/src/sqlbuild/cli/commands/_helpers/clone/invocation.py +++ b/src/sqlbuild/cli/commands/_helpers/clone/invocation.py @@ -14,6 +14,7 @@ from sqlbuild.cli.progress.classes.planning_progress_reporter import PlanningProgressReporter from sqlbuild.compiler.discovery.main.discover import discover_project_inputs from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.presentation.main.supports_color import supports_color from sqlbuild.spec.contracts.main.resolve_effective_adapter_name import ( resolve_effective_adapter_name, @@ -38,6 +39,9 @@ def resolve_clone_invocation(*, request: CloneCommandRequest) -> CloneInvocation project_dir=effective_project_dir ) progress.complete("Discovered project.") + reject_unit_test_selectors( + select=request.select, exclude=request.exclude, discovered_inputs=discovered_inputs + ) destination_target_name: str | None = request.destination_target_name if destination_target_name is None: destination_target_name = resolve_target_name( diff --git a/src/sqlbuild/cli/commands/_helpers/compile/pipeline.py b/src/sqlbuild/cli/commands/_helpers/compile/pipeline.py index 18aca51c83..06a5f6136e 100644 --- a/src/sqlbuild/cli/commands/_helpers/compile/pipeline.py +++ b/src/sqlbuild/cli/commands/_helpers/compile/pipeline.py @@ -47,6 +47,7 @@ from sqlbuild.compiler.discovery.main.discover import discover_project_inputs from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs, DiscoveryCacheRequest from sqlbuild.compiler.lineage.models import ProjectColumnLineage +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.main.selected_graph import ( build_project_graph_with_analysis_selection, ) @@ -126,6 +127,7 @@ def _analyze_compile_project( ), declaration_reuse=COMPILE_RENDER_REUSE.claim_discovery(), ) + reject_unit_test_selectors(select=select, exclude=exclude, discovered_inputs=discovered_inputs) discover_ms: int = elapsed_ms(discover_start) _ = complete_compile_phase( status=status, message=f"Discovered project. ({discover_ms / 1000:.2f}s)" diff --git a/src/sqlbuild/cli/commands/_helpers/diff/invocation.py b/src/sqlbuild/cli/commands/_helpers/diff/invocation.py index 07962d3ebf..db4f00b109 100644 --- a/src/sqlbuild/cli/commands/_helpers/diff/invocation.py +++ b/src/sqlbuild/cli/commands/_helpers/diff/invocation.py @@ -11,6 +11,7 @@ discover_project_configuration, ) from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors def resolve_diff_invocation(*, request: DiffCommandRequest) -> DiffInvocation: @@ -34,6 +35,9 @@ def resolve_diff_invocation(*, request: DiffCommandRequest) -> DiffInvocation: if is_query_diff_request(request=request) else discover_project_inputs(project_dir=effective_project_dir) ) + reject_unit_test_selectors( + select=request.select, exclude=request.exclude, discovered_inputs=discovered_inputs + ) if not request.select and not is_query_diff_request(request=request): raise CliUserError("diff requires --select in v1", code="C204") return DiffInvocation( diff --git a/src/sqlbuild/cli/commands/_helpers/lineage/execution.py b/src/sqlbuild/cli/commands/_helpers/lineage/execution.py index 2c8e1f2f5a..0c7f9107f8 100644 --- a/src/sqlbuild/cli/commands/_helpers/lineage/execution.py +++ b/src/sqlbuild/cli/commands/_helpers/lineage/execution.py @@ -47,7 +47,10 @@ from sqlbuild.compiler.lineage.main.build_semantic_uses import build_direct_semantic_uses from sqlbuild.compiler.lineage.models import DirectSemanticColumnUse from sqlbuild.compiler.pipeline.main.graph import build_project_graph +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.models import ProjectGraph +from sqlbuild.compiler.planner.constants import UNKNOWN_SELECTOR_ERROR_CODE +from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.compiler.planner.main.selection.selection import resolve_project_selectors from sqlbuild.presentation.main.supports_color import supports_color from sqlbuild.spec.contracts.main.resolve_effective_adapter_name import ( @@ -113,6 +116,9 @@ def _prepare_graph( sql_analysis_enabled_override=(None if requires_compiled_graph else False), extract_output_column_locations=False, ) + reject_unit_test_selectors( + select=request.select, exclude=request.exclude, discovered_inputs=discovered + ) adapter: BaseAdapter = resolve_adapter( adapter_name=resolve_effective_adapter_name( project_config=discovered.project_config, @@ -197,13 +203,28 @@ def _render_lineage( depth=parsed_depth, ) else: - lineage_graph = select_selector_lineage( - graph=graph, - select=request.select, - exclude=request.exclude, - depth=parsed_depth, - direction=request.direction, - ) + try: + lineage_graph = select_selector_lineage( + graph=graph, + select=request.select, + exclude=request.exclude, + depth=parsed_depth, + direction=request.direction, + ) + except PlannerInputError as error: + if error.code == UNKNOWN_SELECTOR_ERROR_CODE and isinstance( + graph, RelationLineageIndex + ): + reject_unit_test_selectors( + select=request.select, + exclude=request.exclude, + discovered_inputs=discover_project_inputs( + project_dir=request.project_dir or Path.cwd(), + sql_analysis_enabled_override=False, + extract_output_column_locations=False, + ), + ) + raise return _format_graph(graph=lineage_graph, output_format=request.output_format) diff --git a/src/sqlbuild/cli/commands/_helpers/lint/selection.py b/src/sqlbuild/cli/commands/_helpers/lint/selection.py index e11c2ed751..4998b9d3a5 100644 --- a/src/sqlbuild/cli/commands/_helpers/lint/selection.py +++ b/src/sqlbuild/cli/commands/_helpers/lint/selection.py @@ -14,6 +14,7 @@ from sqlbuild.compiler.discovery.main.discover import discover_project_inputs from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs, DiscoveredSqlModelFile from sqlbuild.compiler.pipeline.main.graph import build_project_graph +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.models import ProjectGraph from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.compiler.planner.main.selection.selection import resolve_project_selectors @@ -61,6 +62,7 @@ def resolve_lint_inputs( sql_analysis_enabled_override=False, extract_output_column_locations=False, ) + reject_unit_test_selectors(select=select, exclude=exclude, discovered_inputs=discovered) adapter = _resolve_discovered_adapter( project_dir=project_dir, discovered=discovered, diff --git a/src/sqlbuild/cli/commands/_helpers/load/invocation.py b/src/sqlbuild/cli/commands/_helpers/load/invocation.py index 5620b315d8..64ef7df106 100644 --- a/src/sqlbuild/cli/commands/_helpers/load/invocation.py +++ b/src/sqlbuild/cli/commands/_helpers/load/invocation.py @@ -17,6 +17,7 @@ from sqlbuild.compiler.compile.main.effective_target import build_effective_target_config from sqlbuild.compiler.discovery.main.discover import discover_project_inputs from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.presentation.main.supports_color import supports_color from sqlbuild.spec.contracts.main.resolve_effective_adapter_name import ( resolve_effective_adapter_name, @@ -33,6 +34,9 @@ def resolve_load_invocation(*, request: LoadCommandRequest) -> LoadInvocation: discovered_inputs: DiscoveredProjectInputs = discover_project_inputs( project_dir=effective_project_dir ) + reject_unit_test_selectors( + select=request.select, exclude=request.exclude, discovered_inputs=discovered_inputs + ) target_config: TargetConfig | None = build_effective_target_config( discovered_inputs=discovered_inputs, selected_target=request.selected_target, diff --git a/src/sqlbuild/cli/commands/_helpers/rules/command.py b/src/sqlbuild/cli/commands/_helpers/rules/command.py index da8aa5ff6b..a199df7e45 100644 --- a/src/sqlbuild/cli/commands/_helpers/rules/command.py +++ b/src/sqlbuild/cli/commands/_helpers/rules/command.py @@ -19,6 +19,7 @@ from sqlbuild.compiler.discovery.main.discover import discover_project_inputs from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.compiler.pipeline.main.graph import build_project_graph +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.models import ProjectGraph from sqlbuild.compiler.planner.main.selection.selection import resolve_project_selectors from sqlbuild.rule_engine.main.build_catalogue import build_catalogue @@ -127,6 +128,9 @@ def _run_rule_selection( return 2 print(f"Evaluating rule selection {selector}...", file=sys.stderr) discovered: DiscoveredProjectInputs = discover_project_inputs(project_dir=project_dir) + reject_unit_test_selectors( + select=request.select, exclude=request.exclude, discovered_inputs=discovered + ) adapter: BaseAdapter = resolve_adapter( adapter_name=resolve_effective_adapter_name( project_config=discovered.project_config, diff --git a/src/sqlbuild/cli/commands/_helpers/scenario_capture/capture.py b/src/sqlbuild/cli/commands/_helpers/scenario_capture/capture.py index d05ac945ec..6b8b7563a0 100644 --- a/src/sqlbuild/cli/commands/_helpers/scenario_capture/capture.py +++ b/src/sqlbuild/cli/commands/_helpers/scenario_capture/capture.py @@ -41,6 +41,7 @@ from sqlbuild.compiler.discovery.main.sql_analysis_off_guidance import sql_analysis_off_guidance from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.compiler.pipeline.main.compile_only import run_compile_only_pipeline +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.models import ( CompilePipelineOptions, CompilePipelineResult, @@ -78,6 +79,9 @@ def run_scenario_capture(request: ScenarioCaptureCommandRequest) -> int: discovered_inputs: DiscoveredProjectInputs = discover_project_inputs( project_dir=effective_project_dir ) + reject_unit_test_selectors( + select=selectors, exclude=exclude, discovered_inputs=discovered_inputs + ) discovered_inputs, namespace = resolve_scenario_namespace( inputs=discovered_inputs, cli_value=request.scenario_namespace ) diff --git a/src/sqlbuild/cli/commands/_helpers/scenario_execution/runner.py b/src/sqlbuild/cli/commands/_helpers/scenario_execution/runner.py index 10b0a072db..cc0f8bebc4 100644 --- a/src/sqlbuild/cli/commands/_helpers/scenario_execution/runner.py +++ b/src/sqlbuild/cli/commands/_helpers/scenario_execution/runner.py @@ -58,6 +58,7 @@ from sqlbuild.compiler.discovery.main.sql_analysis_off_guidance import sql_analysis_off_guidance from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.compiler.pipeline.main.compile_only import run_compile_only_pipeline +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.models import ( CompilePipelineOptions, CompilePipelineResult, @@ -125,6 +126,9 @@ def run_scenario(request: ScenarioTestCommandRequest) -> int: discovered_inputs: DiscoveredProjectInputs = discover_project_inputs( project_dir=effective_project_dir ) + reject_unit_test_selectors( + select=selectors, exclude=exclude, discovered_inputs=discovered_inputs + ) discovered_inputs, namespace = resolve_scenario_namespace( inputs=discovered_inputs, cli_value=request.scenario_namespace ) diff --git a/src/sqlbuild/cli/commands/_helpers/test/planning.py b/src/sqlbuild/cli/commands/_helpers/test/planning.py index 38016b9db5..62c4562208 100644 --- a/src/sqlbuild/cli/commands/_helpers/test/planning.py +++ b/src/sqlbuild/cli/commands/_helpers/test/planning.py @@ -31,6 +31,7 @@ def compile_test_plan( source_deferral_enabled=False, select=request.select, exclude=request.exclude, + accepts_unit_test_selectors=True, connection_config=invocation.connection_config, cli_vars=request.cli_vars, external_sql_reference_resolver=resolve_external_sql_reference_resolver( diff --git a/src/sqlbuild/cli/commands/main/execution/_freshness.py b/src/sqlbuild/cli/commands/main/execution/_freshness.py index ea97f7cf07..d5d57c28f6 100644 --- a/src/sqlbuild/cli/commands/main/execution/_freshness.py +++ b/src/sqlbuild/cli/commands/main/execution/_freshness.py @@ -32,6 +32,7 @@ from sqlbuild.compiler.discovery.main.discover import discover_project_inputs from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.compiler.pipeline.main.graph import build_project_graph +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.models import ProjectGraph from sqlbuild.compiler.source_freshness.models import SourceFreshnessIdentity, SourceFreshnessRecord from sqlbuild.presentation.main.supports_color import supports_color @@ -56,6 +57,9 @@ def run_freshness(request: FreshnessCommandRequest) -> int: discovered_inputs: DiscoveredProjectInputs = discover_project_inputs( project_dir=effective_project_dir ) + _ = reject_unit_test_selectors( + select=select, exclude=request.exclude, discovered_inputs=discovered_inputs + ) adapter_name: str = resolve_effective_adapter_name( project_config=discovered_inputs.project_config, local_config=discovered_inputs.local_config, diff --git a/src/sqlbuild/cli/commands/main/inspection/_contract.py b/src/sqlbuild/cli/commands/main/inspection/_contract.py index 7102544c72..b321563017 100644 --- a/src/sqlbuild/cli/commands/main/inspection/_contract.py +++ b/src/sqlbuild/cli/commands/main/inspection/_contract.py @@ -29,6 +29,7 @@ from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.compiler.pipeline.main.compiled_project import build_compiled_project from sqlbuild.compiler.pipeline.main.project_graph import build_project_graph_from_compiled_project +from sqlbuild.compiler.pipeline.main.reject_unit_test_selectors import reject_unit_test_selectors from sqlbuild.compiler.pipeline.models import ProjectGraph from sqlbuild.compiler.planner.main.selection.selection import ( resolve_project_selectors, @@ -43,6 +44,9 @@ def run_contract(request: ContractCommandRequest) -> int: raise CliUserError("contract --overwrite requires generate --write", code="C471") project_dir: Path = request.project_dir or Path.cwd() discovered: DiscoveredProjectInputs = discover_project_inputs(project_dir=project_dir) + _ = reject_unit_test_selectors( + select=request.select, exclude=request.exclude, discovered_inputs=discovered + ) if request.from_target not in discovered.project_config.targets: raise CliUserError( f"unknown contract source target '{request.from_target}'", diff --git a/src/sqlbuild/compiler/discovery/_helpers/validation/discovery.py b/src/sqlbuild/compiler/discovery/_helpers/validation/discovery.py index e124b9cf7a..ee1920118e 100644 --- a/src/sqlbuild/compiler/discovery/_helpers/validation/discovery.py +++ b/src/sqlbuild/compiler/discovery/_helpers/validation/discovery.py @@ -26,6 +26,8 @@ DiscoveredSqlHookFile, DiscoveredSqlModelFile, DiscoveredSqlScenarioFile, + DiscoveredSqlTestBlock, + DiscoveredSqlTestFile, DiscoveredTaskFunction, ) from sqlbuild.compiler.path_defaults.main._select import select_path_default @@ -644,6 +646,31 @@ def _validate_unique_project_resource_names( path=str(hook_function.relative_path), ) + scenario_file: DiscoveredSqlScenarioFile + for scenario_file in discovered_inputs.scenario_files: + seen_names[scenario_file.name] = _validated_project_resource_entry( + seen_names=seen_names, + name=scenario_file.name, + kind="scenario", + path=str(scenario_file.relative_path), + ) + + test_names: set[str] = set() + test_file: DiscoveredSqlTestFile + for test_file in discovered_inputs.test_files: + block: DiscoveredSqlTestBlock + for block in test_file.blocks: + test_name: str = block.name or test_file.relative_path.stem + if test_name in test_names: + continue + test_names.add(test_name) + seen_names[test_name] = _validated_project_resource_entry( + seen_names=seen_names, + name=test_name, + kind="unit test", + path=str(test_file.relative_path), + ) + def _validated_project_resource_entry( *, seen_names: dict[str, tuple[str, str]], name: str, kind: str, path: str @@ -653,7 +680,8 @@ def _validated_project_resource_entry( raise DiscoveryConflictError( f"Project resource name '{name}' is declared as both {existing_entry[0]} " f"in {existing_entry[1]} and {kind} in {path}; model, source, seed, function, " - "loader, task, asset, check, provider, and hook names must be globally unique" + "loader, task, asset, check, provider, hook, unit test, and scenario names must be " + "globally unique" ) return (kind, path) diff --git a/src/sqlbuild/compiler/pipeline/_helpers/analysis_selection.py b/src/sqlbuild/compiler/pipeline/_helpers/analysis_selection.py index f3be6f1598..850c2996c4 100644 --- a/src/sqlbuild/compiler/pipeline/_helpers/analysis_selection.py +++ b/src/sqlbuild/compiler/pipeline/_helpers/analysis_selection.py @@ -1,7 +1,9 @@ -"""Selection policy for deep model SQL analysis.""" +"""Selection policies applied to compile pipeline options.""" from __future__ import annotations +from dataclasses import replace + from sqlbuild.compiler.compile.models import ( CompileAnalysisSelection, CompiledObjectKey, @@ -12,13 +14,39 @@ from sqlbuild.compiler.planner.main.selection._resolve_planner_scopes import ( resolve_planner_scopes, ) +from sqlbuild.compiler.planner.main.selection._split_unit_test_selectors import ( + split_project_unit_test_selectors, +) from sqlbuild.compiler.planner.models import ( PlannerPolicies, PlannerScopeResolution, PlannerSelection, + UnitTestSelectorSplit, ) +def apply_unit_test_selectors( + *, options: CompilePipelineOptions | None, discovered_inputs: DiscoveredProjectInputs +) -> CompilePipelineOptions: + """Return options whose selectors name only resources, with unit tests selected apart.""" + + resolved: CompilePipelineOptions = options if options is not None else CompilePipelineOptions() + if not resolved.select and not resolved.exclude: + return resolved + split: UnitTestSelectorSplit = split_project_unit_test_selectors( + select=resolved.select, + exclude=resolved.exclude, + discovered_inputs=discovered_inputs, + accepts_unit_tests=resolved.accepts_unit_test_selectors, + ) + return replace( + resolved, + select=split.select, + exclude=split.exclude, + sql_test_selection=split.sql_test_selection, + ) + + def resolve_compile_analysis_selection( *, options: CompilePipelineOptions, @@ -32,7 +60,9 @@ def resolve_compile_analysis_selection( or discovered_inputs.asset_functions or discovered_inputs.check_functions ) - if options.resolve_python_run_selectors and has_python_nodes: + if ( + options.resolve_python_run_selectors and has_python_nodes + ) or options.sql_test_selection.names: return CompileAnalysisSelection(no_cache=options.no_cache) return CompileAnalysisSelection( select=options.select, @@ -50,6 +80,8 @@ def resolve_configured_rule_selection( ) -> frozenset[CompiledObjectKey] | None: """Align configured-rule subjects with models that received deep SQL analysis.""" + if not options.sql_test_selection.models_selected: + return frozenset() analysis_selection: CompileAnalysisSelection | None = resolve_compile_analysis_selection( options=options, discovered_inputs=discovered_inputs, diff --git a/src/sqlbuild/compiler/pipeline/main/compile.py b/src/sqlbuild/compiler/pipeline/main/compile.py index b4a7800262..6bb15d617f 100644 --- a/src/sqlbuild/compiler/pipeline/main/compile.py +++ b/src/sqlbuild/compiler/pipeline/main/compile.py @@ -23,6 +23,7 @@ build_lineage_upstream_deps, ) from sqlbuild.compiler.pipeline._helpers.analysis_selection import ( + apply_unit_test_selectors, resolve_configured_rule_selection, ) from sqlbuild.compiler.pipeline._helpers.deferred_locations import ( @@ -83,8 +84,8 @@ def run_compile_pipeline( ) -> CompilePipelineResult: """Run compile inputs, assembly, planning, and manifest generation.""" - resolved_options: CompilePipelineOptions = ( - options if options is not None else CompilePipelineOptions() + resolved_options: CompilePipelineOptions = apply_unit_test_selectors( + options=options, discovered_inputs=discovered_inputs ) resolved_hooks: ConnectionHooks = hooks if hooks is not None else ConnectionHooks() on_progress: Callable[[str], None] | None = resolved_hooks.on_progress @@ -208,7 +209,9 @@ def _build_result( selected_python_node_names: frozenset[str] = frozenset() python_read_source_names: frozenset[str] = frozenset() run_selection: PythonSqlRunSelection | None = None - if options.resolve_python_run_selectors: + if not options.sql_test_selection.models_selected: + selected_sql_keys = frozenset() + elif options.resolve_python_run_selectors: run_selection = resolve_python_sql_run_selection_from_inputs( select=select, exclude=exclude, @@ -230,6 +233,7 @@ def _build_result( exclude=exclude, selected_keys=selected_sql_keys, python_read_source_names=python_read_source_names, + sql_test_selection=options.sql_test_selection, ), overrides=PlannerOverrides( cursor_overrides=options.cursor_overrides, diff --git a/src/sqlbuild/compiler/pipeline/main/reject_unit_test_selectors.py b/src/sqlbuild/compiler/pipeline/main/reject_unit_test_selectors.py new file mode 100644 index 0000000000..adedaf289c --- /dev/null +++ b/src/sqlbuild/compiler/pipeline/main/reject_unit_test_selectors.py @@ -0,0 +1,25 @@ +"""Public validation that a command which never runs unit tests selects none.""" + +from __future__ import annotations + +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.compiler.planner.main.selection._split_unit_test_selectors import ( + split_project_unit_test_selectors, +) +from sqlbuild.compiler.planner.models import UnitTestSelectorSplit + + +def reject_unit_test_selectors( + *, + select: tuple[str, ...], + exclude: tuple[str, ...], + discovered_inputs: DiscoveredProjectInputs, +) -> UnitTestSelectorSplit: + """Raise when a selector names a unit test; return the unchanged resource selectors.""" + + return split_project_unit_test_selectors( + select=select, + exclude=exclude, + discovered_inputs=discovered_inputs, + accepts_unit_tests=False, + ) diff --git a/src/sqlbuild/compiler/pipeline/main/static_command.py b/src/sqlbuild/compiler/pipeline/main/static_command.py index 9941c2c046..3ce77cf45c 100644 --- a/src/sqlbuild/compiler/pipeline/main/static_command.py +++ b/src/sqlbuild/compiler/pipeline/main/static_command.py @@ -7,6 +7,7 @@ from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter from sqlbuild.compiler.compile.models import CompiledObjectKey, CompiledRelationLocation from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.compiler.pipeline._helpers.analysis_selection import apply_unit_test_selectors from sqlbuild.compiler.pipeline._helpers.deferred_locations import ( build_deferred_locations, resolve_deferred_target_config, @@ -39,6 +40,9 @@ def compile_static_command_context( ) -> StaticCommandContext: """Run canonical compile, selector, and relation phases without build planning.""" + options = apply_unit_test_selectors(options=options, discovered_inputs=discovered_inputs) + if selected_keys is None and not options.sql_test_selection.models_selected: + selected_keys = frozenset() compiled: CompiledProjectPhaseResult = compile_project_phase( discovered_inputs=discovered_inputs, adapter=adapter, @@ -66,6 +70,7 @@ def compile_static_command_context( select=options.select, exclude=options.exclude, selected_keys=selected_keys, + sql_test_selection=options.sql_test_selection, ), auto_load_sources=options.auto_load_sources, ) diff --git a/src/sqlbuild/compiler/pipeline/main/unit_test_selection.py b/src/sqlbuild/compiler/pipeline/main/unit_test_selection.py new file mode 100644 index 0000000000..064b50b1b4 --- /dev/null +++ b/src/sqlbuild/compiler/pipeline/main/unit_test_selection.py @@ -0,0 +1,25 @@ +"""Public split of unit-test selectors from resource selectors for one command.""" + +from __future__ import annotations + +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.compiler.planner.main.selection._split_unit_test_selectors import ( + split_project_unit_test_selectors, +) +from sqlbuild.compiler.planner.models import UnitTestSelectorSplit + + +def split_unit_test_selectors( + *, + select: tuple[str, ...], + exclude: tuple[str, ...], + discovered_inputs: DiscoveredProjectInputs, +) -> UnitTestSelectorSplit: + """Split unit-test selectors from resource selectors for a command that runs unit tests.""" + + return split_project_unit_test_selectors( + select=select, + exclude=exclude, + discovered_inputs=discovered_inputs, + accepts_unit_tests=True, + ) diff --git a/src/sqlbuild/compiler/pipeline/models.py b/src/sqlbuild/compiler/pipeline/models.py index ae91eaec2d..9a00d04fb6 100644 --- a/src/sqlbuild/compiler/pipeline/models.py +++ b/src/sqlbuild/compiler/pipeline/models.py @@ -17,6 +17,7 @@ PlannerScope, PlanOutput, SeedPlanEntry, + SqlTestSelection, ) from sqlbuild.compiler.python_nodes.types import ( PythonIdentityStatus, @@ -62,6 +63,8 @@ class CompilePipelineOptions: selection_diagnostics: bool = False plan_sql_tests: bool = True record_migration_fingerprints: bool = True + accepts_unit_test_selectors: bool = False + sql_test_selection: SqlTestSelection = field(default_factory=SqlTestSelection) @dataclass(frozen=True) diff --git a/src/sqlbuild/compiler/planner/_helpers/command_planning/static.py b/src/sqlbuild/compiler/planner/_helpers/command_planning/static.py index d430ccc4b0..b16b0a068d 100644 --- a/src/sqlbuild/compiler/planner/_helpers/command_planning/static.py +++ b/src/sqlbuild/compiler/planner/_helpers/command_planning/static.py @@ -12,12 +12,12 @@ from sqlbuild.compiler.planner._helpers.output.plan_entry import ( build_planner_relations_context, extract_seed_columns, - scope_overlaps, ) from sqlbuild.compiler.planner._helpers.output.plan_output import ( build_selected_audit_entries, build_selected_test_entries, plan_function, + sql_test_is_selected, ) from sqlbuild.compiler.planner._helpers.output.strategy import get_materialization_type from sqlbuild.compiler.planner._helpers.planning.scopes import resolve_planner_scopes @@ -120,6 +120,7 @@ def build_test_command_plan_impl( project=project, adapter=adapter, selected_keys=scope.selected_keys, + sql_test_selection=scope.sql_test_selection, case_name=case_name, ) available_test_case_names: tuple[str, ...] = tuple( @@ -128,7 +129,11 @@ def build_test_command_plan_impl( test.case_name for test in project.sql_tests if test.case_name is not None - and scope_overlaps(scope_deps=test.scope_deps, selected_keys=scope.selected_keys) + and sql_test_is_selected( + test=test, + selected_keys=scope.selected_keys, + sql_test_selection=scope.sql_test_selection, + ) } ) ) diff --git a/src/sqlbuild/compiler/planner/_helpers/graph/scope.py b/src/sqlbuild/compiler/planner/_helpers/graph/scope.py index 69abcb5a09..194c20e6b8 100644 --- a/src/sqlbuild/compiler/planner/_helpers/graph/scope.py +++ b/src/sqlbuild/compiler/planner/_helpers/graph/scope.py @@ -32,7 +32,12 @@ build_model_tag_index_impl as build_model_tag_index, ) from sqlbuild.compiler.planner._helpers.graph.selectors import parse_selector, resolve_selectors -from sqlbuild.compiler.planner.models import ParsedSelector, PathSelector, PlannerScope +from sqlbuild.compiler.planner.models import ( + ParsedSelector, + PathSelector, + PlannerScope, + SqlTestSelection, +) from sqlbuild.compiler.planner.types import SelectorKind @@ -44,6 +49,7 @@ def build_planner_scope( auto_load_sources: bool, selected_keys: frozenset[CompiledObjectKey] | None = None, python_read_source_names: frozenset[str] = frozenset(), + sql_test_selection: SqlTestSelection | None = None, ) -> PlannerScope: upstream_deps: dict[CompiledObjectKey, tuple[CompiledObjectKey, ...]] = ( build_execution_upstream_deps(project) @@ -102,6 +108,7 @@ def build_planner_scope( ), user_selected_keys=resolved_selected_keys, python_read_source_names=python_read_source_names, + sql_test_selection=sql_test_selection or SqlTestSelection(), ) diff --git a/src/sqlbuild/compiler/planner/_helpers/graph/selectors.py b/src/sqlbuild/compiler/planner/_helpers/graph/selectors.py index b09b9a84c2..d4c0cbbee0 100644 --- a/src/sqlbuild/compiler/planner/_helpers/graph/selectors.py +++ b/src/sqlbuild/compiler/planner/_helpers/graph/selectors.py @@ -2,6 +2,8 @@ from __future__ import annotations +from collections.abc import Iterable +from difflib import get_close_matches from fnmatch import fnmatchcase from sqlbuild.compiler.compile.models import CompiledObjectKey @@ -20,7 +22,11 @@ SELECTOR_KIND_SEPARATOR, SELECTOR_MISSING_NAME_ERROR_FRAGMENT, SELECTOR_PATH_SEPARATOR, + SELECTOR_SUGGESTION_CUTOFF, + SELECTOR_SUGGESTION_LIMIT, SQL_FILE_SELECTOR_SUFFIX, + UNIT_TEST_SELECTOR_ERROR_CODE, + UNIT_TEST_SELECTOR_ONLY_TEST_AND_BUILD, ) from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.compiler.planner.main.selection.selector_expansion import split_selector_expansion @@ -37,6 +43,7 @@ SelectorKind.CHECK: SelectorKind.CHECK, SelectorKind.TAG: SelectorKind.TAG, SelectorKind.PATH: SelectorKind.PATH, + SelectorKind.TEST: SelectorKind.TEST, } _RESOURCE_TYPE_BY_SELECTOR_KIND: dict[SelectorKind, CompiledResourceType] = { @@ -199,10 +206,19 @@ def match_selector_keys( if parsed.kind == SelectorKind.PATH: return _match_path(value=parsed.value, path_index=path_index) + if parsed.kind == SelectorKind.TEST: + raise unit_test_selector_rejection( + selector=f"{SelectorKind.TEST}{SELECTOR_KIND_SEPARATOR}{parsed.value}" + ) + keys: frozenset[CompiledObjectKey] = _lookup_keys(parsed=parsed, all_keys=all_keys) if not keys: label: str = "pattern" if _is_name_pattern(parsed.value) else "name" - raise PlannerInputError(f"unknown selector {label} '{parsed.value}'", code="S007") + raise PlannerInputError( + f"unknown selector {label} '{parsed.value}'", + code="S007", + help=selector_name_help(value=parsed.value, candidates=all_keys), + ) return keys @@ -415,3 +431,25 @@ def _lookup_keys( def _is_name_pattern(value: str) -> bool: return any(character in value for character in "*?[") + + +def selector_name_help(*, value: str, candidates: Iterable[str]) -> str | None: + """Suggest the nearest selectable names for an unknown selector name.""" + + if _is_name_pattern(value): + return None + matches: list[str] = get_close_matches( + value, sorted(candidates), n=SELECTOR_SUGGESTION_LIMIT, cutoff=SELECTOR_SUGGESTION_CUTOFF + ) + if not matches: + return None + return f"did you mean {', '.join(f"'{match}'" for match in matches)}?" + + +def unit_test_selector_rejection(*, selector: str) -> PlannerInputError: + """The error for a unit-test selector given to a command that does not run unit tests.""" + + return PlannerInputError( + f"selector '{selector}' selects a unit test; {UNIT_TEST_SELECTOR_ONLY_TEST_AND_BUILD}", + code=UNIT_TEST_SELECTOR_ERROR_CODE, + ) diff --git a/src/sqlbuild/compiler/planner/_helpers/output/plan_output.py b/src/sqlbuild/compiler/planner/_helpers/output/plan_output.py index a6320c8265..fcb530f027 100644 --- a/src/sqlbuild/compiler/planner/_helpers/output/plan_output.py +++ b/src/sqlbuild/compiler/planner/_helpers/output/plan_output.py @@ -58,6 +58,7 @@ SourceLoadPlanEntry, SqlTestPlanEntry, SqlTestPlanResult, + SqlTestSelection, WarehouseSnapshot, ) from sqlbuild.compiler.planner.types import PlanReason @@ -132,9 +133,16 @@ def build_plan_output( model_materializations=model_materializations, ) planned_tests: PlannedSqlTests | None = resolved_extras.planned_sql_tests - if planned_tests is None or planned_tests.selected_keys != scope.selected_keys: + if ( + planned_tests is None + or planned_tests.selected_keys != scope.selected_keys + or planned_tests.sql_test_selection != scope.sql_test_selection + ): planned_tests = plan_selected_sql_tests( - project=project, adapter=adapter, selected_keys=scope.selected_keys + project=project, + adapter=adapter, + selected_keys=scope.selected_keys, + sql_test_selection=scope.sql_test_selection, ) test_entries: list[SqlTestPlanEntry] = list(planned_tests.entries) test_warnings: list[PlanWarning] = list(planned_tests.warnings) @@ -449,16 +457,39 @@ def plan_selected_sql_tests( project: CompiledProject, adapter: BaseAdapter, selected_keys: frozenset[CompiledObjectKey], + sql_test_selection: SqlTestSelection, ) -> PlannedSqlTests: - """Plan SQL tests overlapping one selection into a reusable result.""" + """Plan SQL tests selected by name or overlapping one selection into a reusable result.""" entries: list[SqlTestPlanEntry] warnings: list[PlanWarning] entries, warnings = build_selected_test_entries( - project=project, adapter=adapter, selected_keys=selected_keys + project=project, + adapter=adapter, + selected_keys=selected_keys, + sql_test_selection=sql_test_selection, ) return PlannedSqlTests( - selected_keys=selected_keys, entries=tuple(entries), warnings=tuple(warnings) + selected_keys=selected_keys, + sql_test_selection=sql_test_selection, + entries=tuple(entries), + warnings=tuple(warnings), + ) + + +def sql_test_is_selected( + *, + test: CompiledSqlTest, + selected_keys: frozenset[CompiledObjectKey], + sql_test_selection: SqlTestSelection, +) -> bool: + """Whether a test is named by the selection or covers a selected resource, and not excluded.""" + + name: str = test.parent_name or test.name + if name in sql_test_selection.excluded_names: + return False + return name in sql_test_selection.names or scope_overlaps( + scope_deps=test.scope_deps, selected_keys=selected_keys ) @@ -467,6 +498,7 @@ def build_selected_test_entries( project: CompiledProject, adapter: BaseAdapter, selected_keys: frozenset[CompiledObjectKey], + sql_test_selection: SqlTestSelection, case_name: str | None = None, ) -> tuple[list[SqlTestPlanEntry], list[PlanWarning]]: entries: list[SqlTestPlanEntry] = [] @@ -475,7 +507,9 @@ def build_selected_test_entries( selected_tests: list[CompiledSqlTest] = [] sql_test: CompiledSqlTest for sql_test in project.sql_tests: - if not scope_overlaps(scope_deps=sql_test.scope_deps, selected_keys=selected_keys): + if not sql_test_is_selected( + test=sql_test, selected_keys=selected_keys, sql_test_selection=sql_test_selection + ): continue if case_name is not None and sql_test.case_name != case_name: continue diff --git a/src/sqlbuild/compiler/planner/_helpers/planning/scopes.py b/src/sqlbuild/compiler/planner/_helpers/planning/scopes.py index 61e5a2aef4..21b80bec6b 100644 --- a/src/sqlbuild/compiler/planner/_helpers/planning/scopes.py +++ b/src/sqlbuild/compiler/planner/_helpers/planning/scopes.py @@ -30,6 +30,7 @@ def resolve_planner_scopes( auto_load_sources=policies.auto_load_sources, selected_keys=selection.selected_keys, python_read_source_names=selection.python_read_source_names, + sql_test_selection=selection.sql_test_selection, ) full_scope: PlannerScope = build_planner_scope( project=project, diff --git a/src/sqlbuild/compiler/planner/_helpers/selection/unit_test_selectors.py b/src/sqlbuild/compiler/planner/_helpers/selection/unit_test_selectors.py new file mode 100644 index 0000000000..17b39561dc --- /dev/null +++ b/src/sqlbuild/compiler/planner/_helpers/selection/unit_test_selectors.py @@ -0,0 +1,207 @@ +"""Unit-test selectors: split them from resource selectors or reject them where tests never run.""" + +from __future__ import annotations + +from fnmatch import fnmatchcase + +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.compiler.planner._helpers.graph.selectors import ( + parse_selector, + selector_name_help, + unit_test_selector_rejection, +) +from sqlbuild.compiler.planner.constants import ( + UNIT_TEST_SELECTOR_ERROR_CODE, + UNKNOWN_SELECTOR_ERROR_CODE, +) +from sqlbuild.compiler.planner.exceptions import PlannerInputError +from sqlbuild.compiler.planner.models import ( + ParsedSelector, + PathSelector, + SqlTestSelection, + UnitTestSelectorNames, + UnitTestSelectorSplit, +) +from sqlbuild.compiler.planner.types import SelectorKind + + +def split_discovered_unit_test_selectors( + *, + select: tuple[str, ...], + exclude: tuple[str, ...], + discovered_inputs: DiscoveredProjectInputs, + accepts_unit_tests: bool, +) -> UnitTestSelectorSplit: + """Split unit-test selectors from resource selectors using discovered project names.""" + + return split_unit_test_selectors( + select=select, + exclude=exclude, + names=UnitTestSelectorNames( + tests=_unit_test_names(discovered_inputs), + resources=_resource_names(discovered_inputs), + ), + accepts_unit_tests=accepts_unit_tests, + ) + + +def split_unit_test_selectors( + *, + select: tuple[str, ...], + exclude: tuple[str, ...], + names: UnitTestSelectorNames, + accepts_unit_tests: bool, +) -> UnitTestSelectorSplit: + """Move unit-test selectors out of resource selectors, or reject them for this command.""" + + selected_tests, resource_select = _split_groups( + groups=select, names=names, accepts_unit_tests=accepts_unit_tests + ) + excluded_tests, resource_exclude = _split_groups( + groups=exclude, names=names, accepts_unit_tests=accepts_unit_tests + ) + return UnitTestSelectorSplit( + select=resource_select, + exclude=resource_exclude, + sql_test_selection=SqlTestSelection( + names=selected_tests, + excluded_names=excluded_tests, + models_selected=not select or bool(resource_select) or not selected_tests, + ), + ) + + +def _split_groups( + *, groups: tuple[str, ...], names: UnitTestSelectorNames, accepts_unit_tests: bool +) -> tuple[frozenset[str], tuple[str, ...]]: + tests: set[str] = set() + kept_groups: list[str] = [] + for group in groups: + tokens: list[str] = group.split() + kept_tokens: list[str] = [] + for token in tokens: + matched, keep = _split_token( + token=token, names=names, accepts_unit_tests=accepts_unit_tests + ) + tests.update(matched) + if keep: + kept_tokens.append(token) + if len(kept_tokens) == len(tokens): + kept_groups.append(group) + elif kept_tokens: + kept_groups.append(" ".join(kept_tokens)) + return frozenset(tests), tuple(kept_groups) + + +def _split_token( + *, token: str, names: UnitTestSelectorNames, accepts_unit_tests: bool +) -> tuple[frozenset[str], bool]: + parts: list[str] = token.split(",") + classified: list[tuple[frozenset[str], bool]] = [ + _classify(part=part, names=names, accepts_unit_tests=accepts_unit_tests) for part in parts + ] + selects_tests: bool = any(matched and not keep for matched, keep in classified) + if len(parts) > 1 and selects_tests: + for part in parts: + parse_selector(part) + raise PlannerInputError( + f"selector '{token}' intersects a unit test with ','; unit tests can only be " + "selected whole, by name", + code=UNIT_TEST_SELECTOR_ERROR_CODE, + ) + if len(parts) > 1: + return frozenset(), True + return classified[0] + + +def _classify( + *, part: str, names: UnitTestSelectorNames, accepts_unit_tests: bool +) -> tuple[frozenset[str], bool]: + """Return the unit tests one selector selects and whether it also selects resources.""" + + try: + parsed: ParsedSelector | PathSelector = parse_selector(part) + except PlannerInputError: + return frozenset(), True + if isinstance(parsed, PathSelector) or parsed.kind not in { + SelectorKind.NAME, + SelectorKind.TEST, + }: + return frozenset(), True + matched_tests: frozenset[str] = _matching(value=parsed.value, names=names.tests) + selects_resources: bool = parsed.kind == SelectorKind.NAME and bool( + _matching(value=parsed.value, names=names.resources) + ) + if selects_resources: + return frozenset(), True + if not matched_tests: + if parsed.kind == SelectorKind.TEST or accepts_unit_tests: + _raise_unknown(parsed=parsed, names=names) + return frozenset(), True + if not accepts_unit_tests: + raise unit_test_selector_rejection(selector=part) + if parsed.upstream or parsed.downstream: + raise PlannerInputError( + f"selector '{part}' expands a unit test with '+'; unit tests have no lineage, so " + "select them by name without '+'", + code=UNIT_TEST_SELECTOR_ERROR_CODE, + ) + return matched_tests, False + + +def _raise_unknown(*, parsed: ParsedSelector, names: UnitTestSelectorNames) -> None: + label: str = "pattern" if _is_pattern(parsed.value) else "name" + candidates: frozenset[str] = names.tests + subject: str = "unit test" + if parsed.kind == SelectorKind.NAME: + candidates = names.tests | names.resources + subject = "selector" + raise PlannerInputError( + f"unknown {subject} {label} '{parsed.value}'", + code=UNKNOWN_SELECTOR_ERROR_CODE, + help=selector_name_help(value=parsed.value, candidates=candidates), + ) + + +def _matching(*, value: str, names: frozenset[str]) -> frozenset[str]: + if not _is_pattern(value): + return frozenset({value}) & names + return frozenset(name for name in names if fnmatchcase(name, value)) + + +def _is_pattern(value: str) -> bool: + return any(character in value for character in "*?[") + + +def _unit_test_names(discovered_inputs: DiscoveredProjectInputs) -> frozenset[str]: + names: set[str] = set() + for test_file in discovered_inputs.test_files: + names.update(block.name or test_file.relative_path.stem for block in test_file.blocks) + return frozenset(names) + + +def _resource_names(discovered_inputs: DiscoveredProjectInputs) -> frozenset[str]: + names: set[str] = {model_file.file_path.stem for model_file in discovered_inputs.model_files} + for schema_file in discovered_inputs.schema_files: + names.update(entry.name for entry in schema_file.model_entries) + names.update(entry.name for entry in schema_file.seed_entries) + for source_file in discovered_inputs.source_files: + names.update(entry.name for entry in source_file.source_entries) + names.update(seed_file.file_path.stem for seed_file in discovered_inputs.seed_files) + names.update( + function_file.file_path.stem + for function_file in ( + *discovered_inputs.sql_function_files, + *discovered_inputs.python_function_files, + ) + ) + names.update( + node.name + for node in ( + *discovered_inputs.loader_functions, + *discovered_inputs.task_functions, + *discovered_inputs.asset_functions, + *discovered_inputs.check_functions, + ) + ) + return frozenset(names) diff --git a/src/sqlbuild/compiler/planner/classes/background_sql_test_planning.py b/src/sqlbuild/compiler/planner/classes/background_sql_test_planning.py index c66c3d28ab..933b9bc193 100644 --- a/src/sqlbuild/compiler/planner/classes/background_sql_test_planning.py +++ b/src/sqlbuild/compiler/planner/classes/background_sql_test_planning.py @@ -9,7 +9,7 @@ from sqlbuild.adapter.contract.classes.base_adapter import BaseAdapter from sqlbuild.compiler.compile.models import CompiledObjectKey, CompiledProject from sqlbuild.compiler.planner._helpers.output.plan_output import plan_selected_sql_tests -from sqlbuild.compiler.planner.models import PlannedSqlTests +from sqlbuild.compiler.planner.models import PlannedSqlTests, SqlTestSelection class BackgroundSqlTestPlanning: @@ -21,6 +21,7 @@ def __init__( project: CompiledProject, adapter: BaseAdapter, selected_keys: frozenset[CompiledObjectKey], + sql_test_selection: SqlTestSelection, enabled: bool, ) -> None: self._future: Future[PlannedSqlTests] | None = None @@ -34,11 +35,13 @@ def __init__( "project": project, "adapter": adapter, "selected_keys": selected_keys, + "sql_test_selection": sql_test_selection, }, name="sqlbuild-test-planning", daemon=True, ).start() self._selected_keys: frozenset[CompiledObjectKey] = selected_keys + self._sql_test_selection: SqlTestSelection = sql_test_selection def __enter__(self) -> BackgroundSqlTestPlanning: return self @@ -55,7 +58,9 @@ def result(self) -> PlannedSqlTests: """Return planned tests, raising any planning error at the original assembly point.""" if self._future is None: - return PlannedSqlTests(selected_keys=self._selected_keys) + return PlannedSqlTests( + selected_keys=self._selected_keys, sql_test_selection=self._sql_test_selection + ) return self._future.result() @@ -65,12 +70,18 @@ def _plan_into( project: CompiledProject, adapter: BaseAdapter, selected_keys: frozenset[CompiledObjectKey], + sql_test_selection: SqlTestSelection, ) -> None: """Run planning on a daemon thread so a failed plan never waits for it at exit.""" try: future.set_result( - plan_selected_sql_tests(project=project, adapter=adapter, selected_keys=selected_keys) + plan_selected_sql_tests( + project=project, + adapter=adapter, + selected_keys=selected_keys, + sql_test_selection=sql_test_selection, + ) ) except BaseException as error: # noqa: BLE001 - re-raised on the joining thread future.set_exception(error) diff --git a/src/sqlbuild/compiler/planner/constants.py b/src/sqlbuild/compiler/planner/constants.py index 5af959ebbd..7aedfb065d 100644 --- a/src/sqlbuild/compiler/planner/constants.py +++ b/src/sqlbuild/compiler/planner/constants.py @@ -33,6 +33,13 @@ UNIFIED_DIFF_REMOVAL_PREFIX: str = "-" UNIFIED_DIFF_REMOVAL_HEADER_PREFIX: str = "---" SELECTOR_EXPANSION_MARKER: str = "+" +SELECTOR_SUGGESTION_CUTOFF: float = 0.8 +SELECTOR_SUGGESTION_LIMIT: int = 3 +UNIT_TEST_SELECTOR_ERROR_CODE: str = "S013" +UNKNOWN_SELECTOR_ERROR_CODE: str = "S007" +UNIT_TEST_SELECTOR_ONLY_TEST_AND_BUILD: str = ( + "only `sqb test` and `sqb build` accept unit-test selectors" +) MODEL_CUSTOM_CONFIG_KEY: str = "config" MODEL_PLACEHOLDERS_CONFIG_KEY: str = "placeholders" MODEL_PRE_HOOKS_CONFIG_KEY: str = "pre_hooks" diff --git a/src/sqlbuild/compiler/planner/main/execution/execution.py b/src/sqlbuild/compiler/planner/main/execution/execution.py index b58abab25e..3be65d53e7 100644 --- a/src/sqlbuild/compiler/planner/main/execution/execution.py +++ b/src/sqlbuild/compiler/planner/main/execution/execution.py @@ -98,6 +98,7 @@ def build_execution_plan( project=project, adapter=adapter, selected_keys=scopes.inspection_scope.selected_keys, + sql_test_selection=scopes.inspection_scope.sql_test_selection, enabled=policies.plan_sql_tests, ) as test_planning, open_inspection_catalog(adapter=adapter, connection=connection), diff --git a/src/sqlbuild/compiler/planner/main/selection/_split_unit_test_selectors.py b/src/sqlbuild/compiler/planner/main/selection/_split_unit_test_selectors.py new file mode 100644 index 0000000000..4d7a8a7270 --- /dev/null +++ b/src/sqlbuild/compiler/planner/main/selection/_split_unit_test_selectors.py @@ -0,0 +1,26 @@ +"""Compiler-internal unit-test selector split entrypoint.""" + +from __future__ import annotations + +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.compiler.planner._helpers.selection.unit_test_selectors import ( + split_discovered_unit_test_selectors, +) +from sqlbuild.compiler.planner.models import UnitTestSelectorSplit + + +def split_project_unit_test_selectors( + *, + select: tuple[str, ...], + exclude: tuple[str, ...], + discovered_inputs: DiscoveredProjectInputs, + accepts_unit_tests: bool, +) -> UnitTestSelectorSplit: + """Move unit-test selectors out of resource selectors, or reject them for this command.""" + + return split_discovered_unit_test_selectors( + select=select, + exclude=exclude, + discovered_inputs=discovered_inputs, + accepts_unit_tests=accepts_unit_tests, + ) diff --git a/src/sqlbuild/compiler/planner/main/selection/unit_test_selector_error.py b/src/sqlbuild/compiler/planner/main/selection/unit_test_selector_error.py new file mode 100644 index 0000000000..2aab683cb6 --- /dev/null +++ b/src/sqlbuild/compiler/planner/main/selection/unit_test_selector_error.py @@ -0,0 +1,12 @@ +"""Public error for unit-test selectors where unit tests never run.""" + +from __future__ import annotations + +from sqlbuild.compiler.planner._helpers.graph.selectors import unit_test_selector_rejection +from sqlbuild.compiler.planner.exceptions import PlannerInputError + + +def unit_test_selector_error(*, selector: str) -> PlannerInputError: + """The error for a unit-test selector given where unit tests never run.""" + + return unit_test_selector_rejection(selector=selector) diff --git a/src/sqlbuild/compiler/planner/models.py b/src/sqlbuild/compiler/planner/models.py index 775c00e55d..b726b8ff07 100644 --- a/src/sqlbuild/compiler/planner/models.py +++ b/src/sqlbuild/compiler/planner/models.py @@ -776,6 +776,32 @@ class ChangeDetectionResult: changed_functions: tuple[str, ...] = field(default_factory=tuple) +@dataclass(frozen=True) +class SqlTestSelection: + """Unit tests selected or excluded by name, beside the tests of the selected resources.""" + + names: frozenset[str] = frozenset() + excluded_names: frozenset[str] = frozenset() + models_selected: bool = True + + +@dataclass(frozen=True) +class UnitTestSelectorNames: + """Names selectors can match: unit tests, and every other selectable resource.""" + + tests: frozenset[str] + resources: frozenset[str] + + +@dataclass(frozen=True) +class UnitTestSelectorSplit: + """Resource selectors left after unit-test selectors moved into a test selection.""" + + select: tuple[str, ...] + exclude: tuple[str, ...] + sql_test_selection: SqlTestSelection + + @dataclass(frozen=True) class PlannerScope: """Resolved graph scope for one planner invocation.""" @@ -788,6 +814,7 @@ class PlannerScope: execution_order: tuple[CompiledObjectKey, ...] user_selected_keys: frozenset[CompiledObjectKey] = frozenset() python_read_source_names: frozenset[str] = frozenset() + sql_test_selection: SqlTestSelection = field(default_factory=SqlTestSelection) @dataclass(frozen=True) @@ -1152,9 +1179,10 @@ class PlanOutputExtras: @dataclass(frozen=True) class PlannedSqlTests: - """SQL test entries planned for one selection; reusable only for the same selected keys.""" + """SQL test entries planned for one selection; reusable only for the same selection.""" selected_keys: frozenset[CompiledObjectKey] + sql_test_selection: SqlTestSelection = field(default_factory=SqlTestSelection) entries: tuple[SqlTestPlanEntry, ...] = () warnings: tuple[PlanWarning, ...] = () @@ -1586,6 +1614,7 @@ class PlannerSelection: exclude: tuple[str, ...] = () selected_keys: frozenset[CompiledObjectKey] | None = None python_read_source_names: frozenset[str] = frozenset() + sql_test_selection: SqlTestSelection = field(default_factory=SqlTestSelection) @dataclass(frozen=True) diff --git a/src/sqlbuild/compiler/planner/types.py b/src/sqlbuild/compiler/planner/types.py index 1ae4ff8cab..b203914048 100644 --- a/src/sqlbuild/compiler/planner/types.py +++ b/src/sqlbuild/compiler/planner/types.py @@ -40,6 +40,7 @@ class SelectorKind(StrEnum): CHECK = "check" TAG = "tag" PATH = "path" + TEST = "test" class GraphResourceKind(StrEnum): diff --git a/src/sqlbuild/compiler/python_nodes/_helpers/selectors.py b/src/sqlbuild/compiler/python_nodes/_helpers/selectors.py index 007fb60e72..f3044420de 100644 --- a/src/sqlbuild/compiler/python_nodes/_helpers/selectors.py +++ b/src/sqlbuild/compiler/python_nodes/_helpers/selectors.py @@ -8,6 +8,9 @@ from sqlbuild.compiler.planner.constants import PATH_SELECTOR_EXPLICIT_ROOT_ERROR from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.compiler.planner.main.selection.selector_parse import parse_project_selector +from sqlbuild.compiler.planner.main.selection.unit_test_selector_error import ( + unit_test_selector_error, +) from sqlbuild.compiler.planner.models import ParsedSelector, PathSelector from sqlbuild.compiler.planner.types import SelectorKind from sqlbuild.compiler.python_nodes.constants import ( @@ -74,6 +77,8 @@ def _resolve_single(*, raw: str, graph: PythonNodeGraph) -> frozenset[str]: code="S011", ) + if parsed.kind == SelectorKind.TEST: + raise unit_test_selector_error(selector=raw) if parsed.kind == SelectorKind.TAG: return _resolve_tag(parsed=parsed, graph=graph) diff --git a/src/sqlbuild/compiler/python_nodes/_helpers/unified_selectors.py b/src/sqlbuild/compiler/python_nodes/_helpers/unified_selectors.py index af24f270b2..965fc95118 100644 --- a/src/sqlbuild/compiler/python_nodes/_helpers/unified_selectors.py +++ b/src/sqlbuild/compiler/python_nodes/_helpers/unified_selectors.py @@ -22,6 +22,9 @@ from sqlbuild.compiler.planner.main.selection.selection import resolve_project_selectors from sqlbuild.compiler.planner.main.selection.selector_expansion import split_selector_expansion from sqlbuild.compiler.planner.main.selection.selector_parse import parse_project_selector +from sqlbuild.compiler.planner.main.selection.unit_test_selector_error import ( + unit_test_selector_error, +) from sqlbuild.compiler.planner.models import ParsedSelector, PathSelector from sqlbuild.compiler.planner.types import SelectorKind from sqlbuild.compiler.python_nodes._helpers.selectors import resolve_python_node_selectors @@ -368,6 +371,8 @@ def _resolve_single_side( ) ) return atoms + if parsed.kind == SelectorKind.TEST: + raise unit_test_selector_error(selector=raw) if parsed.kind == SelectorKind.PATH: atoms = _resolve_path(raw=raw, project_graph=project_graph, python_graph=python_graph) if parsed.upstream: diff --git a/src/sqlbuild/integrations/dagster/_helpers/selection.py b/src/sqlbuild/integrations/dagster/_helpers/selection.py index 05ae1aa30f..ce66f8cbad 100644 --- a/src/sqlbuild/integrations/dagster/_helpers/selection.py +++ b/src/sqlbuild/integrations/dagster/_helpers/selection.py @@ -7,7 +7,11 @@ from sqlbuild.compiler.graph.main.path_nodes import path_nodes from sqlbuild.compiler.graph.main.transitive_closure_many import transitive_closure_many +from sqlbuild.compiler.planner.constants import SELECTOR_KIND_SEPARATOR from sqlbuild.compiler.planner.main.selection.selector_parse import parse_project_selector +from sqlbuild.compiler.planner.main.selection.unit_test_selector_error import ( + unit_test_selector_error, +) from sqlbuild.compiler.planner.models import ParsedSelector, PathSelector from sqlbuild.compiler.planner.types import SelectorKind from sqlbuild.integrations.dagster._helpers.dag import load_sqlbuild_dag @@ -145,6 +149,10 @@ def _match_parsed_selector( dag: Mapping[str, Any], nodes_by_id: Mapping[str, Mapping[str, Any]], ) -> set[str]: + if parsed.kind == SelectorKind.TEST: + raise unit_test_selector_error( + selector=f"{SelectorKind.TEST}{SELECTOR_KIND_SEPARATOR}{parsed.value}" + ) if parsed.kind == SelectorKind.TAG: return { node_id diff --git a/tests/e2e/fixtures/dbt_interop/sqlbuild_project/tests/scenarios/downstream_orders.sql b/tests/e2e/fixtures/dbt_interop/sqlbuild_project/tests/scenarios/downstream_orders_from_mocked_ref.sql similarity index 100% rename from tests/e2e/fixtures/dbt_interop/sqlbuild_project/tests/scenarios/downstream_orders.sql rename to tests/e2e/fixtures/dbt_interop/sqlbuild_project/tests/scenarios/downstream_orders_from_mocked_ref.sql diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/rules/test_unused_cte_contexts.py b/tests/e2e/src/sqlbuild/cli/commands/main/rules/test_unused_cte_contexts.py index 7a7e88507a..5423fcdc15 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/rules/test_unused_cte_contexts.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/rules/test_unused_cte_contexts.py @@ -134,7 +134,7 @@ "models/orders.sql": ( 'MODEL (description "Test model orders.", materialized table);\n\nSELECT * FROM __ref("raw_orders")\n' ), - "tests/unit/orders.sql": dedent( + "tests/unit/orders_test.sql": dedent( """ TEST (); diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_scenario.py b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_scenario.py index 64c8fdf399..70e7a62f47 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_scenario.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/scenario/test_scenario.py @@ -193,8 +193,8 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n expected_count=1, additional_project_files=( ( - "models/local_model_error.sql", - "MODEL (description 'Test model local_model_error.', materialized table);\n\n" + "models/local_model_error_rows.sql", + "MODEL (description 'Test model local_model_error_rows.', materialized table);\n\n" "SELECT CAST('not_a_number_' || amount AS INTEGER) AS bad_value\n" 'FROM __source("raw_orders")\n', ), @@ -205,7 +205,7 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_model_error AS (\n" + "__expected__local_model_error_rows AS (\n" " SELECT 10 AS bad_value\n" ")\n" "SELECT 1\n", @@ -233,8 +233,8 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "FUNCTION (description 'Test function is_large_order.', arguments (amount INTEGER), returns BOOLEAN);\n\namount > 9\n", ), ( - "models/local_sql_function_pass.sql", - "MODEL (description 'Test model local_sql_function_pass.', materialized table);\n\n" + "models/local_sql_function_pass_rows.sql", + "MODEL (description 'Test model local_sql_function_pass_rows.', materialized table);\n\n" 'SELECT id, __udf("is_large_order")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -247,7 +247,7 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n " UNION ALL\n" " SELECT 2 AS id, 5 AS amount\n" "),\n" - "__expected__local_sql_function_pass AS (\n" + "__expected__local_sql_function_pass_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" " UNION ALL\n" " SELECT 2 AS id, FALSE AS is_large_order\n" @@ -281,8 +281,8 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "FUNCTION (description 'Test function bad_sql_function.', arguments (amount INTEGER), returns BOOLEAN);\n\nmissing_col > 9\n", ), ( - "models/local_sql_function_setup_error.sql", - "MODEL (description 'Test model local_sql_function_setup_error.', materialized table);\n\n" + "models/local_sql_function_setup_error_rows.sql", + "MODEL (description 'Test model local_sql_function_setup_error_rows.', materialized table);\n\n" 'SELECT id, __udf("bad_sql_function")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -293,7 +293,7 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_sql_function_setup_error AS (\n" + "__expected__local_sql_function_setup_error_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" ")\n" "SELECT 1\n", @@ -336,8 +336,8 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "FUNCTION (description 'Test function bad_transpile_function.', arguments (amount INTEGER), returns BOOLEAN);\n\namount > 9\n", ), ( - "models/local_sql_function_transpile_error.sql", - "MODEL (description 'Test model local_sql_function_transpile_error.', materialized table);\n\n" + "models/local_sql_function_transpile_error_rows.sql", + "MODEL (description 'Test model local_sql_function_transpile_error_rows.', materialized table);\n\n" 'SELECT id, __udf("bad_transpile_function")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -348,7 +348,7 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_sql_function_transpile_error AS (\n" + "__expected__local_sql_function_transpile_error_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" ")\n" "SELECT 1\n", @@ -380,8 +380,8 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "CAST('bad' AS INTEGER)\n", ), ( - "models/local_sql_function_runtime_error.sql", - "MODEL (description 'Test model local_sql_function_runtime_error.', materialized table);\n\n" + "models/local_sql_function_runtime_error_rows.sql", + "MODEL (description 'Test model local_sql_function_runtime_error_rows.', materialized table);\n\n" 'SELECT id, __udf("sql_runtime_error")(amount) AS bad_value\n' 'FROM __source("raw_orders")\n', ), @@ -392,7 +392,7 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_sql_function_runtime_error AS (\n" + "__expected__local_sql_function_runtime_error_rows AS (\n" " SELECT 1 AS id, 10 AS bad_value\n" ")\n" "SELECT 1\n", @@ -427,8 +427,8 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n " '''Test function main.'''\n return amount is not None and amount > 9\n", ), ( - "models/local_python_function_pass.sql", - "MODEL (description 'Test model local_python_function_pass.', materialized table);\n\n" + "models/local_python_function_pass_rows.sql", + "MODEL (description 'Test model local_python_function_pass_rows.', materialized table);\n\n" 'SELECT id, __udf("is_large_order_py")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -441,7 +441,7 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n " UNION ALL\n" " SELECT 2 AS id, 5 AS amount\n" "),\n" - "__expected__local_python_function_pass AS (\n" + "__expected__local_python_function_pass_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" " UNION ALL\n" " SELECT 2 AS id, FALSE AS is_large_order\n" @@ -482,8 +482,8 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n " '''Test function main.'''\n return amount is not None and amount > 9\n", ), ( - "models/local_python_function_error.sql", - "MODEL (description 'Test model local_python_function_error.', materialized table);\n\n" + "models/local_python_function_error_rows.sql", + "MODEL (description 'Test model local_python_function_error_rows.', materialized table);\n\n" 'SELECT id, __udf("is_large_order_py")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -494,7 +494,7 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_python_function_error AS (\n" + "__expected__local_python_function_error_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" ")\n" "SELECT 1\n", @@ -543,8 +543,8 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n ' """Test function main."""\n raise ValueError("python udf exploded")\n', ), ( - "models/local_python_function_runtime_error.sql", - "MODEL (description 'Test model local_python_function_runtime_error.', materialized table);\n\n" + "models/local_python_function_runtime_error_rows.sql", + "MODEL (description 'Test model local_python_function_runtime_error_rows.', materialized table);\n\n" 'SELECT id, __udf("python_runtime_error")(amount) AS bad_value\n' 'FROM __source("raw_orders")\n', ), @@ -555,7 +555,7 @@ def test_given_partial_scenario_fixture_when_required_type_is_known_then_typed_n "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_python_function_runtime_error AS (\n" + "__expected__local_python_function_runtime_error_rows AS (\n" " SELECT 1 AS id, 10 AS bad_value\n" ")\n" "SELECT 1\n", @@ -901,8 +901,8 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte expected_count=1, additional_project_files=( ( - "models/local_model_error.sql", - "MODEL (description 'Test model local_model_error.', materialized table);\n\n" + "models/local_model_error_rows.sql", + "MODEL (description 'Test model local_model_error_rows.', materialized table);\n\n" "SELECT CAST('not_a_number_' || amount AS INTEGER) AS bad_value\n" 'FROM __source("raw_orders")\n', ), @@ -913,7 +913,7 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_model_error AS (\n" + "__expected__local_model_error_rows AS (\n" " SELECT 10 AS bad_value\n" ")\n" "SELECT 1\n", @@ -941,8 +941,8 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "FUNCTION (description 'Test function is_large_order.', arguments (amount INTEGER), returns BOOLEAN);\n\namount > 9\n", ), ( - "models/local_sql_function_pass.sql", - "MODEL (description 'Test model local_sql_function_pass.', materialized table);\n\n" + "models/local_sql_function_pass_rows.sql", + "MODEL (description 'Test model local_sql_function_pass_rows.', materialized table);\n\n" 'SELECT id, __udf("is_large_order")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -955,7 +955,7 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte " UNION ALL\n" " SELECT 2 AS id, 5 AS amount\n" "),\n" - "__expected__local_sql_function_pass AS (\n" + "__expected__local_sql_function_pass_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" " UNION ALL\n" " SELECT 2 AS id, FALSE AS is_large_order\n" @@ -989,8 +989,8 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "FUNCTION (description 'Test function bad_sql_function.', arguments (amount INTEGER), returns BOOLEAN);\n\nmissing_col > 9\n", ), ( - "models/local_sql_function_setup_error.sql", - "MODEL (description 'Test model local_sql_function_setup_error.', materialized table);\n\n" + "models/local_sql_function_setup_error_rows.sql", + "MODEL (description 'Test model local_sql_function_setup_error_rows.', materialized table);\n\n" 'SELECT id, __udf("bad_sql_function")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -1001,7 +1001,7 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_sql_function_setup_error AS (\n" + "__expected__local_sql_function_setup_error_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" ")\n" "SELECT 1\n", @@ -1044,8 +1044,8 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "FUNCTION (description 'Test function bad_transpile_function.', arguments (amount INTEGER), returns BOOLEAN);\n\namount > 9\n", ), ( - "models/local_sql_function_transpile_error.sql", - "MODEL (description 'Test model local_sql_function_transpile_error.', materialized table);\n\n" + "models/local_sql_function_transpile_error_rows.sql", + "MODEL (description 'Test model local_sql_function_transpile_error_rows.', materialized table);\n\n" 'SELECT id, __udf("bad_transpile_function")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -1056,7 +1056,7 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_sql_function_transpile_error AS (\n" + "__expected__local_sql_function_transpile_error_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" ")\n" "SELECT 1\n", @@ -1099,8 +1099,8 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "CAST('bad' AS INTEGER)\n", ), ( - "models/local_sql_function_runtime_error.sql", - "MODEL (description 'Test model local_sql_function_runtime_error.', materialized table);\n\n" + "models/local_sql_function_runtime_error_rows.sql", + "MODEL (description 'Test model local_sql_function_runtime_error_rows.', materialized table);\n\n" 'SELECT id, __udf("sql_runtime_error")(amount) AS bad_value\n' 'FROM __source("raw_orders")\n', ), @@ -1111,7 +1111,7 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_sql_function_runtime_error AS (\n" + "__expected__local_sql_function_runtime_error_rows AS (\n" " SELECT 1 AS id, 10 AS bad_value\n" ")\n" "SELECT 1\n", @@ -1146,8 +1146,8 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte " '''Test function main.'''\n return amount is not None and amount > 9\n", ), ( - "models/local_python_function_pass.sql", - "MODEL (description 'Test model local_python_function_pass.', materialized table);\n\n" + "models/local_python_function_pass_rows.sql", + "MODEL (description 'Test model local_python_function_pass_rows.', materialized table);\n\n" 'SELECT id, __udf("is_large_order_py")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -1160,7 +1160,7 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte " UNION ALL\n" " SELECT 2 AS id, 5 AS amount\n" "),\n" - "__expected__local_python_function_pass AS (\n" + "__expected__local_python_function_pass_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" " UNION ALL\n" " SELECT 2 AS id, FALSE AS is_large_order\n" @@ -1201,8 +1201,8 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte " '''Test function main.'''\n return amount is not None and amount > 9\n", ), ( - "models/local_python_function_error.sql", - "MODEL (description 'Test model local_python_function_error.', materialized table);\n\n" + "models/local_python_function_error_rows.sql", + "MODEL (description 'Test model local_python_function_error_rows.', materialized table);\n\n" 'SELECT id, __udf("is_large_order_py")(amount) AS is_large_order\n' 'FROM __source("raw_orders")\n', ), @@ -1213,7 +1213,7 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_python_function_error AS (\n" + "__expected__local_python_function_error_rows AS (\n" " SELECT 1 AS id, TRUE AS is_large_order\n" ")\n" "SELECT 1\n", @@ -1262,8 +1262,8 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte ' """Test function main."""\n raise ValueError("python udf exploded")\n', ), ( - "models/local_python_function_runtime_error.sql", - "MODEL (description 'Test model local_python_function_runtime_error.', materialized table);\n\n" + "models/local_python_function_runtime_error_rows.sql", + "MODEL (description 'Test model local_python_function_runtime_error_rows.', materialized table);\n\n" 'SELECT id, __udf("python_runtime_error")(amount) AS bad_value\n' 'FROM __source("raw_orders")\n', ), @@ -1274,7 +1274,7 @@ def test_given_missing_snapshot_when_running_local_scenario_then_reports_expecte "__source__raw_orders AS (\n" " SELECT 1 AS id, 10 AS amount\n" "),\n" - "__expected__local_python_function_runtime_error AS (\n" + "__expected__local_python_function_runtime_error_rows AS (\n" " SELECT 1 AS id, 10 AS bad_value\n" ")\n" "SELECT 1\n", diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/test/_test_types.py b/tests/e2e/src/sqlbuild/cli/commands/main/test/_test_types.py index 1306b8d223..05f2214dd3 100644 --- a/tests/e2e/src/sqlbuild/cli/commands/main/test/_test_types.py +++ b/tests/e2e/src/sqlbuild/cli/commands/main/test/_test_types.py @@ -300,3 +300,14 @@ class FixtureProjectIsolationE2ETestCase: description: str project_dir: Path expected_minimum_tests: int + + +@dataclass(frozen=True) +class SelectTestByNameE2ETestCase: + """One command selecting SQL unit tests by name through the real CLI.""" + + description: str + command: tuple[str, ...] + expected_exit_code: int + expected_output_fragments: tuple[str, ...] + unexpected_output_fragments: tuple[str, ...] = () diff --git a/tests/e2e/src/sqlbuild/cli/commands/main/test/test_select_by_name.py b/tests/e2e/src/sqlbuild/cli/commands/main/test/test_select_by_name.py new file mode 100644 index 0000000000..baf6b94d53 --- /dev/null +++ b/tests/e2e/src/sqlbuild/cli/commands/main/test/test_select_by_name.py @@ -0,0 +1,184 @@ +"""Unit tests are selected by their globally unique names through the real DuckDB CLI.""" + +from pathlib import Path +from subprocess import CompletedProcess + +import pytest + +from tests.e2e.src.sqlbuild.cli.commands.main.test._test_types import ( + SelectTestByNameE2ETestCase, +) +from tests.e2e.src.sqlbuild.cli.commands.main.test.helpers import ( + build_parameterized_test_project_files, +) +from tests.e2e.src.sqlbuild.cli.commands.shared.helpers import prepare_inline_project, run_sqb + +_ORDER_CASES: tuple[str, ...] = ("order_status [open_case]", "order_status [closed_case]") +_CUSTOMER_CASE: str = "customer_status [customer_only]" +_DUPLICATE_TEST_SQL: str = ( + "TEST ();\n\nWITH\n__source__raw_orders AS (SELECT 'open' AS status),\n" + "__expected__orders AS (SELECT 'open' AS status)\nSELECT 1\n" +) + + +@pytest.mark.parametrize( + "test_case", + ( + SelectTestByNameE2ETestCase( + description="a bare test name runs exactly that test", + command=("test", "--select", "order_status"), + expected_exit_code=0, + expected_output_fragments=("PASS=2 FAIL=0", *_ORDER_CASES), + unexpected_output_fragments=(_CUSTOMER_CASE,), + ), + SelectTestByNameE2ETestCase( + description="the test: form runs exactly that test", + command=("test", "--select", "test:customer_status"), + expected_exit_code=0, + expected_output_fragments=("PASS=1 FAIL=0", _CUSTOMER_CASE), + unexpected_output_fragments=_ORDER_CASES, + ), + SelectTestByNameE2ETestCase( + description="a test selector unions with a model selector", + command=("test", "--select", "test:customer_status", "orders"), + expected_exit_code=0, + expected_output_fragments=("PASS=3 FAIL=0", _CUSTOMER_CASE, *_ORDER_CASES), + ), + SelectTestByNameE2ETestCase( + description="case narrows a test selected by name", + command=("test", "--select", "test:order_status", "--case", "open_case"), + expected_exit_code=0, + expected_output_fragments=("PASS=1 FAIL=0", "order_status [open_case]"), + unexpected_output_fragments=("order_status [closed_case]", _CUSTOMER_CASE), + ), + SelectTestByNameE2ETestCase( + description="an unknown name suggests the nearest test", + command=("test", "--select", "ordr_status"), + expected_exit_code=1, + expected_output_fragments=( + "error[S007]: unknown selector name 'ordr_status'", + "did you mean 'order_status'?", + ), + ), + SelectTestByNameE2ETestCase( + description="plan rejects a unit-test name", + command=("plan", "--select", "order_status"), + expected_exit_code=1, + expected_output_fragments=( + "error[S013]: selector 'order_status' selects a unit test; only `sqb test` " + "and `sqb build` accept unit-test selectors", + ), + ), + SelectTestByNameE2ETestCase( + description="build runs the named test and builds no models", + command=("build", "--select", "test:order_status"), + expected_exit_code=0, + expected_output_fragments=("PASS=2", "TOTAL=2", *_ORDER_CASES), + unexpected_output_fragments=(_CUSTOMER_CASE, "read by orders"), + ), + SelectTestByNameE2ETestCase( + description="an empty select value plans nothing", + command=("plan", "--select", ""), + expected_exit_code=0, + expected_output_fragments=("Plan ready 0 selected",), + unexpected_output_fragments=("read by orders",), + ), + SelectTestByNameE2ETestCase( + description="an empty select value builds nothing", + command=("build", "--select", ""), + expected_exit_code=0, + expected_output_fragments=("Plan ready 0 selected",), + unexpected_output_fragments=("read by orders", *_ORDER_CASES), + ), + SelectTestByNameE2ETestCase( + description="an empty select value tests nothing", + command=("test", "--select", ""), + expected_exit_code=0, + expected_output_fragments=("PASS=0 FAIL=0 TOTAL=0",), + unexpected_output_fragments=(_CUSTOMER_CASE, *_ORDER_CASES), + ), + SelectTestByNameE2ETestCase( + description="a blank select value plans nothing", + command=("plan", "--select", " "), + expected_exit_code=0, + expected_output_fragments=("Plan ready 0 selected",), + unexpected_output_fragments=("read by orders",), + ), + SelectTestByNameE2ETestCase( + description="a blank select value builds nothing", + command=("build", "--select", " "), + expected_exit_code=0, + expected_output_fragments=("Plan ready 0 selected",), + unexpected_output_fragments=("read by orders", *_ORDER_CASES), + ), + SelectTestByNameE2ETestCase( + description="a blank select value tests nothing", + command=("test", "--select", " "), + expected_exit_code=0, + expected_output_fragments=("PASS=0 FAIL=0 TOTAL=0",), + unexpected_output_fragments=(_CUSTOMER_CASE, *_ORDER_CASES), + ), + ), + ids=lambda case: case.description, +) +def test_given_test_names_when_selecting_then_exactly_those_tests_run( + tmp_path: Path, test_case: SelectTestByNameE2ETestCase +) -> None: + project: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="select_by_name", + repo_files=build_parameterized_test_project_files(), + ) + + result: CompletedProcess[str] = run_sqb( + command=("--no-color", *test_case.command), project_dir=project + ) + + output: str = result.stdout + result.stderr + assert result.returncode == test_case.expected_exit_code, output + for fragment in test_case.expected_output_fragments: + assert fragment in output, output + for fragment in test_case.unexpected_output_fragments: + assert fragment not in output, output + + +@pytest.mark.parametrize( + "test_case", + ( + SelectTestByNameE2ETestCase( + description="a unit test named like a model is a compile error", + command=("test",), + expected_exit_code=1, + expected_output_fragments=( + "Project resource name 'orders' is declared as both model in models/orders.sql " + "and unit test in tests/unit/orders.sql", + "unit test, and scenario names must be globally unique", + ), + ), + ), + ids=lambda case: case.description, +) +def test_given_test_sharing_a_model_name_when_testing_then_compile_fails( + tmp_path: Path, test_case: SelectTestByNameE2ETestCase +) -> None: + project: Path = prepare_inline_project( + tmp_path=tmp_path, + project_name="duplicate_name", + repo_files={ + **build_parameterized_test_project_files(), + "tests/unit/orders.sql": _DUPLICATE_TEST_SQL, + }, + ) + + result: CompletedProcess[str] = run_sqb( + command=("--no-color", *test_case.command), project_dir=project + ) + + output: str = result.stdout + result.stderr + assert result.returncode == test_case.expected_exit_code, output + for fragment in test_case.expected_output_fragments: + assert fragment in output, output + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/integration/src/sqlbuild/cli/commands/main/test_format.py b/tests/integration/src/sqlbuild/cli/commands/main/test_format.py index 10c641c8ad..4cc892b9d4 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/test_format.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/test_format.py @@ -233,7 +233,7 @@ def test_given_sql_test_string_when_formatting_then_project_still_compiles( "SELECT 1 AS order_id, 'Customer''s order' AS order_label\n", encoding="utf-8", ) - test: Path = tmp_path / "tests" / "unit" / "orders.sql" + test: Path = tmp_path / "tests" / "unit" / "orders_test.sql" test.parent.mkdir(parents=True) test.write_text( "TEST();\n\nWITH __ref__stg_orders AS (" diff --git a/tests/integration/src/sqlbuild/cli/commands/main/test_rules.py b/tests/integration/src/sqlbuild/cli/commands/main/test_rules.py index 743c8ed0cf..a4551570e5 100644 --- a/tests/integration/src/sqlbuild/cli/commands/main/test_rules.py +++ b/tests/integration/src/sqlbuild/cli/commands/main/test_rules.py @@ -2043,7 +2043,7 @@ def test_given_ceremonial_test_select_when_running_alias_rule_then_control_proje model: Path = tmp_path / "models" / "orders.sql" model.parent.mkdir() model.write_text('MODEL (description "Orders");\nSELECT 1 AS order_id\n', encoding="utf-8") - test: Path = tmp_path / "tests" / "unit" / "orders.sql" + test: Path = tmp_path / "tests" / "unit" / "orders_test.sql" test.parent.mkdir(parents=True) test.write_text( "TEST();\n\nWITH __ref__orders AS (SELECT 1 AS order_id), " @@ -2088,7 +2088,7 @@ def test_given_direct_ceremonial_select_when_running_terminal_rule_then_test_is_ model: Path = tmp_path / "models" / "orders.sql" model.parent.mkdir() model.write_text('MODEL (description "Orders");\nSELECT 1 AS order_id\n', encoding="utf-8") - test: Path = tmp_path / "tests" / "unit" / "orders.sql" + test: Path = tmp_path / "tests" / "unit" / "orders_test.sql" test.parent.mkdir(parents=True) test.write_text( "TEST();\n\nWITH __ref__orders AS (SELECT 1 AS order_id), " diff --git a/tests/integration/src/sqlbuild/compiler/pipeline/test_macro_declaration_context.py b/tests/integration/src/sqlbuild/compiler/pipeline/test_macro_declaration_context.py index bf14bcf478..91c6d5289c 100644 --- a/tests/integration/src/sqlbuild/compiler/pipeline/test_macro_declaration_context.py +++ b/tests/integration/src/sqlbuild/compiler/pipeline/test_macro_declaration_context.py @@ -331,7 +331,7 @@ def test_given_macro_context_declarations_when_compiling_resources_then_all_expa "MODEL (description 'Test model summary.', materialized view, audits [minimum_quantity]);\n\n" "SELECT @minimum_quantity() AS minimum_quantity" ), - "tests/unit/summary.sql": ( + "tests/unit/summary_test.sql": ( "TEST (mode macro);\n\nWITH\n__macro_actual__ AS (\n" " SELECT @minimum_quantity() AS minimum_quantity\n),\n" "__macro_expected__ AS (\n SELECT 2 AS minimum_quantity\n)\nSELECT 1" diff --git a/tests/integration/src/sqlbuild/compiler/planner/_helpers/test_fixture_planning.py b/tests/integration/src/sqlbuild/compiler/planner/_helpers/test_fixture_planning.py index 82d1a51f31..57f33b72c8 100644 --- a/tests/integration/src/sqlbuild/compiler/planner/_helpers/test_fixture_planning.py +++ b/tests/integration/src/sqlbuild/compiler/planner/_helpers/test_fixture_planning.py @@ -11,7 +11,7 @@ from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs from sqlbuild.compiler.pipeline.main.project import compile_project from sqlbuild.compiler.planner._helpers.output.plan_output import build_selected_test_entries -from sqlbuild.compiler.planner.models import SqlTestPlanEntry +from sqlbuild.compiler.planner.models import SqlTestPlanEntry, SqlTestSelection from tests.integration.src.sqlbuild.compiler.planner._helpers._test_types import ( FixturePlanningIntegrationTestCase, ) @@ -71,6 +71,7 @@ def test_given_partial_fixture_when_compiling_and_planning_then_completes_requir adapter=adapter, ) entries, warnings = build_selected_test_entries( + sql_test_selection=SqlTestSelection(), project=project, adapter=adapter, selected_keys=frozenset((project.models[0].key,)), @@ -139,6 +140,7 @@ def test_given_untyped_null_fixture_when_planning_then_applies_authoritative_typ adapter=adapter, ) entries, warnings = build_selected_test_entries( + sql_test_selection=SqlTestSelection(), project=project, adapter=adapter, selected_keys=frozenset((project.models[0].key,)), diff --git a/tests/unit/src/sqlbuild/compiler/compile/_helpers/test_declaration_scopes.py b/tests/unit/src/sqlbuild/compiler/compile/_helpers/test_declaration_scopes.py index 922a47a23a..a3781441e2 100644 --- a/tests/unit/src/sqlbuild/compiler/compile/_helpers/test_declaration_scopes.py +++ b/tests/unit/src/sqlbuild/compiler/compile/_helpers/test_declaration_scopes.py @@ -223,19 +223,19 @@ def test_given_scoped_declaration_when_compiling_model_then_only_lexically_visib "model private does not export to test", { "models/orders.sql": "MODEL (description 'Test model orders.', constants (_value 1));\nSELECT 1 AS value", - "tests/unit/orders.sql": ( + "tests/unit/orders_test.sql": ( 'TEST ();\nWITH __expected__orders AS (SELECT @const("_value") AS value) ' "SELECT 1" ), }, - ("known but inaccessible", "models/orders.sql", "tests/unit/orders.sql"), + ("known but inaccessible", "models/orders.sql", "tests/unit/orders_test.sql"), ), ScopedDeclarationErrorTestCase( "filename resemblance without expected relationship", { "models/domain/_constants/value.sql": "CONSTANT (name model_value, value 9);", "models/domain/orders.sql": "MODEL (description 'Test model orders.');\nSELECT 1 AS value", - "tests/unit/orders.sql": ( + "tests/unit/orders_test.sql": ( "TEST ();\nWITH __ref__orders AS (SELECT 1 AS value), " "__assert__valid AS (SELECT @const('model_value') WHERE FALSE) SELECT 1" ), @@ -350,7 +350,7 @@ def test_given_inaccessible_or_unknown_declaration_when_compiling_then_diagnosti "unit test authored path", { "tests/unit/domain/_constants/value.sql": "CONSTANT (name value, value 16);", - "tests/unit/domain/orders.sql": ( + "tests/unit/domain/orders_test.sql": ( "TEST ();\nWITH __ref__orders AS (SELECT 1 AS value), " '__expected__orders AS (SELECT @const("value") AS value) SELECT 1' ), @@ -364,7 +364,7 @@ def test_given_inaccessible_or_unknown_declaration_when_compiling_then_diagnosti "scenario authored path", { "tests/scenarios/domain/_constants/value.sql": ("CONSTANT (name value, value 17);"), - "tests/scenarios/domain/orders.sql": ( + "tests/scenarios/domain/orders_scenario.sql": ( "SCENARIO (description 'Test scenario orders.');\nWITH __ref__orders AS (SELECT 1 AS value), " '__expected__orders AS (SELECT @const("value") AS value) SELECT 1' ), @@ -510,7 +510,7 @@ def test_given_scoped_declaration_when_compiling_sql_surface_then_uses_authored_ description="filename resemblance and no expected model grant nothing", files={ "models/domain/orders.sql": "MODEL (description 'Test model orders.');\nSELECT 1 AS value", - "tests/unit/orders.sql": ( + "tests/unit/orders_test.sql": ( "TEST ();\nWITH __ref__orders AS (SELECT 1 AS value), " "__assert__valid AS (SELECT 1 WHERE FALSE) SELECT 1" ), diff --git a/tests/unit/src/sqlbuild/compiler/compile/_helpers/test_declarations.py b/tests/unit/src/sqlbuild/compiler/compile/_helpers/test_declarations.py index 33c0796275..01b5e45917 100644 --- a/tests/unit/src/sqlbuild/compiler/compile/_helpers/test_declarations.py +++ b/tests/unit/src/sqlbuild/compiler/compile/_helpers/test_declarations.py @@ -121,7 +121,7 @@ def test_given_public_and_private_declarations_when_compiling_then_resolves_scop expression: | SELECT @const("min_items") AS value """, - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH __ref__orders AS ( @@ -132,7 +132,7 @@ def test_given_public_and_private_declarations_when_compiling_then_resolves_scop ) SELECT 1 """, - "tests/scenarios/orders.sql": """ + "tests/scenarios/orders_scenario.sql": """ SCENARIO (description "Test scenario orders."); WITH __ref__orders AS ( @@ -353,13 +353,13 @@ def test_given_typed_constants_when_compiling_then_all_sql_surfaces_use_adapter_ expression: | SELECT * FROM entries WHERE country IN @const("countries") """, - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH __ref__orders AS (SELECT 'GB' IN @const("countries") AS supported), __expected__orders AS (SELECT true AS supported) SELECT 1 """, - "tests/scenarios/orders.sql": """ + "tests/scenarios/orders_scenario.sql": """ SCENARIO (description "Test scenario orders."); WITH __ref__orders AS (SELECT @const("country_array") AS countries), __expected__orders AS (SELECT @const("country_array") AS countries) diff --git a/tests/unit/src/sqlbuild/compiler/compile/test_main.py b/tests/unit/src/sqlbuild/compiler/compile/test_main.py index 70b1c9411c..e7a8b5001f 100644 --- a/tests/unit/src/sqlbuild/compiler/compile/test_main.py +++ b/tests/unit/src/sqlbuild/compiler/compile/test_main.py @@ -1409,7 +1409,7 @@ def grant_select(target: str) -> str: select @project_columns() from __source("raw_orders") """.strip() + "\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -1526,7 +1526,7 @@ def grant_select(target: str) -> str: """.strip() + "\n", "models/staging/orders.sql": "MODEL (description 'Test model orders.');\n\nselect 1\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -1624,7 +1624,7 @@ def source_columns() -> str: amount > @@min_amount AND '@@ENV:USER_NAME' = 'runner' """.strip() + "\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -2218,7 +2218,7 @@ def project_columns() -> str: """.strip() + "\n", "models/staging/orders.sql": "MODEL (description 'Test model orders.');\n\nselect 1\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (name "first"); WITH @@ -3646,7 +3646,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov description="raises when a compiled test body references an unknown macro", repo_files=base_repo_files() | { - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); SELECT @missing_macro() @@ -3662,7 +3662,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov repo_files=base_repo_files() | { "models/staging/orders.sql": "MODEL (description 'Test model orders.');\n\nselect 1\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); SELECT 1 @@ -3684,7 +3684,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov description: Test source raw_orders. """.strip() + "\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -3707,7 +3707,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov repo_files=base_repo_files() | { "models/staging/orders.sql": "MODEL (description 'Test model orders.');\n\nselect 1\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -3730,7 +3730,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov repo_files=base_repo_files() | { "models/staging/orders.sql": "MODEL (description 'Test model orders.');\n\nselect 1\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -3776,7 +3776,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov description: Test source raw__orders. """.strip() + "\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -3827,7 +3827,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov type: VARCHAR """.strip() + "\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -3856,7 +3856,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov description: Test source raw_orders. """.strip() + "\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -3887,7 +3887,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov description: Test source raw_orders. """.strip() + "\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH @@ -3916,7 +3916,7 @@ def test_given_project_and_local_environment_when_resolving_then_local_values_ov description: Test source raw_orders. """.strip() + "\n", - "tests/unit/orders.sql": """ + "tests/unit/orders_test.sql": """ TEST (); WITH diff --git a/tests/unit/src/sqlbuild/compiler/discovery/_test_types.py b/tests/unit/src/sqlbuild/compiler/discovery/_test_types.py index 9a8304483f..cd50322209 100644 --- a/tests/unit/src/sqlbuild/compiler/discovery/_test_types.py +++ b/tests/unit/src/sqlbuild/compiler/discovery/_test_types.py @@ -97,3 +97,11 @@ class DiscoveryRelevantCountTestCase: description: str expected_item_count: int unexpected_root_pattern: str + + +@dataclass(frozen=True) +class DiscoverUniqueTestNameTestCase: + description: str + repo_files: dict[str, str] + expected_test_block_names: tuple[str | None, ...] + expected_case_names: tuple[str, ...] diff --git a/tests/unit/src/sqlbuild/compiler/discovery/test_main.py b/tests/unit/src/sqlbuild/compiler/discovery/test_main.py index e3292c7636..85eea8b12a 100644 --- a/tests/unit/src/sqlbuild/compiler/discovery/test_main.py +++ b/tests/unit/src/sqlbuild/compiler/discovery/test_main.py @@ -8,7 +8,7 @@ from sqlbuild.compiler.discovery.exceptions import ProjectPythonPathError from sqlbuild.compiler.discovery.main.discover import discover_project_inputs -from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs +from sqlbuild.compiler.discovery.models import DiscoveredProjectInputs, DiscoveredSqlTestBlock from sqlbuild.observability import EventDispatcher, LifecycleEvent, dispatcher_scope from tests.unit.src.sqlbuild.compiler.discovery._test_helpers import ( base_repo_files, @@ -17,6 +17,7 @@ DiscoverFactoryValidationTestCase, DiscoverProjectInputsErrorTestCase, DiscoverProjectInputsTestCase, + DiscoverUniqueTestNameTestCase, DiscoveryLifecycleTestCase, DiscoveryRelevantCountTestCase, ProjectPythonPathAcceptanceTestCase, @@ -51,7 +52,7 @@ type: VARCHAR """.strip() + "\n", - "tests/unit/orders.sql": "TEST ();\nSELECT 1\n", + "tests/unit/orders_test.sql": "TEST ();\nSELECT 1\n", "tests/legacy_orders.sql": "TEST ();\nSELECT 2\n", "tests/scenarios/revenue/revenue__customer_refund.sql": """ SCENARIO (description "Customer refund", tags ["revenue"]); @@ -119,7 +120,7 @@ class SlackProvider(Provider): expected_source_paths=("sources/raw.yml",), expected_source_entry_names=(("raw_orders",),), expected_seed_paths=("seeds/country_codes.csv",), - expected_test_paths=("tests/unit/orders.sql",), + expected_test_paths=("tests/unit/orders_test.sql",), expected_test_block_indexes=(1,), expected_test_block_names=(None,), expected_test_block_sql_bodies=("SELECT 1",), @@ -877,6 +878,39 @@ def test_given_invalid_generated_factory_nodes_when_discovering_inputs_then_rais discover_project_inputs(project_dir=tmp_path) +@pytest.mark.parametrize( + "test_case", + [ + DiscoverUniqueTestNameTestCase( + description="parameterized cases share their test's unique name", + repo_files=base_repo_files() + | { + "models/marts/orders.sql": "MODEL ();\n\nselect 1\n", + "tests/unit/order_status.sql": ( + "TEST (parameters (status string), cases " + '(open_case (status "open"), closed_case (status "closed")));\nSELECT 1\n' + ), + }, + expected_test_block_names=(None,), + expected_case_names=("open_case", "closed_case"), + ), + ], + ids=lambda case: case.description, +) +def test_given_parameterized_test_when_discovering_inputs_then_cases_are_not_duplicate_names( + test_case: DiscoverUniqueTestNameTestCase, + tmp_path: Path, + write_repo_files: Callable[[Path, dict[str, str]], None], +) -> None: + write_repo_files(tmp_path, test_case.repo_files) + + discovered: DiscoveredProjectInputs = discover_project_inputs(project_dir=tmp_path) + + blocks: tuple[DiscoveredSqlTestBlock, ...] = discovered.test_files[0].blocks + assert tuple(block.name for block in blocks) == test_case.expected_test_block_names + assert tuple(case.name for case in blocks[0].cases) == test_case.expected_case_names + + @pytest.mark.parametrize( "test_case", [ @@ -962,6 +996,45 @@ def test_given_invalid_generated_factory_nodes_when_discovering_inputs_then_rais }, expected_error_fragment="Duplicate scenario file name found for 'customer_refund'", ), + DiscoverProjectInputsErrorTestCase( + description="raises when a unit test and a model share a name", + repo_files=base_repo_files() + | { + "models/marts/orders.sql": "MODEL ();\n\nselect 1\n", + "tests/unit/orders.sql": "TEST ();\nSELECT 1\n", + }, + expected_error_fragment=( + "Project resource name 'orders' is declared as both model in " + "models/marts/orders.sql and unit test in tests/unit/orders.sql; .*unit test, " + "and scenario names must be globally unique" + ), + ), + DiscoverProjectInputsErrorTestCase( + description="raises when a scenario and a source share a name", + repo_files=base_repo_files() + | { + "sources/raw.yml": ( + "sources:\n - name: raw_orders\n schema: public\n table: orders\n" + ), + "tests/scenarios/raw_orders.sql": "SCENARIO ();\nSELECT 1\n", + }, + expected_error_fragment=( + "Project resource name 'raw_orders' is declared as both source in " + "sources/raw.yml and scenario in tests/scenarios/raw_orders.sql" + ), + ), + DiscoverProjectInputsErrorTestCase( + description="raises when a named unit test and a scenario share a name", + repo_files=base_repo_files() + | { + "tests/scenarios/order_refunds.sql": "SCENARIO ();\nSELECT 1\n", + "tests/unit/refunds.sql": 'TEST (name "order_refunds");\nSELECT 1\n', + }, + expected_error_fragment=( + "Project resource name 'order_refunds' is declared as both scenario in " + "tests/scenarios/order_refunds.sql and unit test in tests/unit/refunds.sql" + ), + ), DiscoverProjectInputsErrorTestCase( description="raises when model and source names collide", repo_files=base_repo_files() diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/selection/__init__.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/selection/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/selection/_test_types.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/selection/_test_types.py new file mode 100644 index 0000000000..213756d6a9 --- /dev/null +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/selection/_test_types.py @@ -0,0 +1,25 @@ +from __future__ import annotations + +from dataclasses import dataclass + +from sqlbuild.compiler.planner.models import SqlTestSelection + + +@dataclass(frozen=True) +class UnitTestSelectorSplitTestCase: + description: str + select: tuple[str, ...] + exclude: tuple[str, ...] + expected_select: tuple[str, ...] + expected_exclude: tuple[str, ...] + expected_selection: SqlTestSelection + + +@dataclass(frozen=True) +class UnitTestSelectorErrorTestCase: + description: str + select: tuple[str, ...] + accepts_unit_tests: bool + expected_code: str + expected_message: str + expected_help: str | None diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/selection/test_unit_test_selectors.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/selection/test_unit_test_selectors.py new file mode 100644 index 0000000000..08a29317dd --- /dev/null +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/selection/test_unit_test_selectors.py @@ -0,0 +1,199 @@ +"""Splitting unit-test selectors from resource selectors, and rejecting them elsewhere.""" + +from __future__ import annotations + +import pytest + +from sqlbuild.compiler.planner._helpers.selection.unit_test_selectors import ( + split_unit_test_selectors, +) +from sqlbuild.compiler.planner.exceptions import PlannerInputError +from sqlbuild.compiler.planner.models import ( + SqlTestSelection, + UnitTestSelectorNames, + UnitTestSelectorSplit, +) +from tests.unit.src.sqlbuild.compiler.planner._helpers.selection._test_types import ( + UnitTestSelectorErrorTestCase, + UnitTestSelectorSplitTestCase, +) + +_NAMES: UnitTestSelectorNames = UnitTestSelectorNames( + tests=frozenset({"order_status", "order_totals_check", "customer_status"}), + resources=frozenset({"orders", "order_totals", "customers", "raw_orders"}), +) + + +@pytest.mark.parametrize( + "test_case", + [ + UnitTestSelectorSplitTestCase( + description="a bare test name selects only that test and no models", + select=("order_status",), + exclude=(), + expected_select=(), + expected_exclude=(), + expected_selection=SqlTestSelection( + names=frozenset({"order_status"}), models_selected=False + ), + ), + UnitTestSelectorSplitTestCase( + description="test: selectors union with model selectors", + select=("test:customer_status orders+", "tag:daily"), + exclude=(), + expected_select=("orders+", "tag:daily"), + expected_exclude=(), + expected_selection=SqlTestSelection(names=frozenset({"customer_status"})), + ), + UnitTestSelectorSplitTestCase( + description="test: patterns match unit tests with fnmatch", + select=("test:order_*",), + exclude=(), + expected_select=(), + expected_exclude=(), + expected_selection=SqlTestSelection( + names=frozenset({"order_status", "order_totals_check"}), models_selected=False + ), + ), + UnitTestSelectorSplitTestCase( + description="a bare pattern that matches resources keeps its resource meaning", + select=("order*",), + exclude=("test:order_totals_check",), + expected_select=("order*",), + expected_exclude=(), + expected_selection=SqlTestSelection(excluded_names=frozenset({"order_totals_check"})), + ), + UnitTestSelectorSplitTestCase( + description="empty and blank select values are kept and select nothing", + select=("", " "), + exclude=(), + expected_select=("", " "), + expected_exclude=(), + expected_selection=SqlTestSelection(), + ), + UnitTestSelectorSplitTestCase( + description="a test-only selector beside an empty value still selects no models", + select=("", "order_status"), + exclude=(), + expected_select=("",), + expected_exclude=(), + expected_selection=SqlTestSelection(names=frozenset({"order_status"})), + ), + UnitTestSelectorSplitTestCase( + description="selectors without unit tests pass through unchanged", + select=("orders", "customers,tag:daily", "models/marts"), + exclude=("raw_orders",), + expected_select=("orders", "customers,tag:daily", "models/marts"), + expected_exclude=("raw_orders",), + expected_selection=SqlTestSelection(), + ), + ], + ids=lambda case: case.description, +) +def test_given_selectors_when_splitting_unit_tests_then_tests_and_resources_separate( + test_case: UnitTestSelectorSplitTestCase, +) -> None: + split: UnitTestSelectorSplit = split_unit_test_selectors( + select=test_case.select, + exclude=test_case.exclude, + names=_NAMES, + accepts_unit_tests=True, + ) + + assert split.select == test_case.expected_select + assert split.exclude == test_case.expected_exclude + assert split.sql_test_selection == test_case.expected_selection + + +@pytest.mark.parametrize( + "test_case", + [ + UnitTestSelectorErrorTestCase( + description="a command that runs no unit tests rejects a bare test name", + select=("order_status",), + accepts_unit_tests=False, + expected_code="S013", + expected_message=( + "selector 'order_status' selects a unit test; only `sqb test` and `sqb build` " + "accept unit-test selectors" + ), + expected_help=None, + ), + UnitTestSelectorErrorTestCase( + description="a command that runs no unit tests rejects test: selectors", + select=("orders test:customer_status",), + accepts_unit_tests=False, + expected_code="S013", + expected_message=( + "selector 'test:customer_status' selects a unit test; only `sqb test` and " + "`sqb build` accept unit-test selectors" + ), + expected_help=None, + ), + UnitTestSelectorErrorTestCase( + description="an unknown test: name suggests the nearest unit tests", + select=("test:order_stats",), + accepts_unit_tests=True, + expected_code="S007", + expected_message="unknown unit test name 'order_stats'", + expected_help="did you mean 'order_status'?", + ), + UnitTestSelectorErrorTestCase( + description="an unknown bare name suggests unit tests and resources", + select=("customer_stat",), + accepts_unit_tests=True, + expected_code="S007", + expected_message="unknown selector name 'customer_stat'", + expected_help="did you mean 'customer_status', 'customers'?", + ), + UnitTestSelectorErrorTestCase( + description="graph expansion of a unit test is refused", + select=("+order_status",), + accepts_unit_tests=True, + expected_code="S013", + expected_message=( + "selector '+order_status' expands a unit test with '+'; unit tests have no " + "lineage, so select them by name without '+'" + ), + expected_help=None, + ), + UnitTestSelectorErrorTestCase( + description="intersecting a unit test is refused", + select=("order_status,tag:daily",), + accepts_unit_tests=True, + expected_code="S013", + expected_message=( + "selector 'order_status,tag:daily' intersects a unit test with ','; unit tests " + "can only be selected whole, by name" + ), + expected_help=None, + ), + UnitTestSelectorErrorTestCase( + description="a stray trailing comma after a unit test reports the empty selector", + select=("test:order_status,",), + accepts_unit_tests=True, + expected_code="S001", + expected_message="empty selector", + expected_help=None, + ), + ], + ids=lambda case: case.description, +) +def test_given_invalid_unit_test_selectors_when_splitting_then_it_raises_clear_errors( + test_case: UnitTestSelectorErrorTestCase, +) -> None: + with pytest.raises(PlannerInputError) as raised: + split_unit_test_selectors( + select=test_case.select, + exclude=(), + names=_NAMES, + accepts_unit_tests=test_case.accepts_unit_tests, + ) + + assert raised.value.code == test_case.expected_code + assert raised.value.message == test_case.expected_message + assert raised.value.help == test_case.expected_help + + +if __name__ == "__main__": + pytest.main([__file__, "-vv"]) diff --git a/tests/unit/src/sqlbuild/compiler/planner/_helpers/test_fixture_planning_work.py b/tests/unit/src/sqlbuild/compiler/planner/_helpers/test_fixture_planning_work.py index 7c1878596c..47549d22bd 100644 --- a/tests/unit/src/sqlbuild/compiler/planner/_helpers/test_fixture_planning_work.py +++ b/tests/unit/src/sqlbuild/compiler/planner/_helpers/test_fixture_planning_work.py @@ -24,6 +24,7 @@ RelationFixtureCompletion, RelationFixturePlanningContext, SqlTestPlanEntry, + SqlTestSelection, ) from sqlbuild.compiler.references.types import SqlReferenceKind from sqlbuild.spec.contracts.models import SettingsConfig @@ -107,6 +108,7 @@ def test_given_many_tests_when_building_entries_then_project_fixture_metadata_is entries: list[SqlTestPlanEntry] warnings: list[PlanWarning] entries, warnings = build_selected_test_entries( + sql_test_selection=SqlTestSelection(), project=test_project, adapter=PlannerTestAdapter(), selected_keys=frozenset((model.key,)), diff --git a/tests/unit/src/sqlbuild/compiler/planner/classes/test_background_sql_test_planning.py b/tests/unit/src/sqlbuild/compiler/planner/classes/test_background_sql_test_planning.py index 09fb602611..83600a755c 100644 --- a/tests/unit/src/sqlbuild/compiler/planner/classes/test_background_sql_test_planning.py +++ b/tests/unit/src/sqlbuild/compiler/planner/classes/test_background_sql_test_planning.py @@ -11,7 +11,7 @@ BackgroundSqlTestPlanning, ) from sqlbuild.compiler.planner.exceptions import PlannerInputError -from sqlbuild.compiler.planner.models import PlannedSqlTests, PlanWarning +from sqlbuild.compiler.planner.models import PlannedSqlTests, PlanWarning, SqlTestSelection from sqlbuild.compiler.planner.types import WarningSeverity from tests.unit.src.sqlbuild.compiler.planner.classes._test_types import ( BackgroundSqlTestPlanningTestCase, @@ -45,15 +45,20 @@ def test_given_enabled_planning_when_joining_then_returns_planned_tests( calls: list[frozenset[CompiledObjectKey]] = [] def plan( - *, project: object, adapter: object, selected_keys: frozenset[CompiledObjectKey] + *, + project: object, + adapter: object, + selected_keys: frozenset[CompiledObjectKey], + sql_test_selection: SqlTestSelection, ) -> PlannedSqlTests: - del project, adapter + del project, adapter, sql_test_selection calls.append(selected_keys) return _PLANNED monkeypatch.setattr(background_module, "plan_selected_sql_tests", plan) with BackgroundSqlTestPlanning( + sql_test_selection=SqlTestSelection(), project=object(), # ty: ignore[invalid-argument-type] adapter=object(), # ty: ignore[invalid-argument-type] selected_keys=_KEYS, @@ -82,15 +87,20 @@ def test_given_disabled_planning_when_joining_then_returns_empty_tests( calls: list[frozenset[CompiledObjectKey]] = [] def plan( - *, project: object, adapter: object, selected_keys: frozenset[CompiledObjectKey] + *, + project: object, + adapter: object, + selected_keys: frozenset[CompiledObjectKey], + sql_test_selection: SqlTestSelection, ) -> PlannedSqlTests: - del project, adapter + del project, adapter, sql_test_selection calls.append(selected_keys) return _PLANNED monkeypatch.setattr(background_module, "plan_selected_sql_tests", plan) with BackgroundSqlTestPlanning( + sql_test_selection=SqlTestSelection(), project=object(), # ty: ignore[invalid-argument-type] adapter=object(), # ty: ignore[invalid-argument-type] selected_keys=_KEYS, @@ -118,14 +128,19 @@ def test_given_failing_planning_when_joining_then_raises_original_error( test_case: BackgroundSqlTestPlanningTestCase, monkeypatch: pytest.MonkeyPatch ) -> None: def plan( - *, project: object, adapter: object, selected_keys: frozenset[CompiledObjectKey] + *, + project: object, + adapter: object, + selected_keys: frozenset[CompiledObjectKey], + sql_test_selection: SqlTestSelection, ) -> PlannedSqlTests: - del project, adapter, selected_keys + del project, adapter, selected_keys, sql_test_selection raise PlannerInputError(str(test_case.expected_error)) monkeypatch.setattr(background_module, "plan_selected_sql_tests", plan) with BackgroundSqlTestPlanning( + sql_test_selection=SqlTestSelection(), project=object(), # ty: ignore[invalid-argument-type] adapter=object(), # ty: ignore[invalid-argument-type] selected_keys=_KEYS, @@ -153,9 +168,13 @@ def test_given_running_planning_when_planner_fails_then_worker_is_daemon( release: threading.Event = threading.Event() def plan( - *, project: object, adapter: object, selected_keys: frozenset[CompiledObjectKey] + *, + project: object, + adapter: object, + selected_keys: frozenset[CompiledObjectKey], + sql_test_selection: SqlTestSelection, ) -> PlannedSqlTests: - del project, adapter + del project, adapter, sql_test_selection started.set() release.wait(timeout=5) return PlannedSqlTests(selected_keys=selected_keys) @@ -163,6 +182,7 @@ def plan( monkeypatch.setattr(background_module, "plan_selected_sql_tests", plan) with BackgroundSqlTestPlanning( + sql_test_selection=SqlTestSelection(), project=object(), # ty: ignore[invalid-argument-type] adapter=object(), # ty: ignore[invalid-argument-type] selected_keys=_KEYS, diff --git a/tests/unit/src/sqlbuild/integrations/dagster/_test_types.py b/tests/unit/src/sqlbuild/integrations/dagster/_test_types.py index 043c02b7af..ce9886ddbd 100644 --- a/tests/unit/src/sqlbuild/integrations/dagster/_test_types.py +++ b/tests/unit/src/sqlbuild/integrations/dagster/_test_types.py @@ -296,3 +296,11 @@ class DagsterAssetSelectionErrorTestCase: description: str select: str expected_error_fragment: str + + +@dataclass(frozen=True) +class DagsterUnitTestSelectorTestCase: + description: str + select: str + expected_code: str + expected_message: str diff --git a/tests/unit/src/sqlbuild/integrations/dagster/test_assets.py b/tests/unit/src/sqlbuild/integrations/dagster/test_assets.py index a6afd656e0..eb881eb96c 100644 --- a/tests/unit/src/sqlbuild/integrations/dagster/test_assets.py +++ b/tests/unit/src/sqlbuild/integrations/dagster/test_assets.py @@ -6,6 +6,7 @@ import pytest +from sqlbuild.compiler.planner.exceptions import PlannerInputError from sqlbuild.integrations.dagster import ( SqlBuildDagsterTranslator, build_sqlbuild_asset_selection, @@ -31,6 +32,7 @@ DagsterPythonArtifactCompatibilityTestCase, DagsterScenarioCheckDecoratorTestCase, DagsterSourceAssetKeyTestCase, + DagsterUnitTestSelectorTestCase, ) from tests.unit.src.sqlbuild.integrations.dagster.helpers import ( build_dagster_selection_graph_dag, @@ -652,3 +654,38 @@ def assets_def() -> dg.MaterializeResult: dag=dag, sqlbuild_select=test_case.select, ) + + +@pytest.mark.parametrize( + "test_case", + [ + DagsterUnitTestSelectorTestCase( + description="rejects a unit-test selector like other non-test commands", + select="test:orders_status_rules", + expected_code="S013", + expected_message=( + "selector 'test:orders_status_rules' selects a unit test; only `sqb test` and " + "`sqb build` accept unit-test selectors" + ), + ), + ], + ids=lambda case: case.description, +) +def test_given_unit_test_selector_when_building_asset_selection_then_raises_s013( + test_case: DagsterUnitTestSelectorTestCase, +) -> None: + dag: Mapping[str, Any] = build_dagster_selection_graph_dag() + + @sqlbuild_assets(dag=dag) + def assets_def() -> dg.MaterializeResult: + return dg.MaterializeResult() + + with pytest.raises(PlannerInputError) as raised: + build_sqlbuild_asset_selection( + sqlbuild_assets=[assets_def], + dag=dag, + sqlbuild_select=test_case.select, + ) + + assert raised.value.code == test_case.expected_code + assert raised.value.message == test_case.expected_message diff --git a/tests/unit/src/sqlbuild/lint/test_expanded_lint.py b/tests/unit/src/sqlbuild/lint/test_expanded_lint.py index 9fb1bf808a..029bf538cd 100644 --- a/tests/unit/src/sqlbuild/lint/test_expanded_lint.py +++ b/tests/unit/src/sqlbuild/lint/test_expanded_lint.py @@ -97,12 +97,12 @@ def counted_discovery( "sqlbuild_project.toml": PROJECT_TOML, "models/domain/_constants/value.sql": "CONSTANT (name value, value 12);\n", "models/domain/orders.sql": f"{HEADER}SELECT 1 AS value\n", - "tests/unit/other/orders.sql": ( + "tests/unit/other/orders_test.sql": ( "TEST ();\nWITH __expected__orders AS " '(SELECT @const("value") AS value) SELECT 1\n' ), }, - model_path="tests/unit/other/orders.sql", + model_path="tests/unit/other/orders_test.sql", authored_sql=('WITH __expected__orders AS (SELECT @const("value") AS value) SELECT 1'), expected_sql="WITH __expected__orders AS (SELECT 12 AS value) SELECT 1", ), diff --git a/website/src/content/docs/docs/cli/build.mdx b/website/src/content/docs/docs/cli/build.mdx index d8047df8a4..f572ec6dbe 100644 --- a/website/src/content/docs/docs/cli/build.mdx +++ b/website/src/content/docs/docs/cli/build.mdx @@ -58,7 +58,7 @@ own option. `--version` works only directly after `sqb`, as in `sqb --version`: | `--selection-diagnostics` | Warn when a selected model will build on changed upstream models that are not selected; see [Selection and staleness](/docs/concepts/planning/selection-and-staleness) | | `--manifest` | Generate `target/manifest.json` with plan-aware project metadata | | `SELECTOR ...` | Positional selectors, the same as `--select` | -| `--select`, `-s` | Select specific models | +| `--select`, `-s` | Select specific models, or unit tests by name (`test:`); selecting only unit tests runs them without building models | | `--exclude` | Exclude specific models | | `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `build` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | diff --git a/website/src/content/docs/docs/cli/test.mdx b/website/src/content/docs/docs/cli/test.mdx index 30741e865c..a5c21bee1c 100644 --- a/website/src/content/docs/docs/cli/test.mdx +++ b/website/src/content/docs/docs/cli/test.mdx @@ -22,12 +22,19 @@ Positional `SELECTOR` arguments work like `--select`, and both forms can be comb |------|-------------| | `--no-sql-analysis` | Disable compile-time SQL analysis (`--no-sql-validation` is an alias) | | `SELECTOR ...` | Positional selectors, the same as `--select` | -| `--select`, `-s` | Select tests targeting specific models | -| `--exclude` | Exclude tests targeting specific models | +| `--select`, `-s` | Select tests by name (`order_status_rules` or `test:order_status_rules`), or the tests targeting specific models | +| `--exclude` | Exclude tests by name, or the tests targeting specific models | | `--warehouse ` | Snowflake warehouse for this invocation; overrides the target's `query` [warehouse group](/docs/concepts/project-configuration#command-group-warehouses) and the connection warehouse | -Parameterized cases use parent-level selection. Selecting a target model includes every case in -each matching `TEST` template; case names are not model selectors. +Test names are globally unique, so a bare test name selects exactly that test. `test:` is the +explicit form and accepts the same glob patterns as names, such as `test:orders_*`. A bare pattern +selects tests only when it matches no other resource. Test selectors are unioned with model +selectors, so `sqb test test:order_status_rules stg_orders` runs that test and every test targeting +`stg_orders`. Test selectors cannot use `+` graph expansion or `,` intersection. Only `sqb test` and +`sqb build` accept test selectors; other commands reject them with `S013`. + +Parameterized cases use parent-level selection. Selecting a test or a target model includes every +case of each matching `TEST` template, and `--case` narrows the run to one case name. Text output identifies each case as ` []` and includes its source path and safe typed parameters. Structured JSON emits one check per case with stable source/block/case identity, @@ -43,4 +50,10 @@ sqb test # Run tests for a specific model sqb test stg_orders + +# Run one test by name +sqb test order_status_rules + +# Run one case of a parameterized test +sqb test test:order_status_rules --case open_case ``` diff --git a/website/src/content/docs/docs/concepts/python-nodes/overview.mdx b/website/src/content/docs/docs/concepts/python-nodes/overview.mdx index 5087778955..7348bbf664 100644 --- a/website/src/content/docs/docs/concepts/python-nodes/overview.mdx +++ b/website/src/content/docs/docs/concepts/python-nodes/overview.mdx @@ -146,7 +146,7 @@ sqb build --select tag:exports # by tag sqb build --select +orders_export # with upstreams ``` -Names are globally unique across models, sources, seeds, functions, loaders, tasks, assets, checks, providers, and hooks. +Names are globally unique across models, sources, seeds, functions, loaders, tasks, assets, checks, providers, hooks, unit tests, and scenarios. ## Lifecycle: run, build, check diff --git a/website/src/content/docs/docs/concepts/scenarios.mdx b/website/src/content/docs/docs/concepts/scenarios.mdx index 49c86dd5fc..5f59d0e742 100644 --- a/website/src/content/docs/docs/concepts/scenarios.mdx +++ b/website/src/content/docs/docs/concepts/scenarios.mdx @@ -32,7 +32,7 @@ tests/ fulfillment__late_shipment.sql ``` -Folders are organizational only. The scenario name is the filename stem and must be globally unique across all discovered scenario files. +Folders are organizational only. The scenario name is the filename stem. Like model, source, seed, and unit-test names, it must be globally unique across the project. ### Scenario format diff --git a/website/src/content/docs/docs/concepts/selectors.mdx b/website/src/content/docs/docs/concepts/selectors.mdx index 3042ae2ae4..0a1ef8470f 100644 --- a/website/src/content/docs/docs/concepts/selectors.mdx +++ b/website/src/content/docs/docs/concepts/selectors.mdx @@ -67,6 +67,19 @@ sqb build --select seed:waffle_types sqb build --select source:raw__orders ``` +### Unit test + +`sqb test` and `sqb build` select SQL unit tests by name. Test names are globally unique, so a bare +name works; `test:` is the explicit form and accepts glob patterns: + +```bash +sqb test order_status_rules +sqb test "test:orders_*" stg_customers +``` + +Test selectors are unioned with other selectors and cannot use `+` or `,`. Other commands reject +them with `S013`. + ## Graph expansion ### Upstream @@ -150,6 +163,14 @@ error[S007]: unknown selector name 'missing_*' error[S009]: no models found under path 'models/nonexistent'. error[S000]: no SQL resources or Python nodes found with tag 'nonexistent_tag' error[S003]: path selector 'fact_orders~' requires names on both sides of '~' +error[S013]: selector 'order_status_rules' selects a unit test; only `sqb test` and `sqb build` accept unit-test selectors +``` + +Unknown names suggest the closest selectable names, including unit tests: + +``` +error[S007]: unknown selector name 'order_status_rule' + = help: did you mean 'order_status_rules'? ``` If a path selector omits the root directory, SQLBuild asks for the explicit form: diff --git a/website/src/content/docs/docs/concepts/testing.mdx b/website/src/content/docs/docs/concepts/testing.mdx index 8bf92b634a..a5eb68167a 100644 --- a/website/src/content/docs/docs/concepts/testing.mdx +++ b/website/src/content/docs/docs/concepts/testing.mdx @@ -524,7 +524,11 @@ CTE prefixes from other modes are not allowed. For example, `__source__` in a ma ## Multiple tests per file -A single test file can contain multiple `TEST()` blocks. Each block must have a unique `name`: +A single test file can contain multiple `TEST()` blocks. A test's name is its `name`, or the file +stem when `name` is omitted. Test names are globally unique: a test cannot share a name with another +test, a scenario, a model, a source, a seed, a function, or a Python node. Name the file or the block +after the behavior it checks, such as `orders_status_rules`, rather than after the model. The cases +of one parameterized test share its name. Each block in a file must have its own `name`: ```sql TEST (name "completed_orders_only");