Repository navigation
Add data_quality_monitors bundle resource (direct engine) - #7094
Merged
Merged
Conversation
Adds the data-quality/v1 Monitor API as a direct-engine bundle resource, alongside the legacy quality_monitors resource it supersedes. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 5a32e8e
Top 14 slowest tests (at least 2 minutes):
|
…ceptance test Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 13 changed tests (limit=20, 0 not selected) 658 tests: regression=14 unreleased=0 coverage=644 failing=0
14 more failure details omitted, see the full report in the run. |
…y monitors Co-authored-by: Isaac <no-reply@databricks.com>
…n alert Co-authored-by: Isaac <no-reply@databricks.com>
kevinlian2it
marked this pull request as ready for review
October 29, 2026 18:23
kevinlian2it
requested review from
janniklasrose and
lennartkats-db
and removed request for
lennartkats-db
October 29, 2026 18:23
Member
There was a problem hiding this comment.
what's the purpose of this folder vs resources/data_quality_monitors/*? let's remove/fold it into the others if this test offers additional coverage
Contributor
Author
There was a problem hiding this comment.
Folded it into resources/data_quality_monitors/data_profiling: it now has a dev target and the script starts by validating it (schedule dropped), while the default target's deploy keeps it. Removed the separate folder.
| # Create:REQ Update:IGN in the data-quality API: the update endpoint ignores it. | ||
| # Same as quality_monitors.assets_dir. | ||
| - field: data_profiling_config.assets_dir | ||
| reason: immutable |
Member
There was a problem hiding this comment.
can this be annotated in the upstream proto to get it out of the box (it'd show up under data_quality_monitors.generated.yml as spec:immutable). Hand-writing it is a smell
Contributor
Author
There was a problem hiding this comment.
Yes: databricks-eng/universe#2963330 marks assets_dir IMMUTABLE in the OpenAPI spec (spec-only, since the backend's ApiUtil.update rejects * masks touching a google.api.field_behavior IMMUTABLE field). I'll swap the hand-written entry for the generated one once an SDK bump picks it up.
Member
There was a problem hiding this comment.
I'll swap the hand-written entry for the generated one once an SDK bump picks it up.
I'm pretty sure the SDK bump will do this automatically (DABs should be set up to error on the field conflict) so no action from you besides merging that upstream PR :-)
Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
approved these changes
Nov 3, 2026
janniklasrose
enabled auto-merge
November 3, 2026 15:52
deco-sdk-tagging Bot
added a commit
that referenced
this pull request
Nov 4, 2026
## Release v1.24.0 ### Notable Changes * A variable used as the value of a secret is now sensitive: referencing it from any other field, such as a job parameter or a secret's `comment`, is an error because the value would be used in plain text. Mark other secret-holding variables with `sensitive: true` to get the same protection. ([#7288](#7288)) ### CLI * Fix `databricks auth login` and `databricks configure` overwriting the parent profile's host and token when saving a profile with a dotted name such as `team.prod` next to an existing `[team]`. ([#7266](#7266)) * `databricks auth login` and `databricks auth token` now reject the name of an existing profile after "Create a new profile" instead of logging in to that profile. ([#7196](#7196)) * `databricks auth login` and `databricks auth token` now reject a host that is not a URL with a hostname, such as `/`, `?o=123` or `https://`, instead of falling back to `DATABRICKS_HOST` or failing later. An empty host after "Enter a host URL manually" in the login profile picker no longer starts discovery login. ([#7195](#7195)) * Prompts that validate their input, such as the `databricks configure` host prompt, no longer open with a red cross before anything is typed. ([#7314](#7314)) * `databricks ssh connect` now documents `--environment-version` for choosing the serverless environment version, starts a separate default-named server for each version, and rejects the flag together with `--cluster`. Default connection names also include the environment version in their hash, so the default name when `--environment-version` is not specified changes with this release. Configuration associated with the old connection name (secret scope, SSH keys and known hosts in your local `.databricks` folder) will no longer be used and can be safely removed. ([#7310](#7310)) ### AI Runtime * Show workspace-supported accelerator types in `databricks air run -h config.compute.accelerator_type`. ([#7275](#7275)) * `air list` now reports `No active runs found.` for an empty active-only result. ([#7276](#7276)) * Stop printing a retry notice on every `air run --watch`; retries are still reported when they happen. ([#7270](#7270)) ### Bundles * Fix `bundle plan` reporting an update to an app's `config.env` values on every run, which also redeployed the app on every `bundle deploy`. The Apps API returns environment variable names only, so `value` and `value_from` are no longer compared against the remote deployment. ([#7326](#7326)) * Add the `data_quality_monitors` resource for data quality monitors on Unity Catalog schemas and tables (direct engine only). ([#7094](#7094)) * `bundle deploy` now asks for confirmation before recreating or deleting a Feature View, and `bundle deploy` and `bundle destroy` warn that FeatureSpecs using a Feature View are not checked. ([#7164](#7164)) * Accept grant `privileges` values that the SDK enum does not list yet: `bundle validate` no longer warns (or errors under `--strict`) on them, and the PyDABs `Privilege` enum accepts unknown strings instead of raising. The JSON schema and PyDABs still advertise the known values for completion. ([#7315](#7315)) * Add the `materialization_sets` resource, which materializes Feature Store Feature Views to offline and online stores. ([#7164](#7164)) * Detect notebooks with the same rule as the workspace: a `.py`, `.sql`, `.scala` or `.r` file is a notebook when its first line contains `Databricks notebook source`, for example after a byte order mark. Previously such a file was tracked as a plain file, so removing it locally left the remote notebook behind. ([#7290](#7290)) * Add `sensitive: true` for bundle variables. A variable used as the value of a secret is sensitive by default. `bundle validate -o json` redacts only sensitive variables instead of every string variable, and a sensitive variable can only be referenced from a sensitive resource field (such as a secret's `value`) or from another sensitive variable. ([#7288](#7288)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#7305](#7305)) * Bump dependencies with known vulnerabilities. ([#7327](#7327)) * Update the Go SDK to [v0.191.1](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.191.1). ([#7277](#7277)) * Update the Go SDK to [v0.193.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.193.0). ([#7303](#7303))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds a
data_quality_monitorsbundle resource for the data-quality/v1 Monitor API (anomaly detection on schemas, data profiling on tables). Direct engine only. The legacyquality_monitorsresource is unchanged; deprecating it is a separate follow-up.bundle/config/resources/data_quality_monitor.goembedsdataquality.Monitor. The ID is<object_type>,<object_id>, the same as the Terraform import ID fordatabricks_data_quality_monitor.bundle/direct/dresources/data_quality_monitor.go: CRUD onw.DataQuality. Update sendsupdate_mask: *.configs/data_quality_monitors.yml:object_typeandobject_idare ID fields;data_profiling_config.assets_dirrecreates on change (the update API ignores it). Output-only fields and the input-onlyskip_builtin_dashboard/warehouse_id(never returned on GET) come from the generated config.quality_monitors:run_asis rejected (the API has none), and presets that pause triggers dropdata_profiling_config.schedule(itspause_statusis read-only). The monitor has no name, so there is no name prefix. It has no permissions and no workspace URL, becauseobject_idis a UUID.Why
Data quality monitoring is moving to GA on the data-quality/v1 API, and
quality_monitorsonly covers the legacy table-monitor API. This resource lets bundles manage v1 monitors, including schema-level anomaly detection, which the legacy resource can't express.Notes for reviewers:
ListMonitoris unimplemented server-side, so lookup variables aren't supported for this resource.snapshot: {}is lost when a monitor is defined in Python.quality_monitorshas the same pre-existing limitation; it's visible in thedata_quality_monitors-supportgolden.*,anomaly_detection_config.notification_settingsis replaced on every update, so omitting it removes an alert set in the Data Quality Monitoring UI. The bundle is the source of truth; this is documented on the resource.data_quality_monitorsto the deployment metadata service, and databricks-eng/universe#2867273 marked the input-only fields in the spec (in SDK v0.191.0).Tests
all_test.go,state_load_test.go,resources_test.go, and the mutator tests.apply_target_mode_testasserts the dev-mode schedule drop.resources/data_quality_monitors/basic: schema monitor that references${resources.schemas.x.schema_id}. Covers create, in-place update, no drift, recreate on an identity change, and destroy.data_profiling: table monitor. Covers output-only fields not drifting, in-place schedule update, andassets_dirrecreate.remote-delete: an out-of-band delete is detected.lifecycle: cloud-capable.bundle/data_quality_monitor: dev-mode target drops the schedule; production keeps it.deployment/bind/data_quality_monitor: bind and unbind, and a malformed ID is rejected.python/data_quality_monitors-support.golangci-lint run ./...reports 0 issues. tidy, whitespace, deadcode, codeowners and lockfile checks pass, and all generators are idempotent. 505 PyDABs tests pass.lifecyclepasses on all clouds, with and without deployment history (DMS).This pull request and its description were written by Isaac.