Skip to content

Add bundle bitmap for per-field telemetry #6104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
denik wants to merge 33 commits into
base: main
Choose a base branch
from
Draft

Add bundle bitmap for per-field telemetry #6104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
50a0602
Add PLAN.md for bundle bitmap
denik Jul 29, 2026
86c8b25
Add bundle bitmap for per-field telemetry
denik Jul 29, 2026
6d576f8
Remove PLAN.md
denik Jul 29, 2026
6ce1905
Add bool value bits, decode round-trip, and template recording
denik Jul 29, 2026
f6aa835
Move boolean telemetry into a typed Telemetry struct
denik Jul 30, 2026
418d7f9
Regenerate goldens after rebase; ignore .databricks in bitmap test
denik Jul 30, 2026
7da4b6c
Fix bitmap test: executable decode script and deterministic cache
denik Jul 30, 2026
d946bb0
Strip CR when parsing embedded bitmap schema
denik Jul 30, 2026
fa3b686
Regenerate bitmap schema from scratch; refresh goldens in generate task
denik Jul 30, 2026
631643b
Make Telemetry additive: mirror bool metrics into a struct on Bundle
denik Jul 31, 2026
71c165d
Record DMS undeclared-writer flags as True/False pairs in bitmap
denik Jul 31, 2026
08cb8c8
Revert DMS flags to single bits: false is not lost here
denik Aug 3, 2026
3ecd228
Fix bitmap encoding every telemetry bit as set
denik Aug 3, 2026
fe465a4
Regenerate bitmap schema and drop removed Local test option
denik Aug 11, 2026
c915a48
Regenerate bitmap schema and goldens after rebase
denik Aug 17, 2026
9d3d0fa
Port bitmap telemetry mirror to current main
denik Nov 3, 2026
03f3c91
Regenerate bitmap schema after rebase (data_quality_monitors)
denik Nov 3, 2026
6e67914
Bitmap: record user-authored config; fix presence detection; simplify…
denik Nov 4, 2026
c069704
Add a dedicated bitmap test that records every schema line
denik Nov 4, 2026
b167722
Make Telemetry the source of bool_values; encode bool values; add decode
denik Nov 4, 2026
0638c0c
Add a self-contained bitmap decoder to databricks-bundles
denik Nov 4, 2026
6c46910
Regenerate bitmap schema after rebase (variables.*.sensitive)
denik Nov 4, 2026
699b48c
Regenerate bitmap schema after rebase (materialization_sets)
denik Nov 4, 2026
1ed42b0
Keep the bitmap decoder out of pydabs codegen cleanup and docs
denik Nov 4, 2026
0436ef8
no_drift: log bitmap stderr instead of diffing it
denik Nov 4, 2026
400921d
Add a local-only invariant test for validate and the bitmap round trip
denik Nov 4, 2026
a42fdc0
Update selftest/selection for the new validate_bitmap invariant test
denik Nov 4, 2026
94d52b3
bitmap: one bitmap command with -o compressed|text|json
denik Nov 5, 2026
ec33305
bitmap/full: split the schema golden per resource type
denik Nov 5, 2026
56f1322
bitmap/presence: lifecycle.started true on one cluster and false on a…
denik Nov 5, 2026
7e048a5
Report each lifecycle.started value seen instead of a random resource's
denik Nov 5, 2026
445b523
Regenerate bitmap schema after rebase (postgres_data_apis)
denik Nov 5, 2026
689a00d
Telemetry: BoolSet.SetBool instead of appending to []bool
denik Nov 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/rules/telemetry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The deploy payload is declared twice: the proto in the universe repo (`proto/log

Other things worth knowing:

- A one-off boolean needs no proto change: add a key constant to `bundle/metrics/metrics.go` (snake_case, matching the constant name) and call `b.Metrics.SetBoolValue(...)`. It lands in `experimental.bool_values`.
- A one-off boolean needs no proto change: add a field to `bundle.Telemetry` (`bundle/telemetry.go`) with a snake_case `metric:"..."` tag and set it. Pick the type by what is measured: `bool` for a flag reported only when true, `*bool` for a measured value (nil when not measured), `BoolSet` for a measurement taken several times (e.g. once per artifact; record each with `SetBool`, and each value taken is reported once). It lands in `experimental.bool_values` and in the bundle bitmap; run `./task generate-bitmap-schema` to append it to the bitmap schema.
- Metrics that may be removed later belong in `BundleDeployExperimental`, which carries no compatibility guarantees.
- Positional encodings such as the `upload_file_sizes` histogram are frozen once the metric has adoption: changing a bucket bound silently changes the meaning of every previously reported entry.
- Per-resource state sizes come from the direct engine's state in `resources.json`.
Expand Down
29 changes: 29 additions & 0 deletions Taskfile.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -802,6 +802,7 @@ tasks:
# Refreshes acceptance/bundle/refschema/output/*.txt, which feeds
# generate-direct-apitypes and generate-direct-resources below.
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand All @@ -820,6 +821,7 @@ tasks:
cmds:
- task: generate-cligen
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand Down Expand Up @@ -989,6 +991,33 @@ tasks:
cmds:
- go test ./acceptance -run TestAccept/bundle/refschema -update

# Appends any newly added bundle config fields to bundle/bitmap/schema.txt, the
# embedded bitmap schema (see cmd/bundle/bitmap). The schema is append-only, so
# update-schema reads the currently embedded schema and only appends. Output is
# written to a temp file first so the redirect does not truncate schema.txt
# before `go run` embeds it at compile time. Also refreshes the schema embedded
# in the databricks-bundles decoder (python/databricks/bundles/_bitmap) and the
# listing in acceptance/bundle/bitmap/full, which tracks every schema line.
generate-bitmap-schema:
desc: Regenerate bundle/bitmap/schema.txt and its acceptance listing
sources:
- bundle/**/*.go
- exclude: bundle/**/*_test.go
- acceptance/bundle/bitmap/full/script
- acceptance/bundle/bitmap/full/split.py
- go.mod
- go.sum
- "{{.EMBED_SOURCES}}"
generates:
- bundle/bitmap/schema.txt
- acceptance/bundle/bitmap/full/output/*.txt
- python/databricks/bundles/_bitmap/_schema.py
cmds:
- go run . bundle bitmap update-schema > bundle/bitmap/schema.txt.tmp
- mv bundle/bitmap/schema.txt.tmp bundle/bitmap/schema.txt
- go run ./bundle/bitmap/genpython python/databricks/bundles/_bitmap/_schema.py
- go test ./acceptance -run TestAccept/bundle/bitmap/full -update

# Upstream field documentation comes from the checked-in .codegen/cli.json;
# bundle/internal/schema/annotations.yml carries the CLI-owned docs and
# overrides and is rewritten in place (synced with the config structure).
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/bitmap/full/out.test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = []
Empty file added acceptance/bundle/bitmap/full/output.txt
View file
Open in desktop
Empty file.
75 changes: 75 additions & 0 deletions acceptance/bundle/bitmap/full/output/alerts.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
resources.alerts 4171
resources.alerts.* 4172
resources.alerts.*.url 4173
resources.alerts.*.modified_status 4174
resources.alerts.*.lifecycle 4175
resources.alerts.*.lifecycle.prevent_destroy 4176
resources.alerts.*.lifecycle.prevent_destroy=true 4177
resources.alerts.*.lifecycle.prevent_destroy=false 4178
resources.alerts.*.lifecycle.depends_on 4179
resources.alerts.*.lifecycle.depends_on[*] 4180
resources.alerts.*.lifecycle.depends_on[*].resource 4181
resources.alerts.*.create_time 4182
resources.alerts.*.custom_description 4183
resources.alerts.*.custom_summary 4184
resources.alerts.*.custom_template_format 4185
resources.alerts.*.display_name 4186
resources.alerts.*.effective_run_as 4187
resources.alerts.*.effective_run_as.service_principal_name 4188
resources.alerts.*.effective_run_as.user_name 4189
resources.alerts.*.evaluation 4190
resources.alerts.*.evaluation.comparison_operator 4191
resources.alerts.*.evaluation.empty_result_state 4192
resources.alerts.*.evaluation.last_evaluated_at 4193
resources.alerts.*.evaluation.notification 4194
resources.alerts.*.evaluation.notification.notify_on_ok 4195
resources.alerts.*.evaluation.notification.notify_on_ok=true 4196
resources.alerts.*.evaluation.notification.notify_on_ok=false 4197
resources.alerts.*.evaluation.notification.retrigger_seconds 4198
resources.alerts.*.evaluation.notification.subscriptions 4199
resources.alerts.*.evaluation.notification.subscriptions[*] 4200
resources.alerts.*.evaluation.notification.subscriptions[*].destination_id 4201
resources.alerts.*.evaluation.notification.subscriptions[*].user_email 4202
resources.alerts.*.evaluation.source 4203
resources.alerts.*.evaluation.source.aggregation 4204
resources.alerts.*.evaluation.source.display 4205
resources.alerts.*.evaluation.source.name 4206
resources.alerts.*.evaluation.state 4207
resources.alerts.*.evaluation.threshold 4208
resources.alerts.*.evaluation.threshold.column 4209
resources.alerts.*.evaluation.threshold.column.aggregation 4210
resources.alerts.*.evaluation.threshold.column.display 4211
resources.alerts.*.evaluation.threshold.column.name 4212
resources.alerts.*.evaluation.threshold.value 4213
resources.alerts.*.evaluation.threshold.value.bool_value 4214
resources.alerts.*.evaluation.threshold.value.bool_value=true 4215
resources.alerts.*.evaluation.threshold.value.bool_value=false 4216
resources.alerts.*.evaluation.threshold.value.double_value 4217
resources.alerts.*.evaluation.threshold.value.string_value 4218
resources.alerts.*.id 4219
resources.alerts.*.lifecycle_state 4220
resources.alerts.*.owner_user_name 4221
resources.alerts.*.parameters 4222
resources.alerts.*.parameters[*] 4223
resources.alerts.*.parameters[*].name 4224
resources.alerts.*.parameters[*].type 4225
resources.alerts.*.parameters[*].value 4226
resources.alerts.*.parent_path 4227
resources.alerts.*.query_text 4228
resources.alerts.*.run_as 4229
resources.alerts.*.run_as.service_principal_name 4230
resources.alerts.*.run_as.user_name 4231
resources.alerts.*.run_as_user_name 4232
resources.alerts.*.schedule 4233
resources.alerts.*.schedule.pause_status 4234
resources.alerts.*.schedule.quartz_cron_schedule 4235
resources.alerts.*.schedule.timezone_id 4236
resources.alerts.*.update_time 4237
resources.alerts.*.warehouse_id 4238
resources.alerts.*.permissions 4239
resources.alerts.*.permissions[*] 4240
resources.alerts.*.permissions[*].level 4241
resources.alerts.*.permissions[*].user_name 4242
resources.alerts.*.permissions[*].service_principal_name 4243
resources.alerts.*.permissions[*].group_name 4244
resources.alerts.*.file_path 4245
215 changes: 215 additions & 0 deletions acceptance/bundle/bitmap/full/output/apps.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
resources.apps 3933
resources.apps.* 3934
resources.apps.*.url 3935
resources.apps.*.modified_status 3936
resources.apps.*.lifecycle 3937
resources.apps.*.lifecycle.prevent_destroy 3938
resources.apps.*.lifecycle.prevent_destroy=true 3939
resources.apps.*.lifecycle.prevent_destroy=false 3940
resources.apps.*.lifecycle.depends_on 3941
resources.apps.*.lifecycle.depends_on[*] 3942
resources.apps.*.lifecycle.depends_on[*].resource 3943
resources.apps.*.active_deployment 3944
resources.apps.*.active_deployment.command 3945
resources.apps.*.active_deployment.command[*] 3946
resources.apps.*.active_deployment.create_time 3947
resources.apps.*.active_deployment.creator 3948
resources.apps.*.active_deployment.deployment_artifacts 3949
resources.apps.*.active_deployment.deployment_artifacts.source_code_path 3950
resources.apps.*.active_deployment.deployment_id 3951
resources.apps.*.active_deployment.env_vars 3952
resources.apps.*.active_deployment.env_vars[*] 3953
resources.apps.*.active_deployment.env_vars[*].name 3954
resources.apps.*.active_deployment.env_vars[*].value 3955
resources.apps.*.active_deployment.env_vars[*].value_from 3956
resources.apps.*.active_deployment.git_source 3957
resources.apps.*.active_deployment.git_source.branch 3958
resources.apps.*.active_deployment.git_source.commit 3959
resources.apps.*.active_deployment.git_source.git_repository 3960
resources.apps.*.active_deployment.git_source.git_repository.auto_deploy 3961
resources.apps.*.active_deployment.git_source.git_repository.auto_deploy=true 3962
resources.apps.*.active_deployment.git_source.git_repository.auto_deploy=false 3963
resources.apps.*.active_deployment.git_source.git_repository.caller_credential_id 3964
resources.apps.*.active_deployment.git_source.git_repository.provider 3965
resources.apps.*.active_deployment.git_source.git_repository.url 3966
resources.apps.*.active_deployment.git_source.resolved_commit 3967
resources.apps.*.active_deployment.git_source.source_code_path 3968
resources.apps.*.active_deployment.git_source.tag 3969
resources.apps.*.active_deployment.health_check 3970
resources.apps.*.active_deployment.health_check.path 3971
resources.apps.*.active_deployment.health_check.timeout 3972
resources.apps.*.active_deployment.mode 3973
resources.apps.*.active_deployment.source_code_path 3974
resources.apps.*.active_deployment.status 3975
resources.apps.*.active_deployment.status.message 3976
resources.apps.*.active_deployment.status.state 3977
resources.apps.*.active_deployment.update_time 3978
resources.apps.*.app_status 3979
resources.apps.*.app_status.message 3980
resources.apps.*.app_status.running_instances 3981
resources.apps.*.app_status.state 3982
resources.apps.*.budget_policy_id 3983
resources.apps.*.compute_max_instances 3984
resources.apps.*.compute_min_instances 3985
resources.apps.*.compute_size 3986
resources.apps.*.compute_status 3987
resources.apps.*.compute_status.active_instances 3988
resources.apps.*.compute_status.message 3989
resources.apps.*.compute_status.state 3990
resources.apps.*.create_time 3991
resources.apps.*.creator 3992
resources.apps.*.default_git_source 3993
resources.apps.*.default_git_source.branch 3994
resources.apps.*.default_git_source.commit 3995
resources.apps.*.default_git_source.git_repository 3996
resources.apps.*.default_git_source.git_repository.auto_deploy 3997
resources.apps.*.default_git_source.git_repository.auto_deploy=true 3998
resources.apps.*.default_git_source.git_repository.auto_deploy=false 3999
resources.apps.*.default_git_source.git_repository.caller_credential_id 4000
resources.apps.*.default_git_source.git_repository.provider 4001
resources.apps.*.default_git_source.git_repository.url 4002
resources.apps.*.default_git_source.resolved_commit 4003
resources.apps.*.default_git_source.source_code_path 4004
resources.apps.*.default_git_source.tag 4005
resources.apps.*.default_source_code_path 4006
resources.apps.*.description 4007
resources.apps.*.effective_budget_policy_id 4008
resources.apps.*.effective_usage_policy_id 4009
resources.apps.*.effective_user_api_scopes 4010
resources.apps.*.effective_user_api_scopes[*] 4011
resources.apps.*.forward_user_access_token 4012
resources.apps.*.forward_user_access_token=true 4013
resources.apps.*.forward_user_access_token=false 4014
resources.apps.*.git_repository 4015
resources.apps.*.git_repository.auto_deploy 4016
resources.apps.*.git_repository.auto_deploy=true 4017
resources.apps.*.git_repository.auto_deploy=false 4018
resources.apps.*.git_repository.caller_credential_id 4019
resources.apps.*.git_repository.provider 4020
resources.apps.*.git_repository.url 4021
resources.apps.*.git_source 4022
resources.apps.*.git_source.branch 4023
resources.apps.*.git_source.commit 4024
resources.apps.*.git_source.git_repository 4025
resources.apps.*.git_source.git_repository.auto_deploy 4026
resources.apps.*.git_source.git_repository.auto_deploy=true 4027
resources.apps.*.git_source.git_repository.auto_deploy=false 4028
resources.apps.*.git_source.git_repository.caller_credential_id 4029
resources.apps.*.git_source.git_repository.provider 4030
resources.apps.*.git_source.git_repository.url 4031
resources.apps.*.git_source.resolved_commit 4032
resources.apps.*.git_source.source_code_path 4033
resources.apps.*.git_source.tag 4034
resources.apps.*.id 4035
resources.apps.*.name 4036
resources.apps.*.oauth2_app_client_id 4037
resources.apps.*.oauth2_app_integration_id 4038
resources.apps.*.pending_deployment 4039
resources.apps.*.pending_deployment.command 4040
resources.apps.*.pending_deployment.command[*] 4041
resources.apps.*.pending_deployment.create_time 4042
resources.apps.*.pending_deployment.creator 4043
resources.apps.*.pending_deployment.deployment_artifacts 4044
resources.apps.*.pending_deployment.deployment_artifacts.source_code_path 4045
resources.apps.*.pending_deployment.deployment_id 4046
resources.apps.*.pending_deployment.env_vars 4047
resources.apps.*.pending_deployment.env_vars[*] 4048
resources.apps.*.pending_deployment.env_vars[*].name 4049
resources.apps.*.pending_deployment.env_vars[*].value 4050
resources.apps.*.pending_deployment.env_vars[*].value_from 4051
resources.apps.*.pending_deployment.git_source 4052
resources.apps.*.pending_deployment.git_source.branch 4053
resources.apps.*.pending_deployment.git_source.commit 4054
resources.apps.*.pending_deployment.git_source.git_repository 4055
resources.apps.*.pending_deployment.git_source.git_repository.auto_deploy 4056
resources.apps.*.pending_deployment.git_source.git_repository.auto_deploy=true 4057
resources.apps.*.pending_deployment.git_source.git_repository.auto_deploy=false 4058
resources.apps.*.pending_deployment.git_source.git_repository.caller_credential_id 4059
resources.apps.*.pending_deployment.git_source.git_repository.provider 4060
resources.apps.*.pending_deployment.git_source.git_repository.url 4061
resources.apps.*.pending_deployment.git_source.resolved_commit 4062
resources.apps.*.pending_deployment.git_source.source_code_path 4063
resources.apps.*.pending_deployment.git_source.tag 4064
resources.apps.*.pending_deployment.health_check 4065
resources.apps.*.pending_deployment.health_check.path 4066
resources.apps.*.pending_deployment.health_check.timeout 4067
resources.apps.*.pending_deployment.mode 4068
resources.apps.*.pending_deployment.source_code_path 4069
resources.apps.*.pending_deployment.status 4070
resources.apps.*.pending_deployment.status.message 4071
resources.apps.*.pending_deployment.status.state 4072
resources.apps.*.pending_deployment.update_time 4073
resources.apps.*.resources 4074
resources.apps.*.resources[*] 4075
resources.apps.*.resources[*].app 4076
resources.apps.*.resources[*].app.name 4077
resources.apps.*.resources[*].app.permission 4078
resources.apps.*.resources[*].database 4079
resources.apps.*.resources[*].database.database_name 4080
resources.apps.*.resources[*].database.instance_name 4081
resources.apps.*.resources[*].database.permission 4082
resources.apps.*.resources[*].description 4083
resources.apps.*.resources[*].experiment 4084
resources.apps.*.resources[*].experiment.experiment_id 4085
resources.apps.*.resources[*].experiment.permission 4086
resources.apps.*.resources[*].genie_space 4087
resources.apps.*.resources[*].genie_space.name 4088
resources.apps.*.resources[*].genie_space.permission 4089
resources.apps.*.resources[*].genie_space.space_id 4090
resources.apps.*.resources[*].job 4091
resources.apps.*.resources[*].job.id 4092
resources.apps.*.resources[*].job.permission 4093
resources.apps.*.resources[*].name 4094
resources.apps.*.resources[*].postgres 4095
resources.apps.*.resources[*].postgres.branch 4096
resources.apps.*.resources[*].postgres.database 4097
resources.apps.*.resources[*].postgres.permission 4098
resources.apps.*.resources[*].secret 4099
resources.apps.*.resources[*].secret.key 4100
resources.apps.*.resources[*].secret.permission 4101
resources.apps.*.resources[*].secret.scope 4102
resources.apps.*.resources[*].serving_endpoint 4103
resources.apps.*.resources[*].serving_endpoint.name 4104
resources.apps.*.resources[*].serving_endpoint.permission 4105
resources.apps.*.resources[*].sql_warehouse 4106
resources.apps.*.resources[*].sql_warehouse.id 4107
resources.apps.*.resources[*].sql_warehouse.permission 4108
resources.apps.*.resources[*].uc_securable 4109
resources.apps.*.resources[*].uc_securable.permission 4110
resources.apps.*.resources[*].uc_securable.securable_full_name 4111
resources.apps.*.resources[*].uc_securable.securable_kind 4112
resources.apps.*.resources[*].uc_securable.securable_type 4113
resources.apps.*.service_principal_client_id 4114
resources.apps.*.service_principal_id 4115
resources.apps.*.service_principal_name 4116
resources.apps.*.source_code_path 4117
resources.apps.*.space 4118
resources.apps.*.telemetry_export_destinations 4119
resources.apps.*.telemetry_export_destinations[*] 4120
resources.apps.*.telemetry_export_destinations[*].unity_catalog 4121
resources.apps.*.telemetry_export_destinations[*].unity_catalog.logs_table 4122
resources.apps.*.telemetry_export_destinations[*].unity_catalog.metrics_table 4123
resources.apps.*.telemetry_export_destinations[*].unity_catalog.traces_table 4124
resources.apps.*.thumbnail_url 4125
resources.apps.*.update_time 4126
resources.apps.*.updater 4127
resources.apps.*.usage_policy_id 4128
resources.apps.*.user_api_scopes 4129
resources.apps.*.user_api_scopes[*] 4130
resources.apps.*.lifecycle.started 4131
resources.apps.*.lifecycle.started=true 4132
resources.apps.*.lifecycle.started=false 4133
resources.apps.*.config 4134
resources.apps.*.config.command 4135
resources.apps.*.config.command[*] 4136
resources.apps.*.config.env 4137
resources.apps.*.config.env[*] 4138
resources.apps.*.config.env[*].name 4139
resources.apps.*.config.env[*].value 4140
resources.apps.*.config.env[*].value_from 4141
resources.apps.*.permissions 4142
resources.apps.*.permissions[*] 4143
resources.apps.*.permissions[*].level 4144
resources.apps.*.permissions[*].user_name 4145
resources.apps.*.permissions[*].service_principal_name 4146
resources.apps.*.permissions[*].group_name 4147
Loading
Loading