Repository navigation
Record deployment history for bundle resource bindings #7400
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| * Record deployment history when binding existing resources with `bundle deployment bind` when deployment history is enabled. ([#7400](https://github.com/databricks/cli/pull/7400)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| bundle: | ||
| name: dms-bind-history-failures-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| jobs: | ||
| create_failed: | ||
| name: create_failed | ||
| record_failed: | ||
| name: record_failed | ||
| complete_failed: | ||
| name: complete_failed | ||
| upload_failed: | ||
| name: upload_failed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = false | ||
| EnvMatrix.DMS = ["true"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
|
|
||
| === A version creation failure does not prevent binding | ||
| >>> fault.py POST /api/2.0/bundle/deployments/*/versions 400 0 1 | ||
|
|
||
| >>> [CLI] bundle deployment bind create_failed [JOB_ID] --auto-approve | ||
| Warn: Failed to record bind history: failed to create deployment version: Fault injected by test. | ||
| Successfully bound job with an id '[JOB_ID]' | ||
| Run 'bundle deploy' to deploy changes to your workspace | ||
|
|
||
| === An operation recording failure does not prevent binding | ||
| >>> fault.py PATCH /api/2.0/bundle/** 400 0 1 | ||
|
|
||
| >>> [CLI] bundle deployment bind record_failed [JOB_ID] --auto-approve | ||
| Warn: Failed to record bind history: Fault injected by test. | ||
| Successfully bound job with an id '[JOB_ID]' | ||
| Run 'bundle deploy' to deploy changes to your workspace | ||
|
|
||
| === A version completion failure does not prevent binding | ||
| >>> fault.py POST /api/2.0/bundle/deployments/*/versions/*/complete 400 0 1 | ||
|
|
||
| >>> [CLI] bundle deployment bind complete_failed [JOB_ID] --auto-approve | ||
| Warn: Failed to complete deployment history version: Fault injected by test. | ||
| Successfully bound job with an id '[JOB_ID]' | ||
| Run 'bundle deploy' to deploy changes to your workspace | ||
|
|
||
| === All bindings survived the remote state round trips | ||
| >>> jq .state | map_values(.__id__) .databricks/bundle/default/resources.json | ||
| { | ||
| "resources.jobs.complete_failed": "[JOB_ID]", | ||
| "resources.jobs.create_failed": "[JOB_ID]", | ||
| "resources.jobs.record_failed": "[JOB_ID]" | ||
| } | ||
|
|
||
| >>> [CLI] jobs get [JOB_ID] | ||
| { | ||
| "job_id": [JOB_ID], | ||
| "name": "bind-target-[UNIQUE_NAME]" | ||
| } | ||
|
|
||
| >>> print_requests.py --dms //versions --oneline | ||
| {"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "3"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_BIND", "resource_key": "jobs.create_failed"}], "version_type": "VERSION_TYPE_DEPLOY"}} | ||
| {"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "5"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_BIND", "resource_key": "jobs.record_failed"}], "version_type": "VERSION_TYPE_DEPLOY"}} | ||
| {"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/5/operations/jobs.record_failed", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-history-failures-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"record_failed\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}} | ||
| {"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/5/complete", "body": {"completion_reason": "VERSION_COMPLETE_FAILURE"}} | ||
| {"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "7"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_BIND", "resource_key": "jobs.complete_failed"}], "previous_version_id": "5", "version_type": "VERSION_TYPE_DEPLOY"}} | ||
| {"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/7/operations/jobs.complete_failed", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-history-failures-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"complete_failed\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}} | ||
| {"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/7/complete", "body": {"completion_reason": "VERSION_COMPLETE_SUCCESS"}} | ||
|
|
||
| === A remote state upload failure fails the bind version | ||
| >>> fault.py POST /api/2.0/workspace-files/import-file/**resources.json 400 0 1 | ||
|
|
||
| >>> musterr [CLI] bundle deployment bind upload_failed [JOB_ID] --auto-approve | ||
| Error: Fault injected by test. (400 INJECTED) | ||
|
|
||
| Endpoint: POST [DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2Fdms-bind-history-failures-[UNIQUE_NAME]%2Fdefault%2Fstate%2Fresources.json?overwrite=true | ||
| HTTP Status: 400 Bad Request | ||
| API error_code: INJECTED | ||
| API message: Fault injected by test. | ||
|
|
||
|
|
||
| >>> print_requests.py --dms //versions --oneline | ||
| {"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions", "q": {"version_id": "9"}, "body": {"cli_version": "[CLI_VERSION]", "operations": [{"action_type": "OPERATION_ACTION_TYPE_BIND", "resource_key": "jobs.upload_failed"}], "previous_version_id": "7", "version_type": "VERSION_TYPE_DEPLOY"}} | ||
| {"method": "PATCH", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/9/operations/jobs.upload_failed", "q": {"update_mask": "state,error_message,resource_id,status"}, "body": {"error_message": "", "resource_id": "[JOB_ID]", "sequence_id": 0, "state": "{\"state\":{\"deployment\":{\"kind\":\"BUNDLE\",\"metadata_file_path\":\"/Workspace/Users/[USERNAME]/.bundle/dms-bind-history-failures-[UNIQUE_NAME]/default/state/metadata.json\"},\"edit_mode\":\"UI_LOCKED\",\"format\":\"MULTI_TASK\",\"max_concurrent_runs\":1,\"name\":\"upload_failed\",\"queue\":{\"enabled\":true}}}", "status": "OPERATION_STATUS_SUCCEEDED"}} | ||
| {"method": "POST", "path": "/api/2.0/bundle/deployments/[DEPLOYMENT_ID]/versions/9/complete", "body": {"completion_reason": "VERSION_COMPLETE_FAILURE"}} | ||
|
|
||
| === Remote state still contains only the successful bindings | ||
| >>> [CLI] bundle summary -o json | ||
| { | ||
| "complete_failed": "[JOB_ID]", | ||
| "create_failed": "[JOB_ID]", | ||
| "record_failed": "[JOB_ID]", | ||
| "upload_failed": null | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
| job_id=$($CLI jobs create --json '{"name": "bind-target-'$UNIQUE_NAME'"}' | jq -r '.job_id') | ||
| add_repl "$job_id" JOB_ID | ||
|
|
||
| title "A version creation failure does not prevent binding" | ||
| trace fault.py "POST /api/2.0/bundle/deployments/*/versions" 400 0 1 | ||
| trace $CLI bundle deployment bind create_failed "$job_id" --auto-approve | ||
| rm -rf .databricks | ||
|
|
||
| title "An operation recording failure does not prevent binding" | ||
| trace fault.py "PATCH /api/2.0/bundle/**" 400 0 1 | ||
| trace $CLI bundle deployment bind record_failed "$job_id" --auto-approve | ||
| rm -rf .databricks | ||
|
|
||
| title "A version completion failure does not prevent binding" | ||
| trace fault.py "POST /api/2.0/bundle/deployments/*/versions/*/complete" 400 0 1 | ||
| trace $CLI bundle deployment bind complete_failed "$job_id" --auto-approve | ||
|
|
||
| title "All bindings survived the remote state round trips" | ||
| trace jq '.state | map_values(.__id__)' .databricks/bundle/default/resources.json | ||
| trace $CLI jobs get "$job_id" | jq '{job_id, name: .settings.name}' | ||
| trace print_requests.py --dms //versions --oneline | ||
|
|
||
|
|
||
| title "A remote state upload failure fails the bind version" | ||
| trace fault.py "POST /api/2.0/workspace-files/import-file/**resources.json" 400 0 1 | ||
| trace musterr $CLI bundle deployment bind upload_failed "$job_id" --auto-approve | ||
| trace print_requests.py --dms //versions --oneline | ||
|
|
||
| rm -rf .databricks | ||
| title "Remote state still contains only the successful bindings" | ||
| trace $CLI bundle summary -o json | jq '.resources.jobs | map_values(.id // null)' | ||
| rm -f "$OUT_REQUESTS" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Cloud = false # Inject failures into the deployment history service. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| bundle: | ||
| name: dms-bind-immutable-$UNIQUE_NAME | ||
| workspace: | ||
| sync_mode: immutable | ||
| resources: | ||
| jobs: | ||
| foo: | ||
| name: foo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = false | ||
| EnvMatrix.DMS = ["true"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
|
|
||
| >>> [CLI] bundle deployment bind foo [JOB_ID] --auto-approve | ||
| Successfully bound job with an id '[JOB_ID]' | ||
| Run 'bundle deploy' to deploy changes to your workspace | ||
|
|
||
| >>> MSYS_NO_PATHCONV=1 [CLI] bundle-deployments list-resources deployments/[DEPLOYMENT_ID] -o json | ||
| [ | ||
| { | ||
| "last_action_type": "OPERATION_ACTION_TYPE_BIND", | ||
| "resource_id": "[JOB_ID]", | ||
| "resource_key": "jobs.foo" | ||
| } | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
| job_id=$($CLI jobs create --json '{"name": "bind-target-'$UNIQUE_NAME'"}' | jq -r '.job_id') | ||
| add_repl "$job_id" JOB_ID | ||
|
|
||
| trace $CLI bundle deployment bind foo "$job_id" --auto-approve | ||
| deployment_id=$(MSYS_NO_PATHCONV=1 $CLI workspace get-status "/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-bind-immutable-${UNIQUE_NAME}/default/state/resources.deployment.json" -o json | jq -r '.object_id') | ||
| trace MSYS_NO_PATHCONV=1 $CLI bundle-deployments list-resources "deployments/${deployment_id}" -o json | jq -S 'map({resource_key, resource_id, last_action_type})' | ||
| rm -f "$OUT_REQUESTS" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| # This test ensures that deployment history works with bind and immutable folders. This case is interesting | ||
| # because by default we record the snapshot path in deploy, but that is not available during bind. So it's important | ||
| # to ensure that we don't do that during deployment bind, and recording history continues to work. | ||
| Cloud = false |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,2 @@ | ||
| Cloud = true | ||
| EnvMatrix.DMS = ["true"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,2 @@ | ||
| # Verify shadow history markers and resource IDs across bind, deploy, unbind, and rebind. | ||
| Cloud = true |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added because now the summary contains deployment and version ID