bundle: record and read deployment state via DMS with server-generated IDs

[shreyas-goenka]

What

Wires the direct deployment engine into the Deployment Metadata Service (DMS) so that a bundle with experimental.record_deployment_history enabled:

1. Records each deploy/destroy as a DMS version (with heartbeat + completion).
2. Records each applied resource operation (create/update/delete) under that version.
3. Reads resource state back from DMS on the next deploy.

The key design point: server-generated deployment IDs

The DMS API assigns the deployment ID server-side. Earlier prototypes used the local state lineage (a client-minted UUID) as the deployment ID; that no longer holds. The flow is now:

• First deploy → CreateDeployment with an empty ID → the server mints one and returns it in deployment.Name → we parse it out and persist it in the direct-engine state header (Header.DeploymentID).
• Later deploys → pass the stored ID back → GetDeployment to compute the next version number.

Because the ID lives in the state file (which is synced to the workspace), it survives deleting the local .databricks cache — the redeploy recovers the same deployment and just increments the version. No new CreateDeployment.

// libs/dms/recorder.go — first deploy lets the server assign the ID
dep, _ := r.svc.CreateDeployment(ctx, bundledeployments.CreateDeploymentRequest{
    Deployment: bundledeployments.Deployment{TargetName: r.targetName},
})
r.deploymentID, _ = deploymentIDFromName(dep.Name) // "deployments/{id}" -> {id}

Reading DMS authority: last_successful_version_id

The read overlay only trusts DMS state when DMS holds a successfully completed version for the deployment. The deployment exposes last_successful_version_id (advanced only on success, unlike last_version_id), so a single GetDeployment answers that — no version listing.

That field is still stage: DEVELOPMENT in the proto, so it's stripped from the generated SDK. Until it's promoted to PRIVATE_PREVIEW and regenerated, the CLI reads it via a temporary raw GET into a local struct:

// bundle/direct/dstate/dms.go — TODO(DMS): drop once the SDK generates the field
var dep struct {
    LastSuccessfulVersionID string `json:"last_successful_version_id"`
}
err = apiClient.Do(ctx, http.MethodGet, "/api/2.0/bundle/deployments/"+deploymentID,
    auth.WorkspaceIDHeaders(cfg), nil, nil, &dep)
return dep.LastSuccessfulVersionID != "", nil

The proto flip to PRIVATE_PREVIEW is a separate universe change; once it lands and the SDK regenerates, the raw call collapses to client.GetDeployment(...).LastSuccessfulVersionId and the threaded cfg argument goes away.

Wiring

• libs/dms — Recorder: create deployment (server-assigned ID) + version, heartbeat the lease, complete it, delete the deployment on destroy.
• bundle/direct/oprecorder.go — operationRecorder: reports each applied operation. The wire resource_key drops the CLI-internal resources. prefix (resources.jobs.foo → jobs.foo); the read path re-adds it.
• bundle/direct/dstate — Open overlays DMS resource state when DMS holds a successful version (last_successful_version_id set); DeploymentID persisted in the state header.
• bundle/phases — create the version after plan approval, complete it under the lock (deferred before lock.Release), record operations during apply.
• libs/testserver — a stateful fake DMS (deployments / versions / operations / resources) with server-generated IDs; GetDeployment serves last_successful_version_id on version completion.

Testing

• Unit tests: recorder (first deploy / redeploy / destroy / nil no-op), op-recorder prefix stripping, deployment-ID state round-trip.
• Acceptance test acceptance/bundle/dms/record covers the full arc: deploy → server assigns ID → redeploy after rm -rf .databricks (ID recovered, version increments, no new CreateDeployment) → destroy (delete op + deployment deletion).

Gated entirely behind experimental.record_deployment_history + the direct engine; terraform and non-opted bundles are untouched.

This pull request and its description were written by Isaac.

[eng-dev-ecosystem-bot]

Integration test report

Commit: da1ed9d

Run: 30102796660

	Env	🔄​flaky	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
💚​	aws linux		4	4	322	1061	10:56
🔄​	aws windows	1	3	4	324	1059	14:40
💚​	azure linux		4	4	322	1060	11:58
💚​	azure windows		4	4	324	1058	8:33
💚​	gcp linux		1	5	321	1062	11:23
🔄​	gcp windows	1	1	5	322	1060	8:15

9 interesting tests: 4 SKIP, 3 RECOVERED, 2 flaky

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
🔄​	TestAccept	💚​R	🔄​f	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🔄​	TestSyncIncrementalFileOverwritesFolder	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f
💚​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R

Top 13 slowest tests (at least 2 minutes):

duration	env	testname
5:18	gcp windows	TestAccept
4:55	azure windows	TestAccept
4:29	aws linux	TestFilerWorkspaceFilesExtensionsReadDir
3:47	aws windows	TestAccept
3:26	azure linux	TestImportDir
3:09	gcp linux	TestFilerWorkspaceFilesExtensionsReadDir
3:09	azure linux	TestFilerWorkspaceFilesExtensionsRead
3:04	azure windows	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct
3:03	aws linux	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform
3:01	azure linux	TestFilerWorkspaceFilesExtensionsReadDir
2:21	aws windows	TestImportDirWithOverwriteFlag
2:05	azure windows	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform
2:01	azure linux	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct