Add experimental support for large file upload. by renaudhartert-db · Pull Request #6018 · databricks/cli

renaudhartert-db · 2026-07-22T04:49:21Z

What

The fs command's UC Volumes filer now runs on the sdk-go/files/v2 client instead of hand-written Files API calls. The filer is shared, so bundle library uploads to Volumes go through the same path.

files/v2 brings its own multipart/resumable upload engine. DATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD gates whether large Volumes uploads are split into parts; it selects multipart-vs-single-shot within files/v2. Default behavior (single-shot PUT) is unchanged.

The CLI takes on four new direct dependencies: github.com/databricks/sdk-go/files (v0.0.0-dev.1), .../auth and .../options (v0.0.0-dev), and .../core bumped to v0.0.1-dev. These are -dev pre-release versions; a follow-up should move to stable tags once sdk-go cuts them.

Notable in the diff

libs/filer/files_client.go is rewritten onto *files.Client. Error mapping keys off the HTTP status (via a httpStatus helper) rather than the SDK's canonical codes.Code, because the Files API reports "path already exists" as 409, which the SDK maps to codes.Aborted (not codes.AlreadyExists).
libs/filer/files_client_auth.go adapts the CLI's resolved config.Config into the auth.Credentials the client expects, signing a request with config.Authenticate and reusing those headers instead of re-reading a profile. Its credentials name is the config's resolved auth type (pat, oauth-m2m, ...).
NewFilesClient gains a ctx parameter, threaded through its callers.
Shared code: the Volumes filer is used by both fs and bundle/libraries, so bundle Volumes uploads are affected. go.mod/NOTICE gain the new sdk-go modules as direct dependencies.
libs/testserver now models "409 when creating a directory over an existing file," which the fake previously did not; this closes the gap that let a Mkdir error-mapping regression through.

Out of scope

Local, DBFS, and workspace filers are unchanged; files/v2 only backs the UC Volumes path.

This pull request and its description were written by Isaac.

github-actions · 2026-07-22T04:50:00Z

Approval status: pending

`/bundle/` - needs approval

Files: bundle/libraries/filer_volume.go
Suggested: @pietern
Also eligible: @denik, @shreyas-goenka, @janniklasrose, @andrewnester, @anton-107, @lennartkats-db

`/cmd/fs/` - needs approval

4 files changed
Suggested: @simonfaltum
Also eligible: @Divyansh-db, @hectorcast-db, @parthban-db, @tanmay-db, @tejaskochar-db, @mihaimitrea-db, @chrisst, @rauchy

`/experimental/air/` - needs approval

Files: experimental/air/cmd/snapshot.go
Suggested: @riddhibhagwat-db
Also eligible: @apeforest, @bfontain, @lu-wang-dl, @panchalhp-db, @vinchenzo-db, @maggiewang-db, @ben-hansen-db, @pardis-beikzadeh-db

`/integration/` - needs approval

Files: integration/cmd/fs/helpers_test.go, integration/libs/filer/helpers_test.go
Suggested: @simonfaltum
Also eligible: @Divyansh-db, @hectorcast-db, @parthban-db, @tanmay-db, @tejaskochar-db, @mihaimitrea-db, @chrisst, @rauchy

`/internal/` - needs approval

Files: internal/build/notice_test.go
Suggested: @simonfaltum
Also eligible: @Divyansh-db, @hectorcast-db, @parthban-db, @tanmay-db, @tejaskochar-db, @mihaimitrea-db, @chrisst, @rauchy

`/libs/filer/` - needs approval

Files: libs/filer/files_client.go, libs/filer/files_client_auth.go, libs/filer/files_client_test.go
Suggested: @simonfaltum
Also eligible: @Divyansh-db, @hectorcast-db, @parthban-db, @tanmay-db, @tejaskochar-db, @mihaimitrea-db, @chrisst, @rauchy

General files (require maintainer)

Files: go.mod, go.sum, libs/testserver/handlers.go
Based on git history:

@pietern -- recent work in libs/filer/, ./, libs/testserver/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @janniklasrose, @lennartkats-db) can approve all areas.

See OWNERS for ownership rules.}

eng-dev-ecosystem-bot · 2026-07-22T05:18:53Z

Integration test report

Commit: 24741c4

Run: 30096615950

	Env	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
🔄	aws linux	1	4	4	321	1059	14:34
🔄	aws windows	1	4	4	323	1057	7:53
🔄	azure linux	2	4	4	320	1058	13:03
💚	azure windows		4	4	324	1056	11:27
💚	gcp linux		1	5	321	1060	13:21
🔄	gcp windows	1	1	5	322	1058	9:14

13 interesting tests: 5 flaky, 4 RECOVERED, 4 SKIP

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🔄	TestFsCpFileToDirWithOverwriteFlag	✅p	✅p	🔄f	✅p	✅p	✅p
🔄	TestFsCpFileToDirWithOverwriteFlag/uc-volumes_to_uc-volumes	✅p	✅p	🔄f	✅p	✅p	✅p
🔄	TestSyncIncrementalFileOverwritesFolder	✅p	✅p	✅p	✅p	✅p	🔄f
🔄	TestSyncNestedFolderDoesntFailOnNonEmptyDirectory	🔄f	✅p	✅p	✅p	✅p	✅p
🔄	TestSyncNestedFolderSync	✅p	🔄f	✅p	✅p	✅p	✅p
💚	TestFetchRepositoryInfoAPI_FromRepo	💚R	💚R	💚R	💚R	🙈S	🙈S
💚	TestFetchRepositoryInfoAPI_FromRepo/root	💚R	💚R	💚R	💚R
💚	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚R	💚R	💚R	💚R

Top 23 slowest tests (at least 2 minutes):

duration	env	testname
7:37	azure windows	TestFilerWorkspaceFilesExtensionsReadDir
4:27	gcp windows	TestFilerWorkspaceFilesExtensionsReadDir
3:50	aws linux	TestFilerWorkspaceFilesExtensionsReadDir
3:20	aws windows	TestFilerWorkspaceFilesExtensionsRead
3:16	azure linux	TestFilerWorkspaceFilesExtensionsRead
3:05	aws windows	TestAccept
3:05	gcp windows	TestAccept
3:04	azure windows	TestAccept
2:45	gcp windows	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform
2:44	azure windows	TestFilerRecursiveDelete/workspace_files_extensions
2:42	gcp windows	TestImportDirWithOverwriteFlag
2:41	azure windows	TestImportDirDoesNotOverwrite
2:39	gcp linux	TestFilerWorkspaceFilesExtensionsReadDir
2:38	aws windows	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct
2:30	azure linux	TestFilerWorkspaceFilesExtensionsStat
2:26	gcp linux	TestFilerReadWrite/workspace_files_extensions
2:23	aws linux	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct
2:13	azure linux	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform
2:08	azure windows	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct
2:05	azure linux	TestSyncFullFileSync
2:03	azure windows	TestSyncIncrementalFileSync
2:01	azure linux	TestImportDir
2:00	azure windows	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform

Reworks the `fs` command's UC Volumes filer (also used by bundle library uploads) to run on the new `sdk-go/files/v2` client instead of hand-rolled Files API calls plus a separate CLI-side multipart upload engine. `files/v2` is not published to the Go proxy yet, so it is vendored under `libs/tmp/` along with the two unpublished sibling modules it needs (`auth` and `options`); the published `sdk-go/core` module stays a normal dependency. The only edits to the vendored copy are the import-path rewrite and two fixes to the generated client (see below). `libs/tmp/README.md` records the copy and the swap-and-delete steps for when the module is published. The filer now delegates upload, download, listing, mkdir, delete, and stat to `files/v2`, which brings its own multipart/resumable engine. The CLI's own `libs/upload` engine is therefore removed. `DATABRICKS_EXPERIMENTAL_MULTIPART_UPLOAD` still gates whether large Volumes uploads are split into parts, but now selects multipart-vs-single-shot within files/v2 rather than a separate engine. This pull request and its description were written by Isaac. Co-authored-by: Isaac

The sdk-go files/v2 client and its auth/options/core dependencies are now published to the Databricks Go proxy, so the temporary in-repo copy under libs/tmp is no longer needed. Replaces it with real module dependencies (files v0.0.0-dev.1, auth v0.0.0-dev, options v0.0.0-dev, core bumped to v0.0.1-dev), rewrites the filer and fs command imports from github.com/databricks/cli/libs/tmp/... back to github.com/databricks/sdk-go/..., and removes libs/tmp along with its carve-outs in .golangci.yaml, tools/check_deadcode.py, and .github/OWNERS. The two local fixes that had been applied to the vendored generated client (empty response bodies and HTTP method casing) are fixed upstream in the published version, so nothing is lost. This pull request and its description were written by Isaac. Co-authored-by: Isaac

The configCredentials adapter reported a hardcoded "databricks-cli" as its credentials name, which the files/v2 client folds into the User-Agent. Report the config's actual resolved auth type (pat, oauth-m2m, azure-cli, ...) instead. The CLI resolves auth when it builds the workspace client, so AuthType is set before the filer is created. This pull request and its description were written by Isaac. Co-authored-by: Isaac

The files/v2 client validates the credential name at construction and rejects an empty value with an "invalid value for \"auth\"" error. configCredentials.Name returned cfg.AuthType, which is empty when auth has not been resolved (as in the bundle/libraries UC Volumes filer tests that build a bare config), breaking task test across all platforms. Co-authored-by: Isaac

renaudhartert-db changed the title ~~fs: reimplement UC Volumes filer on the files/v2 client~~ [WIP] reimplement UC Volumes filer on the files/v2 client Jul 22, 2026

renaudhartert-db force-pushed the fs-files-v2 branch from a392fc5 to 39fe347 Compare July 22, 2026 07:09

renaudhartert-db changed the title ~~[WIP] reimplement UC Volumes filer on the files/v2 client~~ fs: reimplement UC Volumes filer on the sdk-go files/v2 client Jul 24, 2026

renaudhartert-db force-pushed the fs-files-v2 branch from a637d98 to cb8b260 Compare July 24, 2026 12:45

renaudhartert-db changed the title ~~fs: reimplement UC Volumes filer on the sdk-go files/v2 client~~ Add experimental support for large file upload. Jul 24, 2026

renaudhartert-db added 2 commits July 24, 2026 14:47

Merge branch 'main' into fs-files-v2

bc02160

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add experimental support for large file upload.#6018

Add experimental support for large file upload.#6018
renaudhartert-db wants to merge 5 commits into
mainfrom
fs-files-v2

renaudhartert-db commented Jul 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Jul 22, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

renaudhartert-db commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Notable in the diff

Out of scope

Uh oh!

github-actions Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approval status: pending

/bundle/ - needs approval

/cmd/fs/ - needs approval

/experimental/air/ - needs approval

/integration/ - needs approval

/internal/ - needs approval

/libs/filer/ - needs approval

General files (require maintainer)

Uh oh!

eng-dev-ecosystem-bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

renaudhartert-db commented Jul 22, 2026 •

edited

Loading

github-actions Bot commented Jul 22, 2026 •

edited

Loading

`/bundle/` - needs approval

`/cmd/fs/` - needs approval

`/experimental/air/` - needs approval

`/integration/` - needs approval

`/internal/` - needs approval

`/libs/filer/` - needs approval

eng-dev-ecosystem-bot commented Jul 22, 2026 •

edited

Loading