lake: publish dbt integration docs #23489
+70
−33
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,54 +1,89 @@ | ||||||
| --- | ||||||
| title: Use dbt with TiDB Cloud Lake | ||||||
| summary: Learn how to install the TiDB Cloud Lake adapter for dbt, configure a connection profile, and run dbt workflows against your Lake data. | ||||||
| --- | ||||||
|
|
||||||
| # Use dbt with TiDB Cloud Lake | ||||||
|
|
||||||
| [dbt](https://www.getdbt.com/) is an open-source data transformation tool that enables analytics engineers to build, test, and document SQL-based data models. The [TiDB Cloud Lake adapter for dbt](https://github.com/tidbcloud/lake-dbt) connects dbt to {{{ .lake }}} through the `tidbcloudlake-driver` Python package. | ||||||
|
|
||||||
| ## Supported features | ||||||
|
|
||||||
| The `tidbcloudlake-dbt` adapter supports the following dbt features: | ||||||
|
|
||||||
| | Feature | Supported | | ||||||
| |---------|-----------| | ||||||
| | Table materialization | Yes | | ||||||
| | View materialization | Yes | | ||||||
| | Incremental materialization | Yes | | ||||||
| | Ephemeral materialization | No | | ||||||
| | Seeds | Yes | | ||||||
| | Sources | Yes | | ||||||
| | Custom data tests | Yes | | ||||||
| | Docs generate | Yes | | ||||||
| | Snapshots | Yes | | ||||||
| | Connection retry | Yes | | ||||||
|
|
||||||
| Ephemeral materialization is not supported. In output from `dbt docs generate`, owner metadata is not available for views. | ||||||
|
|
||||||
| ## Prerequisites | ||||||
|
|
||||||
| Before you begin, make sure that you have the following: | ||||||
|
|
||||||
| - Python 3.8 or later | ||||||
| - A {{{ .lake }}} account and warehouse | ||||||
| - The host, database, username, password, and warehouse name for your {{{ .lake }}} connection | ||||||
|
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- relevant document sections ---'
sed -n '1,110p' tidb-cloud-lake/guides/load-with-dbt.md
printf '%s\n' '--- warehouse references ---'
rg -n -C 3 '\bwarehouse\b|tidbcloudlake-dbt|lake-dbt' tidb-cloud-lake .github resources 2>/dev/null || true
printf '%s\n' '--- candidate adapter files ---'
git ls-files | rg -i 'dbt|adapter|profile|lake'Repository: pingcap/docs Length of output: 50371 🏁 Script executed: #!/bin/bash
set -eu
python3 - <<'PY'
import json
import urllib.request
def get(url):
req = urllib.request.Request(url, headers={"User-Agent": "review-inspection"})
with urllib.request.urlopen(req, timeout=20) as response:
return json.load(response)
pypi = get("https://pypi.org/pypi/tidbcloudlake-dbt/json")
info = pypi["info"]
print("--- PyPI metadata ---")
for key in ("name", "version", "summary", "home_page", "project_urls", "requires_python", "requires_dist"):
print(f"{key}: {info.get(key)}")
print("--- release files ---")
for item in pypi["releases"].get(info["version"], []):
print(item.get("filename"), item.get("url"))
repo = get("https://api.github.com/repos/tidbcloud/lake-dbt")
print("--- GitHub repository ---")
for key in ("html_url", "default_branch", "visibility", "archived"):
print(f"{key}: {repo.get(key)}")
for path in ("profiles.yml", "dbt/adapters/tidbcloudlake/connections.py",
"dbt/adapters/tidbcloudlake/impl.py",
"dbt/adapters/tidbcloudlake/__init__.py"):
url = "https://api.github.com/repos/tidbcloud/lake-dbt/contents/" + path
try:
item = get(url)
except Exception as exc:
print(f"--- {path}: unavailable ({exc}) ---")
continue
print(f"--- {path} ---")
print(item.get("download_url"))
PYRepository: pingcap/docs Length of output: 2469 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- PyPI project metadata ---'
curl -k -fsSL https://pypi.org/pypi/tidbcloudlake-dbt/json |
jq '{name: .info.name, version: .info.version, project_urls: .info.project_urls,
requires_python: .info.requires_python, requires_dist: .info.requires_dist,
release_urls: [.releases[.info.version][] | .url]}'
printf '%s\n' '--- upstream repository metadata ---'
curl -k -fsSL https://api.github.com/repos/tidbcloud/lake-dbt |
jq '{html_url, default_branch, visibility, archived}'
printf '%s\n' '--- upstream tree paths containing profile/connection/adapter ---'
curl -k -fsSL 'https://api.github.com/repos/tidbcloud/lake-dbt/git/trees/HEAD?recursive=1' |
jq -r '.tree[] | select(.path | test("(?i)(profile|connection|adapter|config)")) | .path'Repository: pingcap/docs Length of output: 1133 🏁 Script executed: #!/bin/bash
set -eu
wheel=/tmp/tidbcloudlake_dbt-1.8.1-py3-none-any.whl
curl -k -fsSL \
'https://files.pythonhosted.org/packages/a7/48/34e912b921839e80f6d722136e9624ab89fe899aa3aef6c781ec56c99a2e/tidbcloudlake_dbt-1.8.1-py3-none-any.whl' \
-o "$wheel"
python3 - "$wheel" <<'PY'
import re
import sys
import zipfile
wheel = sys.argv[1]
with zipfile.ZipFile(wheel) as z:
names = z.namelist()
print("--- package files ---")
for name in names:
if name.endswith((".py", ".yml", ".yaml", ".json", ".toml")):
print(name)
print("--- relevant source lines ---")
for name in names:
if not name.endswith(".py"):
continue
text = z.read(name).decode("utf-8", "replace")
if re.search(r"warehouse|Profile|Connection|credentials|required", text, re.I):
print(f"### {name}")
for number, line in enumerate(text.splitlines(), 1):
if re.search(r"warehouse|Profile|Connection|credentials|required", line, re.I):
print(f"{number}: {line}")
PY
printf '%s\n' '--- upstream repository metadata ---'
curl -k -fsSL https://api.github.com/repos/tidbcloud/lake-dbt |
jq '{html_url, default_branch, visibility, archived}'
printf '%s\n' '--- upstream tree paths ---'
curl -k -fsSL 'https://api.github.com/repos/tidbcloud/lake-dbt/git/trees/HEAD?recursive=1' |
jq -r '.tree[] | select(.path | test("(?i)(profile|connection|adapter|config)")) | .path'Repository: pingcap/docs Length of output: 4497 🏁 Script executed: #!/bin/bash
set -eu
python3 - <<'PY'
import zipfile
wheel = "/tmp/tidbcloudlake_dbt-1.8.1-py3-none-any.whl"
with zipfile.ZipFile(wheel) as z:
for name in (
"dbt/include/tidbcloudlake/profile_template.yml",
"dbt/adapters/tidbcloudlake/connections.py",
):
print(f"--- {name} ---")
for number, line in enumerate(z.read(name).decode().splitlines(), 1):
if name.endswith("profile_template.yml") or 30 <= number <= 170:
print(f"{number}: {line}")
source = z.read("dbt/adapters/tidbcloudlake/connections.py").decode()
namespace = {}
# Extract only the deterministic DSN-building logic for a read-only probe.
start = source.index(" def build_dsn")
end = source.index(" `@classmethod`", start)
print("--- build_dsn source slice ---")
print(source[start:end])
PYRepository: pingcap/docs Length of output: 6419 🏁 Script executed: #!/bin/bash
set -eu
python3 - <<'PY'
import zipfile
wheel = "/tmp/tidbcloudlake_dbt-1.8.1-py3-none-any.whl"
with zipfile.ZipFile(wheel) as z:
for name in (
"dbt/include/tidbcloudlake/profile_template.yml",
"dbt/adapters/tidbcloudlake/connections.py",
):
print(f"--- {name} ---")
for number, line in enumerate(z.read(name).decode().splitlines(), 1):
if name.endswith("profile_template.yml") or 30 <= number <= 170:
print(f"{number}: {line}")
source = z.read("dbt/adapters/tidbcloudlake/connections.py").decode()
start = source.index(" def build_dsn")
end = source.index(" `@classmethod`", start)
print("--- build_dsn source slice ---")
print(source[start:end])
PYRepository: pingcap/docs Length of output: 6419 Treat Keep the warehouse resource in the prerequisites, but do not list the optional Committable suggestion
Suggested change
Source: Path instructions |
||||||
|
|
||||||
| For information about connecting to {{{ .lake }}}, see [Connection Overview](/tidb-cloud-lake/guides/connection-overview.md). | ||||||
|
|
||||||
| ## Install the adapter | ||||||
|
|
||||||
| Install the `tidbcloudlake-dbt` package from PyPI. The package installs dbt Core and the required `tidbcloudlake-driver` dependency. | ||||||
|
|
||||||
| ```shell | ||||||
| python3 -m pip install tidbcloudlake-dbt | ||||||
| ``` | ||||||
|
|
||||||
| Verify the installation: | ||||||
|
|
||||||
| ```shell | ||||||
| dbt --version | ||||||
| ``` | ||||||
|
|
||||||
| ## Configure a dbt profile | ||||||
|
|
||||||
| In `~/.dbt/profiles.yml`, add a profile for your {{{ .lake }}} connection: | ||||||
|
|
||||||
| ```yaml title="~/.dbt/profiles.yml" | ||||||
| lake_dbt: | ||||||
| target: dev | ||||||
| outputs: | ||||||
| dev: | ||||||
| type: tidbcloudlake | ||||||
| host: <tenant>.gw.<region>.default.tidbcloud.com | ||||||
| port: 443 | ||||||
| schema: <database> | ||||||
| user: <username> | ||||||
| pass: <password> | ||||||
| warehouse: <warehouse> | ||||||
| secure: true | ||||||
| ``` | ||||||
|
|
||||||
| Set the `profile` field in your `dbt_project.yml` file to the profile name: | ||||||
|
|
||||||
| ```yaml title="dbt_project.yml" | ||||||
| profile: lake_dbt | ||||||
| ``` | ||||||
|
|
||||||
| The profile supports the following options: | ||||||
|
|
||||||
| | Option | Required | Description | | ||||||
| |--------|----------|-------------| | ||||||
| | `type` | Yes | Adapter type. Set it to `tidbcloudlake`. | | ||||||
| | `host` | Yes | Hostname of your {{{ .lake }}} endpoint. | | ||||||
| | `port` | Yes | Port of the endpoint. Use `443` for {{{ .lake }}}. | | ||||||
| | `schema` | Yes | Target {{{ .lake }}} database where dbt builds models. | | ||||||
| | `user` | Yes | Username used to connect to {{{ .lake }}}. | | ||||||
| | `pass` | Yes | Password used to connect to {{{ .lake }}}. | | ||||||
| | `warehouse` | No | Warehouse used to run dbt queries. | | ||||||
| | `secure` | No | Whether to use TLS. The default value is `true`. | | ||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
Repository: pingcap/docs
Length of output: 41278
🌐 Web query:
Search for the public GitHub repository and published package documentation for "tidbcloud/lake-dbt", "tidbcloudlake-dbt", and "tidbcloudlake-driver".💡 Result:
Based on current technical documentation and repository information as of August 11, 2026, the specific repositories and packages you inquired about are part of the TiDB ecosystem, though their naming conventions differ from your request. 1. tidbcloud/lake-dbt and tidbcloudlake-dbt There is no repository named tidbcloud/lake-dbt or tidbcloudlake-dbt. The official adapter for using dbt with TiDB is pingcap/dbt-tidb [1][2]. - GitHub Repository: https://github.com/pingcap/dbt-tidb [1][3] - Documentation: Available on the dbt Developer Hub [2] and the official TiDB documentation site [4]. - Package: The dbt adapter package is published as dbt-tidb [2]. 2. tidbcloudlake-driver There is no repository precisely named tidbcloudlake-driver. TiDB Cloud Lake provides various language-specific drivers instead of a single monolithic repository with that name [5]. - Core Driver Repository: The main repository for TiDB Cloud Lake native clients (including Rust, Python, and Node.js implementations) is tidbcloud/lakesql [6]. - Rust Driver: The Rust driver is available in the lakesql repository under the /driver directory [7]. It is published on crates.io as lake-driver [8][9]. - Other Languages: TiDB Cloud Lake provides specific drivers for Python, Go (lake-go), and Java (lake-jdbc) [10][5]. The official TiDB documentation maintains a dedicated page for connecting to TiDB Cloud Lake using these drivers [5]. Official documentation for all TiDB Cloud Lake connectivity can be found at the TiDB Docs website [11][5].
Citations:
Make the adapter repository accessible before publication. The
https://github.com/tidbcloud/lake-dbtlink returns 404, so readers cannot access the adapter. Publish the repository or update the link to its canonical public location.Source: Path instructions