diff --git a/TOC-tidb-cloud-lake.md b/TOC-tidb-cloud-lake.md index 4a5a057eae9a1..98bdf2ee26e6e 100644 --- a/TOC-tidb-cloud-lake.md +++ b/TOC-tidb-cloud-lake.md @@ -59,6 +59,8 @@ - [Load from Bucket](/tidb-cloud-lake/guides/load-from-bucket.md) - [Load from Local File](/tidb-cloud-lake/guides/load-from-local-file.md) - [Load from Remote File](/tidb-cloud-lake/guides/load-from-remote-file.md) + - Load with Platforms + - [Load with dbt](/tidb-cloud-lake/guides/load-with-dbt.md) - Load Semi-structured Data - [Overview](/tidb-cloud-lake/guides/load-semi-structured-data.md) - [Load Parquet](/tidb-cloud-lake/guides/load-parquet.md) diff --git a/tidb-cloud-lake/guides/load-with-dbt.md b/tidb-cloud-lake/guides/load-with-dbt.md index 282be6d2b85b8..f06062f66838f 100644 --- a/tidb-cloud-lake/guides/load-with-dbt.md +++ b/tidb-cloud-lake/guides/load-with-dbt.md @@ -1,54 +1,89 @@ --- -title: dbt -summary: dbt is a transformation workflow that helps you get more work done while producing higher quality results. You can use dbt to modularize and centralize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. Collaborate on data models, version them, and test and document your queries before safely deploying them to production, with monitoring and visibility. +title: Use dbt with TiDB Cloud Lake +summary: Learn how to install the TiDB Cloud Lake adapter for dbt, configure a connection profile, and run dbt workflows against your Lake data. --- -# dbt +# Use dbt with TiDB Cloud Lake -[dbt](https://www.getdbt.com/) is a transformation workflow that helps you get more work done while producing higher quality results. You can use dbt to modularize and centralize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. Collaborate on data models, version them, and test and document your queries before safely deploying them to production, with monitoring and visibility. +[dbt](https://www.getdbt.com/) is an open-source data transformation tool that enables analytics engineers to build, test, and document SQL-based data models. The [TiDB Cloud Lake adapter for dbt](https://github.com/tidbcloud/lake-dbt) connects dbt to {{{ .lake }}} through the `tidbcloudlake-driver` Python package. -[dbt-databend-cloud](https://github.com/databendcloud/dbt-databend) is a plugin developed by {{{ .lake }}} with the primary goal of enabling smooth integration between dbt and {{{ .lake }}}. By utilizing this plugin, you can seamlessly perform data modeling, transformation, and cleansing tasks using dbt and conveniently load the output into {{{ .lake }}}. The table below illustrates the level of support that the dbt-databend-cloud plugin offers for commonly used features in dbt: +## Supported features -| Feature | Supported ? | -|----------------------------- |----------- | -| Table Materialization | Yes | -| View Materialization | Yes | -| Incremental Materialization | Yes | -| Ephemeral Materialization | No | -| Seeds | Yes | -| Sources | Yes | -| Custom Data Tests | Yes | -| Docs Generate | Yes | -| Snapshots | No | -| Connection Retry | Yes | +The `tidbcloudlake-dbt` adapter supports the following dbt features: -## Installing dbt-databend-cloud +| Feature | Supported | +|---------|-----------| +| Table materialization | Yes | +| View materialization | Yes | +| Incremental materialization | Yes | +| Ephemeral materialization | No | +| Seeds | Yes | +| Sources | Yes | +| Custom data tests | Yes | +| Docs generate | Yes | +| Snapshots | Yes | +| Connection retry | Yes | -Installing the dbt-databend-cloud plugin has been streamlined for your convenience, as it now includes dbt as a required dependency. To effortlessly set up both dbt and the dbt-databend-cloud plugin, run the following command: +Ephemeral materialization is not supported. In output from `dbt docs generate`, owner metadata is not available for views. + +## Prerequisites + +Before you begin, make sure that you have the following: + +- Python 3.8 or later +- A {{{ .lake }}} account and warehouse +- The host, database, username, password, and warehouse name for your {{{ .lake }}} connection + +For information about connecting to {{{ .lake }}}, see [Connection Overview](/tidb-cloud-lake/guides/connection-overview.md). + +## Install the adapter + +Install the `tidbcloudlake-dbt` package from PyPI. The package installs dbt Core and the required `tidbcloudlake-driver` dependency. ```shell -pip3 install dbt-databend-cloud +python3 -m pip install tidbcloudlake-dbt ``` -However, if you prefer to install dbt separately, you can refer to the official dbt installation guide for detailed instructions. +Verify the installation: -## Tutorial: Run dbt Project jaffle_shop +```shell +dbt --version +``` -If you're new to dbt, {{{ .lake }}} recommends completing the official dbt tutorial available at . Before you start, follow [Installing dbt-databend-cloud](#installing-dbt-databend-cloud) to install dbt and dbt-databend-cloud. +## Configure a dbt profile -This tutorial provides a sample dbt project called "jaffle_shop," offering hands-on experience with the dbt tool. By configuring the default global profile (~/.dbt/profiles.yml) with the necessary information to connect to your {{{ .lake }}} instance, the project will generate tables and views defined in the dbt models directly in your {{{ .lake }}} database. Here's an example of the file profiles.yml that connects to a {{{ .lake }}} instance: +In `~/.dbt/profiles.yml`, add a profile for your {{{ .lake }}} connection: -```yml title="~/.dbt/profiles.yml" -jaffle_shop_databend: +```yaml title="~/.dbt/profiles.yml" +lake_dbt: target: dev outputs: dev: - type: databend - host: 127.0.0.1 - port: 8000 - schema: sjh_dbt - user: databend - pass: ******** + type: tidbcloudlake + host: .gw..default.tidbcloud.com + port: 443 + schema: + user: + pass: + warehouse: + secure: true +``` + +Set the `profile` field in your `dbt_project.yml` file to the profile name: + +```yaml title="dbt_project.yml" +profile: lake_dbt ``` -If you're using {{{ .lake }}}, you can refer to this [Wiki page](https://github.com/databendcloud/dbt-databend/wiki/How-to-use-dbt-with-Databend-Cloud) for step-by-step instructions on how to run the jaffle_shop dbt project. +The profile supports the following options: + +| Option | Required | Description | +|--------|----------|-------------| +| `type` | Yes | Adapter type. Set it to `tidbcloudlake`. | +| `host` | Yes | Hostname of your {{{ .lake }}} endpoint. | +| `port` | Yes | Port of the endpoint. Use `443` for {{{ .lake }}}. | +| `schema` | Yes | Target {{{ .lake }}} database where dbt builds models. | +| `user` | Yes | Username used to connect to {{{ .lake }}}. | +| `pass` | Yes | Password used to connect to {{{ .lake }}}. | +| `warehouse` | No | Warehouse used to run dbt queries. | +| `secure` | No | Whether to use TLS. The default value is `true`. |