Skip to content

Add dbt-deltastream documentation #8014

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: current
Choose a base branch
from
Open

Add dbt-deltastream documentation #8014

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions website/docs/docs/community-adapters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,13 @@ Community adapters are adapter plugins contributed and maintained by members of

| Data platforms (click to view setup guide) |||
| ------------------------------------------ | -------------------------------- | ------------------------------------- |
| [CrateDB](/docs/core/connect-data-platform/cratedb-setup) | [MaxCompute](/docs/core/connect-data-platform/maxcompute-setup) | [Databend Cloud](/docs/core/connect-data-platform/databend-setup) |
| [Doris & SelectDB](/docs/core/connect-data-platform/doris-setup) | [DuckDB](/docs/core/connect-data-platform/duckdb-setup) | [Exasol Analytics](/docs/core/connect-data-platform/exasol-setup) |
|[Extrica](/docs/core/connect-data-platform/extrica-setup) | [Hive](/docs/core/connect-data-platform/hive-setup) | [IBM DB2](/docs/core/connect-data-platform/ibmdb2-setup) |
| [Impala](/docs/core/connect-data-platform/impala-setup) | [Infer](/docs/core/connect-data-platform/infer-setup) | [iomete](/docs/core/connect-data-platform/iomete-setup) |
| [MindsDB](/docs/core/connect-data-platform/mindsdb-setup) | [MySQL](/docs/core/connect-data-platform/mysql-setup) | [RisingWave](/docs/core/connect-data-platform/risingwave-setup) |
| [Rockset](/docs/core/connect-data-platform/rockset-setup) | [SingleStore](/docs/core/connect-data-platform/singlestore-setup)| [SQL Server & Azure SQL](/docs/core/connect-data-platform/mssql-setup) |
| [SQLite](/docs/core/connect-data-platform/sqlite-setup) | [Starrocks](/docs/core/connect-data-platform/starrocks-setup) | [TiDB](/docs/core/connect-data-platform/tidb-setup)|
| [TimescaleDB](https://dbt-timescaledb.debruyn.dev/) | [Upsolver](/docs/core/connect-data-platform/upsolver-setup) | [Vertica](/docs/core/connect-data-platform/vertica-setup) |
| [Watsonx-Presto](/docs/core/connect-data-platform/watsonx-presto-setup) | [IBM watsonx.data - Spark](/docs/core/connect-data-platform/watsonx-spark-setup) | [Yellowbrick](/docs/core/connect-data-platform/yellowbrick-setup) |
| [CrateDB](/docs/core/connect-data-platform/cratedb-setup) | [Databend Cloud](/docs/core/connect-data-platform/databend-setup) | [DeltaStream](/docs/core/connect-data-platform/deltastream-setup) |
| [Doris & SelectDB](/docs/core/connect-data-platform/doris-setup) | [DuckDB](/docs/core/connect-data-platform/duckdb-setup) | [Exasol Analytics](/docs/core/connect-data-platform/exasol-setup) |
| [Extrica](/docs/core/connect-data-platform/extrica-setup) | [Hive](/docs/core/connect-data-platform/hive-setup) | [IBM DB2](/docs/core/connect-data-platform/ibmdb2-setup) |
| [IBM watsonx.data - Spark](/docs/core/connect-data-platform/watsonx-spark-setup) | [Impala](/docs/core/connect-data-platform/impala-setup) | [Infer](/docs/core/connect-data-platform/infer-setup) |
| [iomete](/docs/core/connect-data-platform/iomete-setup) | [MaxCompute](/docs/core/connect-data-platform/maxcompute-setup) | [MindsDB](/docs/core/connect-data-platform/mindsdb-setup) |
| [MySQL](/docs/core/connect-data-platform/mysql-setup) | [RisingWave](/docs/core/connect-data-platform/risingwave-setup) | [Rockset](/docs/core/connect-data-platform/rockset-setup) |
| [SingleStore](/docs/core/connect-data-platform/singlestore-setup) | [SQL Server & Azure SQL](/docs/core/connect-data-platform/mssql-setup) | [SQLite](/docs/core/connect-data-platform/sqlite-setup) |
| [Starrocks](/docs/core/connect-data-platform/starrocks-setup) | [TiDB](/docs/core/connect-data-platform/tidb-setup) | [TimescaleDB](https://dbt-timescaledb.debruyn.dev/) |
| [Upsolver](/docs/core/connect-data-platform/upsolver-setup) | [Vertica](/docs/core/connect-data-platform/vertica-setup) | [Watsonx-Presto](/docs/core/connect-data-platform/watsonx-presto-setup) |
| [Yellowbrick](/docs/core/connect-data-platform/yellowbrick-setup) | | |
96 changes: 96 additions & 0 deletions website/docs/docs/core/connect-data-platform/deltastream-setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
---
title: "DeltaStream setup"
description: "Read this guide to learn about the DeltaStream warehouse setup in dbt."
meta:
maintained_by: Community
authors: 'DeltaStream Team'
github_repo: 'deltastreaminc/dbt-deltastream'
pypi_package: 'dbt-deltastream'
min_core_version: 'v1.10.0'
cloud_support: Not Supported
min_supported_version: '?'
slack_channel_name: '#db-deltastream'
platform_name: 'DeltaStream'
config_page: '/reference/resource-configs/deltastream-configs'
---

import SetUpPages from '/snippets/_setup-pages-intro.md';

<SetUpPages meta={frontMatter.meta} />

## Connecting to DeltaStream with **dbt-deltastream**

To connect to DeltaStream from dbt, you'll need to add a [profile](/docs/core/connect-data-platform/connection-profiles)
to your `profiles.yml` file. A DeltaStream profile conforms to the following syntax:

<File name='profiles.yml'>

```yaml
<profile-name>:
target: <target-name>
outputs:
<target-name>:
type: deltastream

# Required Parameters
token: [ your-api-token ] # Authentication token for DeltaStream API
database: [ your-database ] # Target database name
schema: [ your-schema ] # Target schema name
organization_id: [ your-org-id ] # Organization identifier

# Optional Parameters
url: [ https://api.deltastream.io/v2 ] # DeltaStream API URL, defaults to https://api.deltastream.io/v2
timezone: [ UTC ] # Timezone for operations, defaults to UTC
session_id: [ <empty string> ] # Custom session identifier for debugging purpose
role: [ <empty string> ] # User role
store: [ <empty string> ] # Target store name
compute_pool: [ <empty string> ] # Compute pool name to be used if any else use the default compute pool
```

</File>

### Description of DeltaStream Profile Fields

| Field | Required | Description |
|-------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `type` | ✅ | This must be included either in `profiles.yml` or in the `dbt_project.yml` file. Must be set to `deltastream`. |
| `token` | ✅ | Authentication token for DeltaStream API. This should be stored securely, preferably as an environment variable. |
| `database` | ✅ | Target default database name in DeltaStream where your dbt models will be created. |
| `schema` | ✅ | Target default schema name within the specified database. |
| `organization_id` | ✅ | Organization identifier that determines which DeltaStream organization you're connecting to. |
| `url` | ❌ | DeltaStream API URL. Defaults to `https://api.deltastream.io/v2` if not specified. |
| `timezone` | ❌ | Timezone for operations. Defaults to `UTC` if not specified. |
| `session_id` | ❌ | Custom session identifier for debugging purposes. Helps track operations in DeltaStream logs. |
| `role` | ❌ | User role within the DeltaStream organization. If not specified, uses the default role associated with the token. |
| `store` | ❌ | Target default store name. Stores represent external system connections (Kafka, PostgreSQL, etc.) in DeltaStream. |
| `compute_pool` | ❌ | Compute pool name to be used for models that require computational resources. If not specified, uses the default compute pool. |

## Security Best Practices

When configuring your project for production, it is strongly recommended to use environment variables to store sensitive information such as the authentication token:

<File name='profiles.yml'>

```yaml
your_profile_name:
target: prod
outputs:
prod:
type: deltastream
token: "{{ env_var('DELTASTREAM_API_TOKEN') }}"
database: "{{ env_var('DELTASTREAM_DATABASE') }}"
schema: "{{ env_var('DELTASTREAM_SCHEMA') }}"
organization_id: "{{ env_var('DELTASTREAM_ORG_ID') }}"
```

</File>

## Troubleshooting Connections

If you encounter issues connecting to DeltaStream from dbt, verify the following:

### Authentication Issues

- Ensure your API token is valid and has not expired
- Verify the token has appropriate permissions for the target organization
- Check that the `organization_id` matches your DeltaStream organization
Loading