Skip to content

Commit 4c65dd1

Browse files
natty-linatty-li
authored
Merge branch 'current' into nli/make-gcp-bigquery-ga
2 parents b261f0a + dcc3631 commit 4c65dd1

File tree

4 files changed

+192
-4
lines changed

4 files changed

+192
-4
lines changed

‎website/docs/docs/fusion/new-concepts.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ By rendering and analyzing all models ahead of time, and only beginning executio
5353

5454
### Rendering introspective queries
5555

56-
The exception to AOT rendering is an introspective model: a model whose rendered SQL depends on the results of a database query. Models containg macros like `run_query()` or `dbt_utils.get_column_values()` are introspective. Introspection causes issues with ahead-of-time rendering because:
56+
The exception to AOT rendering is an introspective model: a model whose rendered SQL depends on the results of a database query. Models containing macros like `run_query()` or `dbt_utils.get_column_values()` are introspective. Introspection causes issues with ahead-of-time rendering because:
5757

5858
- Most introspective queries are run against the results of an earlier model in the DAG, which may not yet exist in the database during AOT rendering.
5959
- Even if the model does exist in the database, it might be out of date until after the model has been refreshed.
@@ -64,7 +64,9 @@ Note that macros like `adapter.get_columns_in_relation()` and `dbt_utils.star()`
6464

6565
## Principles of static analysis
6666

67-
[Static analysis](https://en.wikipedia.org/wiki/Static_program_analysis) is meant to guarantee that if a model compiles without error in development, it will also run without compilation errors when deployed. Introspective queries can break this promise by making it possible to modify the rendered query after a model is committed to source control.
67+
The concept of [static analysis](https://en.wikipedia.org/wiki/Static_program_analysis) is meant to guarantee that if a model compiles without error in development, it will also run without compilation errors when deployed. Introspective queries can break this promise by making it possible to modify the rendered query after a model is committed to source control.
68+
69+
The <Constant name="fusion_engine" /> uses the [`static_analysis`](/reference/resource-configs/static-analysis) config to help you control how it performs static analysis for your models.
6870

6971
The <Constant name="fusion_engine" /> is unique in that it can statically analyze not just a single model in isolation, but every query from one end of your DAG to the other. Even your database can only validate the query in front of it! Concepts like [information flow theory](https://roundup.getdbt.com/i/156064124/beyond-cll-information-flow-theory-and-metadata-propagation) &mdash; although not incorporated into the dbt platform [yet](https://www.getdbt.com/blog/where-we-re-headed-with-the-dbt-fusion-engine) &mdash; rely on stable inputs and the ability to trace columns DAG-wide.
7072

@@ -99,15 +101,17 @@ The dbt Fusion engine:
99101

100102
Beyond the default behavior described above, you can always modify the way static analysis is applied for specific models in your project. Remember that **a model is only eligible for static analysis if all of its parents are also eligible.**
101103

102-
The `static_analysis` options are:
104+
The [`static_analysis`](/reference/resource-configs/static-analysis) config options are:
103105

104106
- `on`: Statically analyze SQL. The default for non-introspective models, depends on AOT rendering.
105107
- `unsafe`: Statically analyze SQL. The default for introspective models. Always uses JIT rendering.
106108
- `off`: Skip SQL analysis on this model and its descendants.
107109

108110
When you disable static analysis, features of the VS Code extension which depend on SQL comprehension will be unavailable.
109111

110-
The best place to configure `static_analysis` is as a config on an individual model or group of models. As a debugging aid, you can also use the `--static-analysis off` or `--static-analysis unsafe` CLI flags to override all model-level configuration. Refer to [CLI options](/reference/global-configs/command-line-options) and [Configurations and properties](/reference/configs-and-properties) to learn more about configs.
112+
The best place to configure `static_analysis` is as a config on an individual model or group of models. As a debugging aid, you can also use the [`--static-analysis off` or `--static-analysis unsafe` CLI flags](/reference/global-configs/static-analysis-flag) to override all model-level configuration.
113+
114+
Refer to [CLI options](/reference/global-configs/command-line-options) and [Configurations and properties](/reference/configs-and-properties) to learn more about configs.
111115

112116
### Example configurations
113117

‎website/docs/reference/global-configs/static-analysis-flag.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
---
2+
title: "Static analysis"
3+
id: "static-analysis-flag"
4+
description: "Use the --static-analysis flag to override model-level static_analysis behavior for a single run."
5+
sidebar: "Static analysis"
6+
---
7+
8+
Use the `--static-analysis` flag to override model-level `static_analysis` behavior for a single run. This flag applies to the <Constant name="fusion_engine" /> only; it is ignored by <Constant name="core" />.
9+
10+
Values:
11+
12+
- `off`: Disable static analysis for all models in the run.
13+
- `unsafe`: Use Just-in-time (JIT) static analysis for all models in the run.
14+
15+
If not set, Fusion uses its defaults: Ahead-of-time (AOT) static analysis (`on`) for eligible models and JIT (`unsafe`) for introspective branches. See [Configuring `static_analysis`](/docs/fusion/new-concepts#configuring-static_analysis) for more information.
16+
17+
<File name='Usage'>
18+
19+
```shell
20+
dbt run --static-analysis off
21+
dbt run --static-analysis unsafe
22+
```
23+
24+
</File>
25+
26+
## Related docs
27+
28+
Also check out the model-level [`static_analysis` (resource config)](/reference/resource-configs/static-analysis) and [About flags](/reference/global-configs/about-global-configs) pages for more details.

‎website/docs/reference/resource-configs/static-analysis.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
---
2+
resource_types: [models]
3+
title: "static_analysis"
4+
description: "Use static_analysis config to control how the Fusion engine performs static SQL analysis for models."
5+
datatype: string
6+
default_value: on
7+
sidebar_label: "static_analysis"
8+
---
9+
10+
:::info
11+
12+
The `static_analysis` config is available in the <Constant name="fusion_engine"/> only. It isn't available in <Constant name="core" /> and will be ignored. To upgrade to <Constant name="fusion"/>, refer to [Install <Constant name="fusion"/>](/docs/fusion/install-fusion).
13+
14+
:::
15+
16+
<Tabs>
17+
18+
<TabItem value="dbt_project.yml" label="Project file">
19+
20+
<File name='dbt_project.yml'>
21+
22+
```yml
23+
models:
24+
[resource-path](/reference/resource-configs/resource-path):
25+
+static_analysis: on | unsafe | off
26+
27+
```
28+
29+
</File>
30+
31+
</TabItem>
32+
33+
<TabItem value="Property file">
34+
35+
<File name='models/filename.yml'>
36+
37+
```yml
38+
models:
39+
- name: model_name
40+
[config](/reference/resource-properties/config):
41+
static_analysis: on | unsafe | off
42+
```
43+
44+
</File>
45+
</TabItem>
46+
47+
<TabItem value="SQL config">
48+
49+
<File name='models/model_name.sql'>
50+
51+
```sql
52+
{{ config(static_analysis='on' | 'unsafe' | 'off') }}
53+
54+
select
55+
user_id,
56+
my_cool_udf(ip_address) as cleaned_ip
57+
from {{ ref('my_model') }}
58+
```
59+
60+
</File>
61+
62+
</TabItem>
63+
64+
</Tabs>
65+
66+
## Definition
67+
68+
You can configure if and when the <Constant name="fusion_engine" /> performs static SQL analysis for a model. Configure the `static_analysis` config in your `dbt_project.yml` file, model YAML file, or in the `config` block of your model file. Refer to [rendering strategies](/docs/fusion/new-concepts#rendering-strategies) for more information on how the <Constant name="fusion_engine" /> renders models.
69+
70+
The following values are available for `static_analysis`:
71+
72+
- `on`: Statically analyze SQL ahead-of-time (AOT). Default for non-introspective models, depends on AOT rendering.
73+
- `unsafe`: Statically analyze SQL just-in-time (JIT). The default for when a model (or any of its parents) uses introspective queries. JIT analysis still catches most SQL errors, but [analysis happens]( /docs/fusion/new-concepts#static-analysis-and-introspective-queries) after some upstream execution.
74+
- `off`: Skip SQL analysis for this model and its descendants.
75+
76+
A model is _only_ eligible for static analysis if all of its parents are also eligible.
77+
78+
Refer to the Fusion concepts page for deeper discussion and visuals: [New concepts](/docs/fusion/new-concepts). For more info on the JSON schema, refer to the [dbt-jsonschema file](https://github.com/dbt-labs/dbt-jsonschema/blob/1e2c1536fbdd421e49c8b65c51de619e3cd313ff/schemas/latest_fusion/dbt_project-latest-fusion.json#L4689).
79+
80+
## CLI override
81+
82+
You can override model-level configuration for a run using the following CLI flags. For example, to disable static analysis for a run:
83+
84+
```bash
85+
dbt run --static-analysis off # disable static analysis for all models
86+
dbt run --static-analysis unsafe # use JIT analysis for all models
87+
```
88+
89+
See [static analysis CLI flag](/reference/global-configs/static-analysis-flag).
90+
91+
## Examples
92+
93+
The following examples show how to disable static analysis for all models in a package, for a single model, and for a model that uses a custom UDF.
94+
95+
<!-- no toc -->
96+
- [Disable static analysis for all models in a package](#disable-static-analysis-for-all-models-in-a-package)
97+
- [Disable static analysis in YAML for a single model](#disable-static-analysis-in-yaml-for-a-single-model)
98+
- [Disable static analysis in SQL for a model using a custom UDF](#disable-static-analysis-in-sql-for-a-model-using-a-custom-udf)
99+
100+
#### Disable static analysis for all models in a package
101+
This example shows how to disable static analysis for all models in a package. The [`+` prefix](/reference/resource-configs/plus-prefix) applies the config to all models in the package.
102+
103+
<File name='dbt_project.yml'>
104+
105+
```yml
106+
name: jaffle_shop
107+
108+
models:
109+
jaffle_shop:
110+
marts:
111+
+materialized: table
112+
113+
a_package_with_introspective_queries:
114+
+static_analysis: off
115+
```
116+
117+
</File>
118+
119+
#### Disable static analysis in YAML for a single model
120+
121+
This example shows how to disable static analysis for a single model in YAML.
122+
123+
<File name='models/my_udf_using_model.yml'>
124+
125+
```yml
126+
models:
127+
- name: model_with_static_analysis_off
128+
config:
129+
static_analysis: off
130+
```
131+
132+
</File>
133+
134+
#### Disable static analysis in SQL for a model using a custom UDF
135+
136+
This example shows how to disable static analysis for a model using a custom [user-defined function (UDF)](/docs/build/udfs) in a SQL file.
137+
138+
<File name='models/my_udf_using_model.sql'>
139+
140+
```sql
141+
{{ config(static_analysis='off') }}
142+
143+
select
144+
user_id,
145+
my_cool_udf(ip_address) as cleaned_ip
146+
from {{ ref('my_model') }}
147+
```
148+
149+
</File>
150+
151+
## Considerations
152+
153+
- Disabling static analysis means that features of the VS Code extension that depend on SQL comprehension will be unavailable.
154+
- Static analysis might fail in some cases (for example, dynamic SQL constructs or unrecognized UDFs) and may require setting `static_analysis: off`. For more examples, refer to [When should I turn static analysis off?](/docs/fusion/new-concepts#when-should-i-turn-static-analysis-off).

‎website/sidebars.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1225,6 +1225,7 @@ const sidebarSettings = {
12251225
"reference/resource-configs/plus-prefix",
12261226
"reference/resource-configs/pre-hook-post-hook",
12271227
"reference/resource-configs/schema",
1228+
"reference/resource-configs/static-analysis",
12281229
"reference/resource-configs/tags",
12291230
"reference/resource-configs/unique_key",
12301231
],
@@ -1485,6 +1486,7 @@ const sidebarSettings = {
14851486
"reference/global-configs/print-output",
14861487
"reference/global-configs/record-timing-info",
14871488
"reference/global-configs/resource-type",
1489+
"reference/global-configs/static-analysis-flag",
14881490
"reference/global-configs/warnings",
14891491
],
14901492
},

0 commit comments

Comments
 (0)