Skip to content

Dataform CLI dry run not handling dependent changes #2008

@rsinden

Description

@rsinden

Recently, we added a dataform run --dry-run github action to all of our PRs on our repo containing all of our dataform sqlx files. For simple changes, this works nicely and provides a great CI check before merging to main.

However, recently we have found a few times where this check has failed when adding columns and then using them across 2 dependent views, e.g.

Existing on BigQuery:

  • ViewA
SELECT foo FROM table
  • ViewB
SELECT viewA.foo, table2.bar FROM viewA INNER JOIN table2 USING ...

Then the PR adds baz to both views

  • ViewA
SELECT foo, baz FROM table

(dry run action passes here)

  • ViewB
SELECT viewA.foo, viewA.baz, table2.bar FROM viewA INNER JOIN table2 USING ...

(dry run action fails as viewA.baz does not exist)

Maybe it is our lack of understanding on how we would anticipate this to work, but it seems far from ideal that changes like this do not pass the check we have added.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions