Skip to content

How does one test compilation code changes locally? #1629

@mattkocak

Description

@mattkocak

Hello, this is a question rather than an issue. I'm becoming more familiar with the code base and have been testing changes that I've been making locally. I've been having trouble figuring out how I can test compile step code changes locally, though.

I've completed the setup as described in the contributing documentation and generally things are working well. However, I haven't managed to figure out how to test changes regarding the compile step.

Here's what I've done:

  1. Create a new project with ./scripts/run init bigquery my-project --default-database my-project --default-location EU
  2. Create credentials file with ./scripts/run init-creds bigquery my-project
  3. Add the following test.sqlx file to my-project/definitions:
config {
  type: "table",
  testProperty: "test"
}

SELECT
  CURRENT_TIMESTAMP()
  1. Make some changes to table.ts that I would expect to be reflected when compiling this file (console.logs, updating strings, etc.)
  2. Run ./scripts/run compile my-project
  3. Instead of seeing the changes that I would expect, it appears that the compile command is using code from /my-project/node_modules/@dataform/core/bundle.js (so the published version of Dataform), rather than using the code that I modified. Below is the output that I see after running the command:
Compiling...

Compiled 1 action(s).
1 dataset(s):
  dataform.test [table]

  Compilation errors:
  definitions/test.sqlx: Error: Unexpected property "testProperty" in table config. Supported properties are: ["type","disabled","protected","name","redshift","bigquery","snowflake","sqldatawarehouse","presto","tags","uniqueKey","dependencies","hermetic","schema","assertions","database","columns","description","materialized"]
    at Object.t.checkExcessProperties (/Users/matthewkocak/Documents/GitHub/dataform/my-project/node_modules/@dataform/core/bundle.js:1:209544)
    at a.config (/Users/matthewkocak/Documents/GitHub/dataform/my-project/node_modules/@dataform/core/bundle.js:1:252775)
    at t.Session.sqlxAction (/Users/matthewkocak/Documents/GitHub/dataform/my-project/node_modules/@dataform/core/bundle.js:1:383183)
    at /Users/matthewkocak/Documents/GitHub/dataform/my-project/definitions/test.sqlx:1:24
    at VM2 Wrapper.apply (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/bridge.js:485:11)
    at NodeVM.run (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/nodevm.js:497:23)
    at CustomResolver.loadJS (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/resolver.js:210:6)
    at CustomResolver.loadJS (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/resolver-compat.js:63:65)
    at /Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/resolver.js:84:14
    at VM2 Wrapper.apply (/Users/matthewkocak/Documents/GitHub/dataform/bazel-bin/packages/@dataform/cli/bin.sh.runfiles/npm/node_modules/vm2/lib/bridge.js:485:11)

I expected the error, since I added the testProperty to test.sqlx, but I expected the output text to reflect string changes that I had made in the table.ts. In the call stack, you can see that /my-project/node_modules/@dataform/core/bundle.js is being used to complete the compile step, rather than my local files.

When I run the run command using ./scripts/run run my-project, it appears that the compile step in this command still uses the published version of Dataform, but then the run/execution step of this command uses my local files. So I'm able to see the expected changes that I made locally for this step in the output.

Long post where I'm sure I'm just testing things incorrectly in my local development environment. How can I run the compile command in my local environment and have it reflect changes that I had made to the code?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions