feat(incremental): optimize 'insert_overwrite' strategy (#1409)

[AxelThevenot]

resolves dbt-labs/dbt-adapters#527
docs "N/A"

Problem

The MERGE statement is sub-optimized in BigQuery when it comes to only replace partitions in the 'insert_overwrite' strategy for incremental models

Solution

For the insert_overwrite strategy where we are looking to replace rows at the partition-level, there is a better solution and here is why:

• a DELETE or INSERT statement is cheapest than a MERGE statement.
• incremental tables are the most expensive tables in real-world projects.
• The DELETE statement in BigQuery is free at the partition-level.

This has been tested at Carrefour which is my company.

• On this replacement of the MERGE statement it reduces the cost by 50.4% and the elapsed time by 35.2% (slot based and not on demand)
• On the overall procedure it reduces the cost by 26.1% and the elapsed time by 23.1%

This is wrapped in a transaction to avoid deleting rows if any error occurs.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR --> (using the same existing tests for 'insert_overwrite')
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[joshuao11]

We had a problem with transactions, where other jobs can conflict with it.

For example, if this transaction statement is running and another (normal) statement runs on it, the transaction statement one fails:

https://cloud.google.com/bigquery/docs/transactions#transaction_concurrency

This is different as non transaction queries can run concurrently.

At my company it's relatively common to delete things as part of GDPR, or update late arriving columns in posthooks

I'm not saying this reduction in slot time is not worth this cost of conflicting jobs, but just want to point it out as a past learning! And if there is a non transaction version of this logic, that would swerve the transaction concurrency issue

[joshuao11]

We also had a problem where we tried a separate DELETE + INSERT without a transaction, and jobs ran in between with no data (especially when the DELETE + INSERT was catching up with the context date in airflow)

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: axel_thevenot.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: axel_thevenot.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails