include _airbyte_loaded_at or similar in final tables #69144
bmulh
started this conversation in
Connector Ideas and Features
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
We often have syncs that run for an extended period. The
_airbyte_extracted_atcan be a day or two in the past by the time the data is finally insert into the destination table. This causes problems for our data pipeline if it is ran during a sync as we can miss records. We would like to know when the data landed in the final table, it'd be helpful if the _airbyte_loaded_at or a similar column was included in the final destination schema that is set to the current timestamp when the record is insert.We currently use the postgres destination but this could be helpful across other database destinations.
edit: What we really need is the concept of "originally loaded at" since when the final table is rebuilt, we want to persist the original loaded at concept. Essentially the same concept of the _airbyte_loaded_at in the raw table
Beta Was this translation helpful? Give feedback.
All reactions