-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Description
dbt freshness test results are successfully ingested into DataHub and stored in the schemaMetadata.lastModified field, but they are not visible in the DataHub UI. This is due to a missing implementation in the GraphQL schema and UI layers, even though the backend data model contains the necessary field.
The ingestion pipeline correctly maps dbt.max_loaded_at values to SchemaMetadata.lastModified, but this information is never exposed to the frontend because:
- The GraphQL schema (
entity.graphql) does not include a lastModified field in the SchemaMetadata type - The GraphQL mapper (
SchemaMetadataMapper.java) does not map this field - The UI cannot access data that isn't exposed through GraphQL
To Reproduce
Steps to reproduce the behavior:
- Configure a dbt project with freshness tests on source tables (with loaded_at_field and freshness rules defined)
- Run dbt source freshness to generate freshness test results
- Ingest dbt metadata into DataHub using the dbt ingestion source
- Navigate to a dataset in the DataHub UI that has freshness test results
- Check the Schema tab, Properties, and Validations tab
- Observe that the
max_loaded_at timestampfrom dbt freshness tests is not displayed anywhere in the UI
Expected behavior
dbt freshness test results should be visible in the DataHub UI. Expected implementations could include:
- Display in Schema tab: Show the last modified timestamp (from max_loaded_at) in the Schema tab header or metadata section
Affected Version: v1.2.0
Impact
Users running dbt freshness tests lose visibility into their data freshness monitoring when viewing datasets in DataHub, despite this information being successfully ingested and stored in the backend.
Issue reported on Community Slack: https://datahub.slack.com/archives/CUMUWQU66/p1762404839124209