PowerBI connector not removing stale metadata

[Useserall]

Describe the bug
DataHub PowerBI ingestion is not removing stale metadata for deleted PowerBI assets. Even after assets are deleted from PowerBI, they continue to appear in DataHub with outdated sync timestamps, while only refreshed assets show recent sync dates.

To Reproduce

• Delete assets from PowerBI workspace
• Run DataHub PowerBI ingestion
• Navigate to DataHub
• Search for the deleted PowerBI assets
• Observe that deleted assets still appear with old sync timestamps (e.g., "synced last month")

Expected behavior
Deleted PowerBI assets should be automatically removed from DataHub during ingestion runs, as remove_stale_metadata is true per default. Only active PowerBI assets should remain in the catalog with current sync timestamps.

Environment
DataHub Version: v1.2.0.1

Workaround Applied
As a temporary workaround, we soft-deleted all assets and started a new ingestion run.

Additional context
According to DataHub documentation, remove_stale_metadata should default to true for PowerBI connector.

[deepgarg760]

@Useserall , please share recipe file being used.
remove_stale_metadata works if stateful_ingestion enabled is enabled. Please check if stateful_ingestion enabled and rerun the ingestion.

[Useserall]

Hi @deepgarg760,

Thank you for your reply. Here is the recipe in short form:

source:
  type: "powerbi"
  config:
    tenant_id: "${TENANT_ID}"
    client_id: "${CLIENT_ID}"
    client_secret: "${CLIENT_SECRET}"

    # Optional mapping of PowerBI datasource servers to platform instances
    server_to_platform_instance:
      example.snowflakecomputing.com:
        platform_instance: example_instance

    admin_apis_only: true

    workspace_id_pattern:
      allow:
        - <workspace-id-1>
        - <workspace-id-2>
        - <workspace-id-3>
        # ... add more as needed

    extract_datasets_to_containers: true
    m_query_parse_timeout: 200
    extract_dashboards: true
    extract_dataset_schema: true
    extract_lineage: true
    extract_column_level_lineage: true
    native_query_parsing: true
    extract_ownership: true
    extract_app: true

    stateful_ingestion:
      enabled: true

pipeline_name: powerbi_snowflake_pipeline

transformers:
  - type: simple_add_dataset_ownership
    config:
      owner_urns:
        - "urn:li:corpGroup:<group-id>"
  - type: simple_add_dataset_domain
    config:
      semantics: PATCH
      domains:
        - urn:li:domain:<domain-id>

sink:
  type: "datahub-rest"
  config:
    server: "http://datahub-gms-service:8080"
    token: ${DATAHUB_GMS_TOKEN}
    retry_max_times: 5
    retry_status_codes:
      - 429
      - 500
      - 502
      - 503

Stateful ingestion is enabled and we do not configure remove_stale_metadata directly as it should be the default.

[deepgarg760]

@skrydal , can you please take a look at this

[deepgarg760]

@Useserall , are you still facing the issue ?
Can you try to reproduce it again by adding data to PowerBI, run ingestion with current config, see it appear, then remove it from the source ,run ingestion again - confirm the new entity is deleted