How is Airbyte syncing data from source to destination when triggering sync first time ?

[vikram-bais-1]

How is Airbyte syncing data from source to destination when triggering sync first time ?

Does it first load the complete data into its internal database and then sync it to the destination, or does it read data in batches from the source and sync it to the destination in parallel while reading from the source?

In my case (PostgreSQL to PostgreSQL), Airbyte is first loading the complete data into its internal database, and only after that, it starts syncing the complete data to the destination.

How is it working? Below are the configurations I have used:

The table I am syncing is around 4 GB (12,489,659 records), and it took more than 6 hours to complete the sync.

source

{
  "name": "-",
  "workspaceId": "-",
  "definitionId": "-",
  "configuration": {
    "port": 5432,
    "host": "-",
    "schemas": [
      "-"
    ],
    "database": "-",
    "password": "***",
    "ssl_mode": {
      "mode": "disable"
    },
    "username": "-",
    "tunnel_method": {
      "tunnel_method": "NO_TUNNEL"
    },
    "replication_method": {
      "method": "CDC",
      "plugin": "pgoutput",
      "queue_size": 1000,
      "publication": "-",
      "replication_slot": "-",
      "lsn_commit_behaviour": "After loading Data in the destination",
      "initial_waiting_seconds": 120,
      "initial_load_timeout_hours": 8,
      "invalid_cdc_cursor_position_behavior": "Fail sync"
    }
  }
}

destination

{
  "name": "-",
  "workspaceId": "-",
  "definitionId": "-",
  "configuration": {
    "port": 5432,
    "schema": "-",
    "ssl": false,
    "host": "-",
    "database": "postgres",
    "password": "***",
    "ssl_mode": {
      "mode": "disable"
    },
    "username": "-",
    "drop_cascade": false,
    "tunnel_method": {
      "tunnel_method": "NO_TUNNEL"
    },
    "disable_type_dedupe": false,
    "unconstrained_number": false
  }
}

connection

sync mode - Incremental \ Append