draft: reduce likelihood of occurrence of pagination issue

[maxi297]

What

Mitigate #58666

This seems to be confirmed by various people here and here.

The easiest way to mitigate that would be the set a step under incrementalSync. What it would do is that instead of pagination over all the records from start_date to now, it would iterate over multiple chunk of datetime (for example if this step is yearly, 2023, 2024 and 2025 would be fetch independently and each of those would have its own 100k records limit. Another benefit is that the connector would be faster if we can increase the concurrency. Note that this does not completely solve the issue as if there are more than 100k records within this month, users will still face this issue.

There is a big drawback to this solution which might not make it viable: if a range of 1 month has more than 100k records, the other chunks of 1 month will still be synced but the next sync will start back from the earlier month that failed. It will also probably fail on each subsequent run although it'll try syncing the subsequent months every time from now on. To fix that, we need to:

• allow for partitioned state in low-code (see this). It might be a nice added feature to allow this as source-salesforce is already running on partitioned state for a while without any report of an issue.
• confirm that if we fail for a month, we continue from where we failed (let's way we hit the 100k records mark on April 15th for the range of April 1st to April 30th, then we start on April 15th and not April 1st on the subsequent run)

Possible next steps:

• Make the slice duration configurable by users to that they can have an escape path if this is still not enough. The users will still be at risk of the drawback mentioned above
• [Maybe if we confirm that the state progress for a range] Enable partitioned state
• Add a custom component that resets the pagination (similar to source-google-ads)
• [Long term preferred solution] Fix this issue using a CDK components as we've seen that this is a common pattern for a couple of APIs

How

Add a 1 month step to incremental syncs for streams visits, visitors, visitor_activities, visitor_page_views, and list_membership

Review guide

User Impact

Hopefully, we should see less of incomplete syncs that stops at 100 000k records.

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

• Developing Connectors Locally
• Managing Connector Secrets
• On-Demand Live Tests
• On-Demand Regression Tests
• #connector-ci-issues
• #connector-publish-updates
• #connector-build-statuses

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

• /format-fix - Fixes most formatting issues.
• /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
  Example: /update-connector-cdk-version connector=destination-bigquery
• /bump-version - Bumps connector versions.
  • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
  • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
• /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
• /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
• /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
• /poe source example lock - Alias for /poe connector source-example lock.
• /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
• /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

[github-actions]

source-pardot Connector Test Results

3 tests   1 ✅  3s ⏱️
1 suites  2 💤
1 files    0 ❌

Results for commit 4bff06c.