MLH-1477 add dlq replay service and api endpoint to monitor status #5566
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
webapp/src/main/java/org/apache/atlas/web/service/DLQReplayService.java
Outdated
Show resolved
Hide resolved
sriram-atlan
requested review from
aarshi0301,
hr2904,
nikhilbonte21,
sumandas0 and
suraj5077
webapp/src/main/java/org/apache/atlas/web/service/DLQReplayService.java
Outdated
Show resolved
Hide resolved
sriram-atlan
added a commit
that referenced
this pull request
Nov 14, 2025
…5566) * MLH-1477 dummy commit * MLH-1477 add branch to build * MLH-1477 update to trigger a build * MLH-1477 dummy commit * MLH-1477 add dlq replay service and api endpoint to monitor status * MLH-1477 dummy commit * MLH-1477 wire in only elastic configuration * MLH-1477 lazily connect to elasticsearch index * MLH-1477 load indexProvider like janusgraph * MLH-1477 initialise bootstrap servers * MLH-1477 increase timeout * MLH-1477 add more logs for debugging. update serializer * MLH-1477 reduce poll time * MLH-1477 use pause and resume * MLH-1477 improve DLQ handling * MLH-1477 break on errors * MLH-1477 seek back when error * MLH-1477 retry with exponential backoff * MLH-1477 add tests * MLH-1477 optimise imports * MLH-1477 remove comment on latest * MLH-1477 remove option to start the dlq manually * MLH-1477 change to non daemon thread and improve destroy and cleanup
sriram-atlan
added a commit
that referenced
this pull request
Nov 24, 2025
…stores (#5660) * MLH-1477 add dlq replay service and api endpoint to monitor status (#5566) * MLH-1477 dummy commit * MLH-1477 add branch to build * MLH-1477 update to trigger a build * MLH-1477 dummy commit * MLH-1477 add dlq replay service and api endpoint to monitor status * MLH-1477 dummy commit * MLH-1477 wire in only elastic configuration * MLH-1477 lazily connect to elasticsearch index * MLH-1477 load indexProvider like janusgraph * MLH-1477 initialise bootstrap servers * MLH-1477 increase timeout * MLH-1477 add more logs for debugging. update serializer * MLH-1477 reduce poll time * MLH-1477 use pause and resume * MLH-1477 improve DLQ handling * MLH-1477 break on errors * MLH-1477 seek back when error * MLH-1477 retry with exponential backoff * MLH-1477 add tests * MLH-1477 optimise imports * MLH-1477 remove comment on latest * MLH-1477 remove option to start the dlq manually * MLH-1477 change to non daemon thread and improve destroy and cleanup * MLH-1477 add dependency (#5619) * MLH-1477 add dependency * MLH-1477 add dependency as test * MLH-1477 refactor dlq to use repair flow to keep it idempotent (#5659) * MLH-1477 refactor dlq to use repair flow to keep it idempotent * MLH-1477 handle NPE in mutations map * MLH-1477 set kafka property in atlas startup for DLQ * MLH-1477 not use management system * MLH-1477 remove unused method. also increase poll timeout seconds * MLH-1477 remove cleanUpTransaction since it is not required * MLH-1477 remove cleanUpTransaction since it is not required * MLH-1477 handle NPQ in reindex method
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change description
Type of change
Related issues
https://atlanhq.atlassian.net/browse/MLH-1477
Helm Config Changes for Running Tests (Staging PR)
Does this PR require Helm config changes for testing?
enpla9up36. (You can proceed with the PR.) ✅Checklists
Development
Security
Code review
Note
Adds a Kafka-based DLQ replay to Elasticsearch with a status API, accompanying tests, and minor build/CI/doc updates.
DLQReplayServiceto consume Kafka DLQ (atlas.kafka.dlq.*), replay ES mutations with pause/resume, retries, exponential backoff, metrics, and health tracking.DLQAdminControllerexposingGET /dlq/replay/statusreturning replay status.DLQReplayServiceTestandDLQReplayServiceIntegrationTestcovering replay success/failure, backoff, retry limits, tracker cleanup, metrics, health, and transaction cleanup.webapp/pom.xmltomockito-coreand addmockito-junit-jupiter(Java 17 compatible).janusgraph.versionin rootpom.xmlto1.0.2-atlan(remove-SNAPSHOT).mlh-1477-staging-dlqto.github/workflows/maven.yml.LOCAL_SETUP.md: add repo/artifact download instructions and note on Colima resource tuning.Written by Cursor Bugbot for commit b10576d. This will update automatically on new commits. Configure here.