application: stop schema_registry clients before datalake subsystems #28391
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR addresses a shutdown hang caused by datalake translators repeatedly attempting to query the schema registry during shutdown. The fix ensures the schema registry is stopped before datalake subsystems begin their shutdown sequence.
Key Changes:
- Schema registry shutdown is now invoked earlier in the shutdown sequence, before datalake services
- Schema registry lifecycle management is modified to support early shutdown while maintaining proper destruction order
src/v/redpanda/application.cc
Outdated
Comment on lines
377
to
380
| if (_schema_registry) { | ||
| stop_service(*_schema_registry, "pandaproxy::schema_registry::api"); | ||
| } | ||
|
|
There was a problem hiding this comment.
[nitpick] Adding early shutdown logic for schema_registry while keeping its destruction in the normal order creates a split lifecycle pattern that could be confusing. Consider documenting this pattern more prominently (e.g., in a comment near the _schema_registry member declaration) to prevent future developers from inadvertently breaking this ordering.
src/v/redpanda/application.cc
Outdated
Comment on lines
1364
to
1366
| // NOTE: we'll stop the schema registry out of band before some other | ||
| // subsystems, but destruct it in the "normal" order as it was | ||
| // initialized. |
There was a problem hiding this comment.
[nitpick] The comment should specify which subsystems require schema registry to be stopped first (specifically mentioning datalake subsystems) to make the reasoning clearer and help future maintainers understand the ordering constraint.
Suggested change
| // NOTE: we'll stop the schema registry out of band before some other | |
| // subsystems, but destruct it in the "normal" order as it was | |
| // initialized. | |
| // NOTE: we'll stop the schema registry out of band before certain | |
| // subsystems that depend on it, specifically datalake-related subsystems | |
| // such as data transforms and WASM. This ensures proper shutdown ordering | |
| // and avoids issues with dangling references. The schema registry is | |
| // destructed in the "normal" order as it was initialized. |
Collaborator
CI test resultstest results on build#75717
|
andrwng
force-pushed
the
schema-registry-shutdown
branch
from
93a572c to
88e7917
Compare
andrwng
changed the title
We've seen a shutdown hang that appears to be caused by a datalake translator getting stuck repeatedly trying to query the schema registry. This attempts to fix this by moving schema registry client shutdown to before datalake shutdown.
andrwng
force-pushed
the
schema-registry-shutdown
branch
from
88e7917 to
e96b3ae
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We've seen a shutdown hang that appears to be caused by a datalake translator getting stuck repeatedly trying to query the schema registry.
This attempts to fix this by moving schema registry client shutdown to before datalake shutdown.
Backports Required
Release Notes