KubernetesPodOperator PodManager retries during create pod on too many requests error #58033
+135
−61
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AutomationDev85
requested review from
hussein-awala and
jedcunningham
as code owners
boring-cyborg
bot
added
area:providers
provider:cncf-kubernetes
jscheffl
approved these changes
Nov 7, 2025
Contributor
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
Looks good in general but have a more generic question which is above the scope of a simple fix... probably. But which might be a good further improvement.
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/pod_manager.py
Outdated
Show resolved
Hide resolved
AutomationDev85
force-pushed
the
feature/pod-manager-retry-during-pod-creation
branch
2 times, most recently
from
227ad0f to
2114ae7
Compare
Contributor
Author
|
@jscheffl I switched back to using Tenacity for retries, since retryhttp only handles HTTP exceptions, while the Kubernetes client library can raise various exception types. |
jscheffl
reviewed
Nov 14, 2025
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/utils/test_pod_manager.py
Outdated
Show resolved
Hide resolved
AutomationDev85
force-pushed
the
feature/pod-manager-retry-during-pod-creation
branch
from
ea19e34 to
c22e36a
Compare
aaron-wolmutt
pushed a commit
to aaron-wolmutt/airflow
that referenced
this pull request
Nov 20, 2025
…y requests error (apache#58033) * Retry create pod also on too many requests issue * Fix unit test * fix static checks --------- Co-authored-by: AutomationDev85 <AutomationDev85>
This was referenced Nov 27, 2025
Copilot AI
pushed a commit
to jason810496/airflow
that referenced
this pull request
Dec 5, 2025
…y requests error (apache#58033) * Retry create pod also on too many requests issue * Fix unit test * fix static checks --------- Co-authored-by: AutomationDev85 <AutomationDev85>
itayweb
pushed a commit
to itayweb/airflow
that referenced
this pull request
Dec 6, 2025
…y requests error (apache#58033) * Retry create pod also on too many requests issue * Fix unit test * fix static checks --------- Co-authored-by: AutomationDev85 <AutomationDev85>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
We are sporadically encountering "Too Many Requests" (HTTP 429) errors from the Kubernetes API when scaling up nodes in our Kubernetes cluster. While most PodManager functions already implement retries for various errors, the create_pod function previously only retried on HTTP 409 (Conflict) errors.
With this change, the retry logic is extended to also handle HTTP 429 errors, improving robustness during cluster scaling operations.
We welcome your feedback on this change!
Details of change: