-
Notifications
You must be signed in to change notification settings - Fork 28
🐛 Validate the cache contains the created address before moving on #338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 Validate the cache contains the created address before moving on #338
Conversation
7f4339b
to
9cf17a9
Compare
/cc @schrej |
/kind bug |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good find, just one nit.
IPAM controller relies on controller runtime client, which is cached, to list the existing address before allocating a new IP. When the cache is too slow to reflect the changes, we may end up with an IPAddressClaim reconciliation happening before the previous reconciliation address acknowledge by the cache, which may end with duplicated IP address allocations. This change enforces that the Reconcile function waits for the cached client to be aware of the IPAddress before allowing the reconciliation to move on to the next reconciliation request
9cf17a9
to
404c2e7
Compare
yeah, don't ask me how I was able to find it (spoiler alert: a very slow environment :P ) |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: schrej The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
What this PR does / why we need it:
IPAM controller relies on controller runtime client, which is cached, to list the existing address before allocating a new IP.
When the cache is too slow to reflect the changes, we may end up with an IPAddressClaim reconciliation happening before the previous reconciliation address acknowledge by the cache, which may end with duplicated IP address allocations.
This change enforces that the Reconcile function waits for the cached client to be aware of the IPAddress before allowing the reconciliation to move on to the next reconciliation request
Previous art for a similar problem: https://github.com/kubernetes-sigs/cluster-api/blob/fefd5ac02f9b1497b34ae286108560328e16deef/internal/controllers/topology/cluster/reconcile_state.go#L654
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)
format, will close the issue(s) when PR gets merged):Fixes #