-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Describe the bug
set_last_alert
can lead to a race condition if an alert is received by "duplicate" in the same moment.
To Reproduce
In order to reproduce the follow conditions should be occurred:
- The alert in the Alert table can't be present.
- Hence, in the lastalert table there is not any entry related to that.
- 2 alerts are send in the same moment with some distinct field to avoid the deduplication.
- More than one worker should be up.
As 2 workers are handling the same alert, this will make a race condition, the faster alert will INSERT itself in lastalert and at the same time, the other one will try to INSERT itself but there will be already a lastAlert.
Expected behavior
The expected behavior is to have the real alert in any moment.
Screenshots
Table Alert with same fingerprint:

Add to DB error:


Impact
There are some scenarios with more impact than others, but in the worse case, we can receive a Firing alert just after the Resolve alert, in that case all of the flows that retrieves the lastAlert would show false information.
Moreover, the processes which happen after that call, won't be applied:
- Mapping.
- Resolve incident if it applies.
- Enrichments.
- Pusher notifications.
- WF executions.