Skip to content

[🐛 Bug]: Race condition in set_last_alert #5354

@diegovb-sys

Description

@diegovb-sys

Describe the bug
set_last_alert can lead to a race condition if an alert is received by "duplicate" in the same moment.

To Reproduce
In order to reproduce the follow conditions should be occurred:

  • The alert in the Alert table can't be present.
  • Hence, in the lastalert table there is not any entry related to that.
  • 2 alerts are send in the same moment with some distinct field to avoid the deduplication.
  • More than one worker should be up.

As 2 workers are handling the same alert, this will make a race condition, the faster alert will INSERT itself in lastalert and at the same time, the other one will try to INSERT itself but there will be already a lastAlert.

Expected behavior
The expected behavior is to have the real alert in any moment.

Screenshots
Table Alert with same fingerprint:

Image

Add to DB error:

Image Image

Impact
There are some scenarios with more impact than others, but in the worse case, we can receive a Firing alert just after the Resolve alert, in that case all of the flows that retrieves the lastAlert would show false information.

Moreover, the processes which happen after that call, won't be applied:

  • Mapping.
  • Resolve incident if it applies.
  • Enrichments.
  • Pusher notifications.
  • WF executions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions