💣 Infrequent lag and stalling

[phaistonian]

Problem

When making requests sequentially, occasionally, the average response time (on staging) goes from < 1s to up to 220m.

This affects response times and more.

Solution

• Check if one of the nodes running KMS is acting up
• Check if the analytics update is causing the lag, and if so, remove them (with an option or not)

Context

CleanShot.2024-12-04.at.20.30.37.mp4

[phaistonian]

It's easy to reproduce and has been happening all night, so it's not stress-related.

[phaistonian]

Tried it all servers directly:

10.5.5.6:1338   
10.5.5.98:1338  
10.5.5.7:1338   
10.5.5.23:1338  
10.5.5.3:1338    <<<<<< keepalive
10.5.5.91:1338  <<<<<< keepalive

The issue seems evident in all of them - especially the first one (Jinx).

Let's investigate more, but perhaps, providing a way to not store analytics is, indeed, what we should do.

[phaistonian]

On staging.

[phaistonian]

[phaistonian]

Could be related to what @gabrieltz discovered.

[phaistonian]

[phaistonian]