Problem Statement

When the queue is full, envelopes are dropped when captured, never being added to the queue. Therefore, it's quite possible (especially in a server environment) that a large flood of one type of envelope (such as transactions) can prevent another more important type of envelope (such as error events) from being sent to Sentry.

Solution

Solution adopted by other SDKs from here:

• Project: Backpressure Management team-sdks#10 (comment)

• will make some kind of global Monitor with just the rate limiting feeding into the ‘bad state’ for now
• the Monitor is the central source for tracking if good state / bad state
• we can extend this with more logic later
• this bad state gets picked up in hub.start_transaction and we halve the user’s sample rate in steps (in some time intervals? TBD) till the bad state goes away
• do this with a separate 'downsampling factor' so it also applies to traces_sampler etc
• we track that the sample rate was dynamically tweaked somewhere (TBD)
• use this ^ flag to make a new client report / outcome called TBD
• recovery will be automatic once bad state goes away (monitor’s responsiblilty)

Java Implementation

• Automatically downsample transaction when system is under load (backpressure) sentry-java#3072

Original brainstorming (for posterity)

The internal BackgroundWorker class manages back-pressure for the SDK. When events, transactions, sessions, etc. are captured, the SentryClient adds them to a ConcurrentQueue within the background worker. The background worker then has a separate task that is responsible for reading from the queue and sending events to the ITransport implementation (typically that's the HttpTransport).

This is generally a good design. However, it suffers from the limitation that there's only one queue for all envelope types, non-prioritized, with a single limit set by SentryOptions.MaxQueueItems - defaulting to 30.

We need to support concurrency on the producer side, so the built-in PriortyQueue class is out. We need to be non-blocking on the consumer side, so BlockingCollection is also out. There is no built-in PriortyConcurrentQueue class. Thus, to implement prioritization, we will probably need to use multiple ConcurrentQueue instances in the background worker - perhaps in a dictionary where the key is the envelope type.

We also should have more than one option for controlling the maximum queue size (this might be separate properties, or a dictionary). Currently we just have MaxQueueItems - which is undocumented.

We should probably set higher default queue sizes for ASP.NET and ASP.NET Core. Currently, MaxQueueSize defaults to 30. We do show setting it higher in some of the sample apps, but with no explanation.

References

• Developer docs
• Telemetry Backpressure
• Heap Data Structures (and their use in Priority Queues)
• Experimenting with Greedy Sampling #3167