🔍 Question: How to Reduce Latency Overhead of OpenTelemetry C++ SDK in High-Performance Applications?

[abhinavkumar9962]

Hi team,

I’m planning to use the OpenTelemetry C++ SDK in a high-performance, low-latency application to export metrics.
However, during my POC, I’ve observed non-trivial latency impact when instrumenting even simple counters.

📌 Observations

Shared Metric Objects Across Threads → Latency Increases
My application spawns multiple threads, and when they all use the same Counter<uint64_t> instance, the OpenTelemetry SDK seems to take a spinlock internally, causing contention and latency spikes.

One Metric Object Per Thread → Lower Latency but High Memory Consumption
I attempted creating one counter object per thread to eliminate lock contention.
This helps reduce latency, but each metric object is around ~6 KB, and with many threads this leads to significant memory usage, which is not acceptable for our deployment environment.

❓ Questions

• Is there a recommended pattern for low-latency metric recording using the C++ SDK?
• Are there ways to reduce the memory footprint when creating metric instruments per thread?
• Does the SDK provide (or plan to provide) a lock-free or sharded counter implementation optimized for high-QPS workloads?

📎 Additional Context

My application is extremely latency-sensitive due to the nature of the business logic, and even microsecond-level delays matter. I want to ensure that enabling metrics does not add measurable overhead.

Any insights, recommendations, or best practices from the maintainers or community would be greatly appreciated!