Best practices of Multi-Destination Span Export in OpenTelemetry Java

[anoopknayak]

Hello OpenTelemetry community! I have a question about best practices for exporting spans to multiple destinations. What's the recommended pattern among these options?

Some constraints/considerations about the usecase:

• We are looking for minimal impact on the service in terms of cpu and memory. We don't want to slow down the application drastically. Just enabling tracing even with sampling on has an effect on the service and hence would like to not increase that.
• Minor data loss is acceptable if we are in a state where we will have to allocate more resources for the application.
• This usecase is for a large scale application and allocating 10 MB of RAM could translate to 3 GB of RAM to the overall application (multiplying by 3K)

Option A: Multiple SpanProcessors with Individual Exporters

Implementation: Each destination gets its own SpanProcessor wrapping a dedicated exporter.

Pros

• Isolation: When using BatchSpanProcessor, slow exporters don't impact others since each has independent processing
• Clear separation of concerns: Each export destination is handled independently
• Failure isolation: Issues with one exporter don't affect others

Cons

• Higher memory usage: Multiple queues when using BatchSpanProcessor
• Resource overhead: More threads and processing overhead

Option B: Single SpanProcessor with Custom Multi-Exporter Logic

Implementation: One SpanProcessor that internally loops through multiple exporters.

Pros

• Memory efficient: Single queue with BatchSpanProcessor
• Centralized export logic: Easier to manage export behavior

Cons

• Sequential blocking: Slow exporters block subsequent ones
• Custom implementation required: Need to build multi-exporter logic
• Potential main thread impact: May increase processing time on export thread

Option C: Single SpanProcessor with Composite Exporter

Implementation: Use a composite/multi-exporter that wraps multiple exporters (similar to how MultiSpanProcessor works).

Pros

• Memory efficient: Single queue like Option B
• Reusable pattern: Can leverage existing BatchSpanProcessor
• Clean abstraction: Hides complexity behind standard interface

Cons

• Sequential execution: Exporters run in sequence, blocking on failures
• Error handling complexity: Need strategy for partial failures
• Implementation dependency: Behavior depends on composite exporter

[jkwatson]

Do you have the option of sending the data to a collector, and having the collector do the fan-out?

[anoopknayak]

I don't have the option to use the collector unfortunately.