Skip to content

Commit d44bc2a

Browse files
Add missing spans section (#1100)
* Add missing spans section * Add navTitles to collector and protocol pages while I'm here * decrease image size * improve page navigation structure * remove ordering, since it won't matter * remove unneeded "your" in titles * improve more headers 😅 * Apply suggestions from Susa's review Co-authored-by: Susa Tünker <[email protected]> * improve formatting in headers, and steps in each troubleshoot option * fix indentation --------- Co-authored-by: Susa Tünker <[email protected]>
1 parent db5f4fc commit d44bc2a

File tree

4 files changed

+118
-0
lines changed

4 files changed

+118
-0
lines changed

site/content/docs/traces-open-telemetry/importing-traces/http-vs-grpc-protocols.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
title: HTTP vs. gRPC OpenTelemetry protocols
33
head:
44
title: "Choosing between HTTP and gRPC OpenTelemetry protocols"
5+
navTitle: HTTP vs gRPC OpenTelemetry protocols
56
metatags:
67
title: "Choosing between HTTP and gRPC OpenTelemetry protocols"
78
description: "Learn about the trade-offs between using HTTP and gRPC for sending data to an OpenTelemetry backend."

site/content/docs/traces-open-telemetry/importing-traces/sending-traces-otel-collector.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ title: Setting up the OpenTelemetry collector
44
metatags:
55
title: "Setting up the OpenTelemetry collector"
66
description: "Set up the OpenTelemetry collector and send traces to Checkly."
7+
navTitle: OpenTelemetry Collector configuration
78
weight: 20
89
menu:
910
platform:
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
---
2+
title: Troubleshooting missing or orphan spans
3+
weight: 20
4+
menu:
5+
platform:
6+
parent: "Send your traces to Checkly"
7+
identifier: troubleshooting-missing-spans
8+
navTitle: Missing spans
9+
beta: true
10+
---
11+
12+
This guide will walk you through the process of identifying missing spans, understanding their root causes, and implementing solutions to ensure your OpenTelemetry setup with Checkly provides complete and accurate tracing information.
13+
14+
## Missing spans
15+
16+
Missing spans, also known as orphan spans, are segments of a trace that are disconnected from either their parent spans or from the root span.
17+
18+
An instrumented distributed system will emit spans, creating a hierarchical chain that follows the flow of the request through the different components and services.
19+
20+
When a span is missing, they create a hole in this chain, making it difficult to diagnose issues and understand the full picture. If this happens, you will notice a span missing from the visualization in Checkly.
21+
22+
Checkly uses contextual span data—such as the trace it belongs to and its parent span—to display any missing pieces in the span chain. This helps you identify which application or service may be incorrectly instrumented.
23+
24+
![Trace with missing spans](/docs/images/otel/traces-missing-spans.png)
25+
26+
27+
## Troubleshooting missing spans
28+
29+
Missing spans can happen due to various reasons. Whether you're sending data to Checkly directly from your applications and services, or through the OpenTelemetry Collector, these are steps to confirm everything is properly set up.
30+
31+
### Verify sampling strategy
32+
33+
Using Checkly Traces, we recommend sampling for the tracestate header `checkly=true`, which will reduce your Egress/Ingress costs and ensure that all the spans that were originated through a check are there.
34+
35+
#### Sampling in the OpenTelemetry Collector
36+
37+
* **Step 1**. Make sure that you're filtering out non-checkly spans:
38+
39+
```yaml
40+
processors:
41+
batch:
42+
filter/checkly:
43+
error_mode: ignore
44+
traces:
45+
span:
46+
# filter out spans in which tracestate is not "checkly=true"
47+
- 'trace_state["checkly"] != "true"'
48+
```
49+
* **Step 2**. Confirm that your Collector pipeline uses this processor:
50+
51+
```yaml
52+
service:
53+
pipelines:
54+
traces:
55+
processors: [filter/checkly, batch]
56+
```
57+
58+
#### Sampling directly in your applications' code.
59+
Choose the specific [instrumentation language guide](docs/traces-open-telemetry/instrumenting-code/) and review your configuration against Step 2 in the guides.
60+
61+
### Review OpenTelemetry Exporter configuration
62+
63+
Ensure the right Authorization keys and endpoints are in use.
64+
65+
#### Exporting through the OpenTelemetry Collector
66+
67+
* **Step 1**. Make sure that your exporter uses the right endpoint authorization:
68+
69+
```yaml
70+
exporters:
71+
otlp/checkly:
72+
endpoint: "otel.eu-west-1.checklyhq.com:4317"
73+
headers:
74+
authorization: "${env:CHECKLY_OTEL_API_KEY}"
75+
```
76+
* **Step 2**. Confirm that your Collector pipeline uses this exporter:
77+
78+
```yaml
79+
service:
80+
pipelines:
81+
traces:
82+
exporters: [otlp/checkly]
83+
```
84+
85+
#### Exporting directly from your Application's code
86+
87+
Confirm your environment variables use the correct endpoint and authorization:
88+
89+
```bash
90+
OTEL_EXPORTER_OTLP_HEADERS="authorization=<your-api-key>"
91+
92+
OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.eu-west-1.checklyhq.com"
93+
```
94+
95+
### Checkly Private Location setup
96+
When setting up Traces with a [Checkly Private Location](docs/private-locations/#configuring-a-private-location):
97+
* **Step 1**. Ensure your [Checkly Agent](https://hub.docker.com/r/checkly/agent) version is at least 3.3.5.
98+
99+
* **Step 2**. Review your internal Firewall rules:
100+
* Identify the ports used by OpenTelemetry, commonly:
101+
* Port 4317 for gRPC protocol
102+
* Port 4318 for HTTP protocol
103+
104+
* Access your firewall settings and add rules to allow outgoing traffic on the identified port.
105+
106+
For example, to allow HTTP traces to be sent out:
107+
108+
```bash
109+
A OUTPUT -p tcp --dport 4318 -j ACCEPT
110+
```
111+
112+
113+
### Incomplete instrumentation
114+
115+
To see the full picture, and identify the root cause of a problem faster, ensure your applications and services are [instrumented with the OpenTelemetry SDK](docs/traces-open-telemetry/instrumenting-code/).
116+
147 KB
Loading

0 commit comments

Comments
 (0)