Skip to content

Commit d151aeb

Browse files
juergenfljuergenfl
authored
Had to fix image links and added a further folder with APM content (#84)
* Create README.md * Add files via upload * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md
1 parent 8012c68 commit d151aeb

File tree

8 files changed

+328
-24
lines changed

8 files changed

+328
-24
lines changed

‎knowledge-content/opentelemetry-demo/README.md‎

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -90,24 +90,24 @@ Log in to your Oracle Cloud Infrastructure console and navigate to the Applicati
9090
### Visualising Traces and Spans
9191
Click on the "Traces" tab in the APM console / Trace Explorer. You will see a list of traces generated by the OTel demo application. Each trace represents a user request or an operation within your application:
9292

93-
![Trace-Explorer-05](https://github.com/user-attachments/assets/81e98ec3-cac9-4498-9c58-4a204e361dc0)
93+
![Trace-Explorer-05](https://github.com/user-attachments/assets/2046b3bd-fea5-4ec4-aa80-decec7e2e7ec)
9494

9595
Select a specific trace to view its details, including the topology of the trace and its spans:
9696

97-
![Trace-Explorer-04](https://github.com/user-attachments/assets/e864abb7-3db0-4491-8bde-a3a7e7141c30)
97+
![Trace-Explorer-04](https://github.com/user-attachments/assets/6b7a7c54-2ea0-4af4-a180-40c0f127e5a1)
9898

9999
Click on any of the spans to see the span details, which include kubernetes data, SpanId, TraceID, etc.:
100100

101-
![Trace-Explorer-03](https://github.com/user-attachments/assets/c6244f5c-c8ba-418b-a5dd-eac696e9cea9)
101+
![Trace-Explorer-03](https://github.com/user-attachments/assets/526b6e3f-b5b8-48d6-bcac-2baaa250f67d)
102102

103103
Services Topology view allows to visualize associations between services, several parameters can be used for the arrow width:
104104

105-
![Service-Topology-07](https://github.com/user-attachments/assets/78e97dde-f569-4e18-b9e2-ebc5c95077f5)
105+
![Service-Topology-07](https://github.com/user-attachments/assets/4b25dbe7-004f-46b1-80ad-fa436310aeec)
106106

107-
![Service-Topology-08](https://github.com/user-attachments/assets/be3f58ec-7875-4514-a759-36e8297e2488)
107+
![Service-Topology-08](https://github.com/user-attachments/assets/6e05eca8-96b8-402b-a4e2-594d1a972d70)
108108

109109
## Using OCI Logging Analytics service to collect and analyze Logs from Kubernetes Infrastructure and Pods
110-
OCI Logging Analytics provides a [complete solution](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/kubernetes-solution.html) for monitoring Kubernetes (K8s) cluster deployed in OCI, third party public clouds, private clouds, or on-premises including managed Kubernetes deployments. We will start with discovering the K8s cluster running the OpenTelemetry Demo App.
110+
OCI Logging Analytics (LA) provides a [complete solution](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/kubernetes-solution.html) for monitoring Kubernetes (K8s) cluster deployed in OCI, third party public clouds, private clouds, or on-premises including managed Kubernetes deployments. We will start with discovering the K8s cluster running the OpenTelemetry Demo App.
111111

112112
### Getting Started with monitoring a K8s Cluster using OCI Logging Analytics
113113
Log in to your Oracle Cloud Infrastructure console and navigate to Logging Analytics Administration. Select Solutions -> Kubernetes -> Connect Clusters -> Monitor Kubernetes -> Oracle OKE, here we are assuming that the K8s cluster is running in OCI. Select the Cluster, press *Next* and select the right compartment to be used for telemetry data and related monitoring resources, usually it will be the same like the one used for collecting the logs. Click on *Configure log collection*, this will create all neded dynamic groups and policies to allow collecting logs, metrics, and object information from related Kubernetes components, compute nodes, subnets, and load balancers. The deployed solution will create these statefulsets, daemonsets and cronjobs in namespace *oci-onm*:
@@ -124,28 +124,27 @@ oci-onm-mgmt-agent-0 1/1 Running 2 80d
124124
```
125125
### Screenshots from the Solution Dashboard
126126
Cluster overview:
127-
![K8s-Solution-01](https://github.com/user-attachments/assets/3d767718-60aa-4b47-b84a-93fedf5f9db4)
127+
![K8s-Solution-01](https://github.com/user-attachments/assets/95c10df5-ce67-485d-a5b8-fc6530e0f835)
128128

129129
Workload overview:
130-
![K8s-Solution-02](https://github.com/user-attachments/assets/e8bad199-f6c0-4155-8140-0758cdffab12)
130+
![K8s-Solution-02](https://github.com/user-attachments/assets/010717eb-e42a-4644-9e03-b7b8fe2b3e74)
131131

132132
Node overview:
133-
![K8s-Solution-03](https://github.com/user-attachments/assets/ce809518-70e9-4d09-8cbe-0f3ff4662172)
133+
![K8s-Solution-03](https://github.com/user-attachments/assets/4ead23d9-5d04-412b-8a24-a6e9c7d5a7cd)
134134

135135
Pod overview:
136-
![K8s-Solution-04](https://github.com/user-attachments/assets/a0d7dbe1-9cc9-4219-886e-fa8f4691a382)
136+
![K8s-Solution-04](https://github.com/user-attachments/assets/700f4a9f-d92e-483e-acc7-422165eb0006)
137137

138138
### Screenshots from Log Explorer
139139

140140
List of TOP Log Sources:
141-
![K8s-Log-Sources](https://github.com/user-attachments/assets/a4637601-d932-436a-9921-2f333f0d5450)
141+
![K8s-Log-Sources](https://github.com/user-attachments/assets/18a3d344-6b7c-43dc-99a3-647b13d97ddf)
142142

143143
Log Records by Pod name:
144-
![K8s-Log-by-Pods](https://github.com/user-attachments/assets/47934bca-3084-47e7-87e6-4df986cbe42a)
144+
![K8s-Log-by-Pods](https://github.com/user-attachments/assets/3e3499ef-06ea-4dcf-bfee-a6300015edb1)
145145

146146
Compare Cluster results:
147-
![K8s-Log-Sources_compare-cluster](https://github.com/user-attachments/assets/27d78650-85fe-4c73-9566-08e115e3ebc4)
148-
147+
![K8s-Log-Sources_compare-cluster](https://github.com/user-attachments/assets/59ab42d7-6ffb-4eaf-ba92-a1a928bbddc7)
149148

150149
## Using OCI Monitoring service to inspect the collected metrics
151150

@@ -155,17 +154,20 @@ The collected metrics can be inspected using the Metrics Explorer from the OCI M
155154

156155
Metrics collected by the Management Agent can be found in metric namespace *mgmtagent_kubernetes_metrics*:
157156

158-
![Metrics-Explorer-06](https://github.com/user-attachments/assets/9863f36e-7097-4b34-ae7e-c11454335118)
159-
157+
![Metrics-Explorer-06](https://github.com/user-attachments/assets/05192ea6-091f-4ad2-ba68-2fabc37d3bff)
160158

161159
Metrics sent by the OpenTelemetry Collector can be found in metric namespace *oracle_apm_monitoring*:
162160

163-
![Metrics-Explorer-03](https://github.com/user-attachments/assets/1aa3bea0-4bd1-4bba-81e1-eff9a2d474ea)
161+
![Metrics-Explorer-03](https://github.com/user-attachments/assets/3dfcff66-b2e5-40fc-8d06-50a8e9c311e6)
164162

165163
Finally, out-of-the-box or custom dashboards can be used to visualize the collected metrics:
166164

167-
![k8s-metrics-dashboard-01](https://github.com/user-attachments/assets/5b93d412-4974-4049-a22b-5c8d5be89fc1)
165+
![k8s-metrics-dashboard-01](https://github.com/user-attachments/assets/7d93b237-2b63-40ae-b837-e33269a94fcd)
166+
167+
![k8s-metrics-dashboard-02](https://github.com/user-attachments/assets/1c7173fa-9354-436d-9640-48b04669b235)
168168

169-
![k8s-metrics-dashboard-02](https://github.com/user-attachments/assets/8b6ff3d8-f168-4581-b68e-0baf21b5b986)
169+
## Further Logging Analytics and APM Resources
170170

171+
Further resources for Logging Analytics can be found in folders [log-sources](log-sources/README.md), [logan-lookups](logan-lookups/README.md) and [dashboards](dashboards/README.md). We are providing there an enhanced log source definition to allow queries used in a dashboard with Business Analytics charts.
171172

173+
To maximize the usage value of APM, you can find in folder [apm-configs](apm-configs/README.md) all needed info to enrich the OTel spans with attributes matching the expected APM naming schema.

‎knowledge-content/opentelemetry-demo/apm-configs/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Enriching OTel Span Attributes to maximize OCI APM usage
2+
3+
Attribues of spans generated by OTel libraries and further processed later on by the OTel Collector are following a distinct naming schema. To get the best experience from APM for OTel based spans, we will use the APM feature "Span enrichment" to adjust and add span attribute names.
4+
5+
Prereqs:
6+
- Have a terminal available on a system with OCI CLI configured, e.g. Cloud Shell
7+
- Ensure the provided three JSON files from this folder are available in the terminal/shell
8+
- Create a shell parameter APM_DOMAIN_ID containing the OCID of your APM Domain (available from APM/Administration):
9+
APM_DOMAIN_ID=ocid1.apmdomain.oc1....
10+
11+
First, we will use file "otel_spans.json" to create a "Span filter" with name "OpenTelemetry Spans". By default we have assumed that you used "otel-demo-app" as namespace for the demo shop. If this is not the case you have to adjust "otel_spans.json" to your needs. When done run the following OCI CLI command:
12+
```bash
13+
oci apm-config config create-span-filter --from-json file://otel_spans.json --apm-domain-id $APM_DOMAIN_ID
14+
```
15+
From APM Administration, select your APM Domain and switch on the left to "Span filters", you should see now the following span filter:
16+
17+
![OTel-Span-Filter](https://github.com/user-attachments/assets/7908e14d-fa99-41d2-bd19-4c7b0d02e48d)
18+
19+
Click on the three dots at the right end of the line with the filter and copy its OCID to the clipboard.
20+
21+
Edit now JSON file "otel-span-options.json" at the top and replace the placeholder with the saved OCID:
22+
```bash
23+
{
24+
"filterId": "ocid1.apmconfig.oc1........",
25+
"disabled": false,
26+
...
27+
```
28+
Finally, run the following command:
29+
```bash
30+
oci apm-config config create-options --from-json file://create-span-group.json --apm-domain-id $APM_DOMAIN_ID --options file://otel-span-options.json
31+
```
32+
In the OCI Console select now "Span enrichment" on the left, you should see the following:
33+
34+
![otel-spans-enrichment-group](https://github.com/user-attachments/assets/1f36d454-e0b0-411a-99c5-dc9882676d44)
35+
36+
We are all set :-)

‎knowledge-content/opentelemetry-demo/apm-configs/create-span-group.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"displayName": "OTel Spans Enrichment Group",
3+
"description": "Rules to modify and enhance OTel spans to be in sync with APM schema",
4+
"group": "data-manipulation.spans"
5+
}

‎knowledge-content/opentelemetry-demo/apm-configs/otel-span-options.json‎

Lines changed: 252 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,252 @@
1+
{
2+
"filterId": "<replace with OpenTelemetry Span Filter OCID>",
3+
"disabled": false,
4+
"rules": [
5+
{
6+
"disabled": false,
7+
"displayName": "Apply Oracle provided OTel rule",
8+
"systemRuleId": 500
9+
},
10+
{
11+
"actions": [
12+
{
13+
"config": {
14+
"output": "WebApplicationName",
15+
"value": "OTel Demo App"
16+
},
17+
"disabled": false,
18+
"name": "basic.Set"
19+
},
20+
{
21+
"config": {
22+
"output": "ApmrumPageUpdateType",
23+
"value": "Page Load"
24+
},
25+
"disabled": false,
26+
"name": "basic.Set"
27+
},
28+
{
29+
"config": {
30+
"output": "ApmrumType",
31+
"value": "Page"
32+
},
33+
"disabled": false,
34+
"name": "basic.Set"
35+
},
36+
{
37+
"config": {
38+
"input": "Component",
39+
"output": "OtelComponent"
40+
},
41+
"disabled": false,
42+
"name": "basic.Rename"
43+
},
44+
{
45+
"config": {
46+
"output": "Component",
47+
"value": "BROWSER"
48+
},
49+
"disabled": false,
50+
"name": "basic.Set"
51+
},
52+
{
53+
"config": {
54+
"output": "PageViews",
55+
"value": "1"
56+
},
57+
"disabled": false,
58+
"name": "basic.Set"
59+
},
60+
{
61+
"config": {
62+
"input": "PageViews",
63+
"output": "PageViews"
64+
},
65+
"disabled": false,
66+
"name": "string.ToNumber"
67+
},
68+
{
69+
"config": {
70+
"input": "SpanDuration",
71+
"output": "PageResponseTime",
72+
"pattern": "(.*)",
73+
"valueOnFail": "0"
74+
},
75+
"disabled": false,
76+
"name": "string.Extract"
77+
},
78+
{
79+
"config": {
80+
"input": "PageResponseTime",
81+
"output": "PageResponseTime"
82+
},
83+
"disabled": false,
84+
"name": "string.ToNumber"
85+
},
86+
{
87+
"config": {
88+
"input": "ProcessRuntimeVersion",
89+
"output": "UserName",
90+
"pattern": "^.*Chrome/116.*$",
91+
"replaceWith": "Synthetic_User"
92+
},
93+
"disabled": false,
94+
"name": "string.Replace"
95+
},
96+
{
97+
"config": {
98+
"input": "HttpUserAgent",
99+
"output": "UserName",
100+
"pattern": "^.*HeadlessChrome.*$",
101+
"replaceWith": "Loadgen_User"
102+
},
103+
"disabled": false,
104+
"name": "string.Replace"
105+
},
106+
{
107+
"config": {
108+
"input": "ProcessRuntimeVersion",
109+
"output": "UserName",
110+
"pattern": "^.*Mac OS X.*$",
111+
"replaceWith": "Demo_User"
112+
},
113+
"disabled": false,
114+
"name": "string.Replace"
115+
},
116+
{
117+
"config": {
118+
"input": "ProcessRuntimeVersion",
119+
"output": "UserName",
120+
"pattern": "^.*iPhone.*$",
121+
"replaceWith": "iOS_User"
122+
},
123+
"disabled": false,
124+
"name": "string.Replace"
125+
},
126+
{
127+
"config": {
128+
"input": "ProcessRuntimeVersion",
129+
"output": "UserName",
130+
"pattern": "^.*Android.*$",
131+
"replaceWith": "Android_User"
132+
},
133+
"disabled": false,
134+
"name": "string.Replace"
135+
}
136+
],
137+
"disabled": false,
138+
"displayName": "Add attributes to browser agent spans",
139+
"filterText": "ServiceName = 'frontend-web'"
140+
},
141+
{
142+
"actions": [
143+
{
144+
"config": {
145+
"input": "HttpUrl",
146+
"output": "HttpUrl",
147+
"pattern": "/_next/static.*$",
148+
"replaceWith": "/_next/static"
149+
},
150+
"disabled": false,
151+
"name": "string.Replace"
152+
},
153+
{
154+
"config": {
155+
"input": "HttpUrl",
156+
"output": "HttpUrl",
157+
"pattern": "/api/cart.*$",
158+
"replaceWith": "/api/cart"
159+
},
160+
"disabled": false,
161+
"name": "string.Replace"
162+
},
163+
{
164+
"config": {
165+
"input": "HttpUrl",
166+
"output": "HttpUrl",
167+
"pattern": "/api/recommendations.*$",
168+
"replaceWith": "/api/recommendations"
169+
},
170+
"disabled": false,
171+
"name": "string.Replace"
172+
},
173+
{
174+
"config": {
175+
"input": "HttpUrl",
176+
"output": "HttpUrl",
177+
"pattern": "/images.*$",
178+
"replaceWith": "/images"
179+
},
180+
"disabled": false,
181+
"name": "string.Replace"
182+
},
183+
{
184+
"config": {
185+
"input": "HttpUrl",
186+
"output": "HttpUrl",
187+
"pattern": "/api/data.*$",
188+
"replaceWith": "/api/data"
189+
},
190+
"disabled": false,
191+
"name": "string.Replace"
192+
},
193+
{
194+
"config": {
195+
"input": "HttpUrl",
196+
"output": "HttpUrl",
197+
"pattern": "/api/products.*$",
198+
"replaceWith": "/api/products"
199+
},
200+
"disabled": false,
201+
"name": "string.Replace"
202+
},
203+
{
204+
"config": {
205+
"input": "HttpUrl",
206+
"output": "HttpUrl",
207+
"pattern": "/icons.*$",
208+
"replaceWith": "/icons"
209+
},
210+
"disabled": false,
211+
"name": "string.Replace"
212+
}
213+
],
214+
"disabled": false,
215+
"displayName": "Create short unique HttpUrl patterns"
216+
},
217+
{
218+
"actions": [
219+
{
220+
"config": {
221+
"input": "K8sDeploymentName",
222+
"output": "DisplayName",
223+
"pattern": "otel-demo-app-(.*$)"
224+
},
225+
"disabled": false,
226+
"name": "string.Extract"
227+
}
228+
],
229+
"disabled": false,
230+
"displayName": "Add attributes to application services spans",
231+
"filterText": "ServiceName like '%service' or ServiceName = 'imageprovider'"
232+
},
233+
{
234+
"actions": [
235+
{
236+
"config": {
237+
"input": "ProcessRuntimeVersion",
238+
"output": "UserAgent",
239+
"pattern": "(.*)"
240+
},
241+
"disabled": false,
242+
"name": "string.Extract"
243+
}
244+
],
245+
"description": "Recognize load generation scripts to capture userAgent for load data.",
246+
"disabled": false,
247+
"displayName": "Mark browser",
248+
"filterText": "ProcessRuntimeName = 'browser'"
249+
}
250+
],
251+
"type": "data-manipulation.group"
252+
}

‎knowledge-content/opentelemetry-demo/apm-configs/otel_spans.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"displayName": "OpenTelemetry Spans",
3+
"description": "All spans coming from OTel Collector",
4+
"filterText": "\"k8s.namespace.name\" = 'otel-demo-app'"
5+
}

0 commit comments

Comments
 (0)