From 1fbebeb6bc6f157529a92773d1ffee56a0c23c28 Mon Sep 17 00:00:00 2001 From: Adam Gardner Date: Wed, 25 Sep 2024 14:04:16 +1000 Subject: [PATCH] Deployed d7248be with MkDocs version: 1.6.0 --- getting-started/index.html | 2 +- introduce-change/index.html | 23 ++++++++--------------- review-problem/index.html | 4 ++-- search/search_index.json | 2 +- view-observability-data/index.html | 8 +++++--- 5 files changed, 17 insertions(+), 22 deletions(-) diff --git a/getting-started/index.html b/getting-started/index.html index 2623306..a9baabc 100755 --- a/getting-started/index.html +++ b/getting-started/index.html @@ -627,7 +627,7 @@

Create API TokenAPI Token Permissions Explained#

Start Demo#

diff --git a/introduce-change/index.html b/introduce-change/index.html index 1b7d850..ce4f83e 100755 --- a/introduce-change/index.html +++ b/introduce-change/index.html @@ -574,30 +574,23 @@

6. Introduce System Change

This simulates releasing new functionality to your users in production.

Inform Dynatrace#

First, inform Dynatrace that a change is about to occur. -Namely, you are going to change the cartServiceFailure feature flag from off to on.

+Namely, you are going to make a change to the my-otel-demo-cartservice service +by changing the cartServiceFailure feature flag from off to on.

Tell Dynatrace about the upcoming change by sending an event (note: This event does not actually make the change; you need to do this).

-

A wrapper script to help you with this.

Run the following:

-
./runtimeChange.sh cartservice cartServiceFailure on
+
./runtimeChange.sh my-otel-demo-cartservice cartServiceFailure on
 

-
You should see output signifying an event has been successfully sent:

-
{
-  "reportCount":1,
-  "eventIngestResults":[{
-    "correlationId":"abcd1234abcd1234",
-    "status":"OK"
-  }]
-}
-

-
Refresh the cartservice page and near the bottom you should see the configuration change event.

+
Refresh the my-otel-demo-cartservice page and near the bottom you should see the configuration change event.

 
configuration changed event

 
Make Change#

-
Open this file: .devcontainer/otel-demo/flags.yaml

+
Open this file: flags.yaml

 
Change the defaultValue of cartServiceFailure from "off" to "on" (line Scroll to line 75)

 
feature flag YAML

 
Now apply the change by running this command:

-
kubectl apply -f .devcontainer/otel-demo/flags.yaml
+
kubectl apply -f $CODESPACE_VSCODE_FOLDER/flags.yaml
 

+
The application will now generate errors when emptying the users cart.
+It will do this 1/10th of the time, so be patient, it can take a few moments for the errors to occur.

 

 
    
 
  • Click Here to Continue 
    • 
diff --git a/review-problem/index.html b/review-problem/index.html
index 5fad459..d7db5b8 100755
--- a/review-problem/index.html
+++ b/review-problem/index.html
@@ -598,7 +598,7 @@ 
    7. Review Problem
    
 
    problem screen
    
 
    Explore Problem Details#
    
 
    problem details
    
-
    Opening the problem record, you will see that it has effected one entity: the cartservice.
    
+
    Opening the problem record, you will see that it has effected one entity: the my-otel-demo-cartservice.
    
 
    Navigate to the Logs panel. Click Run query next to Show x errors (your number of errors may differ from the screenshot)
    
 
    logs panel 1
    
 
    Expand the log entry and notice you have some key metadata available:
    
@@ -668,7 +668,7 @@ 
    Reveal the root cause
 
 

 
diff --git a/search/search_index.json b/search/search_index.json
index f43273f..921c6e6 100755
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Using Dynatrace to Detect Problems in Logs","text":"
In this hands-on demo, you will send logs from the OpenTelemetry demo application to Dynatrace.
 
You will induce a problem which Dynatrace DAVIS will detect and thus raise a problem report based on the Observability data.
 
The logs include span and trace IDs meaning you will be easily able to drill between signal types to see logs in the context of the distributed trace and vice versa.
"},{"location":"#how-is-the-problem-created","title":"How is the problem created?","text":"
You will release a new feature into production. For demo purposes, this new feature intentionally introduces failure into the system.
 
First you will inform Dynatrace that a change is incoming. This will be done by sending a CUSTOM_CONFIGURATION event to Dynatrace. Then the feature will be enabled by toggling a feature flag.
 
After a few moments, the error will occur. The ERROR logs flowing into Dynatrace will trigger the problem.
 
This demo uses the OpenTelemetry demo application and the Dynatrace OpenTelemetry collector distribution (why might I want to use the Dynatrace OTEL Collector?).
"},{"location":"#logical-flow","title":"Logical Flow","text":""},{"location":"#compatibility","title":"Compatibility","text":"Deployment Tutorial Compatible Dynatrace Managed \u2714\ufe0f Dynatrace SaaS \u2714\ufe0f 
     
  • Click Here to Begin 
    •  
"},{"location":"access-ui/","title":"4. Access User Interface","text":""},{"location":"access-ui/#access-the-demo-user-interface","title":"Access The Demo User Interface","text":"
Tip
 
This step is optional because there is a load generator already running. Observability data will be flowing into Dynatrace.
 
Expose the user interface on port 8080 by port-forwarding:
 
kubectl -n default port-forward svc/my-otel-demo-frontendproxy 8080:8080\n
 
Go to the Ports tab, right click the port 8080 and choose \"Open in Browser\".
 
You should see the OpenTelemetry demo application.
 
 
     
  • Click Here to Continue 
    •  
"},{"location":"cleanup/","title":"8. Cleanup","text":"
To cleanup resources, go to https://github.com/codespaces and delete the codespace.
 
You may also want to deactivate or delete the API token.
 
     
  • What's Next? 
    •  
"},{"location":"getting-started/","title":"Getting Started","text":"
You must have the following to use this hands on demo.
 
     
  • A Dynatrace environment (sign up here)
    •  
  • A Dynatrace API token (see below)
    •  
 
Save the Dynatrace environment URL without the trailing slash and without the .apps. in the URL:
 
https://abc12345.live.dynatrace.com\n
"},{"location":"getting-started/#create-api-token","title":"Create API Token","text":"
In Dynatrace:
 
     
  • Press ctrl + k. Search for access tokens.
    •  
  • Create a new access token with the following permissions:
       
    • logs.ingest
      •  
    • metrics.ingest
      •  
    • openTelemetryTrace.ingest
      •  
    • events.ingest
      •  
     
    •  
"},{"location":"getting-started/#api-token-permissions-explained","title":"API Token Permissions Explained","text":"
     
  • logs.ingest, metrics.ingest, and openTelemetryTrace.ingest are required to send OpenTelemetry data into Dynatrace
    •  
  • events.ingest is required to send the CUSTOM_CONFIGURATION event into Dynatrace
    •  
"},{"location":"getting-started/#start-demo","title":"Start Demo","text":"
Click this button to open the demo environment. This will open in a new tab.
 
 
     
  • Click Here to Continue 
    •  
"},{"location":"installation-explained/","title":"3. Installation Explained","text":"
The OpenTelemetry demo and the Dynatrace collector will be installed automatically.
 
The Dynatrace details you provided during startup will be encrypted, stored in GitHub secrets and made available as environment variables (hint: printenv to see).
 
They will also be stored in a Kubernetes secret:
 
kubectl get secret/dynatrace-otelcol-dt-api-credentials -o yaml\n
"},{"location":"installation-explained/#wait-for-system","title":"Wait for System","text":"
Wait for all pods to be Ready (can take up to 10mins)
 
kubectl wait --for condition=Ready pod --timeout=10m --all\n
 
     
  • Click Here to Continue 
    •  
"},{"location":"introduce-change/","title":"6. Introduce System Change","text":"
The application is running correctly. It is time to introduce a change into the system.
 
This simulates releasing new functionality to your users in production.
"},{"location":"introduce-change/#inform-dynatrace","title":"Inform Dynatrace","text":"
First, inform Dynatrace that a change is about to occur. Namely, you are going to change the cartServiceFailure feature flag from off to on.
 
Tell Dynatrace about the upcoming change by sending an event (note: This event does not actually make the change; you need to do this).
 
A wrapper script to help you with this.
 
Run the following:
 
./runtimeChange.sh cartservice cartServiceFailure on\n
 
You should see output signifying an event has been successfully sent:
 
{\n  \"reportCount\":1,\n  \"eventIngestResults\":[{\n    \"correlationId\":\"abcd1234abcd1234\",\n    \"status\":\"OK\"\n  }]\n}\n
 
Refresh the cartservice page and near the bottom you should see the configuration change event.
 
"},{"location":"introduce-change/#make-change","title":"Make Change","text":"
Open this file: .devcontainer/otel-demo/flags.yaml
 
Change the defaultValue of cartServiceFailure from \"off\" to \"on\" (line Scroll to line 75)
 
 
Now apply the change by running this command:
 
kubectl apply -f .devcontainer/otel-demo/flags.yaml\n
 
     
  • Click Here to Continue 
    •  
"},{"location":"resources/","title":"Resources","text":""},{"location":"resources/#why-would-i-use-the-dynatrace-otel-collector","title":"Why would I use the Dynatrace OTEL Collector?","text":"
See here.
"},{"location":"review-problem/","title":"7. Review Problem","text":"
Tip
 
Right click each image and \"Open image in new tab\" to see a larger version.
 
After a few moments, DAVIS will detect the issue and create a problem.
 
Press ctrl + k and search for Problems this will open the problems screen.
 
You should see a problem titled: Redis connection errors
 
"},{"location":"review-problem/#explore-problem-details","title":"Explore Problem Details","text":"
Opening the problem record, you will see that it has effected one entity: the cartservice.
 
Navigate to the Logs panel. Click Run query next to Show x errors (your number of errors may differ from the screenshot)
 
 
Expand the log entry and notice you have some key metadata available:
 
     
  • Timestamp of this log line
    •  
  • host.name (which equates to the container name)
    •  
  • loglevel ie. ERROR
    •  
  • OpenTelemetry span_id and trace_id
    •  
 
 
Now click Show surrounding logs this shows ALL log lines with the same trace_id.
 
You can also choose based on topology to see the error in context of all other logs on that service at the time of the error.
 
This means you can see precisely what led up to the failure. In this case:
 
     
  • The system retrieved the cart contents
    •  
  • A pricing quote was calculated
    •  
  • The system performed a currency conversion
    •  
  • The user requested to empty their cart
    •  
  • The error occured whilst trying to empty the cart
    •  
 
 
Notice that an Error status code and detailed message is also available:
 
     
  • The statuscode is FailedPrecondition
    •  
  • The detail provides an error message: Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis...
    •  
  • The detail field also provides a reference to the line of code (LOC) where this error occured.
    •  
 
"},{"location":"review-problem/#navigate-from-log-to-trace","title":"Navigate from Log to Trace","text":"
In this demo application logs, spans and traces are all correctly instrumented with the span_id and trace_id field. Meaning logs can be correlated and linked to traces.
 
Let's navigate from the log line to the trace view to get a wider view of the error and what hte user was trying to do during this action.
 
     
  • Click the value field of the trace_id. This should open the Explore context menu.
    •  
  • Select Open field with (open record with also opens the trace but \"jumps\" you down the trace to the error location)
    •  
  • Choose to open with the Distributed traces app
    •  
 
 
 
The trace view gives a deeper, more contextual view of what we've already seen from the logs.
 
The user tries to place an order, there are currency conversions and quotations occurring. Finally the EmptyCart method is called, which fails.
 
"},{"location":"review-problem/#reveal-the-root-cause","title":"Reveal the root cause","text":"
DAVIS told us (and our investigation confirmed) that the problem originated in the cartservice.
 
We know that the problem was caused by a failure to connect to Redis. We can guess that it was due to the feature flag change we made.
 
However, beyond a demo scenario, in a realistic, complex system, there are many moving parts. Let's see what Dynatrace can tell us about what was happening with (and to) the cartservice around the time of the failure.
 
Navigate back to the cartservice screen. You can do so either by:
 
     
  • Selecting a span in the trace which is attributed to cartservice. Then click the blue hyperlink to take you to the cartservice page
    •  
  • Press ctrl + k search for Services and navigate to the cartservice entity
    •  
 
Scroll to the bottom of the cartservice page until you see the Events table.
 
Notice the configuration change event which immediately preceeds the App cannot connect to redis event.
 
Change is the cause of most failures
 
Something changed on the cartservice immediately prior to an issue occuring.
 
Thanks to the \"configuration changed\" event we have all necessary information to understand the true root cause.
 
 
 
\ud83c\udf89Congratulations \ud83c\udf89
 
You have successfully completed this Observability Lab. Continue below to clean up your environment.
 
     
  • Click Here to Continue 
    •  
"},{"location":"view-observability-data/","title":"5. Validate Observability Data","text":"
In Dynatrace, press ctrl + k and search for Services. Dynatrace creates service entities based on the incoming span data. The logs are also available for each service.
 
You can also query data via notebooks and dashboards (ctrl + k and search for notebooks or dashboards).
 
During this tutorial we will be working with the cartservice.
 
 
     
  • Click Here to Continue 
    •  
"},{"location":"whats-next/","title":"10. What's Next?","text":"
Content here about what the user should do, where they should and what they could learn next.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Using Dynatrace to Detect Problems in Logs","text":"
In this hands-on demo, you will send logs from the OpenTelemetry demo application to Dynatrace.
 
You will induce a problem which Dynatrace DAVIS will detect and thus raise a problem report based on the Observability data.
 
The logs include span and trace IDs meaning you will be easily able to drill between signal types to see logs in the context of the distributed trace and vice versa.
"},{"location":"#how-is-the-problem-created","title":"How is the problem created?","text":"
You will release a new feature into production. For demo purposes, this new feature intentionally introduces failure into the system.
 
First you will inform Dynatrace that a change is incoming. This will be done by sending a CUSTOM_CONFIGURATION event to Dynatrace. Then the feature will be enabled by toggling a feature flag.
 
After a few moments, the error will occur. The ERROR logs flowing into Dynatrace will trigger the problem.
 
This demo uses the OpenTelemetry demo application and the Dynatrace OpenTelemetry collector distribution (why might I want to use the Dynatrace OTEL Collector?).
"},{"location":"#logical-flow","title":"Logical Flow","text":""},{"location":"#compatibility","title":"Compatibility","text":"Deployment Tutorial Compatible Dynatrace Managed \u2714\ufe0f Dynatrace SaaS \u2714\ufe0f 
     
  • Click Here to Begin 
    •  
"},{"location":"access-ui/","title":"4. Access User Interface","text":""},{"location":"access-ui/#access-the-demo-user-interface","title":"Access The Demo User Interface","text":"
Tip
 
This step is optional because there is a load generator already running. Observability data will be flowing into Dynatrace.
 
Expose the user interface on port 8080 by port-forwarding:
 
kubectl -n default port-forward svc/my-otel-demo-frontendproxy 8080:8080\n
 
Go to the Ports tab, right click the port 8080 and choose \"Open in Browser\".
 
You should see the OpenTelemetry demo application.
 
 
     
  • Click Here to Continue 
    •  
"},{"location":"cleanup/","title":"8. Cleanup","text":"
To cleanup resources, go to https://github.com/codespaces and delete the codespace.
 
You may also want to deactivate or delete the API token.
 
     
  • What's Next? 
    •  
"},{"location":"getting-started/","title":"Getting Started","text":"
You must have the following to use this hands on demo.
 
     
  • A Dynatrace environment (sign up here)
    •  
  • A Dynatrace API token (see below)
    •  
 
Save the Dynatrace environment URL without the trailing slash and without the .apps. in the URL:
 
https://abc12345.live.dynatrace.com\n
"},{"location":"getting-started/#create-api-token","title":"Create API Token","text":"
In Dynatrace:
 
     
  • Press ctrl + k. Search for access tokens.
    •  
  • Create a new access token with the following permissions:
       
    • logs.ingest
      •  
    • metrics.ingest
      •  
    • openTelemetryTrace.ingest
      •  
    • events.ingest
      •  
     
    •  
"},{"location":"getting-started/#api-token-permissions-explained","title":"API Token Permissions Explained","text":"
     
  • logs.ingest is required to send log entries into Dynatrace
    •  
  • events.ingest is required to send the CUSTOM_CONFIGURATION event into Dynatrace
    •  
"},{"location":"getting-started/#start-demo","title":"Start Demo","text":"
Click this button to open the demo environment. This will open in a new tab.
 
 
     
  • Click Here to Continue 
    •  
"},{"location":"installation-explained/","title":"3. Installation Explained","text":"
The OpenTelemetry demo and the Dynatrace collector will be installed automatically.
 
The Dynatrace details you provided during startup will be encrypted, stored in GitHub secrets and made available as environment variables (hint: printenv to see).
 
They will also be stored in a Kubernetes secret:
 
kubectl get secret/dynatrace-otelcol-dt-api-credentials -o yaml\n
"},{"location":"installation-explained/#wait-for-system","title":"Wait for System","text":"
Wait for all pods to be Ready (can take up to 10mins)
 
kubectl wait --for condition=Ready pod --timeout=10m --all\n
 
     
  • Click Here to Continue 
    •  
"},{"location":"introduce-change/","title":"6. Introduce System Change","text":"
The application is running correctly. It is time to introduce a change into the system.
 
This simulates releasing new functionality to your users in production.
"},{"location":"introduce-change/#inform-dynatrace","title":"Inform Dynatrace","text":"
First, inform Dynatrace that a change is about to occur. Namely, you are going to make a change to the my-otel-demo-cartservice service  by changing the cartServiceFailure feature flag from off to on.
 
Tell Dynatrace about the upcoming change by sending an event (note: This event does not actually make the change; you need to do this).
 
Run the following:
 
./runtimeChange.sh my-otel-demo-cartservice cartServiceFailure on\n
 
Refresh the my-otel-demo-cartservice page and near the bottom you should see the configuration change event.
 
"},{"location":"introduce-change/#make-change","title":"Make Change","text":"
Open this file: flags.yaml
 
Change the defaultValue of cartServiceFailure from \"off\" to \"on\" (line Scroll to line 75)
 
 
Now apply the change by running this command:
 
kubectl apply -f $CODESPACE_VSCODE_FOLDER/flags.yaml\n
 
The application will now generate errors when emptying the users cart. It will do this 1/10th of the time, so be patient, it can take a few moments for the errors to occur.
 
     
  • Click Here to Continue 
    •  
"},{"location":"resources/","title":"Resources","text":""},{"location":"resources/#why-would-i-use-the-dynatrace-otel-collector","title":"Why would I use the Dynatrace OTEL Collector?","text":"
See here.
"},{"location":"review-problem/","title":"7. Review Problem","text":"
Tip
 
Right click each image and \"Open image in new tab\" to see a larger version.
 
After a few moments, DAVIS will detect the issue and create a problem.
 
Press ctrl + k and search for Problems this will open the problems screen.
 
You should see a problem titled: Redis connection errors
 
"},{"location":"review-problem/#explore-problem-details","title":"Explore Problem Details","text":"
Opening the problem record, you will see that it has effected one entity: the my-otel-demo-cartservice.
 
Navigate to the Logs panel. Click Run query next to Show x errors (your number of errors may differ from the screenshot)
 
 
Expand the log entry and notice you have some key metadata available:
 
     
  • Timestamp of this log line
    •  
  • host.name (which equates to the container name)
    •  
  • loglevel ie. ERROR
    •  
  • OpenTelemetry span_id and trace_id
    •  
 
 
Now click Show surrounding logs this shows ALL log lines with the same trace_id.
 
You can also choose based on topology to see the error in context of all other logs on that service at the time of the error.
 
This means you can see precisely what led up to the failure. In this case:
 
     
  • The system retrieved the cart contents
    •  
  • A pricing quote was calculated
    •  
  • The system performed a currency conversion
    •  
  • The user requested to empty their cart
    •  
  • The error occured whilst trying to empty the cart
    •  
 
 
Notice that an Error status code and detailed message is also available:
 
     
  • The statuscode is FailedPrecondition
    •  
  • The detail provides an error message: Can't access cart storage. System.ApplicationException: Wasn't able to connect to redis...
    •  
  • The detail field also provides a reference to the line of code (LOC) where this error occured.
    •  
 
"},{"location":"review-problem/#navigate-from-log-to-trace","title":"Navigate from Log to Trace","text":"
In this demo application logs, spans and traces are all correctly instrumented with the span_id and trace_id field. Meaning logs can be correlated and linked to traces.
 
Let's navigate from the log line to the trace view to get a wider view of the error and what hte user was trying to do during this action.
 
     
  • Click the value field of the trace_id. This should open the Explore context menu.
    •  
  • Select Open field with (open record with also opens the trace but \"jumps\" you down the trace to the error location)
    •  
  • Choose to open with the Distributed traces app
    •  
 
 
 
The trace view gives a deeper, more contextual view of what we've already seen from the logs.
 
The user tries to place an order, there are currency conversions and quotations occurring. Finally the EmptyCart method is called, which fails.
 
"},{"location":"review-problem/#reveal-the-root-cause","title":"Reveal the root cause","text":"
DAVIS told us (and our investigation confirmed) that the problem originated in the cartservice.
 
We know that the problem was caused by a failure to connect to Redis. We can guess that it was due to the feature flag change we made.
 
However, beyond a demo scenario, in a realistic, complex system, there are many moving parts. Let's see what Dynatrace can tell us about what was happening with (and to) the cartservice around the time of the failure.
 
Navigate back to the cartservice screen. You can do so either by:
 
     
  • Selecting a span in the trace which is attributed to cartservice. Then click the blue hyperlink to take you to the cartservice page
    •  
  • Press ctrl + k search for Services and navigate to the cartservice entity
    •  
 
Scroll to the bottom of the cartservice page until you see the Events table.
 
Notice the configuration change event which immediately preceeds the App cannot connect to redis event.
 
Change is the cause of most failures
 
Something changed on the cartservice immediately prior to an issue occuring.
 
Thanks to the \"configuration changed\" event we have all necessary information to understand the true root cause.
 
 
 
\ud83c\udf89Congratulations \ud83c\udf89
 
You have successfully completed this Observability Lab. Continue below to clean up your environment.
 
     
  • Click Here to Begin 
    •  
"},{"location":"view-observability-data/","title":"5. Validate Observability Data","text":"
In Dynatrace, press ctrl + k and search for Services. Dynatrace creates service entities based on the incoming span data. The logs are also available for each service.
 
You can also query data via notebooks and dashboards (ctrl + k and search for notebooks or dashboards).
 
For example, to validate logs are available for cartservice:
 
fetch logs\n| filter service.name == \"cartservice\"\n| limit 10\n
 
     
  • Click Here to Continue 
    •  
"},{"location":"whats-next/","title":"10. What's Next?","text":"
Content here about what the user should do, where they should and what they could learn next.
"}]}
\ No newline at end of file
diff --git a/view-observability-data/index.html b/view-observability-data/index.html
index ef65dfe..bafe0c5 100755
--- a/view-observability-data/index.html
+++ b/view-observability-data/index.html
@@ -493,13 +493,15 @@
 
   
5. Validate Observability Data

 
-
services screen

 
In Dynatrace, press ctrl + k and search for Services. Dynatrace creates service entities based on the incoming span data.
 The logs are also available for each service.

 
You can also query data via notebooks
 and dashboards (ctrl + k and search for notebooks or dashboards).

-
During this tutorial we will be working with the cartservice.

-
cartservice screen

+
For example, to validate logs are available for cartservice:

+
fetch logs
+| filter service.name == "cartservice"
+| limit 10
+