-
Notifications
You must be signed in to change notification settings - Fork 376
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: split troubleshooting page into subpages
It makes more sense now that we have multiple topics to create subpages. Signed-off-by: Mahe Tardy <[email protected]>
- Loading branch information
Showing
3 changed files
with
148 additions
and
122 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| --- | ||
| title: "Log level" | ||
| weight: 2 | ||
| description: "Learn how to configure log levels." | ||
| --- | ||
|
|
||
| When debugging, it might be useful to change the log level. The default log | ||
| level is controlled by the log-level option at startup: | ||
|
|
||
| * Enable debug level with `--log-level=debug` | ||
| * Enable trace level with `--log-level=trace` | ||
|
|
||
| ## Change log level on Kubernetes | ||
|
|
||
| {{< warning >}} | ||
| The Pods of the Tetragon DaemonSet will be restarted automatically after | ||
| changing the debug Helm value. | ||
| {{< /warning >}} | ||
|
|
||
| It is possible to change the log level of Tetragon's DaemonSet Pods by setting | ||
| `tetragon.debug` to `true`. | ||
|
|
||
| ## Change log level dynamically | ||
|
|
||
| It is possible to change the log level dynamically by using the `tetra loglevel` | ||
| sub-command. `tetra` needs access to Tetragon's gRPC server endpoint which can | ||
| be configured via `--server-address`. | ||
|
|
||
| {{< warning >}} | ||
| Keep in mind that Tetragon's gRPC server is (by default) only exposed on | ||
| `localhost`. Also, it's important to understand that this only changes the log | ||
| level of the single Tetragon instance targeted with `--server-address` and not | ||
| all Tetragon instances when it's, for example, running as DaemonSet in a | ||
| Kubernetes environment. | ||
| {{< /warning >}} | ||
|
|
||
| * Get the current log level: | ||
|
|
||
| ```shell | ||
| tetra loglevel get | ||
| ``` | ||
|
|
||
| * Dynamically change the log level. Allowed values are | ||
| `[trace|debug|info|warning|error|fatal|panic]`: | ||
|
|
||
| ```shell | ||
| tetra loglevel set debug | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| --- | ||
| title: "System dump" | ||
| weight: 1 | ||
| description: "Learn how to collect system dumps." | ||
| --- | ||
|
|
||
| Before you report a problem, make sure to retrieve the necessary information | ||
| from your cluster. | ||
|
|
||
| Tetragon's bugtool captures potentially useful information about your | ||
| environment for debugging. The tool is meant to be used for debugging a single | ||
| Tetragon agent node but can be run automatically in a cluster. Note that in the | ||
| context of Kubernetes, the command needs to be run from inside the Tetragon | ||
| Pod's container. | ||
|
|
||
| Key information collected by bugtool: | ||
| - Tetragon configuration | ||
| - Network configuration | ||
| - Kernel configuration | ||
| - eBPF maps | ||
| - Process traces (if tracing is enabled) | ||
|
|
||
| ## Automatic Kubernetes cluster sysdump | ||
|
|
||
| You can collect information in a Kubernetes cluster using the Cilium CLI: | ||
|
|
||
| ```shell | ||
| cilium-cli sysdump | ||
| ``` | ||
|
|
||
| More details can be found in the [Cilium docs](https://docs.cilium.io/en/stable/operations/troubleshooting/#automatic-log-state-collection). | ||
| The Cilium CLI `sysdump` command will automatically run `tetra bugtool` on each | ||
| nodes where Tetragon is running. | ||
|
|
||
| ## Manual single node sysdump | ||
|
|
||
| It's also possible to run the bug collection tool manually with the scope of a | ||
| single node using `tetra bugtool`. | ||
|
|
||
| ### Kubernetes installation | ||
|
|
||
| 1. Identify the Tetragon Pod (`<tetragon-namespace>` is likely to be `kube-system` | ||
| with the default install): | ||
|
|
||
| ```bash | ||
| kubectl get pods -n <tetragon-namespace> -l app.kubernetes.io/name=tetragon | ||
| ``` | ||
|
|
||
| 2. Execute tetra bugtool within the Pod: | ||
|
|
||
| ```bash | ||
| kubectl exec -n <tetragon-namespace> <tetragon-pod-name> -c tetragon -- tetra bugtool | ||
| ``` | ||
|
|
||
| 3. Retrieve the created archive from the Pod's filesystem: | ||
|
|
||
| ```bash | ||
| kubectl cp -c tetragon <tetragon-namespace>/<tetragon-pod-name>:tetragon-bugtool.tar.gz tetragon-bugtool.tar.gz | ||
| ``` | ||
|
|
||
| ### Container installation | ||
|
|
||
| 1. Enter the Tetragon Container: | ||
|
|
||
| ```bash | ||
| docker exec -it <tetragon-container-id> tetra bugtool | ||
| ``` | ||
|
|
||
| 2. Retrieve the archive using docker cp: | ||
|
|
||
| ```bash | ||
| docker cp <tetragon-container-id>:/tetragon-bugtool.tar.gz tetragon-bugtool.tar.gz | ||
| ``` | ||
|
|
||
| ### Systemd host installation | ||
|
|
||
| 1. Execute tetra bugtool with Elevated Permissions: | ||
|
|
||
| ```bash | ||
| sudo tetra bugtool | ||
| ``` | ||
|
|
||
| ## Enable debug log level | ||
|
|
||
| When debugging, it might be useful to change the log level. The default log | ||
| level is controlled by the log-level option at startup: | ||
|
|
||
| * Enable debug level with `--log-level=debug` | ||
| * Enable trace level with `--log-level=trace` | ||
|
|
||
| ### Change log level on Kubernetes | ||
|
|
||
| {{< warning >}} | ||
| The Pods of the Tetragon DaemonSet will be restarted automatically after | ||
| changing the debug Helm value. | ||
| {{< /warning >}} | ||
|
|
||
| It is possible to change the log level of Tetragon's DaemonSet Pods by setting | ||
| `tetragon.debug` to `true`. | ||
|
|