Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add doc about profiling Alloy #1781

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions docs/sources/troubleshoot/profile.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
---
canonical: https://grafana.com/docs/alloy/latest/troubleshoot/profile/
description: Learn how to profile resource consumption
title: Profile Grafana Alloy resource consumption
menuTitle: Profile resource consumption
weight: 300
---

# Profile {{% param "FULL_PRODUCT_NAME" %}} resource consumption

{{< param "PRODUCT_NAME" >}} is written in the Go programming language, which offers [built-in][pprof-pkg] support for profiling.
Like other applications written in Go, you can profile {{< param "PRODUCT_NAME" >}} by sending an HTTP request, which returns a pprof Go profile.

Once you have the pprof file, visualize it as a [flame graph][flame-graph] in [Grafana Pyroscope][pyroscope-getstarted].
Alternatively, you could visualize it on [Grafana Play][pyroscope-adhoc] or locally by using Go's built-in [pprof tool][go-pprof].

{{< admonition type="note" >}}
A profile may contain sensitive information about your environment.
You may not want to upload your profiles to a public location.
{{< /admonition >}}

The port you use to send the HTTP request is governed by Alloy's `--server.http.listen-addr` [command line argument][cmd-cli].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The port you use to send the HTTP request is governed by Alloy's `--server.http.listen-addr` [command line argument][cmd-cli].
The port you use to send the HTTP request is controlled by the `--server.http.listen-addr` [command line argument][cmd-cli].

Missed this one in the first pass. I think we can drop the Alloy's and assume the reader knows we are talking about the CLI arg in Alloy here.

It is set to `127.0.0.1:12345` by default.

[pprof-pkg]: https://pkg.go.dev/net/http/pprof
[pyroscope-adhoc]: https://play.grafana.org/a/grafana-pyroscope-app/ad-hoc
[go-pprof]: https://go.dev/blog/pprof
[pyroscope-getstarted]: https://grafana.com/docs/pyroscope/latest/get-started/
[flame-graph]: https://grafana.com/docs/pyroscope/latest/view-and-analyze-profile-data/flamegraphs/
[cmd-cli]: ../../reference/cli/run

## Obtaining a single profile

Different types of HTTP requests will retrieve different profiles.

### Memory consumption

Goroutine leaks often cause memory leaks.
This is why obtaining a goroutine profile is usually necessary when investigating memory issues.
For example:

```bash
curl localhost:12345/debug/pprof/heap -o heap.pprof
curl localhost:12345/debug/pprof/goroutine -o goroutine.pprof
```

It is often helpful to collect profiles both when memory usage is low and when it is high.
You can compare the profiles, and it may be easier to identify what caused the memory consumption to increase.

### CPU consumption

If you are experiencing high CPU consumption, you can collect a CPU profile:

```bash
curl http://localhost:12345/debug/pprof/profile?seconds=30 -o cpu.pprof
```

The `?seconds=30` part of the URL above means the profiling will continue for 30 seconds.

## Continuous profiling

You don't have to send manual `curl` commands each time you want to collect profiles.
You can also profile continuously using the [pyroscope components][] in {{< param "PRODUCT_NAME" >}}.

If you have very few {{< param "PRODUCT_NAME" >}} instances, you can even configure them to profile themselves.
However, if you have a large cluster of collectors, it is best to set up {{< param "PRODUCT_NAME" >}} instances whose sole job is to profile other {{< param "PRODUCT_NAME" >}} instances.

The following is an example of an {{< param "PRODUCT_NAME" >}} instance profiling itself:

```alloy
pyroscope.scrape "default" {
targets = [{"__address__" = "localhost:12345", "service_name"="alloy"}]
forward_to = [pyroscope.write.default.receiver]
}

pyroscope.write "default" {
endpoint {
url = "https://profiles-prod-014.grafana.net"
basic_auth {
username = sys.env("PYROSCOPE_USERNAME")
password = sys.env("PYROSCOPE_PASSWORD")
}
}
}
```

[pyroscope components]: ../../reference/components/pyroscope

## Expected resource consumption

Refer to [Estimate resource usage][res-usage] for more information about the expected resource consumption in {{< param "PRODUCT_NAME" >}}.

[res-usage]: ../../introduction/estimate-resource-usage

## {{% param "PRODUCT_NAME" %}} consumes an abnormally large amount of resources

If {{< param "PRODUCT_NAME" >}} consumes an abnormally large amount of resources, you can open an issue in the [Alloy repository][alloy-repo].
Attach your pprof files and your {{< param "PRODUCT_NAME" >}} configuration file.
Make sure you redact any secrets in the attachments.

[alloy-repo]: https://github.com/grafana/alloy/issues