-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add doc about profiling Alloy #1781
Open
ptodev
wants to merge
2
commits into
main
Choose a base branch
from
ptodev/memory-usage-docs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
--- | ||
canonical: https://grafana.com/docs/alloy/latest/troubleshoot/profile/ | ||
description: Learn how to profile resource consumption | ||
title: Profile Grafana Alloy resource consumption | ||
menuTitle: Profile resource consumption | ||
weight: 300 | ||
--- | ||
|
||
# Profile {{% param "FULL_PRODUCT_NAME" %}} resource consumption | ||
|
||
{{< param "PRODUCT_NAME" >}} is written in the Go programming language, which offers [built-in][pprof-pkg] support for profiling. | ||
Like other applications written in Go, you can profile {{< param "PRODUCT_NAME" >}} by sending an HTTP request, which returns a pprof Go profile. | ||
|
||
Once you have the pprof file, visualize it as a [flame graph][flame-graph] in [Grafana Pyroscope][pyroscope-getstarted]. | ||
Alternatively, you could visualize it on [Grafana Play][pyroscope-adhoc] or locally by using Go's built-in [pprof tool][go-pprof]. | ||
|
||
{{< admonition type="note" >}} | ||
A profile may contain sensitive information about your environment. | ||
You may not want to upload your profiles to a public location. | ||
{{< /admonition >}} | ||
|
||
The port you use to send the HTTP request is governed by Alloy's `--server.http.listen-addr` [command line argument][cmd-cli]. | ||
It is set to `127.0.0.1:12345` by default. | ||
|
||
[pprof-pkg]: https://pkg.go.dev/net/http/pprof | ||
[pyroscope-adhoc]: https://play.grafana.org/a/grafana-pyroscope-app/ad-hoc | ||
[go-pprof]: https://go.dev/blog/pprof | ||
[pyroscope-getstarted]: https://grafana.com/docs/pyroscope/latest/get-started/ | ||
[flame-graph]: https://grafana.com/docs/pyroscope/latest/view-and-analyze-profile-data/flamegraphs/ | ||
[cmd-cli]: ../../reference/cli/run | ||
|
||
## Obtaining a single profile | ||
|
||
Different types of HTTP requests will retrieve different profiles. | ||
|
||
### Memory consumption | ||
|
||
Goroutine leaks often cause memory leaks. | ||
This is why obtaining a goroutine profile is usually necessary when investigating memory issues. | ||
For example: | ||
|
||
```bash | ||
curl localhost:12345/debug/pprof/heap -o heap.pprof | ||
curl localhost:12345/debug/pprof/goroutine -o goroutine.pprof | ||
``` | ||
|
||
It is often helpful to collect profiles both when memory usage is low and when it is high. | ||
You can compare the profiles, and it may be easier to identify what caused the memory consumption to increase. | ||
|
||
### CPU consumption | ||
|
||
If you are experiencing high CPU consumption, you can collect a CPU profile: | ||
|
||
```bash | ||
curl http://localhost:12345/debug/pprof/profile?seconds=30 -o cpu.pprof | ||
``` | ||
|
||
The `?seconds=30` part of the URL above means the profiling will continue for 30 seconds. | ||
|
||
## Continuous profiling | ||
|
||
You don't have to send manual `curl` commands each time you want to collect profiles. | ||
You can also profile continuously using the [pyroscope components][] in {{< param "PRODUCT_NAME" >}}. | ||
|
||
If you have very few {{< param "PRODUCT_NAME" >}} instances, you can even configure them to profile themselves. | ||
However, if you have a large cluster of collectors, it is best to set up {{< param "PRODUCT_NAME" >}} instances whose sole job is to profile other {{< param "PRODUCT_NAME" >}} instances. | ||
|
||
The following is an example of an {{< param "PRODUCT_NAME" >}} instance profiling itself: | ||
|
||
```alloy | ||
pyroscope.scrape "default" { | ||
targets = [{"__address__" = "localhost:12345", "service_name"="alloy"}] | ||
forward_to = [pyroscope.write.default.receiver] | ||
} | ||
|
||
pyroscope.write "default" { | ||
endpoint { | ||
url = "https://profiles-prod-014.grafana.net" | ||
basic_auth { | ||
username = sys.env("PYROSCOPE_USERNAME") | ||
password = sys.env("PYROSCOPE_PASSWORD") | ||
} | ||
} | ||
} | ||
``` | ||
|
||
[pyroscope components]: ../../reference/components/pyroscope | ||
|
||
## Expected resource consumption | ||
|
||
Refer to [Estimate resource usage][res-usage] for more information about the expected resource consumption in {{< param "PRODUCT_NAME" >}}. | ||
|
||
[res-usage]: ../../introduction/estimate-resource-usage | ||
|
||
## {{% param "PRODUCT_NAME" %}} consumes an abnormally large amount of resources | ||
|
||
If {{< param "PRODUCT_NAME" >}} consumes an abnormally large amount of resources, you can open an issue in the [Alloy repository][alloy-repo]. | ||
Attach your pprof files and your {{< param "PRODUCT_NAME" >}} configuration file. | ||
Make sure you redact any secrets in the attachments. | ||
|
||
[alloy-repo]: https://github.com/grafana/alloy/issues |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missed this one in the first pass. I think we can drop the
Alloy's
and assume the reader knows we are talking about the CLI arg in Alloy here.