Add Prometheus metric support #236
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nv-braf
reviewed
Jan 7, 2025
|
|
||
| # Separate function that can raise exceptions used for testing | ||
| # to assert correct errors and messages. | ||
| def run(): | ||
| # TMA-1900: refactor CLI handler | ||
| logging.init_logging() |
There was a problem hiding this comment.
This should not have been deleted, it is needed for the logger to function properly
| @@ -46,6 +50,31 @@ def run(): | |||
| else: # profile | |||
| args.func(args, extra_args) | |||
|
|
|||
| if getattr(args, "enable_prometheus", False): | |||
| # Fix: This doesn't actually get logged. | |||
There was a problem hiding this comment.
You can fix this by adding main to logger.py
| @@ -70,6 +70,7 @@ def to_lowercase(self): | |||
| DEFAULT_SYNTHETIC_FILENAME = "synthetic_data.json" | |||
| DEFAULT_WARMUP_REQUEST_COUNT = 0 | |||
| DEFAULT_BACKEND = "tensorrtllm" | |||
| DEFAULT_PROMETHEUS_PORT = 8002 | |||
There was a problem hiding this comment.
This is the default port for the Tritonserver metrics. I would change this to be 9090 which is what prometheus uses in its getting started configuration file.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support a Prometheus metric endpoint in GenAI-Perf. This is a proof of concept. With this design, Prometheus metrics can be enabled via the CLI. If they are enabled, the metrics are exported at that endpoint. This would only be supported for the
profilesubcommand to begin with, though it may be useful to expand it toanalyze. To avoid breaking anything, that is out of scope for this proof of concept.When Prometheus metrics are enabled, the user must terminate the program when they want GenAI-Perf to exit. This design is necessary because otherwise, the metrics would be up for a very short amount of time, since the endpoint is closed when GenAI-Perf exits.
TODO: Only system metrics are displayed right now, so add the other metrics. Also, it may be good to omit request_goodput if goodput is not used.
Example command:
genai-perf profile -v -m gpt2 --service-kind openai --endpoint-type completions --num-requests 5 --num-prompts 5 --enable-prometheus --prometheus-port 8002Example metrics:
