Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Misc] Log spec decode metrics #6454

Merged
merged 7 commits into from
Jul 16, 2024
Merged

Conversation

comaniac
Copy link
Collaborator

This PR adds speculative decoding metrics to engine metrics so that they can be dumped and evaluated. Two unit tests were added accordingly:

  1. In test_metrics.py, added a unit test to verify theres metrics are correctly logged.
  2. In spec decode conftest, override log interval to 0 (log per request) to log all metrics, and check the acceptance rate to be 100% when target and draft morel are the same.

cc @cadedaniel @alexm-neuralmagic

Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only trigger fastcheck CI to run, which consists only a small and essential subset of tests to quickly catch errors with the flexibility to run extra individual tests on top (you can do this by unblocking test steps in the Buildkite run).

Full CI run is still required to merge this PR so once the PR is ready to go, please make sure to run it. If you need all test signals in between PR commits, you can trigger full CI as well.

To run full CI, you can do one of these:

  • Comment /ready on the PR
  • Add ready label to the PR
  • Enable auto-merge.

🚀

@comaniac
Copy link
Collaborator Author

/ready

@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 15, 2024
Copy link
Collaborator

@cadedaniel cadedaniel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks so much for adding this

tests/spec_decode/e2e/conftest.py Outdated Show resolved Hide resolved
@comaniac
Copy link
Collaborator Author

CI blocked by #6453. Will rebase once it merged.

@comaniac comaniac force-pushed the spec_deocde_metric branch from 8b67a8b to 5680452 Compare July 16, 2024 16:41
@comaniac comaniac enabled auto-merge (squash) July 16, 2024 16:43
@comaniac comaniac merged commit 160e1d8 into vllm-project:main Jul 16, 2024
73 checks passed
@comaniac comaniac deleted the spec_deocde_metric branch July 16, 2024 21:01
dtrifiro pushed a commit to opendatahub-io/vllm that referenced this pull request Jul 17, 2024
fialhocoelho pushed a commit to opendatahub-io/vllm that referenced this pull request Jul 19, 2024
xjpang pushed a commit to xjpang/vllm that referenced this pull request Jul 24, 2024
Alvant pushed a commit to compressa-ai/vllm that referenced this pull request Oct 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants