Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Misc] Improve simulator&&api_server performance #6

Merged
merged 3 commits into from
Jul 25, 2024
Merged

[Misc] Improve simulator&&api_server performance #6

merged 3 commits into from
Jul 25, 2024

Conversation

ZeldaHuang
Copy link
Contributor

  1. Fix typo in benchmark_serving.py.
  2. Improve api server throughput.
  3. Record inference latency in executor to improve simulator accuracy.
  4. Fix TP hang by changing worker max_concurrency to 2.
  5. Use real profiling latency data by default(rather than estimated data) to improve simulator accuracy.

@CLAassistant
Copy link

CLAassistant commented Jul 25, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

zhypku
zhypku approved these changes Jul 25, 2024
View reviewed changes
Copy link
Collaborator

@zhypku zhypku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@s5u13b s5u13b self-requested a review July 25, 2024 08:09
@s5u13b
Copy link
Contributor

s5u13b commented Jul 25, 2024

LGTM

@ZeldaHuang ZeldaHuang merged commit fa393d4 into main Jul 25, 2024
3 of 4 checks passed
@ZeldaHuang ZeldaHuang deleted the misc-llumnix branch August 19, 2024 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@s5u13b s5u13b s5u13b approved these changes

@zhypku zhypku zhypku approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ZeldaHuang @CLAassistant @s5u13b @zhypku