Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep #9994

Merged
merged 1 commit into from
Nov 5, 2024
Merged

[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep #9994

merged 1 commit into from
Nov 5, 2024

Conversation

tlrmchlsmth
Copy link
Collaborator

While profiling #9856, I found that the calls to time.sleep in shm_broadcast.py were never taking under 50 microseconds, even if RINGBUFFER_SLEEP_INTERVAL is set to 0.

This PR switches to use os.sched_yield() instead.

Using this gist for benchmarking code, os.sched_yield() gives us close to what we wanted, at least on my system.

Benchmarking sleep(1.000000e-07) for 10000 iterations...

Results:
Requested sleep duration: 1.000000e-07s
Mean actual duration:     5.247413e-05s
Std dev:                  2.300718e-06s
Min duration:             8.199364e-06s
Max duration:             6.059092e-05s

Benchmarking os.sched_yield() for 10000 iterations...

Results:
Requested sleep duration: 1.000000e-07s
Mean actual duration:     3.195110e-07s
Std dev:                  1.182954e-07s
Min duration:             2.998859e-07s
Max duration:             1.141988e-05s

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 4, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@tlrmchlsmth tlrmchlsmth force-pushed the tms/shm_broadcast_yield branch from 6863795 to 02f6c96 Compare November 4, 2024 15:35
mgoin
mgoin approved these changes Nov 4, 2024
View reviewed changes
Copy link
Member

@mgoin mgoin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find

youkaichao
youkaichao approved these changes Nov 4, 2024
View reviewed changes
Copy link
Member

@youkaichao youkaichao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great finding!

@tlrmchlsmth tlrmchlsmth added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 4, 2024
@tlrmchlsmth tlrmchlsmth enabled auto-merge (squash) November 4, 2024 20:37
@tlrmchlsmth tlrmchlsmth merged commit 04bbf38 into vllm-project:main Nov 5, 2024
74 checks passed
bigPYJ1151 pushed a commit to bigPYJ1151/vllm that referenced this pull request Nov 5, 2024
@tlrmchlsmth @bigPYJ1151
[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep (vll…
66dd8d5 
…m-project#9994)

Signed-off-by: Tyler Michael Smith <[email protected]>
DarkLight1337 pushed a commit that referenced this pull request Nov 5, 2024
@tlrmchlsmth @DarkLight1337
[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep (#9994)
857e913 
Signed-off-by: Tyler Michael Smith <[email protected]>
JC1DA pushed a commit to JC1DA/vllm that referenced this pull request Nov 11, 2024
@tlrmchlsmth @JC1DA
[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep (vll…
8e2ef2a 
…m-project#9994)

Signed-off-by: Tyler Michael Smith <[email protected]>
Signed-off-by: Loc Huynh <[email protected]>
sumitd2 pushed a commit to sumitd2/vllm that referenced this pull request Nov 14, 2024
@tlrmchlsmth @sumitd2
[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep (vll…
e2b00d7 
…m-project#9994)

Signed-off-by: Tyler Michael Smith <[email protected]>
Signed-off-by: Sumit Dubey <[email protected]>
KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024
@tlrmchlsmth
[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep (vll…
a4dd0da 
…m-project#9994)

Signed-off-by: Tyler Michael Smith <[email protected]>
mfournioux pushed a commit to mfournioux/vllm that referenced this pull request Nov 20, 2024
@tlrmchlsmth @mfournioux
[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep (vll…
ea8da04 
…m-project#9994)

Signed-off-by: Tyler Michael Smith <[email protected]>
Signed-off-by: Maxime Fournioux <[email protected]>
tlrmchlsmth added a commit to neuralmagic/vllm that referenced this pull request Nov 23, 2024
@tlrmchlsmth
[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep (vll…
d9d1ade 
…m-project#9994)

Signed-off-by: Tyler Michael Smith <[email protected]>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
@tlrmchlsmth
[Core] Use os.sched_yield in ShmRingBuffer instead of time.sleep (vll…
3004f75 
…m-project#9994)

Signed-off-by: Tyler Michael Smith <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bnellnm bnellnm bnellnm approved these changes

@youkaichao youkaichao youkaichao approved these changes

@mgoin mgoin mgoin approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tlrmchlsmth @mgoin @youkaichao @bnellnm