Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] upgrade vllm to v0.6.3.post1 #69

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open

[WIP] upgrade vllm to v0.6.3.post1 #69

wants to merge 28 commits into from

Conversation

ZeldaHuang
Copy link
Contributor

  1. support pipeline parallelism.
  2. support multi-step scheduling.
  3. support chunked prefill.

@xixingya
Copy link

Does it work properly in version 0.6.3.post1 now?

@s5u13b
Copy link
Contributor

s5u13b commented Nov 21, 2024

Does it work properly in version 0.6.3.post1 now?

Not yet, we will merge it into main once it work properly.

@xixingya
Copy link

Does it work properly in version 0.6.3.post1 now?

Not yet, we will merge it into main once it work properly.

Thank you, I look forward to it, and it is very helpful to me.

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 11197.03 63520.64 129841.45 175147.71 179215.11 70996.51
decode p25 p50 p75 p95 p99 mean
latency(ms) 52.27 56.92 73.23 111.80 206.68 66.02

@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
rayrpc_speed(GB/s) 0.88 1.35 1.64 1.84 1.90 2.02 2.10 2.16 2.18 2.27 2.27 2.26 2.26 2.13 2.21 2.32
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
gloo_speed(GB/s) 0.90 1.45 1.84 2.06 2.29 2.58 2.49 2.58 2.67 2.76 2.83 2.81 2.45 3.05 3.05 2.74

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 3948.71 85906.31 124175.78 176010.55 188518.26 75108.50
decode p25 p50 p75 p95 p99 mean
latency(ms) 49.34 53.94 62.80 98.12 192.23 63.98

@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
rayrpc_speed(GB/s) 0.88 1.37 1.64 1.84 1.95 2.06 2.12 2.17 2.26 2.34 2.29 2.35 2.23 2.07 2.34 2.31
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
nccl_speed(GB/s) 0.29 0.49 0.75 0.92 1.25 1.44 1.51 1.61 1.68 2.13 1.56 2.18 2.01 2.15 2.08 2.07

@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
rayrpc_speed(GB/s) 0.89 1.33 1.62 1.77 1.87 2.06 2.12 2.11 2.24 2.25 2.24 2.25 2.08 2.19 2.30 2.41
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
nccl_speed(GB/s) 0.27 0.52 0.86 0.97 1.05 1.29 1.49 1.46 1.65 1.56 2.10 1.66 1.77 2.23 2.25 2.24

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 3723.53 62014.10 148679.22 210168.42 211821.17 82627.49
decode p25 p50 p75 p95 p99 mean
latency(ms) 43.46 53.77 67.40 109.24 279.27 63.64

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 8766.36 52324.69 154991.52 200430.38 201702.30 79930.10
decode p25 p50 p75 p95 p99 mean
latency(ms) 46.18 54.15 64.30 96.71 191.41 60.81

@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
rayrpc_speed(GB/s) 0.88 1.36 1.64 1.82 1.99 2.06 2.14 2.15 2.25 2.34 2.40 2.24 2.24 2.16 2.38 2.35
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
nccl_speed(GB/s) 0.27 0.49 0.71 0.99 1.19 1.43 1.47 1.63 1.83 1.53 1.81 1.82 1.66 1.94 2.23 2.33

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 4726.40 68807.46 112719.23 174126.51 192091.00 70264.87
decode p25 p50 p75 p95 p99 mean
latency(ms) 51.65 56.39 71.75 120.82 197.16 70.07

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 13137.96 72987.44 129198.76 162482.13 170966.83 74459.20
decode p25 p50 p75 p95 p99 mean
latency(ms) 49.63 54.50 63.51 105.61 214.30 63.24

@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
rayrpc_speed(GB/s) 0.92 1.40 1.69 1.90 2.04 2.16 2.20 2.19 2.28 2.24 2.45 2.22 2.21 2.40 2.21 2.42
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
gloo_speed(GB/s) 0.90 1.46 1.86 2.07 2.33 2.49 2.63 2.60 2.85 2.70 3.01 3.07 2.99 2.48 2.98 2.74
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
nccl_speed(GB/s) 0.28 0.51 0.76 0.96 1.01 1.18 1.44 1.47 1.43 1.69 1.91 1.87 1.91 1.73 1.79 2.54

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 5882.92 76656.49 119550.95 153106.65 160328.33 69794.48
decode p25 p50 p75 p95 p99 mean
latency(ms) 51.21 57.34 67.58 101.74 202.53 70.10

@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
rayrpc_speed(GB/s) 0.89 1.34 1.62 1.81 1.98 2.09 2.14 2.13 2.20 2.25 2.33 2.19 2.11 2.21 2.29 2.40
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
gloo_speed(GB/s) 0.89 1.44 1.91 2.11 2.18 2.38 2.45 2.57 2.53 2.47 3.00 2.56 2.37 2.41 2.59 2.21
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
nccl_speed(GB/s) 0.26 0.49 0.69 0.96 1.04 1.26 1.38 1.41 1.78 1.74 1.76 2.07 1.97 1.87 2.36 1.87

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 7679.99 71422.06 124939.18 161609.09 166007.80 70801.88
decode p25 p50 p75 p95 p99 mean
latency(ms) 51.17 56.58 71.19 121.14 319.88 71.02

@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
rayrpc_speed(GB/s) 0.91 1.42 1.66 1.88 2.00 2.11 2.20 2.12 2.28 2.28 2.42 2.33 2.18 2.20 2.24 2.51
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
gloo_speed(GB/s) 0.88 1.45 1.75 2.04 2.15 2.30 2.49 2.42 2.35 2.59 2.32 2.82 2.33 2.51 2.65 2.56
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
nccl_speed(GB/s) 0.26 0.49 0.72 0.86 1.05 1.13 1.40 1.42 1.82 1.77 2.09 1.82 2.47 2.16 2.48 2.58

@github-actions GitHub Actions
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 11101.48 65265.98 116765.41 173497.34 203737.40 71233.68
decode p25 p50 p75 p95 p99 mean
latency(ms) 51.58 57.06 67.44 105.18 156.22 61.95

@github-actions GitHub Actions
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
rayrpc_speed(GB/s) 0.90 1.34 1.63 1.82 1.94 2.05 2.12 2.15 2.20 2.25 2.29 2.21 2.17 1.93 2.34 2.38
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
gloo_speed(GB/s) 0.92 1.49 1.81 2.11 2.24 2.26 2.48 2.64 2.57 2.66 2.73 2.95 2.37 2.68 2.61 2.58
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB
nccl_speed(GB/s) 0.27 0.45 0.70 0.96 1.00 1.32 1.50 1.42 1.42 1.73 1.71 1.86 1.81 1.66 2.59 2.64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ZeldaHuang @xixingya @s5u13b