Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Add comprehensive testing: migration, e2e, and bench #30

Merged
merged 3 commits into from
Sep 19, 2024
Merged

Conversation

KuilongCui
Copy link
Contributor

No description provided.

@KuilongCui KuilongCui force-pushed the action branch 10 times, most recently from 0dd1aa5 to 28ad93f Compare September 12, 2024 08:57
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@KuilongCui KuilongCui force-pushed the action branch 2 times, most recently from 8f48278 to 4e67e50 Compare September 12, 2024 12:02
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 12, 2024
@KuilongCui KuilongCui changed the title [WIP] ci workflow [CI] Added migration, e2e, and bench tests. Sep 12, 2024
@KuilongCui KuilongCui changed the title [CI] Added migration, e2e, and bench tests. [CI] Added migration, e2e, and bench tests Sep 12, 2024
@KuilongCui KuilongCui changed the title [CI] Added migration, e2e, and bench tests [CI] Added Migration, E2E, and Bench CI Test Sep 12, 2024
@KuilongCui KuilongCui changed the title [CI] Added Migration, E2E, and Bench CI Test [CI] Add Migration, E2E, and Bench ci test Sep 12, 2024
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 128.00 MB 136.00 MB 184.00 MB 312.00 MB
rpc_speed(GB/s) 1.02 1.45 1.66 1.72 1.77 1.95 2.08 1.92 2.01 1.91 2.18 2.21 2.22 2.18 2.24 2.28 2.57 2.39
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 128.00 MB 488.00 MB 552.00 MB
gloo_speed(GB/s) 0.96 1.66 2.04 2.22 2.44 2.54 2.97 3.02 3.17 2.63 2.78 3.18 2.76 2.63 2.23 2.51
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 128.00 MB 136.00 MB
nccl_speed(GB/s) 0.18 0.41 0.83 0.93 1.13 1.22 1.47 1.70 0.82 2.50 1.78 4.89 3.47 3.25

Copy link

decode p25 p50 p75 p95 p99 mean
latency(ms) 51.26 56.06 67.29 145.34 287.28 69.71

.github/workflows/migration.yml Outdated Show resolved Hide resolved
llumnix/backends/vllm/worker.py Outdated Show resolved Hide resolved
conftest.py Outdated Show resolved Hide resolved
llumnix/backends/vllm/worker.py Show resolved Hide resolved
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 176.00 MB 208.00 MB 312.00 MB 536.00 MB
rpc_speed(GB/s) 1.03 1.52 1.80 1.94 1.96 2.08 2.14 2.16 2.22 2.30 2.27 2.38 2.21 2.34 2.44 2.49 2.47 2.46 2.52 2.48 2.38 2.44 2.66 3.20
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 160.00 MB 168.00 MB 176.00 MB 184.00 MB 192.00 MB 312.00 MB 472.00 MB 536.00 MB 560.00 MB
gloo_speed(GB/s) 0.96 1.57 1.90 2.18 2.52 2.61 2.81 3.02 2.70 2.68 2.87 3.20 3.03 2.97 3.13 2.65 2.56 2.93 2.60 2.38 2.53 2.65 2.58 2.55 2.57 2.48 2.64
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 160.00 MB 176.00 MB 192.00 MB 312.00 MB 480.00 MB 536.00 MB 544.00 MB 688.00 MB
nccl_speed(GB/s) 0.19 0.47 0.74 0.88 1.05 1.28 1.32 1.77 1.97 1.67 2.25 2.09 2.52 2.72 2.13 2.37 2.68 2.72 3.33 1.29 2.59 3.97 4.54 4.36 4.06 2.72

Copy link

decode p25 p50 p75 p95 p99 mean
latency(ms) 51.84 57.56 70.29 135.34 364.53 75.70

.github/workflows/bench.yml Outdated Show resolved Hide resolved
conftest.py Outdated Show resolved Hide resolved
llumnix/entrypoints/llumnix_utils.py Show resolved Hide resolved
tests/unit_test/entrypoints/vllm/test_api_server.py Outdated Show resolved Hide resolved
Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 184.00 MB 192.00 MB 200.00 MB 208.00 MB 232.00 MB 312.00 MB 320.00 MB 384.00 MB 416.00 MB 424.00 MB 432.00 MB 440.00 MB 464.00 MB 472.00 MB 488.00 MB 560.00 MB 568.00 MB
rpc_speed(GB/s) 1.04 1.54 1.76 1.91 1.99 2.04 2.09 2.22 2.21 2.26 2.33 2.26 2.29 2.41 2.42 2.45 2.43 2.49 2.51 2.54 2.44 2.52 2.43 2.55 2.34 2.50 2.48 2.51 2.81 2.84 3.05 2.72 2.86 3.21 3.01 2.94 3.27 3.01 3.00
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 184.00 MB 192.00 MB 224.00 MB 264.00 MB 304.00 MB 312.00 MB 320.00 MB 416.00 MB 424.00 MB 432.00 MB 480.00 MB 536.00 MB 696.00 MB
gloo_speed(GB/s) 0.95 1.55 1.96 2.27 2.27 2.50 2.68 2.73 2.85 3.08 2.47 3.31 3.26 2.60 3.12 2.73 2.63 2.59 2.76 1.98 2.63 1.76 1.83 2.65 2.62 1.88 2.85 2.25 2.29 2.52 2.71 2.59 1.75 2.71 2.03
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 184.00 MB 192.00 MB 208.00 MB 232.00 MB 248.00 MB 288.00 MB 312.00 MB 320.00 MB 336.00 MB 416.00 MB 480.00 MB 536.00 MB
nccl_speed(GB/s) 0.19 0.47 0.70 0.89 1.02 1.18 1.50 1.67 2.03 2.02 1.84 2.15 2.30 2.83 2.32 2.57 3.08 3.18 3.16 3.81 2.99 3.70 3.76 3.35 3.67 4.23 4.58 2.51 4.33 3.13 0.80 4.51 5.04 5.46

Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 65477.25 193169.00 343420.00 418448.75 431154.08 205713.51
decode p25 p50 p75 p95 p99 mean
latency(ms) 51.66 58.32 69.41 118.22 197.39 68.10

@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 19, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 19, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 19, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 19, 2024
@AlibabaPAI AlibabaPAI deleted a comment from github-actions bot Sep 19, 2024
Copy link

prefill p25 p50 p75 p95 p99 mean
latency(ms) 28560.50 109577.50 195929.25 243959.40 263478.02 112795.95
decode p25 p50 p75 p95 p99 mean
latency(ms) 53.59 61.01 78.79 147.31 326.56 75.75

Copy link

migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 184.00 MB 192.00 MB 200.00 MB 208.00 MB 224.00 MB 280.00 MB 312.00 MB 416.00 MB 424.00 MB 464.00 MB 472.00 MB 488.00 MB 560.00 MB 912.00 MB
rpc_speed(GB/s) 1.05 1.56 1.78 1.93 2.03 2.15 2.20 2.14 2.21 2.31 2.39 2.35 2.40 2.33 2.46 2.45 2.46 2.53 2.44 2.60 2.72 2.21 2.48 2.53 2.73 2.64 2.50 2.76 2.59 3.02 3.18 3.16 2.86 3.03 3.23 3.35
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 184.00 MB 192.00 MB 200.00 MB 232.00 MB 256.00 MB 280.00 MB 312.00 MB 328.00 MB 336.00 MB 416.00 MB 424.00 MB 432.00 MB 480.00 MB 560.00 MB
gloo_speed(GB/s) 0.89 1.55 2.05 2.27 2.38 2.60 2.73 2.84 2.71 2.72 3.02 3.01 2.77 3.32 2.85 2.65 2.44 2.15 2.54 2.41 2.07 2.03 2.21 2.83 2.39 2.69 1.21 2.58 2.38 2.73 2.56 2.75 2.81 2.66 1.91 2.44
migration_size 8.00 MB 16.00 MB 24.00 MB 32.00 MB 40.00 MB 48.00 MB 56.00 MB 64.00 MB 72.00 MB 80.00 MB 88.00 MB 96.00 MB 104.00 MB 112.00 MB 120.00 MB 128.00 MB 136.00 MB 144.00 MB 152.00 MB 160.00 MB 168.00 MB 176.00 MB 184.00 MB 192.00 MB 200.00 MB 216.00 MB 224.00 MB 312.00 MB 416.00 MB 424.00 MB 472.00 MB 536.00 MB 544.00 MB 560.00 MB
nccl_speed(GB/s) 0.19 0.49 0.76 0.92 1.26 1.37 1.50 1.61 1.76 1.85 2.26 2.20 2.10 2.74 1.98 2.86 3.07 2.98 4.23 2.89 2.44 4.57 3.21 4.31 3.39 1.29 3.88 3.70 2.49 5.66 5.75 4.07 4.36 5.99

@KuilongCui KuilongCui changed the title [CI] Add Migration, E2E, and Bench ci test [CI] Add Comprehensive Testing: Migration, E2E, and Bench Sep 19, 2024
@KuilongCui KuilongCui changed the title [CI] Add Comprehensive Testing: Migration, E2E, and Bench [CI] Add comprehensive testing: migration, e2e, and bench Sep 19, 2024
@KuilongCui KuilongCui merged commit 91a6454 into main Sep 19, 2024
14 checks passed
@KuilongCui KuilongCui deleted the action branch September 19, 2024 07:30
KuilongCui added a commit that referenced this pull request Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants