-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
6c4963c
commit c0bc50e
Showing
5 changed files
with
62 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,12 @@ | ||
vllm_address generated 1032 tokens with 40.715563518001495 tps (with warm up) | ||
vllm_address unconstrained generated 1500 tokens with 42.01920504622742 tps | ||
linkedlist generated 1262 tokens with 40.568280943625346 tps (with warm up) | ||
linkedlist unconstrained generated 1337 tokens with 41.946869871332254 tps | ||
orders generated 4271 tokens with 40.09933827742811 tps (with warm up) | ||
orders unconstrained generated 4589 tokens with 41.564089177085926 tps | ||
llama3_8b_vllm_address generated 1085 tokens with 40.811995725723065 tps (with warm up) | ||
llama3_8b_vllm_address unconstrained generated 1526 tokens with 41.99365646425085 tps | ||
llama3_8b_linkedlist generated 1252 tokens with 40.58724636960033 tps (with warm up) | ||
llama3_8b_linkedlist unconstrained generated 1389 tokens with 41.977055386201535 tps | ||
llama3_8b_orders generated 4266 tokens with 40.109222217293876 tps (with warm up) | ||
llama3_8b_orders unconstrained generated 4595 tokens with 41.5902565932466 tps | ||
llama2_7b_vllm_address generated 1571 tokens with 46.65812840219695 tps (with warm up) | ||
llama2_7b_vllm_address unconstrained generated 1748 tokens with 47.65099657371631 tps | ||
llama2_7b_linkedlist generated 1648 tokens with 46.55372314285624 tps (with warm up) | ||
llama2_7b_linkedlist unconstrained generated 1917 tokens with 47.682641280112215 tps | ||
llama2_7b_orders generated 5120 tokens with 45.76177824108742 tps (with warm up) | ||
llama2_7b_orders unconstrained generated 5114 tokens with 46.81601673468687 tps |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,7 +5,6 @@ | |
import typing | ||
|
||
import kbnf | ||
import torch | ||
from vllm import LLM | ||
|
||
from config import EngineGenerationConfig | ||
|