Fix issues in README on feedback (#15)

* Update README.md * Update README.md * Update README.md * Update README.md --------- Co-authored-by: Jeremy Song <[email protected]>
SJTU-IPADS · Dec 19, 2023 · ded0613 · ded0613
1 parent e3b4b85
commit ded0613
Showing 1 changed file with 6 additions and 11 deletions.
diff --git a/README.md b/README.md
@@ -5,7 +5,7 @@ PowerInfer is a CPU/GPU LLM inference engine leveraging **activation locality**
 
 ## Demo 🔥
 
-https://github.com/SJTU-IPADS/PowerInfer/assets/34213478/d26ae05b-d0cf-40b6-8788-bda3fe447e28
+https://github.com/SJTU-IPADS/PowerInfer/assets/34213478/fe441a42-5fce-448b-a3e5-ea4abb43ba23
 
 PowerInfer v.s. llama.cpp on a single RTX 4090(24G) running Falcon(ReLU)-40B-FP16 with a 11x speedup!
 
@@ -75,8 +75,8 @@ cd PowerInfer
 ### Build
 In order to build PowerInfer you have two different options. These commands are supposed to be run from the root directory of the project.
 
-Using `CMake` on Linux or macOS:
-* If you have one GPU:
+Using `CMake`(3.13+) on Linux or macOS:
+* If you have an NVIDIA GPU:
 ```bash
 cmake -S . -B build -DLLAMA_CUBLAS=ON
 cmake --build build --config Release
@@ -109,14 +109,7 @@ If you want to limit the VRAM usage of GPU:
   ./build/bin/main -m /PATH/TO/MODEL -n $output_token_count -t $thread_num -p $prompt --vram-budget $vram_gb
 ```
 
-As for now, it requires an offline-generated "GPU index" file to split FFNs on GPU. If you want to try it, please use the following instructions to generate the GPU index file:
-```bash
-python scripts/export-gpu-split.py $activation_count_path $output_idx_path solver
-```
-Then, you can use the following instructions to run PowerInfer with GPU index:
-```bash
-./build/bin/main -m /PATH/TO/MODEL -n $output_token_count -t $thread_num -p $prompt --gpu-index $split_path
-```
+As for now, it requires an offline-generated "GPU index" file to split FFNs on GPU. And we found these files are hard to maintain and distribute. We will ship automatic FFN split based on VRAM capacity via [#11](https://github.com/SJTU-IPADS/PowerInfer/pull/11) very soon.
 
 ## Evaluation
 
@@ -131,6 +124,8 @@ We will release the code and data in the following order, please stay tuned!
 
 - [x] Release core code of PowerInfer, supporting Llama-2, Falcon-40B.
 - [ ] Support Mistral-7B
+- [ ] Support Windows
+- [ ] Support text-generation-webui
 - [ ] Release perplexity evaluation code
 - [ ] Support Metal for Mac
 - [ ] Release code for OPT models