Merge branch 'master' of https://github.com/internlm/huixiangdou

InternLM · Jan 12, 2024 · 9b9f200 · 9b9f200
2 parents 1d35ae0 + ffbc401
commit 9b9f200
Show file tree

Hide file tree

Showing 6 changed files with 39 additions and 12 deletions.
diff --git a/README.md b/README.md
@@ -19,11 +19,11 @@ View [HuixiangDou inside](./huixiangdou-inside.md).
 
 The following are the hardware requirements for running. It is suggested to follow this document, starting with the basic version and gradually experiencing advanced features.
 
-| Version | GPU Memory Requirements | Remarks | Tested on |
+| Version | GPU Memory Requirements | Features | Tested on Linux |
 | :-: | :-: | :-: | :-: |
-| Basic Version | 20GB | Able to answer basic domain knowledge questions, zero cost operation | ![](https://img.shields.io/badge/linux%203090%2024G-passed-blue?style=for-the-badge) |
-| Advanced Version | 40GB | Able to answer source code level questions, zero cost operation | ![](https://img.shields.io/badge/linux%20A100%2080G-passed-blue?style=for-the-badge) |
-| Modified Version | 4GB | Using openai API to replace local LLM, basic development capability required, operation involves cost | ![](https://img.shields.io/badge/linux%201660ti%206GB-passed-blue?style=for-the-badge) |
+| Basic Version | 20GB | Answer basic domain knowledge questions, zero cost | ![](https://img.shields.io/badge/3090%2024G-passed-blue?style=for-the-badge) |
+| Advanced Version | 40GB | Answer source code level questions, zero cost | ![](https://img.shields.io/badge/A100%2080G-passed-blue?style=for-the-badge) |
+| Modified Version | 4GB | Using openai API, operation involves cost | ![](https://img.shields.io/badge/1660ti%206G-passed-blue?style=for-the-badge) |
 
 # 🔥 Run
 

diff --git a/README_zh.md b/README_zh.md
@@ -18,11 +18,11 @@
 
 以下是运行茴香豆的硬件需求。建议遵循部署流程，从基础版开始，逐渐体验高级特性。
 
-| 版本 | 硬件需求 | 备注 | 已验证设备 |
+| 版本 | 硬件需求 | 描述 | Linux 系统已验证设备 |
 | :-: | :-: | :-: | :-: |
-| 基础版 | 20GB | 能够回答领域知识的基础问题，零成本运行 | ![](https://img.shields.io/badge/linux%203090%2024G-passed-blue?style=for-the-badge) |
-| 高级版 | 40GB | 能够回答源码级问题，零成本运行 | ![](https://img.shields.io/badge/linux%20A100%2080G-passed-blue?style=for-the-badge) |
-| 魔改版 | 4GB| 用 openai API 替代本地 LLM，需要基础开发能力，运行需要费用 | ![](https://img.shields.io/badge/linux%201660ti%206GB-passed-blue?style=for-the-badge) |
+| 基础版 | 20GB | 能回答领域知识的基础问题，零成本运行 | ![](https://img.shields.io/badge/3090%2024G-passed-blue?style=for-the-badge) |
+| 高级版 | 40GB | 能够回答源码级问题，零成本运行 | ![](https://img.shields.io/badge/A100%2080G-passed-blue?style=for-the-badge) |
+| 魔改版 | 4GB| 用 openai API 替代本地 LLM，需要基础开发能力，运行需要费用 | ![](https://img.shields.io/badge/1660ti%206G-passed-blue?style=for-the-badge) |
 
 # 🔥 运行
 
@@ -40,9 +40,16 @@ cd HuixiangDou && mkdir workdir # 创建工作目录
 python3 -m pip install -r requirements.txt # 安装依赖，python3.11 需要 `conda install conda-forge::faiss-gpu`
 python3 service/feature_store.py # 把 repodir 的特征保存到 workdir
 ```
-首次运行将自动下载配置中的 [text2vec-large-chinese](https://huggingface.co/GanymedeNil/text2vec-large-chinese)，如果自动下载失败，可以手动下载到本地，然后在 `config.ini` 设置模型路径。
+首次运行将自动下载配置中的 [text2vec-large-chinese](https://huggingface.co/GanymedeNil/text2vec-large-chinese)。考虑到不同地区 huggingface 连接问题，建议先手动下载到本地，然后在 `config.ini` 设置模型路径。例如：
 
-结束后，茴香豆能够区分应该处理哪些用户话题，哪些闲聊应该拒绝。请编辑 [good_questions](./resource/good_questions.json) 和 [bad_questions](./resource/bad_questions.json)，尝试自己的领域知识（医疗，金融，电力等）。
+```shell
+# config.ini
+[feature_store]
+..
+model_path = "/path/to/text2vec-large-chinese"
+```
+
+运行结束后，茴香豆能够区分应该处理哪些用户话题，哪些闲聊应该拒绝。请编辑 [good_questions](./resource/good_questions.json) 和 [bad_questions](./resource/bad_questions.json)，尝试自己的领域知识（医疗，金融，电力等）。
 
 ```shell
 # 接受技术话题

diff --git a/requirements.txt b/requirements.txt
@@ -6,3 +6,10 @@ sentence_transformers
 readability-lxml
 faiss-gpu
 transformers>=4.33.2
+openai>=1.0.0
+pydantic>=1.10.13
+transformers_stream_generator
+accelerate>=0.26.1
+beautifulsoup4
+tiktoken
+einops
diff --git a/service/llm_server_hybrid.py b/service/llm_server_hybrid.py
@@ -43,7 +43,8 @@ def __init__(self,
                 model_path,
                 trust_remote_code=True,
                 device_map="auto",
-                # fp16=True,
+#                torch_dtype="auto",
+#                fp16=True,
             ).eval()
         else:
             logger.warning(f'local LLM disabled.')

diff --git a/service/worker.py b/service/worker.py
@@ -136,7 +136,7 @@ def generate(self, query, history, groupname):
             response = self.llm.generate_response(prompt=prompt,
                                              history=history,
                                              remote=True)
-            tracker.track('feature store doc', [db_context_part, response])
+            tracker.log('feature store doc', [db_context_part, response])
             return ErrorCode.SUCCESS, response
 
         else:

diff --git a/tests/test_hf_import_accelerate.py b/tests/test_hf_import_accelerate.py
@@ -0,0 +1,12 @@
+from accelerate import dispatch_model, infer_auto_device_map, init_empty_weights
+from accelerate.hooks import add_hook_to_module
+from accelerate.utils import (
+    check_tied_parameters_on_same_device,
+    find_tied_parameters,
+    get_balanced_memory,
+    get_max_memory,
+    load_offloaded_weights,
+    offload_weight,
+    save_offload_index,
+    set_module_tensor_to_device,
+)