Skip to content

HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance

License

Notifications You must be signed in to change notification settings

Modas-Li/HuixiangDou

 
 

Repository files navigation

"HuixiangDou" is a domain-specific knowledge assistant based on the LLM. Features:

  1. Deal with complex scenarios like group chats, answer user questions without causing message flooding.
  2. Propose an algorithm pipeline for answering technical questions.
  3. Low deployment cost, only need the LLM model to meet 4 traits can answer most of the user's questions, see arxiv2401.08772.

Check out the scenes in which HuixiangDou are running

🆕 What's new

📦 Hardware Requirements

The following are the hardware requirements for running. It is suggested to follow this document, starting with the basic version and gradually experiencing advanced features.

Version GPU Memory Requirements Features Tested on Linux
Experience Version 2.3GB Use openai API (e.g., deepseek) to handle source code-level issues
Free within quota
Basic Version 19GB Deploy local LLM can answer basic questions
Advanced Version 40GB Fully utilizing search + long-text, answer source code-level questions

🔥 Run

We will take mmpose as examples to explain how to deploy the knowledge assistant to Feishu group chat.

STEP1. Establish Topic Feature Repository

Huggingface login

huggingface-cli login

Execute all the commands below (including the '#' symbol).

# Download the repo
git clone https://github.com/internlm/huixiangdou --depth=1 && cd huixiangdou

# Download chatting topics
mkdir repodir
git clone https://github.com/open-mmlab/mmpose --depth=1 repodir/mmpose

# Build a feature store
mkdir workdir # create a working directory
python3 -m pip install -r requirements.txt # install dependencies
python3 -m huixiangdou.service.feature_store # save the features of repodir to workdir

The first run will automatically download text2vec model, you can also manually download it and update model path in config.ini.

After running, HuixiangDou can distinguish which user topics should be dealt with and which chitchats should be rejected. Please edit good_questions and bad_questions, and try your own domain knowledge (medical, finance, electricity, etc.).

# Accept technical topics
process query: Does mmdeploy support mmtrack model conversion now?
process query: Are there any Chinese text to speech models?
# Reject chitchat
reject query: What to eat for lunch today?
reject query: How to make HuixiangDou?

STEP2. Run Basic Technical Assistant

Configure free TOKEN

HuixiangDou uses a search engine. Click Serper to obtain a quota-limited TOKEN and fill it in config.ini.

# config.ini
..
[web_search]
x_api_key = "${YOUR-X-API-KEY}"
..

Test Q&A Effect

[Experience Version] If your GPU memory is insufficient to locally run the 7B LLM (less than 15GB), try deepseek for 30 million free token. See config-experience.ini

# config.ini

[llm]
enable_local = 0
enable_remote = 1
..
[llm.server]
..
remote_type = "deepseek"
remote_api_key = "YOUR-API-KEY"
remote_llm_max_text_length = 16000
remote_llm_model = "deepseek-chat"

By default, with enable_local=1, the LLM will be automatically downloaded on your first run depending on GPU.

  • Non-docker users. If you don't use docker, you can start all services at once.

    # standalone
    python3 -m huixiangdou.main --standalone
    ..
    ErrorCode.SUCCESS,
    Query: Could you please advise if there is any good optimization method for video stream detection flickering caused by frame skipping?
    Reply:
    1. Frame rate control and frame skipping strategy are key to optimizing video stream detection performance, but you need to pay attention to the impact of frame skipping on detection results.
    2. Multithreading processing and caching mechanism can improve detection efficiency, but you need to pay attention to the stability of detection results.
    3. The use of sliding window method can reduce the impact of frame skipping and caching on detection results.
  • Docker users. If you are using docker, HuixiangDou's Hybrid LLM Service needs to be deployed separately.

    # First start LLM service listening the port 8888
    python3 -m huixiangdou.service.llm_server_hybrid
    ..
    ======== Running on http://0.0.0.0:8888 ========
    (Press CTRL+C to quit)

    Then open a new docker container, configure the host IP (not container IP) in config.ini, and run python3 -m huixiangdou.main

    # config.ini
    [llm]
    ..
    client_url = "http://10.140.24.142:8888/inference" # example, use your real host IP here
    
    # run
    python3 -m huixiangdou.main
    ..
    ErrorCode.SUCCESS

STEP3. Send to Feishu/Personal Wechat [Optional]

Click Create a Feishu Custom Robot to get the WEBHOOK_URL callback, and fill it in the config.ini.

# config.ini
..
[frontend]
type = "lark"
webhook_url = "${YOUR-LARK-WEBHOOK-URL}"

Run. After it ends, the technical assistant's reply will be sent to the Feishu group chat.

python3 -m huixiangdou.main --standalone # for non-docker users
python3 -m huixiangdou.main # for docker users

STEP4. Advanced Version [Optional]

The basic version may not perform well. You can enable these features to enhance performance. The more features you turn on, the better.

  1. Use higher accuracy local LLM

    Adjust the llm.local model in config.ini to internlm2-chat-20b. This option has a significant effect, but requires more GPU memory.

  2. Hybrid LLM Service

    For LLM services that support the openai interface, HuixiangDou can utilize its Long Context ability. Using kimi as an example, below is an example of config.ini configuration:

    # config.ini
    [llm]
    enable_local = 1
    enable_remote = 1
    ..
    [llm.server]
    ..
    # open https://platform.moonshot.cn/
    remote_type = "kimi"
    remote_api_key = "YOUR-KIMI-API-KEY"
    remote_llm_max_text_length = 128000
    remote_llm_model = "moonshot-v1-128k"

    We also support chatgpt. Note that this feature will increase response time and operating costs.

  3. Repo search enhancement

    This feature is suitable for handling difficult questions and requires basic development capabilities to adjust the prompt.

    • Click sourcegraph-account-access to get token

      # open https://github.com/sourcegraph/src-cli#installation
      sudo curl -L https://sourcegraph.com/.api/src-cli/src_linux_amd64 -o /usr/local/bin/src && chmod +x /usr/local/bin/src
      
      # Enable search and fill the token
      [worker]
      enable_sg_search = 1
      ..
      [sg_search]
      ..
      src_access_token = "${YOUR_ACCESS_TOKEN}"
    • Edit the name and introduction of the repo, we take opencompass as an example

      # config.ini
      # add your repo here, we just take opencompass and lmdeploy as example
      [sg_search.opencompass]
      github_repo_id = "open-compass/opencompass"
      introduction = "Used for evaluating large language models (LLM) .."
    • Use python3 -m huixiangdou.service.sg_search for unit test, the returned content should include opencompass source code and documentation

      python3 -m huixiangdou.service.sg_search
      ..
      "filepath": "opencompass/datasets/longbench/longbench_trivia_qa.py",
      "content": "from datasets import Dataset..

    Run main.py, HuixiangDou will enable search enhancement when appropriate.

  4. Tune Parameters

    It is often unavoidable to adjust parameters with respect to business scenarios.

🛠️ FAQ

  1. What if the robot is too cold/too chatty?

    • Fill in the questions that should be answered in the real scenario into resource/good_questions.json, and fill the ones that should be rejected into resource/bad_questions.json.
    • Adjust the theme content in repodir to ensure that the markdown documents in the main library do not contain irrelevant content.

    Re-run feature_store to update thresholds and feature libraries.

  2. Launch is normal, but out of memory during runtime?

    LLM long text based on transformers structure requires more memory. At this time, kv cache quantization needs to be done on the model, such as lmdeploy quantization description. Then use docker to independently deploy Hybrid LLM Service.

  3. How to access other local LLM / After access, the effect is not ideal?

  4. What if the response is too slow/request always fails?

    • Refer to hybrid llm service to add exponential backoff and retransmission.
    • Replace local LLM with an inference framework such as lmdeploy, instead of the native huggingface/transformers.
  5. What if the GPU memory is too low?

    At this time, it is impossible to run local LLM, and only remote LLM can be used in conjunction with text2vec to execute the pipeline. Please make sure that config.ini only uses remote LLM and turn off local LLM.

🍀 Acknowledgements

  • kimi-chat: long context LLM
  • BCEmbeding: Bilingual and Crosslingual Embedding (BCEmbedding) in English and Chinese

📝 Citation

@misc{kong2024huixiangdou,
      title={HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance},
      author={Huanjun Kong and Songyang Zhang and Kai Chen},
      year={2024},
      eprint={2401.08772},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

About

HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 99.6%
  • Shell 0.4%