diff --git a/README.md b/README.md index 5cbb0d6..5d3c6a1 100644 --- a/README.md +++ b/README.md @@ -237,13 +237,15 @@ deepspeed --master_port=20001 toolbench/train/train_long_seq_lora.py \ ``` -## Inference -First prepare your toolbench key: +## Inference With Our RapidAPI Server +Please fill out the [form](https://forms.gle/oCHHc8DQzhGfiT9r6) first and after reviewing we will send you the toolbench key. Then prepare your toolbench key by: ```bash export TOOLBENCH_KEY="your_toolbench_key" ``` -Then run the following commands: +### For ToolLLaMA + +To inference with ToolLLaMA, run the following commands: ```bash export PYTHONPATH=./ python toolbench/inference/qa_pipeline.py \ @@ -258,7 +260,9 @@ python toolbench/inference/qa_pipeline.py \ --toolbench_key $TOOLBENCH_KEY ``` -For **lora** version: +To use your own RapidAPI key, run: + +For **ToolLLaMA-LoRA**: ```bash export PYTHONPATH=./ python toolbench/inference/qa_pipeline.py \ @@ -275,7 +279,7 @@ python toolbench/inference/qa_pipeline.py \ --toolbench_key $TOOLBENCH_KEY ``` -For lora version under **open-domain** setting, run: +For ToolLLaMA-LoRA under **open-domain** setting, run: ```bash export PYTHONPATH=./ python toolbench/inference/qa_pipeline_open_domain.py \ @@ -295,6 +299,57 @@ python toolbench/inference/qa_pipeline_open_domain.py \ --toolbench_key $TOOLBENCH_KEY ``` +### For OpenAI Models +To use ChatGPT, run: +```bash +export TOOLBENCH_KEY="" +export OPENAI_KEY="" +export PYTHONPATH=./ +python toolbench/inference/qa_pipeline.py \ + --tool_root_dir data/toolenv/tools/ \ + --backbone_model chatgpt_function \ + --openai_key $OPENAI_KEY \ + --max_observation_length 1024 \ + --method DFS_woFilter_w2 \ + --input_query_file data/instruction/inference_query_demo.json \ + --output_answer_file data/answer/chatgpt_dfs \ + --toolbench_key $TOOLBENCH_KEY +``` + +To use Text-Davinci-003, run: +```bash +export TOOLBENCH_KEY="" +export OPENAI_KEY="" +export PYTHONPATH=./ +python toolbench/inference/qa_pipeline.py \ + --tool_root_dir data/toolenv/tools/ \ + --backbone_model davinci \ + --openai_key $OPENAI_KEY \ + --max_observation_length 1024 \ + --method DFS_woFilter_w2 \ + --input_query_file data/instruction/inference_query_demo.json \ + --output_answer_file data/answer/davinci_dfs \ + --toolbench_key $TOOLBENCH_KEY +``` + +## Inference With Your Own RapidAPI Account +To do inference with customized RapidAPI account, pass your **rapidapi key** and specify the `use_rapidapi_key` argument in the script: +```bash +export RAPIDAPI_KEY="" +export OPENAI_KEY="" +export PYTHONPATH=./ +python toolbench/inference/qa_pipeline.py \ + --tool_root_dir data/toolenv/tools/ \ + --backbone_model chatgpt_function \ + --openai_key $OPENAI_KEY \ + --max_observation_length 1024 \ + --method DFS_woFilter_w2 \ + --input_query_file data/instruction/inference_query_demo.json \ + --output_answer_file data/answer/chatgpt_dfs \ + --rapidapi_key $RAPIDAPI_KEY \ + --use_rapidapi_key +``` + ## Setting up and running the interface ToolBench contains a Web UI based on [Chatbot UI](https://github.com/mckaywrigley/chatbot-ui), forked to include the use of tools in the interface. It comes in two parts: the backend server, and [chatbot-ui-toolllama](https://github.com/lilbillybiscuit/chatbot-ui-toolllama). Here is a [video demo](assets/toolbench-demo.mp4). diff --git a/README_ZH.md b/README_ZH.md index 6e4ee65..47399c5 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -241,13 +241,13 @@ deepspeed --master_port=20001 toolbench/train/train_long_seq_lora.py \ ``` -## Inference -首先准备您的ToolBench key: +## 用我们的RapidAPI服务进行推理 +请先填写[问卷](https://forms.gle/oCHHc8DQzhGfiT9r6)，我们会尽快审核然后给您发送toolbench key。然后初始化您的toolbench key: ```bash export TOOLBENCH_KEY="your_toolbench_key" ``` - -然后用以下命令做inference: +### ToolLLaMA +用以下命令用ToolLLaMA做推理: ```bash export PYTHONPATH=./ python toolbench/inference/qa_pipeline.py \ @@ -298,6 +298,56 @@ python toolbench/inference/qa_pipeline_open_domain.py \ --output_answer_file data/answer/toolllama_lora_dfs_open_domain \ --toolbench_key $TOOLBENCH_KEY ``` +### OpenAI模型 +用ChatGPT: +```bash +export TOOLBENCH_KEY="" +export OPENAI_KEY="" +export PYTHONPATH=./ +python toolbench/inference/qa_pipeline.py \ + --tool_root_dir data/toolenv/tools/ \ + --backbone_model chatgpt_function \ + --openai_key $OPENAI_KEY \ + --max_observation_length 1024 \ + --method DFS_woFilter_w2 \ + --input_query_file data/instruction/inference_query_demo.json \ + --output_answer_file data/answer/chatgpt_dfs \ + --toolbench_key $TOOLBENCH_KEY +``` + +用Text-Davinci-003: +```bash +export TOOLBENCH_KEY="" +export OPENAI_KEY="" +export PYTHONPATH=./ +python toolbench/inference/qa_pipeline.py \ + --tool_root_dir data/toolenv/tools/ \ + --backbone_model davinci \ + --openai_key $OPENAI_KEY \ + --max_observation_length 1024 \ + --method DFS_woFilter_w2 \ + --input_query_file data/instruction/inference_query_demo.json \ + --output_answer_file data/answer/davinci_dfs \ + --toolbench_key $TOOLBENCH_KEY +``` + +## 用您自己的RapidAPI账号做推理 +要用定制化的RapidAPI账号进行推理，请在脚本中传入您的**rapidapi key**并指定`use_rapidapi_key`参数: +```bash +export RAPIDAPI_KEY="" +export OPENAI_KEY="" +export PYTHONPATH=./ +python toolbench/inference/qa_pipeline.py \ + --tool_root_dir data/toolenv/tools/ \ + --backbone_model chatgpt_function \ + --openai_key $OPENAI_KEY \ + --max_observation_length 1024 \ + --method DFS_woFilter_w2 \ + --input_query_file data/instruction/inference_query_demo.json \ + --output_answer_file data/answer/chatgpt_dfs \ + --rapidapi_key $RAPIDAPI_KEY \ + --use_rapidapi_key +``` ## Setting up and running the interface diff --git a/scripts/inference_chatgpt_pipeline_w_rapidapi_key.sh b/scripts/inference_chatgpt_pipeline_w_rapidapi_key.sh new file mode 100644 index 0000000..bb079ac --- /dev/null +++ b/scripts/inference_chatgpt_pipeline_w_rapidapi_key.sh @@ -0,0 +1,17 @@ +export RAPIDAPI_KEY="" +export OUTPUT_DIR="data/answer/chatgpt_dfs" +export OPENAI_KEY="" +export PYTHONPATH=./ + +mkdir $OUTPUT_DIR +python toolbench/inference/qa_pipeline.py \ + --tool_root_dir data/toolenv/tools/ \ + --backbone_model chatgpt_function \ + --openai_key $OPENAI_KEY \ + --max_observation_length 1024 \ + --method DFS_woFilter_w2 \ + --input_query_file data/instruction/inference_query_demo.json \ + --output_answer_file $OUTPUT_DIR \ + --rapidapi_key $RAPIDAPI_KEY \ + --use_rapidapi_key + diff --git a/toolbench/inference/Downstream_tasks/rapidapi.py b/toolbench/inference/Downstream_tasks/rapidapi.py index 89c158b..b342d1b 100644 --- a/toolbench/inference/Downstream_tasks/rapidapi.py +++ b/toolbench/inference/Downstream_tasks/rapidapi.py @@ -54,6 +54,8 @@ def __init__(self, query_json, tool_descriptions, retriever, args, process_id=0) self.tool_root_dir = args.tool_root_dir self.toolbench_key = args.toolbench_key + self.rapidapi_key = args.rapidapi_key + self.use_rapidapi_key = args.use_rapidapi_key self.service_url = "http://8.218.239.54:8080/rapidapi" self.max_observation_length = args.max_observation_length self.observ_compress_method = args.observ_compress_method @@ -328,14 +330,18 @@ def _step(self, action_name="", action_input=""): } if self.process_id == 0: print(colored(f"query to {self.cate_names[k]}-->{self.tool_names[k]}-->{action_name}",color="yellow")) - response = requests.post(self.service_url, json=payload,timeout=30) - if response.status_code != 200: - return json.dumps({"error": f"request invalid, data error. status_code={response.status_code}", "response": ""}), 12 - try: - response = response.json() - except: - print(response) - return json.dumps({"error": f"request invalid, data error", "response": ""}), 12 + if self.use_rapidapi_key: + payload["rapidapi_key"] = self.rapidapi_key + response = get_rapidapi_response(payload) + else: + response = requests.post(self.service_url, json=payload,timeout=30) + if response.status_code != 200: + return json.dumps({"error": f"request invalid, data error. status_code={response.status_code}", "response": ""}), 12 + try: + response = response.json() + except: + print(response) + return json.dumps({"error": f"request invalid, data error", "response": ""}), 12 # 1 Hallucinating function names # 4 means that the model decides to pruning by itself # 5 represents api call timeout diff --git a/toolbench/inference/qa_pipeline.py b/toolbench/inference/qa_pipeline.py index b05788c..ad98756 100644 --- a/toolbench/inference/qa_pipeline.py +++ b/toolbench/inference/qa_pipeline.py @@ -21,6 +21,8 @@ parser.add_argument('--input_query_file', type=str, default="", required=False, help='input path') parser.add_argument('--output_answer_file', type=str, default="",required=False, help='output path') parser.add_argument('--toolbench_key', type=str, default="",required=False, help='your toolbench key to request rapidapi service') + parser.add_argument('--rapidapi_key', type=str, default="",required=False, help='your rapidapi key to request rapidapi service') + parser.add_argument('--use_rapidapi_key', action="store_true", help="To use customized rapidapi service or not.") args = parser.parse_args() diff --git a/toolbench/inference/qa_pipeline_open_domain.py b/toolbench/inference/qa_pipeline_open_domain.py index eebf5d3..3dcd8d1 100644 --- a/toolbench/inference/qa_pipeline_open_domain.py +++ b/toolbench/inference/qa_pipeline_open_domain.py @@ -23,6 +23,8 @@ parser.add_argument('--input_query_file', type=str, default="", required=False, help='input path') parser.add_argument('--output_answer_file', type=str, default="",required=False, help='output path') parser.add_argument('--toolbench_key', type=str, default="",required=False, help='your toolbench key to request rapidapi service') + parser.add_argument('--rapidapi_key', type=str, default="",required=False, help='your rapidapi key to request rapidapi service') + parser.add_argument('--use_rapidapi_key', action="store_true", help="To use customized rapidapi service or not.") args = parser.parse_args() diff --git a/toolbench/inference/toolbench_server.py b/toolbench/inference/toolbench_server.py index 4c6c195..71f5af4 100644 --- a/toolbench/inference/toolbench_server.py +++ b/toolbench/inference/toolbench_server.py @@ -88,6 +88,8 @@ def get_args(self): parser.add_argument('--input_query_file', type=str, default="", required=False, help='input path') parser.add_argument('--output_answer_file', type=str, default="", required=False, help='output path') parser.add_argument('--toolbench_key', type=str, default="", required=False, help='your toolbench key') + parser.add_argument('--rapidapi_key', type=str, default="",required=False, help='your rapidapi key to request rapidapi service') + parser.add_argument('--use_rapidapi_key', action="store_true", help="To use customized rapidapi service or not.") args = parser.parse_args() return args