Skip to content

Commit b3596b1

Browse files
committed
update readme, remove useless link
1 parent aa7c889 commit b3596b1

File tree

1 file changed

+7
-5
lines changed

1 file changed

+7
-5
lines changed

README.md

+7-5
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,18 @@
11
# CIBench: Evaluating Your LLMs with a Code Interpreter Plugin
22

3-
[![arXiv](https://img.shields.io/badge/arXiv-2312.14033-b31b1b.svg)](https://arxiv.org/abs/2312.14033)
3+
<!-- [![arXiv](https://img.shields.io/badge/arXiv-2312.14033-b31b1b.svg)](https://arxiv.org/abs/2312.14033) -->
44
[![license](https://img.shields.io/github/license/InternLM/opencompass.svg)](./LICENSE)
55

66
## ✨ Introduction
77

8-
This is an evaluation harness for the benchmark described in [CIBench: Evaluating Your LLMs with a Code Interpreter Plugin](https://arxiv.org/abs/2312.14033).
8+
This is an evaluation harness for the benchmark described in CIBench: Evaluating Your LLMs with a Code Interpreter Plugin.
99

10-
[[Paper](https://arxiv.org/abs/2312.14033)]
10+
<!-- [CIBench: Evaluating Your LLMs with a Code Interpreter Plugin](https://arxiv.org/abs/2312.14033). -->
11+
12+
<!-- [[Paper](https://arxiv.org/abs/2312.14033)]
1113
[[Project Page](https://open-compass.github.io/CIBench/)]
1214
[[LeaderBoard](https://open-compass.github.io/CIBench/leaderboard.html)]
13-
[[HuggingFace](https://huggingface.co/datasets/lovesnowbest/CIBench)]
15+
[[HuggingFace](https://huggingface.co/datasets/lovesnowbest/CIBench)] -->
1416

1517
> While LLM-Based agents, which use external tools to solve complex problems, have made significant progress, benchmarking their ability is challenging, thereby hindering a clear understanding of their limitations. In this paper, we propose an interactive evaluation framework, named CIBench, to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks. Our evaluation framework includes an evaluation dataset and two evaluation modes. The evaluation dataset is constructed using an LLM-human cooperative approach and simulates an authentic workflow by leveraging consecutive and interactive IPython sessions. The two evaluation modes assess LLMs' ability with and without human assistance. We conduct extensive experiments to analyze the ability of 24 LLMs on CIBench and provide valuable insights for future LLMs in code interpreter utilization.
1618
@@ -100,7 +102,7 @@ python test.py --model_type api --model_path gpt-4-1106-preview --resume --out_n
100102
```bash
101103
python run.py config/cibench_eval/eval_cibench_hf.py
102104
```
103-
105+
Note that the currently accelerator config (-a lmdeploy) doesnot support CodeAgent model. If you want to use lmdeploy to acclerate the evaluation, please refer to [lmdeploy_internlm2_chat_7b](https://github.com/open-compass/opencompass/blob/main/configs/models/hf_internlm/lmdeploy_internlm2_chat_7b.py) to write the model config by yourself.
104106
<!-- Note: You can install [lmdeploy](https://github.com/InternLM/lmdeploy) and add '-a lmdeploy' to acclerate the evaluation. -->
105107
### 💫 Final Results
106108
Once you finish all tested samples, you can check the results in *outputs/cibench*.

0 commit comments

Comments
 (0)