diff --git a/README.md b/README.md index a729909..f0d0f45 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,7 @@ For the data preprocessing part, simply **run the following script** : sh dbgpt_hub/scripts/gen_train_eval_data.sh ``` -In the directory `dbgpt_hub/data/`, you will find the newly generated training file example_text2sql_train.json and testing file example_text2sql_dev.json, containing 8659 and 1034 entries respectively. +In the directory `dbgpt_hub/data/`, you will find the newly generated training file example_text2sql_train.json and testing file example_text2sql_dev.json, containing 8659 and 1034 entries respectively. For the data used in subsequent fine-tuning, set the parameter `file_name` value to the file name of the training set in dbgpt_hub/data/dataset_info.json, such as example_text2sql_train.json The data in the generated JSON looks something like this: @@ -168,7 +168,7 @@ deepspeed --num_gpus 2 dbgpt_hub/train/sft_train.py \ ... ``` -The other parts that are omitted (…) can be kept consistent. If you want to change the default deepseed configuration, go into the `dbgpt_hub/configs` directory and make changes to ds_config.json as needed. +The other parts that are omitted (…) can be kept consistent. If you want to change the default deepseed configuration, go into the `dbgpt_hub/configs` directory and make changes to ds_config.json as needed,the default is stage2. In the script, during fine-tuning, different models correspond to key parameters lora_target and template, as shown in the following table: @@ -251,9 +251,9 @@ The whole process we will divide into three phases: - [x] internlm * Stage 2: - * Optimize model performance, and support fine-tuning more different models in various ways before `20231010` - * Optimize `prompts` - * Release evaluation results, and optimized models open to peers. + - [x] Optidmize model performance, and support fine-tuning more different models in various ways before `20231010` + - [x] Optimize `prompts` + - [x] Release evaluation results, and optimized models open to peers. * Stage 3: - [ ] Inference speed optimization and improvement - [ ] Targeted optimization and improvement of business scenarios and Chinese effects diff --git a/README.zh.md b/README.zh.md index 62bee43..28f6369 100644 --- a/README.zh.md +++ b/README.zh.md @@ -118,7 +118,7 @@ DB-GPT-Hub使用的是信息匹配生成法进行数据准备,即结合表信 ## 生成train数据 和dev(eval)数据, sh dbgpt_hub/scripts/gen_train_eval_data.sh ``` -在`dbgpt_hub/data/`目录你会得到新生成的训练文件example_text2sql_train.json 和测试文件example_text2sql_dev.json ,数据量分别为8659和1034条。 +在`dbgpt_hub/data/`目录你会得到新生成的训练文件example_text2sql_train.json 和测试文件example_text2sql_dev.json ,数据量分别为8659和1034条。 对于后面微调时的数据使用在dbgpt_hub/data/dataset_info.json中将参数`file_name`值给为训练集的文件名,如example_text2sql_train.json。 生成的json中的数据形如: ``` @@ -156,7 +156,7 @@ deepspeed --num_gpus 2 dbgpt_hub/train/sft_train.py \ --quantization_bit 4 \ ... ``` -其他省略(...)的部分均保持一致即可。 如果想要更改默认的deepseed配置,进入 `dbgpt_hub/configs` 目录,在ds_config.json 更改即可。 +其他省略(...)的部分均保持一致即可。 如果想要更改默认的deepseed配置,进入 `dbgpt_hub/configs` 目录,在ds_config.json 更改即可,默认为stage2的策略。 脚本中微调时不同模型对应的关键参数lora_target 和 template,如下表: diff --git a/dbgpt_hub/configs/stage2.json b/dbgpt_hub/configs/ds_config.json similarity index 100% rename from dbgpt_hub/configs/stage2.json rename to dbgpt_hub/configs/ds_config.json diff --git a/dbgpt_hub/configs/stage3.json b/dbgpt_hub/configs/ds_config_stage3.json similarity index 100% rename from dbgpt_hub/configs/stage3.json rename to dbgpt_hub/configs/ds_config_stage3.json diff --git a/dbgpt_hub/data/dataset_info.json b/dbgpt_hub/data/dataset_info.json index 02994b6..3cef624 100644 --- a/dbgpt_hub/data/dataset_info.json +++ b/dbgpt_hub/data/dataset_info.json @@ -1,6 +1,6 @@ { "example_text2sql": { - "file_name": "example_text2sql_train.json", + "file_name": "example_text2sql.json", "columns": { "prompt": "instruction", "query": "input",