-
Notifications
You must be signed in to change notification settings - Fork 143
GLUE
zhezhaoa edited this page Sep 26, 2022
·
6 revisions
Here is a short summary of our solution on GLUE classification benchmark. This section mainly focuses on single model.
The example of fine-tuning and doing inference on CoLA dataset with BERT-Base-uncased:
python3 finetune/run_classifier.py --pretrained_model_path models/bert_base_uncased.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/CoLA/train.tsv \
--dev_path datasets/CoLA/dev.tsv \
--output_model_path models/cola_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/cola_classifier_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/CoLA/test_nolabel.tsv \
--prediction_path datasets/CoLA/prediction.tsv \
--seq_length 128 --labels_num 2
The example of fine-tuning and doing inference on CoLA dataset with RoBERTa-Base: Since RoBERTa-Base uses different special tokens, it is necessary to change the path of special token mapping from models/special_tokens_map.json to models/xlmroberta_special_tokens_map.json in tencentpretrain/utils/constants.py
python3 finetune/run_classifier.py --pretrained_model_path models/roberta_base.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/CoLA/train.tsv \
--dev_path datasets/CoLA/dev.tsv \
--output_model_path models/cola_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/cola_classifier_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/CoLA/test_nolabel.tsv \
--prediction_path datasets/CoLA/prediction.tsv \
--seq_length 128 --labels_num 2
The example of fine-tuning and doing inference on SST-2 dataset with BERT-Base-uncased:
python3 finetune/run_classifier.py --pretrained_model_path models/bert_base_uncased.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/SST-2/train.tsv \
--dev_path datasets/SST-2/dev.tsv \
--output_model_path models/sst-2_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/sst-2_classifier_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/SST-2/test_nolabel.tsv \
--prediction_path datasets/SST-2/prediction.tsv \
--seq_length 128 --labels_num 2
The example of fine-tuning and doing inference on SST-2 dataset with RoBERTa-Base:
python3 finetune/run_classifier.py --pretrained_model_path models/roberta_base.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/SST-2/train.tsv \
--dev_path datasets/SST-2/dev.tsv \
--output_model_path models/sst-2_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/sst-2_classifier_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/SST-2/test_nolabel.tsv \
--prediction_path datasets/SST-2/prediction.tsv \
--seq_length 128 --labels_num 2
The example of fine-tuning and doing inference on QQP dataset with BERT-Base-uncased:
python3 finetune/run_classifier.py --pretrained_model_path models/bert_base_uncased.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/QQP/train.tsv \
--dev_path datasets/QQP/dev.tsv \
--output_model_path models/qqp_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/qqp_classifier_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/QQP/test_nolabel.tsv \
--prediction_path datasets/QQP/prediction.tsv \
--seq_length 128 --labels_num 2
The example of fine-tuning and doing inference on QQP dataset with RoBERTa-Base:
python3 finetune/run_classifier.py --pretrained_model_path models/roberta_base.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/QQP/train.tsv \
--dev_path datasets/QQP/dev.tsv \
--output_model_path models/qqp_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/qqp_classifier_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/QQP/test_nolabel.tsv \
--prediction_path datasets/QQP/prediction.tsv \
--seq_length 128 --labels_num 2
The example of fine-tuning and doing inference on QNLI dataset with BERT-Base-uncased:
python3 finetune/run_classifier.py --pretrained_model_path models/bert_base_uncased.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/QNLI/train.tsv \
--dev_path datasets/QNLI/dev.tsv \
--output_model_path models/qnli_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/qnli_classifier_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/QNLI/test_nolabel.tsv \
--prediction_path datasets/QNLI/prediction.tsv \
--seq_length 128 --labels_num 2
The example of fine-tuning and doing inference on QNLI dataset with RoBERTa-Base:
python3 finetune/run_classifier.py --pretrained_model_path models/roberta_base.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/QNLI/train.tsv \
--dev_path datasets/QNLI/dev.tsv \
--output_model_path models/qnli_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/qnli_classifier_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/QNLI/test_nolabel.tsv \
--prediction_path datasets/QNLI/prediction.tsv \
--seq_length 128 --labels_num 2
利用BERT-Base-uncased在WNLI数据集上做微调和预测示例：
python3 finetune/run_classifier.py --pretrained_model_path models/bert_base_uncased.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/WNLI/train.tsv \
--dev_path datasets/WNLI/dev.tsv \
--output_model_path models/wnli_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/wnli_classifier_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/WNLI/test_nolabel.tsv \
--prediction_path datasets/WNLI/prediction.tsv \
--seq_length 128 --labels_num 2
利用RoBERTa-Base在WNLI数据集上做微调和预测示例：
python3 finetune/run_classifier.py --pretrained_model_path models/roberta_base.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/WNLI/train.tsv \
--dev_path datasets/WNLI/dev.tsv \
--output_model_path models/wnli_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/wnli_classifier_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/WNLI/test_nolabel.tsv \
--prediction_path datasets/WNLI/prediction.tsv \
--seq_length 128 --labels_num 2
在MNLI数据集上做微调和预测示例： MNLI有两个验证集和测试集，分别为MNLI-m和MNLI-mm 利用BERT-Base-uncased在MNLI-m和MNLI-mm做微调和预测示例：
python3 finetune/run_classifier.py --pretrained_model_path models/bert_base_uncased.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/MNLI/train.tsv \
--dev_path datasets/MNLI/dev_m.tsv \
--output_model_path models/mnli-m_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/mnli-m_classifier_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/MNLI/test_nolabel_m.tsv \
--prediction_path datasets/MNLI/prediction_m.tsv \
--seq_length 128 --labels_num 3
python3 finetune/run_classifier.py --pretrained_model_path models/bert_base_uncased.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/MNLI/train.tsv \
--dev_path datasets/MNLI/dev_mm.tsv \
--output_model_path models/mnli_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/mnli-mm_classifier_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/MNLI/test_nolabel_mm.tsv \
--prediction_path datasets/MNLI/prediction_mm.tsv \
--seq_length 128 --labels_num 3
利用RoBERTa-Base在MNLI-m和MNLI-mm做微调和预测示例：
python3 finetune/run_classifier.py --pretrained_model_path models/roberta_base.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/MNLI/train.tsv \
--dev_path datasets/MNLI/dev_m.tsv \
--output_model_path models/mnli-m_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/mnli-m_classifier_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/MNLI/test_nolabel_m.tsv \
--prediction_path datasets/MNLI/prediction_m.tsv \
--seq_length 128 --labels_num 3
python3 finetune/run_classifier.py --pretrained_model_path models/roberta_base.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/MNLI/train.tsv \
--dev_path datasets/MNLI/dev_mm.tsv \
--output_model_path models/mnli_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/mnli-mm_classifier_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/MNLI/test_nolabel_mm.tsv \
--prediction_path datasets/MNLI/prediction_mm.tsv \
--seq_length 128 --labels_num 3
RoBERTa的论文中指出，MRPC、RTE和STS-B任务使用在MNLI-m上微调过的模型替换原始模型进行微调，可以取得更好的效果，我们沿用了这一设定。 由于MNLI是三分类任务，MRPC、RTE是二分类任务，STS-B是回归任务，因此在使用MNLI-m上微调过的模型前需要去除掉该模型的预测层。
import torch
input_model = torch.load("models/mnli-m_classifier_model.bin", map_location="cpu")
for key in list(input_model.keys()):
if "output_layer_2" in key:
del input_model[key]
torch.save(input_model, "models/mnli-m_classifier_del_model.bin")
利用在MNLI-m上微调过的BERT-Base-uncased模型，在MRPC数据集上做微调和预测示例：
python3 finetune/run_classifier.py --pretrained_model_path models/mnli-m_classifier_del_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/MRPC/train.tsv \
--dev_path datasets/MRPC/dev.tsv \
--output_model_path models/mrpc_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/mrpc_classifier_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/MRPC/test_nolabel.tsv \
--prediction_path datasets/MRPC/prediction.tsv \
--seq_length 128 --labels_num 2
利用在MNLI-m上微调过的RoBERTa-Base模型，在MRPC数据集上做微调和预测示例：
python3 finetune/run_classifier.py --pretrained_model_path models/mnli-m_classifier_del_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/MRPC/train.tsv \
--dev_path datasets/MRPC/dev.tsv \
--output_model_path models/mrpc_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/mrpc_classifier_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/MRPC/test_nolabel.tsv \
--prediction_path datasets/MRPC/prediction.tsv \
--seq_length 128 --labels_num 2
利用在MNLI-m上微调过的BERT-Base-uncased模型，在RTE数据集上做微调和预测示例：
python3 finetune/run_classifier.py --pretrained_model_path models/mnli-m_classifier_del_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/RTE/train.tsv \
--dev_path datasets/RTE/dev.tsv \
--output_model_path models/rte_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/rte_classifier_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/RTE/test_nolabel.tsv \
--prediction_path datasets/RTE/prediction.tsv \
--seq_length 128 --labels_num 2
利用在MNLI-m上微调过的RoBERTa-Base模型，在RTE数据集上做微调和预测示例：
python3 finetune/run_classifier.py --pretrained_model_path models/mnli-m_classifier_del_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/RTE/train.tsv \
--dev_path datasets/RTE/dev.tsv \
--output_model_path models/rte_classifier_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_classifier_infer.py --load_model_path models/rte_classifier_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/RTE/test_nolabel.tsv \
--prediction_path datasets/RTE/prediction.tsv \
--seq_length 128 --labels_num 2
利用在MNLI-m上微调过的BERT-Base-uncased模型，在STS-B数据集上做微调和预测示例：
python3 finetune/run_regression.py --pretrained_model_path models/mnli-m_classifier_del_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--train_path datasets/STS-B/train.tsv \
--dev_path datasets/STS-B/dev.tsv \
--output_model_path models/sts-b_regression_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_regression_infer.py --load_model_path models/sts-b_regression_model.bin \
--vocab_path models/google_uncased_en_vocab.txt \
--config_path models/bert/base_config.json \
--test_path datasets/STS-B/test_nolabel.tsv \
--prediction_path datasets/STS-B/prediction.tsv \
--seq_length 128
利用在MNLI-m上微调过的RoBERTa-Base模型，在STS-B数据集上做微调和预测示例：
python3 finetune/run_regression.py --pretrained_model_path models/mnli-m_classifier_del_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--train_path datasets/STS-B/train.tsv \
--dev_path datasets/STS-B/dev.tsv \
--output_model_path models/sts-b_regression_model.bin \
--epochs_num 3 --batch_size 32
python3 inference/run_regression_infer.py --load_model_path models/sts-b_regression_model.bin \
--vocab_path models/huggingface_gpt2_vocab.txt \
--merges_path models/huggingface_gpt2_merges.txt \
--tokenizer bpe \
--config_path models/xlm-roberta/base_config.json \
--test_path datasets/STS-B/test_nolabel.tsv \
--prediction_path datasets/STS-B/prediction.tsv \
--seq_length 128