NotImplementedError #25

masir110 · 2023-06-08T07:09:01Z

Is your feature request related to a problem?

2023-06-08 14:43:53,296 - INFO - modelscope - Checkpoints will be saved to NER/koubei/230608144347.471030
2023-06-08 14:43:53,296 - INFO - adaseq.training.hooks.text_logger_hook - Text logs will be saved to: NER/koubei/230608144347.471030/metrics.json
2023-06-08 14:43:53,369 - INFO - modelscope - tensorboard files will be saved to NER/koubei/230608144347.471030/tensorboard_output
/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/adaseq/modules/decoders/crf.py:293: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:402.)
score = torch.where(mask[i].unsqueeze(1), next_score, score)
2023-06-08 14:44:00,504 - INFO - modelscope - epoch [1][50/478] lr: 5.000e-05, eta: 0:05:33, iter_time: 0.143, data_load_time: 0.013, memory: 2636, loss: 24.8958
2023-06-08 14:44:05,806 - INFO - modelscope - epoch [1][100/478] lr: 5.000e-05, eta: 0:04:44, iter_time: 0.106, data_load_time: 0.011, memory: 2689, loss: 6.3790
2023-06-08 14:44:11,301 - INFO - modelscope - epoch [1][150/478] lr: 5.000e-05, eta: 0:04:27, iter_time: 0.110, data_load_time: 0.012, memory: 2689, loss: 4.3439
2023-06-08 14:44:16,351 - INFO - modelscope - epoch [1][200/478] lr: 5.000e-05, eta: 0:04:11, iter_time: 0.101, data_load_time: 0.012, memory: 2689, loss: 3.1956
2023-06-08 14:44:22,130 - INFO - modelscope - epoch [1][250/478] lr: 5.000e-05, eta: 0:04:06, iter_time: 0.116, data_load_time: 0.014, memory: 2800, loss: 3.9149
2023-06-08 14:44:28,207 - INFO - modelscope - epoch [1][300/478] lr: 5.000e-05, eta: 0:04:02, iter_time: 0.122, data_load_time: 0.015, memory: 2800, loss: 3.5845
2023-06-08 14:44:33,858 - INFO - modelscope - epoch [1][350/478] lr: 5.000e-05, eta: 0:03:55, iter_time: 0.113, data_load_time: 0.014, memory: 2800, loss: 2.7284
2023-06-08 14:44:38,887 - INFO - modelscope - epoch [1][400/478] lr: 5.000e-05, eta: 0:03:46, iter_time: 0.101, data_load_time: 0.012, memory: 2800, loss: 2.6073
2023-06-08 14:44:44,202 - INFO - modelscope - epoch [1][450/478] lr: 5.000e-05, eta: 0:03:39, iter_time: 0.106, data_load_time: 0.012, memory: 2800, loss: 2.7159
Total test samples: 0%|▏ | 1/463 [00:00<00:46, 9.89it/sTotal test samples: 4%|███▏ | 17/463 [00:00<00:14, 30.77iTotal test samples: 11%|█████████▎ | 49/463 [00:00<00:04, Total test samples: 17%|███████████████▏ | 81/463 [00:00<00Total test samples: 24%|████████████████████▉ | 113/463 [00:Total test samples: 31%|██████████████████████████▉ | 145/46Total test samples: 38%|████████████████████████████████▉ | Total test samples: 45%|██████████████████████████████████████▊ Total test samples: 52%|████████████████████████████████████████████▊ Total test samples: 62%|█████████████████████████████████████████████████████▋ Total test samples: 69%|███████████████████████████████████████████████████████████▌ Total test samples: 76%|████████████████████████████████████████████████████████████Total test samples: 87%|████████████████████████████████████████████████████████████Total test samples: 97%|████████████████████████████████████████████████████████████Total test samples: 100%|██████████████████████████████████████████████████████████████████████████████████████| 463/463 [00:02<00:00, 227.71it/s]
Traceback (most recent call last):
File "/data/ningfeim/.myconda/envs/modelscope/bin/adaseq", line 8, in
sys.exit(run())
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/adaseq/main.py", line 13, in run
main(prog='adaseq')
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/adaseq/commands/init.py", line 29, in main
args.func(args)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/adaseq/commands/train.py", line 85, in train_model_from_args
checkpoint_path=args.checkpoint_path,
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/adaseq/commands/train.py", line 144, in train_model
trainer.train(checkpoint_path)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/adaseq/training/default_trainer.py", line 146, in train
return super().train(checkpoint_path=checkpoint_path, *args, **kwargs)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/modelscope/trainers/trainer.py", line 676, in train
self.train_loop(self.train_dataloader)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/modelscope/trainers/trainer.py", line 1181, in train_loop
self.invoke_hook(TrainerStages.after_train_epoch)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/modelscope/trainers/trainer.py", line 1328, in invoke_hook
getattr(hook, fn_name)(self)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/modelscope/trainers/hooks/evaluation_hook.py", line 35, in after_train_epoch
self.do_evaluate(trainer)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/modelscope/trainers/hooks/evaluation_hook.py", line 47, in do_evaluate
eval_res = trainer.evaluate()
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/modelscope/trainers/trainer.py", line 764, in evaluate
metric_classes)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/modelscope/trainers/trainer.py", line 1239, in evaluation_loop
data_loader_iters=self._eval_iters_per_epoch)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/modelscope/trainers/utils/inference.py", line 77, in single_gpu_test
return get_metric_values(metric_classes)
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/modelscope/trainers/utils/inference.py", line 194, in get_metric_values
metric_values.update(metric_cls.evaluate())
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/adaseq/data/dataset_dumpers/base.py", line 26, in evaluate
self.dump()
File "/data/ningfeim/.myconda/envs/modelscope/lib/python3.7/site-packages/adaseq/data/dataset_dumpers/named_entity_recognition_dataset_dumper.py", line 40, in dump
raise NotImplementedError
NotImplementedError

环境：
python3.7
ubuntu 22.04
cuda11.7

Describe the solution you'd like.

最后一行依赖对应报错行：

这块是在判断训练集类型conll

yaml训练内如下：
experiment:
exp_dir: NER/ # 所有实验的根文件夹
exp_name: koubei # 本次配置文件的实验名称
seed: 42 # 随机种子

dataset:
data_file: # 数据文件
train: '/data/ningfeim/test/NER/koubei/datas/train.txt'
dev: '/data/ningfeim/test/NER/koubei/datas/dev.txt'
test: '/data/ningfeim/test/NER/koubei/datas/test.txt'
data_type: conll # 数据格式

task: named-entity-recognition # 任务名称，用于加载内建的 DatasetBuilder（如果需要的话）

preprocessor:
type: sequence-labeling-preprocessor # 预处理器名称
model_dir: /data/ningfeim/project/Bert-ner-yiwei/bert #huggingface/modelscope 模型名字或路径，用于初始化 Tokenizer，可缺省
max_length: 512 # 预训练模型支持的最大输入长度

data_collator: SequenceLabelingDataCollatorWithPadding # 用于 batch 转换的 data_collator 名称

model:
type: sequence-labeling-model # 模型名称
embedder:
model_name_or_path: /data/ningfeim/test/NER/koubei/nlp_raner_named-entity-recognition_chinese-base-news # 预训练模型名称或路径，可以是huggingface/modelscope的backbone模型，或者也可以加载modelscope上的任务模型
dropout: 0.1 # dropout 概率
use_crf: true # 是否使用CRF

train:
hooks:
- type: TensorboardHook
max_epochs: 5 # 最大训练轮数
dataloader:
batch_size_per_gpu: 8 # 训练batch_size
optimizer:
type: AdamW # pytorch 优化器名称
lr: 5.0e-5 # 全局学习率
param_groups:
- regex: crf
lr: 5.0e-1
lr_scheduler:
type: LinearLR # transformers 或 pytorch 的 lr_scheduler 名称
start_factor: 1.0
end_factor: 0.0
total_iters: 20

evaluation:
dataloader:
batch_size_per_gpu: 16 # 评估batch_size
metrics:
- type: ner-metric # 所有已实现的metric见 adaseq/metainfo.py 的 Metrics 类。
- type: ner-dumper # 输出预测结果
model_type: sequence_labeling
dump_format: column

Describe alternatives you've considered.

No response

Additional context.

No response

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

masir110 · 2023-06-08T07:10:32Z

请问这是什么问题呢，如有相关讨论社区麻烦也给发一下

wanshichenguang · 2024-01-12T03:21:49Z

它的配置文件不能照抄，这里说dump_format只实现了两种格式的输出：conll和jsonline，，给的配置文件里面是column

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NotImplementedError #25

NotImplementedError #25

masir110 commented Jun 8, 2023

masir110 commented Jun 8, 2023

wanshichenguang commented Jan 12, 2024

NotImplementedError #25

NotImplementedError #25

Comments

masir110 commented Jun 8, 2023

Is your feature request related to a problem?

Name Version Build Channel

Describe the solution you'd like.

Describe alternatives you've considered.

Additional context.

Code of Conduct

masir110 commented Jun 8, 2023

wanshichenguang commented Jan 12, 2024