Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mindyolo FAQ #325

Open
Ash-Lee233 opened this issue Jul 25, 2024 · 2 comments
Open

mindyolo FAQ #325

Ash-Lee233 opened this issue Jul 25, 2024 · 2 comments

Comments

@Ash-Lee233
Copy link
Collaborator

Here is a summary of common training and inference issues.

@Ash-Lee233 Ash-Lee233 pinned this issue Jul 25, 2024
@WongGawa
Copy link
Collaborator

WongGawa commented Jul 25, 2024

When you are training with yolo on Ascend910,you may encounter the following similar problem: the loss suddenly increases.
Solution: Remove the overflow_still_update: False parameter in yolo‘s yaml configuration and re-execute training.
e.g. https://github.com/mindspore-lab/mindyolo/blob/master/configs/yolov8/yolov8s.yaml/#L7

当你在Ascend910上使用yolo全量训练时可能会遇到以下类似问题:训练过程中loss会突然增加。
解决方法:删除yolo的yaml里的overflow_still_update: False参数配置,然后重新执行全量训练。
e.g. https://github.com/mindspore-lab/mindyolo/blob/master/configs/yolov8/yolov8s.yaml/#L7

@WongGawa
Copy link
Collaborator

When you are training a model on 2 NPU,you may encounter the following similar problem: compared with the benchmark, the performance results are poor.
Solution: When using mpirun command to train a model, add the --bind-to numa parameter to improve performance. For example:

  mpirun --allow-run-as-root -n 2 --bind-to numa python train.py --config ./configs/yolov7/yolov7.yaml  --is_parallel True

如果您在双卡设备上使用mpirun进行分布式模型训练时,可能会遇到以下类似问题:相比较benchmark,性能结果较差。
解决方法:使用mpirun指令启动时,请添加--bind-to numa参数以提升性能表现。例如:

  mpirun --allow-run-as-root -n 2 --bind-to numa python train.py --config ./configs/yolov7/yolov7.yaml  --is_parallel True

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants