Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

训练初期显存异常大 #128

Open
TinaChen95 opened this issue Sep 22, 2023 · 2 comments
Open

训练初期显存异常大 #128

TinaChen95 opened this issue Sep 22, 2023 · 2 comments

Comments

@TinaChen95
Copy link

发现刚开始训练的时候，显存变化剧烈，且容易爆显存；过了一段时间之后，显存降下去且显存利用率比较低。
有大佬观察到这个现象吗？这是为什么呢？
@suzhenghang
Copy link

是因为音频时长不等吧。如果你的音频都是同等时长应该不会

@Isshang
Copy link

Isshang commented Jan 10, 2024

发现刚开始训练的时候，显存变化剧烈，且容易爆显存；过了一段时间之后，显存降下去且显存利用率比较低。 有大佬观察到这个现象吗？这是为什么呢？

佬，麻烦可以用一下你的训练log吗？条件限制训练不了，但是需要需要一些log图。可以的话，麻烦[email protected]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@suzhenghang @TinaChen95 @Isshang