about diffVC on Mandarin datasets

[Theweekfoolish229]

Hello, I adapted the diffvc code on Mandarin datasets. However, the audio after VC has the problem of tone sandhi. I want to ask the performance is normal ?

[Theweekfoolish229]

It's diffvc performance on Mandarin datasetshttps://www.yuque.com/qinjitao/te3orn/zfgbp9

[yaoxunji]

Hello, I adapted the diffvc code on Mandarin datasets. However, the audio after VC has the problem of tone sandhi. I want to ask the performance is normal ?

Hi, I meet the same problem, did you solve it？

[Theweekfoolish229]

I didn't solve the problem, but lost my job because of it. Finally, I chose this solution: ppg+speaker_ id+decoder ----- 原始邮件 ----- 发件人：落日的悲哀 ***@***.***> 收件人：huawei-noah/Speech-Backbones ***@***.***> 抄送人：Theweekfoolish229 ***@***.***>, Author ***@***.***> 主题：Re: [huawei-noah/Speech-Backbones] about diffVC on Mandarin datasets (Issue #24) 日期：2023年03月22日 16点07分 Hello, I adapted the diffvc code on Mandarin datasets. However, the audio after VC has the problem of tone sandhi. I want to ask the performance is normal ? Hi, I meet the same problem, did you solve it？ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[li1jkdaw]

Hi, @Theweekfoolish229 !

Sorry to hear about that. Actually, we tested our voice conversion model only on the English dataset, and even for this language, despite the overall good quality, there were some problems with the source prosody preservation (including mispronunciation issues as discussed in Section 4.2 of our paper). This is because our DiffVC model has only the so-called "average mel-spectrograms" as the information about the content of the source utterance. For tone languages like Mandarin it may be insufficient.

However, in our recent paper we proposed a sampling method based on the optimal transport property that keeps source prosody better compared to vanilla sampling from a diffusion model. Although we didn't test it on Mandarin, we suppose it should work better on it as well and help to reduce the problem with tones. I hope that in the nearest future we'll add the sampling method described in the mentioned paper to our repo.