Finetuned large-v3 inference problem.

[sinisha]

I have finetuned large-v3 model on 1k hours of domain-specific data. When I run standard inference the results are ok.
Finetuned model is converted using ctranslate2 but the results obtained with whisperx are almost all hallucinations (maybe first couple of phonemes at the beginning are correct). I used same ctranslate2 command to convert original large-v3 model and whisperx inference is also correct.
Model is finetuned using Transformers 4.45.2. I have tried a couple of different Transformers version in inference and the results are similar.
Has anyone encountered similar problem?

[DaddyCodesAlot]

How long were your training samples? finetuning whisper is best when the samples are relatively short