Suggestions for prompts/prefix to include repetitions and false starts

[praat-enthusiast]

Hi everyone!

I'm using Whisper on a research project where we're hoping to use it as a first step in transcribing data verbatim according to a strict transcription protocol. This data is real, spontaneous speech rather than subtitles for a TV show or similar, so there's a lot of hesitations, filler words, repetitions of words, etc, and we want these transcribed. I've managed to write a prompt to include 'um's and 'uh's, but I was wondering if anyone has previously managed to get Whisper to successfully transcribe false starts and repetitions of words using prompts or any other features/settings?

Our current prompt is something like this:

Okay, uh, there's a boy who's um playing football and two two trees. And uh it's it's red.

This has helped a lot, but it doesn't catch all the repetitions, and doesn't do anything for false starts. Ideally, we want to end up with transcripts that include repetitions, as well as false starts in brackets followed by an underscore, so that the transcript look a bit like this:

(Th_) there's a there's a cat
So so so the boy is playing (f_) football
No we we didn't have (any_) anything like that in our school

Has anyone had any success with getting Whisper to do anything like this, or have any suggestions for what I could try? I've also tried a prompt modelled off the above examples with includes things like '(th_) there's a cat', but unfortunately this doesn't work at all, as well as having the completely unintended effect of swear words being censored with underscores.

Any help or thoughts would be much appreciated - thanks in advance!

[birknu75]

Hi Joe,
I had the same question like you and tried the same by giving whisper some prompts, but it does not work. I am particularly interested in hesitations etc, so I need spontaneous speech to be transcribed and aligned as accurate as possible. The best way to do this (for large amount of recordings you do not want to do all manually), at the moment, as far as I know, is to have whisperx do the transcription, then correct the output by filling in the hesitations etc and then use webmouse to get the oneset/offset marking (alignment). That is still a lot of work if you have about 1500 recordings.
I was wondering, did you work out something better in the meantime? I understood that whisper includes an algorithm that is ignoring hesitations etc (for subtitle purposes).

Many thanks in advance!