Support for AMD GPUs

[rishikanthc]

This thread tracks the development of adding rocm support to run transcription on AMD GPUs

[BarbellDwarf]

I think I may have found the issue. I much have fat fingered it when I put it in:
Line 107-109 of whisperx.go

if device == "rocm" {
    device = "cuda"  
}

I just pushed a new commit to the branch. Let me know if it works.

[MadDogTen]

I updated my "working" branch with your fix and rebuilt, and again tested the same command with both 'cuda' and 'rocm'. 'cuda' was the same (RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version), and the 'rocm':
uv run whisperx "/app/data/uploads/9cd7adc9-7b6d-482d-9a35-59d64cd5e51c.webm" --device rocm

INFO:speechbrain.utils.quirks:Applied quirks (see `speechbrain.utils.quirks`): [allow_tf32, disable_jit_profiling]
INFO:speechbrain.utils.quirks:Excluded quirks specified by the `SB_DISABLE_QUIRKS` environment (comma-separated list): []
Traceback (most recent call last):
  File "/app/whisperx-env/WhisperX/.venv/bin/whisperx", line 10, in <module>
    sys.exit(cli())
             ^^^^^
  File "/app/whisperx-env/WhisperX/whisperx/__main__.py", line 85, in cli
    transcribe_task(args, parser)
  File "/app/whisperx-env/WhisperX/whisperx/transcribe.py", line 124, in transcribe_task
    model = load_model(
            ^^^^^^^^^^^
  File "/app/whisperx-env/WhisperX/whisperx/asr.py", line 336, in load_model
    model = model or WhisperModel(whisper_arch,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/whisperx-env/WhisperX/.venv/lib/python3.12/site-packages/faster_whisper/transcribe.py", line 647, in __init__
    self.model = ctranslate2.models.Whisper(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: unsupported device rocm

[MadDogTen]

Any progress on figuring out the issue, or just currently busy irl / with other stuff?

[BarbellDwarf]

@rishikanthc, I did implement that patched version of the ctranslate2 library (ctranslate2-rocm), but it did not seem to correct the issue.

But I see in @MadDogTen's logs, that it is not detecting the GPU correctly.

Also. @MadDogTen , those needed to be run on your system directly. That way the logic for detecting ROCM could be properly tested on something that we know has an AMD GPU.

I wish there was a way to rent a cheap AMD GPU for testing in the cloud somewhere. But, alas.

I will look at it some more today if time permits.

[BarbellDwarf]

@SpirusNox can we try to get the nvidia models working ? If we can support them I think that would be a good start. Anyways the nvidia models are superior to whisper in every way. Just a thought..

Are we talking about Nvidia transcription models? Or getting acceleration working on NVidia devices?

[MadDogTen]

@rishikanthc, I did implement that patched version of the ctranslate2 library (ctranslate2-rocm), but it did not seem to correct the issue.

But I see in @MadDogTen's logs, that it is not detecting the GPU correctly.

Also. @MadDogTen , those needed to be run on your system directly. That way the logic for detecting ROCM could be properly tested on something that we know has an AMD GPU.

I wish there was a way to rent a cheap AMD GPU for testing in the cloud somewhere. But, alas.

I will look at it some more today if time permits.

@SpirusNox
I'll run them soon, I planned on testing out ROCm7, and cleaned up what I had installed to start clean, Left for a bit, and when I came back to actually install it, openSUSE download repositories are down. ROCm is a bit of a pain to install on Tumbleweed as it is, So I'd rather wait for them to come back up.

I do want to note, That in the Original Issue, 'rocminfo' info did work (and showed the GPU).

Another example of a project that has use's whisper for transcription (I've tested it, it works on my system no issue) is Storyteller. Once again, I have no idea if their method will work in this case, but hopefully it can give you some ideas.

[rishikanthc]

@SpirusNox i meant getting nvidia models to work on amd.
the acceleration for cuda seems to be working now after I merged a contributors PR. So we can for now consider running on nvidia gpu kinda resolved. Would be great if you could test the latest update on main branch and build and test the image.

[BarbellDwarf]

@MadDogTen, I checked out a few of the other projects, and there is a difference. they are using Whisper.cpp, which is a different version of Whisper than what we are using. Which is WhisperX. That is why there are capabilities that are present there in that version that are not present here in this application/implementation of Whisper.

Although, let me look a this issue. I must have changed some of the logic incorrectly where it is not detecting the GPU anymore. I will look into that.

[BarbellDwarf]

@rishikanthc, I am going to need some more information on what you are stating as Nvidia Models. CUDA acceleration is only available to CUDA GPUs, which are only NVidia cards. CUDA acceleration will not work on AMD GPUs because they do not have CUDA cores to be able to perform the task.

AMD GPUs have ROCM as their acceleration abilities. Some people have basically patched in AMD GPU functionality, but they are basically translating the CUDA API calls into ROCM API calls to the GPU, and thus allowing the GPU to perform the conversion.

But we are using WhisperX, not Faster-Whisper, or Whisper.cpp, which currently does not seem to support AMD GPUs due to using a non ROCM supported version of cTranslate2.

If we were using Whisper.cpp instead of WhisperX, this would be a much easier implementation since it already supports ROCM via the Vulkan API that AMD GPUs support.

I am just trying to explain why this may not be able to work as we intended due to the complexity of these different versions of Whisper that have been created, which we are using one of.

[MadDogTen]

That I did not know, and is very unfortunate. Hopefully someone makes a way for NVIDIA NeMo to be used with ROCm at some point.

Regardless, Thanks for trying to figure things out.

[rishikanthc]

@BarbellDwarf hi ! So I recently added additional models for transcription and diarization. So you can now choose between whisper models or nvidia parakeet or nvidia canary for transcription and for diarization choose between pyannote and nvidia sortformer. The code has already been merged to the main branch.

When I meant nvidia models i was referring to the newly added models. What I was wondering is if these new nvidia models can be run on AMD gpus with rocm.

Hope that makes sense. Apologies for not providing full context.

[ndrewpj]

Any chance of this being done with Vulkan and not ROCm?

[BarbellDwarf]

@ndrewpj, This is not something that would be handled on the side of Scriberr. This would be something that would need to be handled by the developer of WhisperX. Moving to WhisperX was great as it made a slew of things easier to handle overall. But it has caused other issues like this to crop up. So its a bit of the "6 in one hand, half a dozen in the other" kind of deal. But I have been continuing to look into ways to get this to work. Because if ROCm works, then Vulkan almost assuredly will work. But so far, I have no found anything to be profitable.

But I will keep looking. And will update here once I am able to.

[BarbellDwarf]

@BarbellDwarf hi ! So I recently added additional models for transcription and diarization. So you can now choose between whisper models or nvidia parakeet or nvidia canary for transcription and for diarization choose between pyannote and nvidia sortformer. The code has already been merged to the main branch.

When I meant nvidia models i was referring to the newly added models. What I was wondering is if these new nvidia models can be run on AMD gpus with rocm.

Hope that makes sense. Apologies for not providing full context.

Gotcha! I will get those tested tomorrow if I get the chance, that should be a fairly simple test.