NVidia Model Testing

[BarbellDwarf]

So. I started testing out the new NVidia Parakeet and Canary models. And there are some issues. Here are the logs:

Canary Execution

INFO  00:54:30 GET /api/v1/transcription/list?page=1&limit=10 200 0.73ms

time=00:54:31 level=ERROR msg="Canary execution failed" output="Traceback (most recent call last):\n  File \"/app/whisperx-env/parakeet/canary_transcribe.py\", line 11, in <module>\n    import nemo.collections.asr as nemo_asr\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/collections/asr/__init__.py\", line 15, in <module>\n    from nemo.collections.asr import data, losses, models, modules\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/collections/asr/losses/__init__.py\", line 15, in <module>\n    from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/collections/asr/losses/angularloss.py\", line 18, in <module>\n    from nemo.core.classes import Loss, Typing, typecheck\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/core/__init__.py\", line 16, in <module>\n    from nemo.core.classes import *\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/core/classes/__init__.py\", line 33, in <module>\n    from nemo.core.classes.modelPT import ModelPT\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/core/classes/modelPT.py\", line 51, in <module>\n    from nemo.lightning.callback_group import CallbackGroup\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/__init__.py\", line 26, in <module>\n    from nemo.lightning.fabric.fabric import Fabric\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/fabric/fabric.py\", line 26, in <module>\n    from nemo.lightning.io.mixin import IOMixin, serialization, track_io\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/io/__init__.py\", line 2, in <module>\n    from nemo.lightning.io.api import export_ckpt, import_ckpt, load, load_context, model_exporter, model_importer\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/io/api.py\", line 22, in <module>\n    from nemo.lightning.io.pl import TrainerContext\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/io/pl.py\", line 25, in <module>\n    from megatron.core.dist_checkpointing.serialization import (\nModuleNotFoundError: No module named 'megatron'\n" error="exit status 1"

time=00:54:31 level="INFO " msg="Model processing completed" model_id=canary job_id=23167d1f-19fe-42ee-8d81-71001469f61d processing_time=6.595846541s

time=00:54:31 level="INFO " msg="Cleaned up temporary file" file=/app/data/uploads/23167d1f-19fe-42ee-8d81-71001469f61d_converted.wav

time=00:54:31 level=ERROR msg="Job processing failed" worker_id=0 job_id=23167d1f-19fe-42ee-8d81-71001469f61d error="single-track processing failed: transcription failed: Canary execution failed: exit status 1"

Parakeet Execution:

INFO  00:54:30 GET /api/v1/transcription/list?page=1&limit=10 200 0.73ms

time=00:54:31 level=ERROR msg="Canary execution failed" output="Traceback (most recent call last):\n  File \"/app/whisperx-env/parakeet/canary_transcribe.py\", line 11, in <module>\n    import nemo.collections.asr as nemo_asr\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/collections/asr/__init__.py\", line 15, in <module>\n    from nemo.collections.asr import data, losses, models, modules\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/collections/asr/losses/__init__.py\", line 15, in <module>\n    from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/collections/asr/losses/angularloss.py\", line 18, in <module>\n    from nemo.core.classes import Loss, Typing, typecheck\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/core/__init__.py\", line 16, in <module>\n    from nemo.core.classes import *\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/core/classes/__init__.py\", line 33, in <module>\n    from nemo.core.classes.modelPT import ModelPT\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/core/classes/modelPT.py\", line 51, in <module>\n    from nemo.lightning.callback_group import CallbackGroup\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/__init__.py\", line 26, in <module>\n    from nemo.lightning.fabric.fabric import Fabric\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/fabric/fabric.py\", line 26, in <module>\n    from nemo.lightning.io.mixin import IOMixin, serialization, track_io\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/io/__init__.py\", line 2, in <module>\n    from nemo.lightning.io.api import export_ckpt, import_ckpt, load, load_context, model_exporter, model_importer\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/io/api.py\", line 22, in <module>\n    from nemo.lightning.io.pl import TrainerContext\n  File \"/app/whisperx-env/parakeet/.venv/lib/python3.12/site-packages/nemo/lightning/io/pl.py\", line 25, in <module>\n    from megatron.core.dist_checkpointing.serialization import (\nModuleNotFoundError: No module named 'megatron'\n" error="exit status 1"

time=00:54:31 level="INFO " msg="Model processing completed" model_id=canary job_id=23167d1f-19fe-42ee-8d81-71001469f61d processing_time=6.595846541s

time=00:54:31 level="INFO " msg="Cleaned up temporary file" file=/app/data/uploads/23167d1f-19fe-42ee-8d81-71001469f61d_converted.wav

time=00:54:31 level=ERROR msg="Job processing failed" worker_id=0 job_id=23167d1f-19fe-42ee-8d81-71001469f61d error="single-track processing failed: transcription failed: Canary execution failed: exit status 1"

it looks to be the same error for both ModuleNotFoundError: No module named 'megatron, But wanted to get this up and ready before I begin troubleshooting.

[BarbellDwarf]

@rishikanthc, it just looks like some python modules are missing. I am adding them and will see if it works.

[BarbellDwarf]

So I manually installed the megatron module in the parakeet venv. And it works! Pretty well too. Although. I cannot run Canary at all. my system just runs out of memory. So we may need to give more options in the UI to help limit the memory usage if possible:

Also. The diarization using the nvidia transformers is just not working/not displaying correctly

@rishikanthc, I will try to get to work on the module install and getting the backend setup for accepting some inputs.

[BarbellDwarf]

As an aside. Large-v3 runs great. Very resource hungry though. To be expected.

[rishikanthc]

@BarbellDwarf eait until you try parakeet or canary. You will be mind blown. Both are way faster and better than whisper.

[BarbellDwarf]

Okay. Parakeet is insane how good and fast it is. I have still not been able to get Canary to work due to a memory issue. Even though I have 8GB of VRAM.

[chipwilson]

Looks like we are working in the same area - I am also getting the No module named 'megatron' error.

I'm trying to fix this by using a different base image in Dockerfile.cuda for the base cuda dependencies, from the NGC catalog. Takes a long time to download and rebuild but I'll update here if I find a solution.

@BarbellDwarf can you be a little more specific about how you manually installed the megatron module in the parakeet venv? Was this in the running container?

[BarbellDwarf]

That is correct. I just manually ran the install commands:
cd /app/whisperx-env/parakeet
then
uv pip install megatron

and then ran it again. It looks like there might be more to add to the install script. I also had to install megatron.core

[chipwilson]

The configuration is currently initializing 4 models in the same parakeet environment folder - is this intentional? I can see that the models are supposed to initialize in parallel, but uv is preventing that from happening in parallel because of the uv.lock file in the parakeet folder.

time=14:20:07 level="INFO " msg="Initializing registered models in parallel..."
time=14:20:07 level="INFO " msg="Preparing NVIDIA Sortformer environment" env_path=whisperx-env/parakeet
time=14:20:07 level="INFO " msg="Preparing NVIDIA Parakeet environment" env_path=whisperx-env/parakeet
time=14:20:07 level="INFO " msg="Preparing WhisperX environment" env_path=whisperx-env
time=14:20:07 level="INFO " msg="Preparing PyAnnote environment" env_path=whisperx-env/parakeet
time=14:20:07 level="INFO " msg="Preparing NVIDIA Canary environment" env_path=whisperx-env/parakeet

[chipwilson]

I see these comments now indicating it is intentional:

func NewPyAnnoteAdapter() *PyAnnoteAdapter {
	envPath := "whisperx-env/parakeet" // Shares environment with NVIDIA models

[chipwilson]

I was able to resolve most/many of the dependency issues by adding a few lines to internal/transcription/adapters/parakeet_adapter.go

dependencies = [
    "nemo-toolkit[asr]",
    "packaging",
    "Cython",
    "megatron",
    "megatron.core",
    "cuda-python>=12.3",

There does seem to be a memory problem after the dependencies are fixed, though - it seems that the current code tells cuda to load the entire file to be transcribed into memory - I validated this behavior by trying different sized audio files and the amount of VRAM requested grew with the size of the file. I'm not sure how to address this.

The error message says
Error during transcription: CUDA out of memory. Tried to allocate 3.56 GiB. GPU 0 has a total capacity of 23.56 GiB of which 1.70 GiB is free. Including non-PyTorch memory, this process has 21.85 GiB memory in use. Of the allocated memory 21.45 GiB is allocated by PyTorch, and 86.01 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables

It occurs to me that this is not a good starting point for VRAM utilization "Including non-PyTorch memory, this process has 21.85 GiB memory in use. Of the allocated memory 21.45 GiB is allocated by PyTorch, and 86.01 MiB is reserved by PyTorch but unallocated. "

In NVTop, I see the process quickly grab all the VRAM and then exit.

[rishikanthc]

@chipwilson the problem is if you chunk audio files then transcription accuracy would drop during the transitions between the audio. I can check if nvidia parakeet exposes a batch size parameter which I think it does. Then we can probably reduce batch size to tackle this.

[rishikanthc]

I tested it on my Mac cpu with 64gigs ram which is why I don't hit it I suppose. One other user was able to get parakeet working on gpu I think. Let me find that thread and link it so we can discuss there as well

[chipwilson]

I think there are capabilities of the nvidia container toolkit that are not being enabled. For example, I frequently work with an ollama server in a docker container behind an Open WebUI front end, and when I try to load something that won't fit into VRAM, such as gpt-oss:20b with a large context, the container toolkit splits the layers of the model between the GPU and the CPU, utilizing the CPU's RAM in addition to the GPU's VRAM. You can watch the model move back and forth between the CPU and the GPU using HTop and NVTop. I don't think this CPU Offloading is currently enabled by Scriberr.

[rishikanthc]

I'm aware of model splitting but unfortunately I don't think nvidia parakeet currently supports that. Here's the official documentation

https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2

Let me know if I'm missing something. If the model supports it I can definitely look into it.

[chipwilson]

• parakeet-sortformer and canary-sortformer are both working in-GPU for me for very short audio files.
• pyannote is complaining about missing pyannote.audio so that probably needs to be in the dependencies too.

[chipwilson]

After adding pyannote.audio to the dependencies in internal/transcription/adapters/parakeet_adapter.go, I now have parakeet-pyannote and canary-pyannote working in-GPU as well for very short audio files (a few seconds).