Using WhisperKit Without Requesting Huggingface

[cunoe]

When developing with WhisperKit, we found that it defaults to connecting to HuggingFace's servers. This may cause issues such as inability to use in scenarios with poor network environments or limited network traffic.

To address this, I explored the related information provided by #81 and ultimately implemented the requirement.

Setup

WhisperKit requires loading two things when loading models: one is the model itself, such as openai_whisper-large-v3-v20240930_547MB, and the other is the tokenizer corresponding to the model. To achieve local loading, we need to provide both files simultaneously. Refer to the loading function below

func setupWhisper() async {
        do {
            await MainActor.run {
                loadingState = .loading
            }
            guard let modelPath = Bundle.main.url(forResource: "openai_whisper-large-v3-v20240930_547MB", withExtension: nil),
                  let tokenizerPath = Bundle.main.url(forResource: "tokenizerFolder", withExtension: nil) else {
                print("Failed to find model or tokenizer files")
                await MainActor.run {
                    loadingState = .notLoaded
                }
                return
            }

            whisperKit = try await WhisperKit(modelFolder: modelPath.path, tokenizerFolder: tokenizerPath)
            try await whisperKit?.loadModels(prewarmMode: true)

            await MainActor.run {
                loadingState = .loaded
                if let variant = whisperKit?.modelVariant {
                    currentModel = variant.description
                }
            }
            print("Whisper model loaded successfully: \(currentModel)")
        } catch {
            print("Whisper initialization failed: \(error)")
            await MainActor.run {
                loadingState = .notLoaded
            }
        }
    }

Note that you need to add the model file and tokenizer file to the bundle in the Copy Bundle Resources of Build Phases.

script

To facilitate model downloading and management, the following ci script was written

#!/bin/sh
# ci_scripts/ci_pre_xcodebuild.sh
echo "setup whisperkit"
export WHISPER_VARIANT=large-v3-v20240930_547MB
export TOKENIZER_VARIANT=openai/whisper-large-v3
git clone https://github.com/argmaxinc/whisperkit.git
cd whisperkit
make setup
make download-model MODEL=$WHISPER_VARIANT
mkdir -p tokenizerFolder/models/$TOKENIZER_VARIANT
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/$TOKENIZER_VARIANT tokenizerFolder/models/$TOKENIZER_VARIANT
echo "whisperkit setup done"

Where WHISPER_VARIANT can be found in the HuggingFace repository provided by WhisperKit, i.e., https://huggingface.co/argmaxinc/whisperkit-coreml

TOKENIZER_VARIANT needs to be confirmed by referring to the case in

WhisperKit/Sources/WhisperKit/Core/Utils.swift

Lines 366 to 394 in 0af7146

	func tokenizerNameForVariant(_ variant: ModelVariant) -> String {
	var tokenizerName: String
	switch variant {
	case .tiny:
	tokenizerName = "openai/whisper-tiny"
	case .tinyEn:
	tokenizerName = "openai/whisper-tiny.en"
	case .base:
	tokenizerName = "openai/whisper-base"
	case .baseEn:
	tokenizerName = "openai/whisper-base.en"
	case .small:
	tokenizerName = "openai/whisper-small"
	case .smallEn:
	tokenizerName = "openai/whisper-small.en"
	case .medium:
	tokenizerName = "openai/whisper-medium"
	case .mediumEn:
	tokenizerName = "openai/whisper-medium.en"
	case .large:
	tokenizerName = "openai/whisper-large"
	case .largev2:
	tokenizerName = "openai/whisper-large-v2"
	case .largev3:
	tokenizerName = "openai/whisper-large-v3"
	}
	return tokenizerName
	}

; its specific tokenizer is determined by the variant of the model. You can learn about different models' variants through whisperKit?.modelVariant after loading.

It's worth noting that when mounting tokenizerFolder, you must directly mount tokenizerFolder and preserve the file structure of tokenizerFolder/models/openai/whisper-large-v3.

[ZachNagengast]

Great example, thanks for sharing 💯 Open to any patches to make this more straightforward, offline access is a key focus for this repo.