502 Bad Gateway Error When Using Whisper.php in Laravel with Herd

[mrfelipemartins]

Hi! First of all, thank you for the amazing work on this package!

I am integrating Whisper.php into my Laravel project to implement a transcription API endpoint. However, I am encountering a strange issue that results in a 502 Bad Gateway error when processing requests.

Debugging Steps Taken

1. Initially, the endpoint returned a 502 Bad Gateway response.
2. Upon checking the Laravel Herd Nginx error logs, I found the following error message:

[error] 77725#32945537: *3 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: *****.test, request: "POST /api/v1/audio/transcriptions HTTP/1.1",

3. To investigate further, I enabled logging in Whisper.php. However, the logs consistently stop at a specific point, and no further output or errors are shown.

Logs and Code

Whisper.php Logs:

[2024-12-17 14:00:03] local.INFO: whisper_init_from_file_with_params_no_state: loading model from '/Users/test/*****/*****/storage/app/models/ggml-base.bin'
  
[2024-12-17 14:00:03] local.INFO: whisper_init_with_params_no_state: use gpu    = 0
  
[2024-12-17 14:00:03] local.INFO: whisper_init_with_params_no_state: flash attn = 0
  
[2024-12-17 14:00:03] local.INFO: whisper_init_with_params_no_state: gpu_device = 0
  
[2024-12-17 14:00:03] local.INFO: whisper_init_with_params_no_state: dtw        = 0
  
[2024-12-17 14:00:03] local.INFO: whisper_init_with_params_no_state: devices    = 3
  
[2024-12-17 14:00:03] local.INFO: whisper_init_with_params_no_state: backends   = 3
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: loading model
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_vocab       = 51865
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_audio_ctx   = 1500
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_audio_state = 512
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_audio_head  = 8
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_audio_layer = 6
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_text_ctx    = 448
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_text_state  = 512
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_text_head   = 8
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_text_layer  = 6
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_mels        = 80
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: ftype         = 1
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: qntvr         = 0
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: type          = 2 (base)
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: adding 1608 extra tokens
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: n_langs       = 99
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load:      CPU total size =   147.37 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_model_load: model size    =  147.37 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_backend_init: using BLAS backend
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: kv self size  =    6.29 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: kv cross size =   18.87 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: kv pad  size  =    3.15 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: compute buffer (conv)   =   16.26 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: compute buffer (encode) =   85.86 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: compute buffer (cross)  =    4.65 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: compute buffer (decode) =   96.35 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_backend_init: using BLAS backend
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: kv self size  =    6.29 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: kv cross size =   18.87 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: kv pad  size  =    3.15 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: compute buffer (conv)   =   16.26 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: compute buffer (encode) =   85.86 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: compute buffer (cross)  =    4.65 MB
  
[2024-12-17 14:00:03] local.INFO: whisper_init_state: compute buffer (decode) =   96.35 MB
  

Current Implementation Code:

$file = $request->file('file')->store('audios');

Whisper::setLogger(Log::getLogger());

$whisper = Whisper::fromPretrained('base', baseDir: storage_path('app/models'));

$audio = readAudio(storage_path('app/private/' . $file));

$segments = $whisper->transcribe($audio, 4);

foreach ($segments as $segment) {
    dump(toTimestamp($segment->startTimestamp) . ': ' . $segment->text . "\n");
}

I also tried increasing php memory limits with no luck.

Environment Details:

• Laravel Version: v11.26.0
• PHP Version: 8.3
• Server: Laravel Herd
• System: Apple Silicon M2 Pro

[CodeWithKyrian]

From the logs, we can confirm that the model loaded successfully, so we can rule that out as the cause of the issue.

My best guess is that the problem lies with the readAudio function, and the application is running out of memory. Let me explain:

• The readAudio function samples the audio data at 16kHz and, in this initial version of the package, stores the sampled data in a PHP array.
• For longer audio files, there are far more samples to hold in memory. Because PHP imposes a default memory limit (typically 128MB), processing large audio files can easily exceed this limit, leading to errors.

To check if this is the issue, set the memory_limit to unlimited in your script: ini_set('memory_limit', '-1'); then run your script again and see if it works.

Once confirmed, you can adjust the memory_limit to a reasonable value that works for your use case, for example: ini_set('memory_limit', '1G');

In the future, I plan to store the sampled data in a C array or tensor instead of a PHP array to reduce the memory usage and improve performance.

Let me know if this solves the issue or if you need further assistance.

[mrfelipemartins]

I already tried increasing the PHP memory_limit to -1 as suggested, but unfortunately, it made no difference. The audio file I’m testing is very small (approximately 5 seconds), so I am not sure memory exhaustion is the cause here.

Additionally, the request fails very quickly, within 300ms, which seems too short for a memory-related issue to occur. This makes me suspect that the process might be exiting prematurely, possibly due to an unexpected crash or another error that isn’t being logged.

[CodeWithKyrian]

Oh. I see. So does the readAudio function complete successfully? Maybe dump out the count? I'm trying to figure out if the issue is from it, or perhaps from the transcribe method itself (though it should've shown in the logs if so)

[mrfelipemartins]

I can confirm the issue lies within the transcribe method. I dumped the output of the readAudio function, it completes successfully and I can see the generated array without any problems.

[CodeWithKyrian]

Okay. Just took a second look at the logs and realized that the transcribe method did run, just up to a point before it failed. Hold on

[mrfelipemartins]

Alright, I’ll dig into this further, but I suspect the issue is related to Laravel Herd. I tested the same endpoint using php artisan serve, and it worked perfectly as expected. This suggests that something in Herd’s configuration might be prematurely terminating the process or aborting the request.

I’ll continue debugging and investigate Herd’s behavior in more detail. I’ll report back with any findings or results.

[CodeWithKyrian]

I have a very similar setup—Apple Silicon M1, PHP 8.3 (via Herd), and Laravel 11—and I just tested it using artisan serve, and it worked perfectly. Thanks for mentioning the Herd stuff, cos I was already wondering what else. I'll continue testing on my end as well

[CodeWithKyrian]

Hey. I wanted to drop an update for you. So I’ve tracked down the issue, though I’m still working on a proper solution. It turns out the problem is related to Metal on Apple Silicon Macs. Even though Metal isn’t actively used unless you pass the useGpu option when creating a context, its libraries are still loaded.

When running through php artisan serve, the dependencies for Metal load correctly. However, when using Herd, something in the nginx and php-fpm configurations prevents certain dependencies from loading properly, causing the issue.

One workaround that I’ve tested is recompiling whisper.cpp and disabling Metal, which does solve the problem. But I’d prefer to dig deeper and fully resolve this without sacrificing GPU support on Mac.

I’ll keep you posted as I figure out more!

[mrfelipemartins]

Thanks for the update! That explanation makes a lot of sense, and I really appreciate you looking into this.

As a temporary workaround, I’ve been using the following snippet to get things working in Herd with Laravel Concurrency. While I know it’s not the ideal solution, it does the job for now. I opted to use concurrency with a single callback instead of defer, as I need to wait for the result:

[$result] = Concurrency::run([
    function() use ($responseFormat, $file) {
        Whisper::setLogger(Log::getLogger());

        $whisper = Whisper::fromPretrained('large-v3', baseDir: storage_path('app/models'));

        $audio = readAudio(storage_path('app/private/' . $file));

        $segmentsJson = [];

        return $whisper->transcribe($audio, 4);
    }
]);

This approach is far from perfect, but it’s working for me at the moment.

Thanks again for your efforts!