The size of tensor a must match the size of tensor at non-singleton dimension

[atrdev-rgb]

Good afternoon. There was a problem when I send 16 curl requests at the same time the application crashes
with an error:

Error: The size of tensor a (99) must match the size of tensor b (3) at non-singleton dimension 3
The size of tensor a (11) must match the size of tensor b (3) at non-singleton dimension 3
The size of tensor a (27) must match the size of tensor b (3) at non-singleton dimension 3

As the error appears, no more requests are processed
however, if I run from in turn or use locker.acquire(). Then there are no problems.

Flask + Python 3.10

help

[jongwook]

Hi, it appears that you're calling the model from different threads. The model is not equipped for that, mainly because of the kv cache logic using the hooks. I'd suggest keep using the lock, if that's not too much of a slowdown.

[atrdev-rgb]

Thanks for the advice. But I have a server running, and I have to process audio and video files from two or more at the same time. Maybe there are some private solutions?

[Guneetsinghtuli]

@jongwook @atrdev-rgb Did you find a solution to this issue as lock and key concept is working fine but for many concurrent request the users have to wait for a long time not the best possible way to solve this.

[itsomar278]

did you find a solution ?