Multiple Embedding-Models within one server instance

[ChristophRaab]

Hi,

i want to host multiple embeddings models, serving different purposes (fine tuned on mutual exclusive data). I was wondering, although i have not found anything in the docs, if it is possible to run multiple embedding models at the same time within one inference server. Currently, one would start an instance of an inference server for every model right?

Is something possible or planned?

Thanks in advance!

Best
Chris

[OlivierDehaene]

Hello!

No, that's not something that is possible/planned.
It's completely fine to launch multiple instances of TEI on the same GPU. You might need to limit the number of tokenization workers though as we spawn one per cpu core by default (arg --tokenization-workers or TOKENIZATION_WORKERS env var).

The reason we do not plan to add such a feature is because we feel that other parts of the stack already cover it (for example MIG).