Is there a way to get the Physical Batch Size from the REST API?

[teleprint-me]

Is there a way to get the Physical Batch Size from the REST API? This would be useful for chunking large inputs for downstream tasks.

[slaren]

I don't think so, changing the batch size requires creating a new llama_context.

Sorry, I misunderstood the question. I think you should look into the /props endpoint for that, but it does not seem to report the batch size at the moment.

[teleprint-me]

I couldn't find it in the README.md. I took a peek at the code endpoints (including props) and scanned the server source and didn't see it. I could have missed it.

I'm trying to do RAG with embeddings, but ran into issues once I saw the logs were producing errors and figured out that it was linked to the physical batch size.

It'd be nice to be able to know what the user set it to. I've been working around it by using an optional parameter for it instead.