Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify batch size for MME #134

Open
austinmw opened this issue Nov 23, 2022 · 0 comments
Open

Specify batch size for MME #134

austinmw opened this issue Nov 23, 2022 · 0 comments

Comments

@austinmw
Copy link

What did you find confusing? Please describe.
How do you specify batch size for MME models?

Describe how documentation can be improved
This blog describes using env vars to set batch size and other parameters for a single-model endpoint, however, I haven't found any documentation on setting batch size for individual models within a MME.

Additional context
Each model in my MME has a MAR-INF/MANIFEST.json within its model.tar.gz, so I tried to specify batchSize in these files, but I don't think it's being applied.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant