New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] batch similar requests for NLP tasks #277

Closed

kemingy opened this issue Feb 17, 2023 · 1 comment

Labels

Member

kemingy commented Feb 17, 2023

Is your feature request related to a problem? Please describe.

NLP tasks usually require padding. Thus batching the requests with similar lengths might also help.

Describe the solution you'd like

Since we don't parse the request body on the Rust side, this feature may require adding some special HTTP request headers.

kemingy added the enhancement label

Member Author

kemingy commented Jun 14, 2023

refer to #383

kemingy closed this as completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment