Expose ignore_eos_token on /generate and/or /v1/completions endpoints

[martinbomio]

Feature request

Hi! Seems like internally, the lorax gRPC server allows setting ignore_eos_token, but there's no way to set that param in the /generate endpoint of the router.

Motivation

This parameter is useful for benchmarking and being able to consistently and correctly compare runs.

Your contribution

Would need some guidance but could contribute it

[tgaddair]

Hey @martinbomio, thanks for the feature request! This should be a pretty simple one to add. I can definitely take a look soon if no one else picks it up first.

[martinbomio]

@tgaddair @jeffreyftang thanks for the quick addition! When could we expect to have this released? (we are using the docker image right now)

[tgaddair]

Hey @martinbomio, this should be available in the latest docker image to try out now :)

Let me know if you need a new version of the Python client to be pushed to PyPI.

[martinbomio]

@tgaddair that's great!

I tested passing ignore_eos_token to the openai compatible api /v1/completions and I still do not get the expected output tokens on some prompts. Is this expected? I can always switch to the /generate one and try

[tgaddair]

Hey @martinbomio , I think the current implementation didn't update the OpenAI endpoints as this param isn't native to their spec.

@jeffreyftang what do you think about adding this param to the OpenAI endpoints as well?

[jeffreyftang]

I think the current implementation didn't update the OpenAI endpoints as this param isn't native to their spec.

Yep, that was my reasoning. However, happy to add to the OpenAI endpoints as well so long as we're not concerned with going beyond the OpenAI spec.

[martinbomio]

@jeffreyftang thanks for the quick PR! Would you mind sharing which base model you used for the example you posted in this PR #340? I am trying the latest version and I can't seem get the max tokens on certain prompts even if the ignore_eos_token is set

[martinbomio]

nvm, I can confirm it is working as expected!

[jeffreyftang]

Glad it's working! For posterity, I was using mistralai/Mistral-7B-Instruct-v0.1.