Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this PR I introduce an
outlines
command-line interface which allows users to serve locally JSON-structured generation workflows. The workflows consists in a prompt template, a LLM and a Pydantic model. The API's parameters are the prompt template's arguments and it returns a JSON object that respect the JSON Schema implicitly defined by the Pydantic model.The use of the CLI is as follows:
I am still not sure whether serving should happen via llama.cpp or vLLM. The interface to define the API using Outlines is also not completely defined:
It should also be possible to call the thus defined function from another script.