Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track and format chat history #4

Open
rlouf opened this issue Jul 26, 2024 · 0 comments · May be fixed by #9
Open

Track and format chat history #4

rlouf opened this issue Jul 26, 2024 · 0 comments · May be fixed by #9
Labels
enhancement New feature or request

Comments

@rlouf
Copy link
Member

rlouf commented Jul 26, 2024

The chat format consists in multi-turn conversations between the user and the large language model. Each turn is associated with a role, either system, user or assistant. A chat history can thus be represented as an iterable of tuples:

history = [
    ("system", "This is a system message"),
    ("user", "A user asking a question to the model"),
    ("assistant", "A model responding to the user")
]

We need a function which, given a model, will return the corresponding prompt in the correct format. For instance, for Zephyr the corresponding prompt should be:

<|system|>
This is a system message</s> 
<|user|>
A user asking a question to the model</s> 
<|assistant|>
A model responding to the user.

The simplest way to go about this is to define a function which, given a model name and a sequence of interactions returns the corresponding prompt.

from prompts import to_chat_format

history = [
    ("system", "This is a system message"),
    ("user", "A user asking a question to the model"),
]
prompt = to_chat_format("HuggingFaceH4/zephyr-7b-beta", history)
print(prompt)
# <|system|>
# This is a system message</s> 
# <|user|>
# A user asking a question to the model</s> 

However, when we want to pass the prompt to the model for generation we need to append the <|assistant|> token. For this we can define a new function, to_chat_prompt:

from prompts import to_chat_prompt

history = [
    ("system", "This is a system message"),
    ("user", "A user asking a question to the model"),
]
prompt = to_chat_prompt("HuggingFaceH4/zephyr-7b-beta", history)
print(prompt)
# <|system|>
# This is a system message</s> 
# <|user|>
# A user asking a question to the model</s> 
# <|assistant|>

This system is still cumbersome, since we need to append </s> to the response, and need to call to_chat_format with the model name before every call to the model. Could we simplify this?

class Chat:
    history: List[Tuple[str, str]]

    def __init__(self, model_name: str, system: str=None, tools=None, documents=None):
        raise NotImplementedError

    def __str__(self):
        """Render the prompt corresponding to the history.

        If we want to be able to use the prompt with any library this should return the prompt with the "assistant" keyword?

        As with `Template` we need to
be able to specify the model we are working with.

        """
        raise NotImplementedError

    def __getitem__(self, i):
        return self.history[i]

    def user(self, string: str):
        """Add user input to history"""
        self.history.append(("user", string))

    def assistant(self, string: str):
        """Add user input to history"""
        self.history.append(("user", string))

So a session would look like:

from outlines import generate, models


model = models.transformers("HuggingFaceH4/zephyr-7b-beta")

session = Chat("HuggingFaceH4/zephyr-7b-beta", "This is a system message")
session.user("A user asking a question to the model")

result = generate.text(model)(session)
session.assistant(result)

The obvious downside of this design is the necessity to specify the model name twice.

str(chat) must return the correctly formatted templates to be able to interface with any library

This issue (and arguably the repository) was triggered by this issue in Outlines and inspired by this repository opened by a member of the Outlines community.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant