Output probabilities

[dnhkng]

The library CAPPR allows the probabilities for each completion to be calculated. Although these are not usable as a measure of certainty in LLMs, I have a strong need to get access to them.

Currently, we can use this code to get categorical answers:

import outlines

model = outlines.models.transformers("mistralai/Mistral-7B-v0.1")

prompt = """You are a sentiment-labelling assistant.
Is the following review positive or negative?

Review: This restaurant is just awesome!
"""

generator = outlines.generate.choice(model, ["Positive", "Negative"])
answer = generator(prompt)

print(answer) # Positive

I want this "probability" method:

import outlines

model = outlines.models.transformers("mistralai/Mistral-7B-v0.1")

prompt = """You are a sentiment-labelling assistant.
Is the following review positive or negative?

Review: This restaurant is just awesome!
"""

generator = outlines.generate.probability(model, ["Positive", "Negative"])
answer = generator(prompt)

print(answer) # [0.8. 0.2]

[rlouf]

Possible duplicate of #479

[dnhkng]

@rlouf Point me to the right place in the code, and I'll try to implement it.

[dnhkng]

Yes, its a dupe, but rather than modifying "choice", I would like a new method. But I'll close this issue.