You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It would be great to be able to natively use model providers other than OpenAI for the LLMEvaluator and related evaluators i.e. any that make use of LLMs.
Describe the solution you'd like
Not sure of design but maybe the generator could be passed in?
Describe alternatives you've considered
Just swapping out the generator can work for some of the models but is a bit hacky / not guaranteed to work. But it does mean that users, if they really want to, do have this option right now.
fromtypingimportListfromhaystack.components.evaluatorsimportLLMEvaluatorfromhaystack_integrations.components.generators.amazon_bedrockimportAmazonBedrockGeneratorclaude_generator=AmazonBedrockGenerator(
model="anthropic.claude-3-5-sonnet-20241022-v2:0",
max_length=600,
)
# To prevent the Evaluator from complainingos.environ['OPENAI_API_KEY'] =''llm_evaluator=LLMEvaluator(
instructions="Respond with valid json. Is this answer problematic for children?",
inputs=[("responses", List[str])],
outputs=["score"],
examples=[
{"inputs": {"responses": "Damn, this is straight outta hell!!!"}, "outputs": {"score": 1}},
{"inputs": {"responses": "Football is the most popular sport."}, "outputs": {"score": 0}},
],
)
# Swap out the generatorllm_evaluator.generator=claude_generatorresponses= [
"Football is the most popular sport with around 4 billion followers worldwide",
"Python language was created by Guido van Rossum.",
]
results=llm_evaluator.run(responses=responses)
print(results)
# {'results': [{'score': 0}, {'score': 0}]}
The text was updated successfully, but these errors were encountered:
It would be great to be able to natively use model providers other than OpenAI for the
LLMEvaluator
and related evaluators i.e. any that make use of LLMs.Right now it's fixed to the OpenAIGenerator.
Describe the solution you'd like
Not sure of design but maybe the generator could be passed in?
Describe alternatives you've considered
Just swapping out the generator can work for some of the models but is a bit hacky / not guaranteed to work. But it does mean that users, if they really want to, do have this option right now.
The text was updated successfully, but these errors were encountered: