Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expanded Model Support in LLMEvaluator #8918

Open
bglearning opened this issue Feb 25, 2025 · 0 comments
Open

Expanded Model Support in LLMEvaluator #8918

bglearning opened this issue Feb 25, 2025 · 0 comments

Comments

@bglearning
Copy link
Contributor

bglearning commented Feb 25, 2025

It would be great to be able to natively use model providers other than OpenAI for the LLMEvaluator and related evaluators i.e. any that make use of LLMs.

Right now it's fixed to the OpenAIGenerator.

Describe the solution you'd like
Not sure of design but maybe the generator could be passed in?

Describe alternatives you've considered

Just swapping out the generator can work for some of the models but is a bit hacky / not guaranteed to work. But it does mean that users, if they really want to, do have this option right now.

from typing import List
from haystack.components.evaluators import LLMEvaluator

from haystack_integrations.components.generators.amazon_bedrock import AmazonBedrockGenerator

claude_generator = AmazonBedrockGenerator(
    model="anthropic.claude-3-5-sonnet-20241022-v2:0",
    max_length=600,
)

# To prevent the Evaluator from complaining
os.environ['OPENAI_API_KEY'] = ''

llm_evaluator = LLMEvaluator(
    instructions="Respond with valid json. Is this answer problematic for children?",
    inputs=[("responses", List[str])],
    outputs=["score"],
    examples=[
        {"inputs": {"responses": "Damn, this is straight outta hell!!!"}, "outputs": {"score": 1}},
        {"inputs": {"responses": "Football is the most popular sport."}, "outputs": {"score": 0}},
    ],
)

# Swap out the generator
llm_evaluator.generator = claude_generator

responses = [
    "Football is the most popular sport with around 4 billion followers worldwide",
    "Python language was created by Guido van Rossum.",
]
results = llm_evaluator.run(responses=responses)
print(results)
# {'results': [{'score': 0}, {'score': 0}]}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant