Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ellipsis] LLMBasedFaithfulness does not work when not using LLMs other than OpenAI #70

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ellipsis-dev[bot]
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot commented Jun 11, 2024

Summary:

⚠️ We couldn't build/test your project to verify our changes. Add a Dockerfile to significantly improve code quality.

Issue: resolves #69

Implementation:

Step 1: Modify LLMBasedFaithfulness class in continuous_eval/metrics/generation/text/llm_based.py

In the __call__ method of LLMBasedFaithfulness class, modify the instantiation of LLMBasedContextCoverage to pass the model parameter. The model parameter should be self.model if it is not None, otherwise it should be None. The modified code should look like this:

model = self.model if self.model is not None else None
context_coverage = LLMBasedContextCoverage(model=model, use_few_shot=self.use_few_shot)

Report:

Step Result Details
1
Modify LLMBasedFaithfulness class in continuous_eval/metrics/generation/text/llm_based.pyModified the call method of the LLMBasedFaithfulness class in the file continuous_eval/metrics/generation/text/llm_based.py to pass the model parameter to the LLMBasedContextCoverage class.

Something look wrong?: If this Pull Request doesn't contain the expected changes, add more information to #69. Then, try again. For more information, check the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LLMBasedFaithfulness does not work when using LLMs other than OpenAI
0 participants