Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When I set local ollama , but still need the Open api key #1243

Open
lijinxinKing opened this issue Dec 31, 2024 · 1 comment
Open

When I set local ollama , but still need the Open api key #1243

lijinxinKing opened this issue Dec 31, 2024 · 1 comment

Comments

@lijinxinKing
Copy link

lijinxinKing commented Dec 31, 2024

❗BEFORE YOU BEGIN❗
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

Describe the bug
A clear and concise description of what the bug is.

when set the local succeed

image

But when use the DeepEval test , still the OpenAI Key ?

image

`import pytest
from deepeval.test_case import LLMTestCase
from deepeval.metrics.ragas import (
RagasMetric,
RAGASContextualPrecisionMetric,
RAGASContextualRecallMetric,
RAGASContextualEntitiesRecall,
RAGASAnswerRelevancyMetric,
RAGASFaithfulnessMetric,
)
from deepeval import assert_test
from langchain_openai import OpenAIEmbeddings, ChatOpenAI

query = "Who won the FIFA World Cup in 2018 and what was the score?"
output = (
"The winners of the FIFA World Cup in 2018 were the French national football team. "
"They defeated the Croatian national football team with a score of 4-2 in the final match. "
"The final was held at Luzhniki Stadium in Moscow, Russia on July 15, 2018. "
"The French team played exceptionally well, with goals scored by Mario Mandzukic (own goal), Antoine Griezmann, Paul Pogba, and Kylian Mbappe. "
"Croatia fought valiantly, with goals by Ivan Perisic and Mario Mandzukic, but ultimately could not overcome France's lead. "
"This victory marked France's second World Cup title, with their first win being in 1998 when they hosted the tournament."
)
expected_output = "French national football team"
context = [
"The FIFA World Cup in 2018 was won by the French national football team.",
"I am birdy.",
"I am a froggy.",
"The French defeated Croatia 4-2 in the final FIFA match to claim the championship.",
"The final match took place on July 15, 2018, in Moscow, Russia at Luzhniki Stadium.",
]

def test_everything():
# embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
test_case = LLMTestCase(
input=query,
actual_output=output,
expected_output=expected_output,
retrieval_context=context,
context=context,
)
model_local = ChatOpenAI(model="llama3.2:3b", openai_api_key="ollama", openai_api_base="http://localhost:11434/v1/")

contextual_precision = RAGASContextualPrecisionMetric(model=model_local)
context_recall = RAGASContextualRecallMetric(model=model_local)
context_entities_recall = RAGASContextualEntitiesRecall(model=model_local)
# answer_relevancy = RAGASAnswerRelevancyMetric(embeddings=embeddings)
faithfulness = RAGASFaithfulnessMetric(model=model_local)
# 初始化 ChatOpenAI 模型
ragas = RagasMetric(
    model = model_local
)
assert_test(test_case, [ragas])

test_everything()
`

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@lijinxinKing lijinxinKing changed the title When I set local ollama , but need the The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable When I set local ollama , but still need the Open api key Dec 31, 2024
@penguine-ip
Copy link
Contributor

@lijinxinKing its because the ragas metrics don't natively support all types of LLMs and so you'll want to use Deepeval's rag metrics instead: https://docs.confident-ai.com/docs/metrics-ragas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants