Skip to content

Commit

Permalink
chore(llmobs): fix tests to be less noisy locally (#11890)
Browse files Browse the repository at this point in the history
These tests rely on env vars that may already be present on dev envs and
will result in noisy errors as a result.

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
  • Loading branch information
Yun-Kim authored Jan 10, 2025
1 parent 1b223aa commit 4beeccc
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 22 deletions.
12 changes: 2 additions & 10 deletions tests/llmobs/test_llmobs_evaluator_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,15 +64,7 @@ def test_evaluator_runner_on_exit(mock_writer_logs, run_python_code_in_subproces
pypath = [os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__))))]
if "PYTHONPATH" in env:
pypath.append(env["PYTHONPATH"])
env.update(
{
"DD_API_KEY": os.getenv("DD_API_KEY", "dummy-api-key"),
"DD_SITE": "datad0g.com",
"PYTHONPATH": ":".join(pypath),
"DD_LLMOBS_ML_APP": "unnamed-ml-app",
"_DD_LLMOBS_EVALUATOR_INTERVAL": "5",
}
)
env.update({"PYTHONPATH": ":".join(pypath), "_DD_LLMOBS_EVALUATOR_INTERVAL": "5"})
out, err, status, pid = run_python_code_in_subprocess(
"""
import os
Expand All @@ -87,7 +79,7 @@ def test_evaluator_runner_on_exit(mock_writer_logs, run_python_code_in_subproces
ctx = logs_vcr.use_cassette("tests.llmobs.test_llmobs_evaluator_runner.send_score_metric.yaml")
ctx.__enter__()
atexit.register(lambda: ctx.__exit__())
LLMObs.enable()
LLMObs.enable(api_key="dummy-api-key", site="datad0g.com", ml_app="unnamed-ml-app")
LLMObs._instance._evaluator_runner.evaluators.append(DummyEvaluator(llmobs_service=LLMObs))
LLMObs._instance._evaluator_runner.start()
LLMObs._instance._evaluator_runner.enqueue({"span_id": "123", "trace_id": "1234"}, None)
Expand Down
15 changes: 3 additions & 12 deletions tests/llmobs/test_llmobs_ragas_evaluators.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,14 +206,11 @@ def test_llmobs_with_faithfulness_emits_traces_and_evals_on_exit(mock_writer_log
pypath.append(env["PYTHONPATH"])
env.update(
{
"DD_API_KEY": os.getenv("DD_API_KEY", "dummy-api-key"),
"DD_SITE": "datad0g.com",
"PYTHONPATH": ":".join(pypath),
"OPENAI_API_KEY": os.getenv("OPENAI_API_KEY", "dummy-openai-api-key"),
"DD_LLMOBS_ML_APP": "unnamed-ml-app",
"_DD_LLMOBS_EVALUATOR_INTERVAL": "5",
"_DD_LLMOBS_EVALUATORS": "ragas_faithfulness",
"DD_LLMOBS_AGENTLESS_ENABLED": "1",
"DD_TRACE_ENABLED": "0",
}
)
out, err, status, pid = run_python_code_in_subprocess(
Expand All @@ -232,14 +229,8 @@ def test_llmobs_with_faithfulness_emits_traces_and_evals_on_exit(mock_writer_log
)
ctx.__enter__()
atexit.register(lambda: ctx.__exit__())
with mock.patch(
"ddtrace.internal.writer.HTTPWriter._send_payload",
return_value=Response(
status=200,
body="{}",
),
):
LLMObs.enable()
with mock.patch("ddtrace.internal.writer.HTTPWriter._send_payload", return_value=Response(status=200, body="{}")):
LLMObs.enable(api_key="dummy-api-key", site="datad0g.com", ml_app="unnamed-ml-app", agentless_enabled=True)
LLMObs._instance._evaluator_runner.enqueue(_llm_span_with_expected_ragas_inputs_in_messages(), None)
""",
env=env,
Expand Down

0 comments on commit 4beeccc

Please sign in to comment.