Update template.py (#1096)

* Update template.py * Update hallucination.md * Update hallucination.md
comet-ml · Jan 21, 2025 · 8a665a7 · 8a665a7
1 parent 7b59c2c
commit 8a665a7
Show file tree

Hide file tree

Showing 2 changed files with 19 additions and 21 deletions.
diff --git a/apps/opik-documentation/documentation/docs/evaluation/metrics/hallucination.md b/apps/opik-documentation/documentation/docs/evaluation/metrics/hallucination.md
@@ -50,15 +50,16 @@ The template uses a few-shot prompting technique to detect hallucinations. The t
 Guidelines:
 1. The OUTPUT must not introduce new information beyond what's provided in the CONTEXT.
 2. The OUTPUT must not contradict any information given in the CONTEXT.
-3. Ignore the INPUT when evaluating faithfulness; it's provided for context only.
-4. Consider partial hallucinations where some information is correct but other parts are not.
-5. Pay close attention to the subject of statements. Ensure that attributes, actions, or dates are correctly associated with the right entities (e.g., a person vs. a TV show they star in).
-6. Be vigilant for subtle misattributions or conflations of information, even if the date or other details are correct.
-7. Check that the OUTPUT doesn't oversimplify or generalize information in a way that changes its meaning or accuracy.
+3. The OUTPUT should not contradict well-established facts or general knowledge.
+4. Ignore the INPUT when evaluating faithfulness; it's provided for context only.
+5. Consider partial hallucinations where some information is correct but other parts are not.
+6. Pay close attention to the subject of statements. Ensure that attributes, actions, or dates are correctly associated with the right entities (e.g., a person vs. a TV show they star in).
+7. Be vigilant for subtle misattributions or conflations of information, even if the date or other details are correct.
+8. Check that the OUTPUT doesn't oversimplify or generalize information in a way that changes its meaning or accuracy.
 
-Verdict options:
-- "{FACTUAL_VERDICT}": The OUTPUT is entirely faithful to the CONTEXT.
-- "{HALLUCINATION_VERDICT}": The OUTPUT contains hallucinations or unfaithful information.
+Analyze the text thoroughly and assign a hallucination score between 0 and 1, where:
+- 0.0: The OUTPUT is entirely faithful to the CONTEXT
+- 1.0: The OUTPUT is entirely unfaithful to the CONTEXT
 
 {examples_str}
 
@@ -71,13 +72,10 @@ CONTEXT:
 OUTPUT:
 {output}
 
-Provide your verdict in JSON format:
+It is crucial that you provide your answer in the following JSON format:
 {{
-    "{VERDICT_KEY}": <your verdict>,
-    "{REASON_KEY}": [
-        <list your reasoning as bullet points>
-    ]
+    "score": <your score between 0.0 and 1.0>,
+    "reason": ["reason 1", "reason 2"]
 }}
+Reasons amount is not restricted. Output must be JSON format only.
 ```
-
-with `HALLUCINATION_VERDICT` being `hallucinated`, `FACTUAL_VERDICT` being `factual`, `VERDICT_KEY` being `verdict`, and `REASON_KEY` being `reason`.
diff --git a/sdks/python/src/opik/evaluation/metrics/llm_judges/hallucination/template.py b/sdks/python/src/opik/evaluation/metrics/llm_judges/hallucination/template.py
@@ -15,12 +15,12 @@ class FewShotExampleHallucination(TypedDict):
 Guidelines:
 1. The OUTPUT must not introduce new information beyond what's provided in the CONTEXT.
 2. The OUTPUT must not contradict any information given in the CONTEXT.
-2. The OUTPUT should not contradict well-established facts or general knowledge.
-3. Ignore the INPUT when evaluating faithfulness; it's provided for context only.
-4. Consider partial hallucinations where some information is correct but other parts are not.
-5. Pay close attention to the subject of statements. Ensure that attributes, actions, or dates are correctly associated with the right entities (e.g., a person vs. a TV show they star in).
-6. Be vigilant for subtle misattributions or conflations of information, even if the date or other details are correct.
-7. Check that the OUTPUT doesn't oversimplify or generalize information in a way that changes its meaning or accuracy.
+3. The OUTPUT should not contradict well-established facts or general knowledge.
+4. Ignore the INPUT when evaluating faithfulness; it's provided for context only.
+5. Consider partial hallucinations where some information is correct but other parts are not.
+6. Pay close attention to the subject of statements. Ensure that attributes, actions, or dates are correctly associated with the right entities (e.g., a person vs. a TV show they star in).
+7. Be vigilant for subtle misattributions or conflations of information, even if the date or other details are correct.
+8. Check that the OUTPUT doesn't oversimplify or generalize information in a way that changes its meaning or accuracy.
 
 Analyze the text thoroughly and assign a hallucination score between 0 and 1, where:
 - 0.0: The OUTPUT is entirely faithful to the CONTEXT