Skip to content

Commit

Permalink
For simple matching scorers, only include explanation if it differs f…
Browse files Browse the repository at this point in the history
…rom answer
  • Loading branch information
jjallaire committed Oct 5, 2024
1 parent 0cf3292 commit 0af777a
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 2 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
- Add optional `content` parameter to `ModelOutput.for_tool_call()`.
- Display total samples in Inspect View
- Prune `sample_reductions` when returning eval logs with `header_only=True`.
- For simple matching scorers, only include explanation if it differs from answer.

## v0.3.39 (3 October 2024)

Expand Down
10 changes: 8 additions & 2 deletions src/inspect_ai/scorer/_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,19 @@ async def score(state: TaskState, target: Target) -> Score:
for value in target:
answer, matched = match(state.output.completion, value)
if matched:
explanation = (
state.output.completion
if state.output.completion != answer
else None
)
return Score(
value=CORRECT, answer=answer, explanation=state.output.completion
)

return Score(
value=INCORRECT, answer=answer, explanation=state.output.completion
explanation = (
state.output.completion if state.output.completion != answer else None
)
return Score(value=INCORRECT, answer=answer, explanation=explanation)

return score

Expand Down

0 comments on commit 0af777a

Please sign in to comment.