Add proposed exact and f1 scorers.
#306
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`exact()` Scorer which will normalize the text of the answer and target(s) and perform an exact matching comparison of the text. This scorer will return `CORRECT` when the answer is an exact match to one or more targets. `f1()` Scorer which computes the `F1` score for the answer (which balances recall precision by taking the harmonic mean between recall and precision).
jjallaire-aisi
approved these changes
Aug 31, 2024
5 tasks
…ature/f1 # Conflicts: # CHANGELOG.md # src/inspect_ai/_view/www/dist/assets/index.js
Tested against main baseline using 100 samples ### Gpt-4o | Branch | Accuracy | StdErr | | main | 0.856 | 0.0257 | | f1 | 0.879 | 0.0229 | ### Gpt-4o-mini | Branch | Accuracy | StdErr | | main | 0.868 | 0.031 | | f1 | 0.877 | 0.0285 |
|
@xeon27 I made a change to the DROP benchmark implementation to allow it to use our new built in inspect_ai/benchmarks/drop/drop.py Line 175 in b8d60fa Rather than using Tuples, I'm just forcing the lower level answers to strings which can then be easily handled by the rest of the code (with no special processing required when scoring). If you have a moment to review this, I'd appreciate any thoughts or alternatives if you think there is a better way! |
max-kaufmann
pushed a commit
that referenced
this pull request
Sep 9, 2024
* Add proposed `exact` and `f1` scorers. `exact()` Scorer which will normalize the text of the answer and target(s) and perform an exact matching comparison of the text. This scorer will return `CORRECT` when the answer is an exact match to one or more targets. `f1()` Scorer which computes the `F1` score for the answer (which balances recall precision by taking the harmonic mean between recall and precision). * Add changelog entry * Round values to match f1 behavior exactly * Improve list of targets display * Convert the drop benchmark to using build in f1 * Optionally show error card * cleanup drop imports * correct test for rounding * Improve F1 scorer to accept an answer extraction function * Update drop benchmark to simplify data reading Tested against main baseline using 100 samples ### Gpt-4o | Branch | Accuracy | StdErr | | main | 0.856 | 0.0257 | | f1 | 0.879 | 0.0229 | ### Gpt-4o-mini | Branch | Accuracy | StdErr | | main | 0.868 | 0.031 | | f1 | 0.877 | 0.0285 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
exact()Scorer which will normalize the text of the answer and target(s) and perform an exact matching comparison of the text. This scorer will return
CORRECTwhen the answer is an exact match to one or more targets.f1()Scorer which computes the
F1score for the answer (which balances recall precision by taking the harmonic mean between recall and precision).