You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I was debugging an issue with our model outputting empty responses for all questions and noticed the accuracy score was still 22% when I expected it should be 0%.
I debug further and found that for multi_choice questions there is a path that computes Levenshtein distance but doesn't guard against empty inputs meaning it would output a valid choice regardless.
(Likely the choice with the least amount of characters which would be the minimum edit distance, or first choice if all equal length)
I also saw there was a questionable Exception handling when coercing the input value to a string.
It assigns an empty string and continues. I think it should exiting early and return None.
This assignment of empty string could further contribute to the issue above, for multiple choice problems where the extraction is not a string
I was debugging an issue with our model outputting empty responses for all questions and noticed the accuracy score was still 22% when I expected it should be 0%.
I debug further and found that for
multi_choice
questions there is a path that computes Levenshtein distance but doesn't guard against empty inputs meaning it would output a valid choice regardless.(Likely the choice with the least amount of characters which would be the minimum edit distance, or first choice if all equal length)
MathVista/evaluation/calculate_score.py
Lines 45 to 51 in 82f68d0
MathVista/evaluation/calculate_score.py
Lines 14 to 20 in 82f68d0
I also saw there was a questionable Exception handling when coercing the input value to a string.
It assigns an empty string and continues. I think it should exiting early and return None.
This assignment of empty string could further contribute to the issue above, for multiple choice problems where the extraction is not a string
MathVista/evaluation/calculate_score.py
Lines 30 to 36 in 82f68d0
Video Demonstration
https://youtu.be/vj07WRvcLDw
The text was updated successfully, but these errors were encountered: