You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Does the elether_eval recipe call model.eval() somewhere? I tried to find it but could not. It's there in the lm evaluation harness but we have overriden the constructor in eval wrapper. Also, my results change when I switch model to eval mode.
The text was updated successfully, but these errors were encountered:
Thanks @joecummings. Do you mind sharing why does eleuther_eval recipe use evaluate instead of simple_evaluate method from lm-eval-harness? The reason I am asking is evaluate does not allow few shot evaluation whereas simple_evaluate accepts num_fewshot argument for the same.
Does the elether_eval recipe call model.eval() somewhere? I tried to find it but could not. It's there in the lm evaluation harness but we have overriden the constructor in eval wrapper. Also, my results change when I switch model to eval mode.
The text was updated successfully, but these errors were encountered: