-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
make pytorch an optional dependency #2004
base: main
Are you sure you want to change the base?
Conversation
textsynth, | ||
vllm_causallms, | ||
) | ||
|
||
|
||
# TODO: implement __all__ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
drive-by fix since I needed to do some mucking around in here anyway
@@ -57,8 +55,10 @@ Homepage = "https://github.com/EleutherAI/lm-evaluation-harness" | |||
Repository = "https://github.com/EleutherAI/lm-evaluation-harness" | |||
|
|||
[project.optional-dependencies] | |||
torch = ["torch>=1.8"] | |||
hf = ["transformers", "torch>=1.8", "accelerate>=0.26.0", "peft>=0.2.0"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
two new extras. My guess is that people will generally want these, but pip has no way of specifying a "default" install and a "minimal" install.
@@ -15,6 +16,25 @@ | |||
# test once we break evaluator into smaller, more manageable pieces | |||
|
|||
|
|||
def test_evaluator_with_dummy_lm(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ensured this test passes with torch not installed.
Thanks for the PR, would you mind sharing a snippet of how you would use LM-Eval with JAX? Just so that I can understand better how folks use this without PyTorch. |
cc @haileyschoelkopf @lintangsutawika
Fixes #1959 . The main idea is to just remove top-level torch imports, which disables many LMs (unless torch is installed) but other than a few nuisances it was pretty easy! (Thanks for making a clear code base!)
I'll add a few notes inline.