make pytorch an optional dependency #2004
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
dlwh
commented
Jun 20, 2024
| textsynth, | ||
| vllm_causallms, | ||
| ) | ||
|
|
||
|
|
||
| # TODO: implement __all__ |
There was a problem hiding this comment.
drive-by fix since I needed to do some mucking around in here anyway
dlwh
commented
Jun 20, 2024
| @@ -57,8 +55,10 @@ Homepage = "https://github.com/EleutherAI/lm-evaluation-harness" | |||
| Repository = "https://github.com/EleutherAI/lm-evaluation-harness" | |||
|
|
|||
| [project.optional-dependencies] | |||
| torch = ["torch>=1.8"] | |||
| hf = ["transformers", "torch>=1.8", "accelerate>=0.26.0", "peft>=0.2.0"] | |||
There was a problem hiding this comment.
two new extras. My guess is that people will generally want these, but pip has no way of specifying a "default" install and a "minimal" install.
dlwh
commented
Jun 20, 2024
| @@ -15,6 +16,25 @@ | |||
| # test once we break evaluator into smaller, more manageable pieces | |||
|
|
|||
|
|
|||
| def test_evaluator_with_dummy_lm(): | |||
There was a problem hiding this comment.
ensured this test passes with torch not installed.
|
Thanks for the PR, would you mind sharing a snippet of how you would use LM-Eval with JAX? Just so that I can understand better how folks use this without PyTorch. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cc @haileyschoelkopf @lintangsutawika
Fixes #1959 . The main idea is to just remove top-level torch imports, which disables many LMs (unless torch is installed) but other than a few nuisances it was pretty easy! (Thanks for making a clear code base!)
I'll add a few notes inline.