Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LM Eval Testing #945

Merged
merged 8 commits into from
Dec 10, 2024
Merged

Add LM Eval Testing #945

merged 8 commits into from
Dec 10, 2024

Conversation

dsikka
Copy link
Collaborator

@dsikka dsikka commented Nov 29, 2024

Summary:

  • Add LM Eval Teseting to run on a weekly cadence
  • Model is quantized after which lm_eval runs and the metrics produced are asserted against metrics defined in the config for the particular test case
  • Updates the bash script to take in configs folder and test script, allowing it to work for both the vLLM e2e tests and LM Eval tests

Testing

  • Testing has to be updated to use the following commands:
bash tests/e2e/vLLM/run_tests.sh -c tests/e2e/vLLM/lm_eval_configs -t tests/e2e/vLLM/test_lmeval.py 
bash tests/e2e/vLLM/run_tests.sh -c tests/e2e/vLLM/configs -t tests/e2e/vLLM/test_vllm.py 
  • LM Eval testing requires CADENCE=weekly

Copy link

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

@dsikka dsikka self-assigned this Nov 29, 2024
@dsikka dsikka added the ready When a PR is ready for review label Nov 29, 2024
@dsikka dsikka requested review from dbarbuzzi and dhuangnm November 29, 2024 20:35
setup.py Outdated Show resolved Hide resolved
@dsikka dsikka requested a review from anmarques December 2, 2024 20:12
@dsikka
Copy link
Collaborator Author

dsikka commented Dec 2, 2024

  • To add in future PRs:
  • w4 + act order
  • int8 + smoothquant

@horheynm
Copy link
Collaborator

horheynm commented Dec 9, 2024

one ocmment otherwise lgtm

rahul-tuli
rahul-tuli previously approved these changes Dec 9, 2024
Copy link
Collaborator

@rahul-tuli rahul-tuli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending Domenic's comment about "usage" function in the bash script

tests/e2e/vLLM/test_lmeval.py Show resolved Hide resolved
kylesayrs
kylesayrs previously approved these changes Dec 9, 2024
@dsikka dsikka dismissed stale reviews from kylesayrs and rahul-tuli via 52c4226 December 10, 2024 03:07
@dsikka dsikka merged commit 606aab2 into main Dec 10, 2024
6 of 7 checks passed
@dsikka dsikka deleted the add_lm_eval branch December 10, 2024 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready When a PR is ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants