Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding inference endpoints models #12

Merged
merged 18 commits into from
Feb 7, 2024
Merged

Conversation

clefourrier
Copy link
Member

@clefourrier clefourrier commented Feb 5, 2024

Inference endpoints models are these ones: https://huggingface.co/docs/huggingface_hub/v0.20.3/en/package_reference/inference_endpoints
Not to be confused with TGI models (which need a local deployment)

This PR:

  • uses Requests instead of passing tuples (which are more error prones) in the Datasets
  • introduces an Abstract Model class which defines the minimum functions we need to have in a model for it to be lighteval compatible
  • cleans up the BaseModel code
  • introduces inference endpoints models

@clefourrier clefourrier requested a review from NathanHB February 7, 2024 12:18
src/lighteval/data.py Outdated Show resolved Hide resolved
src/main.py Show resolved Hide resolved
@clefourrier clefourrier merged commit 1e837a9 into main Feb 7, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants