Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace HuggingFaceTokenizer in AmazonSageMakerJumpstartPromptDriver with SimpleTokenizer #1460

Open
1 task done
igor-2lemetry opened this issue Dec 18, 2024 · 0 comments
Open
1 task done

Replace HuggingFaceTokenizer in AmazonSageMakerJumpstartPromptDriver with SimpleTokenizer #1460

igor-2lemetry opened this issue Dec 18, 2024 · 0 comments
Labels
type:enhancement Improvements to existing features
Milestone
2.0

Comments

@igor-2lemetry
Copy link
Contributor

Is your feature request related to a problem? Please describe.
HuggingFaceTokenizer making requests to https://huggingface.co/{model_name} which is restricted without an access token.

Describe the solution you'd like
Replace HuggingFaceTokenizer with SimpleTokenizer in AmazonSageMakerJumpstartPromptDriver.

Describe alternatives you've considered
Describe in docs and example how to get and pass token which using AmazonSageMakerJumpstartPromptDriver.

Additional context
This replacement also will allow to remove 'transformers' from drivers-prompt-amazon-sagemaker extra.
@igor-2lemetry igor-2lemetry added the type:enhancement Improvements to existing features label Dec 18, 2024
@collindutter collindutter added this to the 2.0 milestone Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:enhancement Improvements to existing features
Projects
None yet
Milestone
2.0
Development

No branches or pull requests
2 participants
@collindutter @igor-2lemetry