Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Gemma #87

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add Gemma #87

wants to merge 5 commits into from

Conversation

yisi-wang-slalom
Copy link

@yisi-wang-slalom yisi-wang-slalom commented May 23, 2024 •
edited
Loading

Description of changes:

We are excited to help adding support for Gemma model architectured by Google.

A few highlights, mainly referencing: huggingface/transformers#29402

  • Using gelu_new activation function, we've validated it against the PyTorch implementation of gelu_pytorch_tanh to ensure same result, and recommend implementing gelu_pytorch_tanh when possible.
  • Add Layernorm (w+1) of RMS Layernorm for gemma
  • Include additional normalization in embedding (multiplies the embeddings by sqrt(hidden_dim)) for gemma

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aws-maens aws-maens Awaiting requested review from aws-maens aws-maens is a code owner

@musunita musunita Awaiting requested review from musunita musunita is a code owner

@mmcclean-aws mmcclean-aws Awaiting requested review from mmcclean-aws

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@yisi-wang-slalom