Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run init_weights under no_grad #747

Merged
merged 1 commit into from
Dec 17, 2024
Merged

Conversation

carmocca
Copy link
Contributor

The initializations in init_weights can create gradients. This is almost always not intended

The alternative would be to add the decorator to model.init_weights directly. Then the responsibility is moved to the model writer. I can change that if that's preferred.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 17, 2024
Copy link
Contributor

@tianyu-l tianyu-l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. I think it's fine, compared to having a decorator on every single init_weights() definition (reasoning: technically we probably only need it on the outermost one, but what if users would like to call it explicitly for a submodule -- they still have to use the context manager).

@awgu awgu merged commit 5ce8a0c into pytorch:main Dec 17, 2024
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants