Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Brandon/cast unquantized flux to bfloat16 #6815

Merged
merged 5 commits into from
Sep 5, 2024

Conversation

brandonrising
Copy link
Collaborator

@brandonrising brandonrising commented Sep 5, 2024

Summary

This PR forces unquantized flux model state dicts to be converted to bfloat16 since that's all we support for inference currently. This can be removed or minimized as we support more data types in the future. This will allow us to support the seemingly common float8 flux fine tunes available on sites like huggingface and civitAI.

QA Instructions

Attempt to perform inference with a model similar to this one

Merge Plan

Can be merged when tested and approved

Checklist

  • The PR has a short but descriptive title, suitable for a changelog

@github-actions github-actions bot added python PRs that change python files backend PRs that change backend files labels Sep 5, 2024
Copy link
Collaborator

@RyanJDick RyanJDick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a handful of comments - mostly around all of the explicit memory management we are doing with (del and gc.collect()).

I tested it as-is and it works for me. The quality seems pretty clearly worse than a 4-bit quantized model though 😅

invokeai/backend/model_manager/load/model_loaders/flux.py Outdated Show resolved Hide resolved
invokeai/backend/model_manager/load/model_loaders/flux.py Outdated Show resolved Hide resolved
invokeai/backend/model_manager/load/model_loaders/flux.py Outdated Show resolved Hide resolved
invokeai/backend/model_manager/load/model_loaders/flux.py Outdated Show resolved Hide resolved
invokeai/backend/model_manager/load/model_loaders/flux.py Outdated Show resolved Hide resolved
invokeai/backend/model_manager/load/model_loaders/flux.py Outdated Show resolved Hide resolved
invokeai/backend/model_manager/util/model_util.py Outdated Show resolved Hide resolved
@brandonrising brandonrising force-pushed the brandon/cast-unquantized-flux-to-bfloat16 branch from 117e3e0 to f08e942 Compare September 5, 2024 19:38
@brandonrising brandonrising enabled auto-merge (rebase) September 5, 2024 19:39
@brandonrising brandonrising merged commit a16b555 into main Sep 5, 2024
14 checks passed
@brandonrising brandonrising deleted the brandon/cast-unquantized-flux-to-bfloat16 branch September 5, 2024 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files python PRs that change python files
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants