Fix Nonetype attribute error when loading multiple Flux loras

[jonathanyin12]

What does this PR do?

Fixes #10180 to allow loading multiple loras (issue originally introduced by #9999)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@a-r-r-o-w

[sayakpaul]

Let's make this more explicit:

module_bias = None
if getattr(module, "bias", None) is not None:
    module_bias = module.bias.data

WDYT? Additionally, do you think we should add a test for this? When we merged the Control LoRA PR, we did run all the integration tests too and we didn't face an issue.

[a-r-r-o-w]

Since this is under an if statement where we know the following layer is nn.Linear, we know that there is already a module.bias parameter. So I think the current condition is okay.

Yes, the integration tests when I last tested did not fail either. I think we can investigate again.

We should add one more fast test here IMO that tests for a failure when:

• First lora loaded expands the shape
• Second lora is of normal shape without the expansion

This usecase is not supported yet, so until it is, we expect that an error will be raised.

Another fast test that we should probably have is loading lora into nn.Linear with/without bias present, for a total of 4 tests (lora with/without, linear with/without - some are already covered with existing test suite).

WDYT @sayakpaul? We can do this in a separate PR since the change here is minimal and the actual correct thing to do

[sayakpaul]

Works for me but let's high-prioritize the tests then (immediately after this merge) given how important LoRAs are.

[a-r-r-o-w]

Sounds good. I'll open a PR after merging this with the mentioned tests

[sayakpaul]

Thanks so much! I have a single comment.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

Thank you for spotting this and fixing! 🤗

[sayakpaul]

Okay for me to merge as is