allow models to run with a user-provided dtype map instead of a single dtype

[hlky]

What does this PR do?

Example

import torch
from diffusers import HunyuanVideoPipeline

model_id = "tencent/HunyuanVideo"
pipe = HunyuanVideoPipeline.from_pretrained(model_id, torch_dtype={'transformer': torch.bfloat16, '_': torch.float16}, revision="refs/pr/18")
pipe.transformer.dtype, pipe.vae.dtype

(torch.bfloat16, torch.float16)

_ is used as a default dtype for components that are not specified, otherwise the current default of torch.float32 is used.

Haven't looked at from_pipe case yet and we'll need to add tests but ready for a first review in case there's something missing because it's simpler than expected.

Fixes #10108

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc @DN6 @sayakpaul @yiyixuxu

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Thanks! Do we not have to handle the typecasts? I think for sharded checkpoints, we might have to.

[sayakpaul]

I feel like _ might be a bit unintuitive. Better to expose full dtype maps or in case partial ones are provided we default to torch.float32 for the rest of the components.

[hlky]

Could be default? Considering how it will work for integrations, instead of say {'transformer': torch.bfloat16, 'text_encoder': torch.float16, 'text_encoder_2': torch.float16, 'text_encoder_3': torch.float16} for SD3 and {'transformer': torch.bfloat16, 'text_encoder': torch.float16, 'text_encoder_2': torch.float16} for Flux. Not a big issue because components can be got from cls._get_signature_types().

[sayakpaul]

Yeah no strong opinions.

[hlky]

For now it's renamed to default to be clearer, we can remove later if its not needed.

[hlky]

Thanks for the review @sayakpaul. Will look into sharded checkpoints.

[hlky]

HunyuanVideo is sharded so I think it's ok.