should we allow passing image embedding as ip_adapter_image?

[yiyixuxu]

Currently, all our pipelines only accept ip_adapter_image as a PIL.Image or a list of PIL.Images - I wonder if it makes sense to allow passing embeddings directly?
It should be very easy to implement. However, we would only want to do so if it's meaningful. Let me know what you all think.

[asomoza]

This should be a must, there are huge benefits, with the current implementation of diffusers even if you don't change the images the pipeline encodes the images over and over again, this could potentially take a lot of time if you use a lot of images with multiple adapters, so the first benefit is that it would make generations faster in those cases.

There's another potential benefit with this, currently in the ComfyUI node you can save the embeddings to disk, if diffusers allows this and make them both compatible the embeddings can be used and shared in both systems, this is particularly useful for styles with a lot of images, you can just share the embeddings to other people or reuse them without the need to find the images again and encode them. Also this could enable the use of the negative noise images without the need to implement it into diffusers, you can just create all the embeddings in others systems that have UIs and more features and use them in diffusers.

I was also planning to add this, and perhaps more UIs can add them too so this could be shared universally, the same as LoRAs

[asomoza]

also I forgot the third benefit, it saves a huge amount of space, 6MB of input images can be saved in 35 KB

[yiyixuxu]

great! thanks for the insights:)
I think we can quickly support passing image embedding as ip_adapter_image and look into compatibility with the ComfyUI as next step

[fabiorigano]

I also agree about passing image embeddings as parameter. Is it perhaps more consistent to use the same design used for text (prompt and prompt_embeds) instead of one parameter for both types of input?