DeepFloydIF officially released

[Alchete]

https://github.com/deep-floyd/IF
https://huggingface.co/DeepFloyd

Thoughts on if it's possible to roll this into the UI? This is the first model that'll do text!

[vladmandic]

That's exactly the stuff is like to be working on - if I didn't have to deal with issues backlog. 😁😔

[Alchete]

Well, the backlog is called the backlog for a reason! It'll always be there! 😅😂

[vladmandic]

True.
I'm hoping to deal with 99th percentile of installs by fixing backlog and for borderline cases, leave as backlog and focus on features. But that's just an idea for now.

[econundrum]

Have you seen the VRAM requirements for these models? Not many people will be using these locally.

[FullBleed]

Have you seen the VRAM requirements for these models? Not many people will be using these locally.

Minimum requirements to use all IF models:

• 16GB vRAM for IF-I-XL (4.3B text to 64x64 base module) & IF-II-L (1.2B to 256x256 upscaler module)
  

• 24GB vRAM for IF-I-XL (4.3B text to 64x64 base module) & IF-II-L (1.2B to 256x256 upscaler module) & Stable x4 (to 1024x1024 upscaler)
  

• xformers and set env variable FORCE_MEM_EFFICIENT_ATTN=1
  

Demanding, but not out of reach.

[baaoh]

Have you seen the VRAM requirements for these models? Not many people will be using these locally.

Minimum requirements to use all IF models:

• 16GB vRAM for IF-I-XL (4.3B text to 64x64 base module) & IF-II-L (1.2B to 256x256 upscaler module)
  

• 24GB vRAM for IF-I-XL (4.3B text to 64x64 base module) & IF-II-L (1.2B to 256x256 upscaler module) & Stable x4 (to 1024x1024 upscaler)
  

• xformers and set env variable FORCE_MEM_EFFICIENT_ATTN=1
  

Demanding, but not out of reach.

Somebody somewhere mentioned model 4bit quantization, they said such model would take less vram to run?

[drax-xard]

I'm confident that the community will find ways to make it fit into a toaster given time.

[froger-me]

https://github.com/neonsecret/IF

Tested on rtx 3070, 8 gb vram

stage 1: 1.30 min for 27 steps, ~3.35 sec per iteration
stage 2: 11 seconds for 27 steps
stage 3: 30 seconds for 40 steps

[Alchete]

Update: multiple rumors that the model weights will be released today. So, hopefully we'll have all the details soon.

[Alchete]

Official release: https://huggingface.co/DeepFloyd

Need to log in with a Hugging Face account. Multiple models from 740MB to 8.6GB.

@vladmandic are you able to tell what would be needed to incorporate these? Maybe others can assist in that effort.

[vladmandic]

first step is to bring in proper support for diffusers, once that is done, its going to be much easier to support any models that are build on top of diffusers. and that first step is has just kicked off :)

[ke1ne]

@vladmandic https://github.com/huggingface/diffusers/releases/tag/v0.16.0

[vladmandic]

@ke1ne work-in-progress: https://github.com/vladmandic/automatic/tree/diffusers

[pkuliyi2015]

I'm also VERY interested in the new model. Compared to the SD, the new model is in pixel space, meaning that many traditional operations & constraint can be applied to it.

However the model is quite different from SD. First of all, It doesn't need a VAE. Second it may deal with much longer prompts so that the hacks in the automatic1111 will no longer need. Third it may not support i2i easily. Fourth it is not compatible with any existing extensions, LoRA, Embeddings, Dreambooth, ...

There are huge differences. I wonder if a totally new UI should be developed for it?

[cibernicola]

These are the questions:

As a source of inspiration for potential research, we pose several questions divided into three groups: technical, academic and ethical. 

1. Technical research questions:

a) How can users optimize the IF model by identifying potential improvements that can enhance its performance, scalability, and efficiency?

b) How can output quality be improved with better sampling, guiding, or even fine-tuning the DeepFloyd IF mode?

c) How can users apply certain techniques used to modify Stable Diffusion output, such as DreamBooth, ControlNet and LoRA on DeepFloyd IF?

https://stability.ai/blog/deepfloyd-if-text-to-image-model

[elchupacabrinski]

oh man that would be so cool if it could be integrated

[ilikenwf]

I noticed that there's a mention of it now in one of the img2img config files in our repo here now?