Add `TiledMultiDiffusionDenoiseLatents` invocation (for upscaling workflows) #6522

RyanJDick · 2024-06-19T19:20:55Z

Summary

This PR adds a TiledMultiDiffusionDenoiseLatents invocation. This invocation has an interface similar to that of the DenoiseLatentsInvocation, but uses MultiDiffusion internally to perform tiled denoising. The primary intended use case for this node is in tiled upscaling workflows.

This PR refactors (and tidies) the StableDiffusionGeneratorPipeline class to facilitate sharing most of the core logic between standard and MultiDiffusion denoising.

Workflows

Here are some test workflows that use the new invocation:

Tiled upscaling with ESRGAN x2 as the initial upscaler:
sd15_multi_diffusion_esrgan_x2_upscale.json
A simple panorama workflow for more direct testing of MultiDiffusion: multi_diffusion_panorama.json

Screenshot of multi_diffusion_esrgan_x2_upscaling.json:

Sample Outputs from Upscaling

Before (352x352):

After 4x (1408x1408):

Before (842x842):

After 4x (3360x3360):

QA Instructions

Test plan:

SD 1.5 regression tests (no change expected on these workflows with this PR)
- Text-to-image
- Image-to-image
- Inpainting with non-inpainting model
- Inpainting with inpainting model
- Text-to-image with an inpainting model
- Regional text conditioning and regional IP-Adapter (just a smoke test - this shouldn't be affected)
SDXL regression tests (no change expected on these workflows with this PR)
- Text-to-image
- Image-to-image
- Inpainting with non-inpainting model
Multi-Diffusion tests
- SD 1.5 upscaling workflow
- SDXL upscaling workflow
- Try to break it with awkward input image dimensions

Merge Plan

Checklist

The PR has a short but descriptive title, suitable for a changelog
Tests added / updated (if applicable)
Documentation added / updated (if applicable)

invokeai/app/invocations/denoise_latents.py

psychedelicious · 2024-06-21T06:35:19Z

I focused on the code changes that don't interact with ML stuff - LGTM.

…nvocation so that it can be called from other invocations.

…so that they can be called externally.

…iled refine.

…hat it can be called externally.

…ork with more tile shapes and overlaps.

…edMultiDiffusionDenoiseLatents.py

…o that it can be called externally.

…ultiDiffusionPipeline.

…s not yet supported.

…ne backend.

…t that has been superseded by TiledMultiDiffusionDenoiseLatents.

…ocation." This reverts commit 753239b.

…ound. It is simply the first element of the timesteps array.

…cation.

…ding any value.

…with MultiDiffusion.

…amount of overlap rather than being a lower bound.

…nts invocation. This is more natural for many users.

@StAlKeR7779

## Summary #6522 introduced a change in behavior in cases where start/end were set such that there are 0 timesteps. This PR reverts that change. cc @StAlKeR7779 ## QA Instructions Run with euler, 5 steps, start: 0.0, end: 0.05. I ran this test before #6522, after #6522, and on this branch. This branch restores the behavior to pre-#6522 i.e. noise is injected even if no denoising steps are applied. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [x] _Tests added / updated (if applicable)_ - [x] _Documentation added / updated (if applicable)_

github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files labels Jun 19, 2024

psychedelicious reviewed Jun 21, 2024

View reviewed changes

invokeai/app/invocations/denoise_latents.py Show resolved Hide resolved

RyanJDick marked this pull request as ready for review June 25, 2024 17:55

RyanJDick requested review from lstein, blessedcoolant, brandonrising and hipsterusername as code owners June 25, 2024 17:55

hipsterusername approved these changes Jun 25, 2024

View reviewed changes

RyanJDick added 19 commits June 25, 2024 11:23

(minor) Add a TODO note to get_scheduler(...).

abbc867

Split out the prepare_noise_and_latents(...) logic in DenoiseLatentsI…

1657d1c

…nvocation so that it can be called from other invocations.

Simplify the logic in prepare_noise_and_latents(...).

87ffb1e

Convert several methods in DenoiseLatentsInvocation to staticmethods …

8f7281a

…so that they can be called externally.

WIP on TiledStableDiffusionRefine

6e12754

Only allow a single positive/negative prompt conditioning input for t…

3037dd4

…iled refine.

Make init_scheduler() a staticmethod on DenoiseLatentsInvocation so t…

91f92e6

…hat it can be called externally.

Fix return type of prepare_noise_and_latents(...).

98ede8d

Expose vae_decode(...) as a staticmethod on LatentsToImageInvocation.

8122c32

Minor improvements to LatentsToImageInvocation type hints.

6629f97

WIP - TiledStableDiffusionRefine

c1ebe2e

Rough prototype of TiledStableDiffusionRefineInvocation is working.

c118167

Fix ControlNetModel type hint import source.

a9fc02d

Add naive ControlNet support to TiledStableDiffusionRefineInvocation

3e4b851

Add support for LoRA models in TiledStableDiffusionRefineInvocation.

390018d

Expose a few more params from TiledStableDiffusionRefineInvocation.

329d20e

Change tiling strategy to make TiledStableDiffusionRefineInvocation w…

f43c94b

…ork with more tile shapes and overlaps.

Copy TiledStableDiffusionRefineInvocation as a starting point for Til…

9ddfaf0

…edMultiDiffusionDenoiseLatents.py

Make DenoiseLatentsInvocation.prep_control_data(...) a staticmethod s…

431c713

…o that it can be called externally.

RyanJDick added 24 commits June 25, 2024 11:23

Copy StableDiffusionGeneratorPipeline as a starting point for a new M…

e634a1c

…ultiDiffusionPipeline.

Add detailed docstring to latents_from_embeddings().

067de31

Document plan for the rest of the MultiDiffusion implementation.

948bd45

Remove IP-Adapter and T2I-Adapter support from MultiDiffusionPipeline.

aa2c9d8

Remove inpainting support from MultiDiffusionPipeline.

e39ec09

Initial (untested) implementation of MultiDiffusionPipeline.

2893f5c

Remove regional conditioning logic from MultiDiffusionPipeline - it i…

e6452b5

…s not yet supported.

Connect TiledMultiDiffusionDenoiseLatents to the MultiDiffusionPipeli…

94494b5

…ne backend.

Fix handling of stateful schedulers in MultiDiffusionPipeline.

a9d1054

Fix advanced scheduler behaviour in MultiDiffusionPipeline.

68d7906

Delete rough notes.

0ea0f0b

(minor) typo

c207c03

Remove TiledStableDiffusionRefineInvocation. It was a proof-of-concep…

b3b95e7

…t that has been superseded by TiledMultiDiffusionDenoiseLatents.

(minor) Address a few small TODOs.

3790328

Revert "Expose vae_decode(...) as a staticmethod on LatentsToImageInv…

372f867

…ocation." This reverts commit 753239b.

Remove the redundant init_timestep parameter that was being passed ar…

5fd7d26

…ound. It is simply the first element of the timesteps array.

Add progress image callbacks to TiledMultiDiffusionDenoiseLatentsInvo…

7296a17

…cation.

(minor) Fix some documentation typos.

d33d4c6

Consolidate _region_step() function - the separation wasn't really ad…

ceb9830

…ding any value.

Add TODO comment explaining why some schedulers do not interact well …

ad94bcd

…with MultiDiffusion.

Make the tile_overlap input to MultiDiffusion *strictly* control the …

446e94a

…amount of overlap rather than being a lower bound.

Mark TiledMultiDiffusionDenoiseLatents as a Beta node.

013771b

Use image-space tile dimensions on the TiledMultiDiffusionDenoiseLate…

c3f8c43

…nts invocation. This is more natural for many users.

Fix inpainting mask shape assertions.

6b1f620

hipsterusername force-pushed the ryan/multi-diffusion branch from b05299e to 6b1f620 Compare June 25, 2024 18:23

hipsterusername enabled auto-merge (rebase) June 25, 2024 18:24

hipsterusername merged commit cd9dfef into main Jun 25, 2024
14 checks passed

hipsterusername deleted the ryan/multi-diffusion branch June 25, 2024 18:31

RyanJDick mentioned this pull request Jun 25, 2024

Fix handling handling of 0-step denoising process #6544

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `TiledMultiDiffusionDenoiseLatents` invocation (for upscaling workflows) #6522

Add `TiledMultiDiffusionDenoiseLatents` invocation (for upscaling workflows) #6522

RyanJDick commented Jun 19, 2024 •

edited

Loading

psychedelicious commented Jun 21, 2024

Add TiledMultiDiffusionDenoiseLatents invocation (for upscaling workflows) #6522

Add TiledMultiDiffusionDenoiseLatents invocation (for upscaling workflows) #6522

Conversation

RyanJDick commented Jun 19, 2024 • edited Loading

Summary

Workflows

Sample Outputs from Upscaling

QA Instructions

Merge Plan

Checklist

psychedelicious commented Jun 21, 2024

Add `TiledMultiDiffusionDenoiseLatents` invocation (for upscaling workflows) #6522

Add `TiledMultiDiffusionDenoiseLatents` invocation (for upscaling workflows) #6522

RyanJDick commented Jun 19, 2024 •

edited

Loading