Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add gradient blending to tile seams in MultiDiffusion #6635

Merged
merged 1 commit into from
Jul 19, 2024

Conversation

RyanJDick
Copy link
Collaborator

@RyanJDick RyanJDick commented Jul 19, 2024

Summary

This PR adds gradient blending between tiles during MultiDiffusion denoising. Previously, no gradient was applied and the sharp cut-off caused visible seams in some case (particularly evident on smooth single-color regions).

Before

Upsampled to 2240x1872
ce075863-72a1-450b-bcac-5d5446f9b829

Upsampled to 4480x3744
bff991dd-d0e0-46d7-bcb1-2f5bd7446ccf

After

Upsampled to 2240x1872
bfa197fc-1e52-4db4-b68a-5e303aaba74a

Upsampled to 4480x3744
d5bcb459-ff8f-4389-a3e4-28017f95cd81

Other Notes:

  • The non-gradient blending was based on the official MultiDiffusion implementation that accompanied the paper - this deviates from that.
  • Other ideas that would likely help with this problem:
    • Large overlaps between tiles so that each tile edge is covered by multiple other tiles. (This is what the official MultiDiffusion implementation seems to do.)
    • Jitter the tile locations so that seams aren't in the same place in every step.

Aside: Tile color shift

As seen in the sample images, there is a remaining issue with tile color shift. We currently rely on a ControlNet to maintain this consistency, but clearly it is not very effective. Future improvements that might help with this:

  • Larger MultiDiffusion tile sizes and larger overlaps
  • Shift latent tile distributions close to original latent tile distributions
  • Use noise inversion to guide the process (i.e. like SUPIR, DemoFusion, and others)
  • Apply color correction in pixel space on the final image

QA Instructions

Speed tests on RTX4090:

  • Denoising during upscaling to 2240x1872:
    • Before: 36s
    • After 36s
  • Denoising during upscaling to 4480x3744
    • Before: 150s
    • After: 150s
      I.e. no change in speed on an RTX4090.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files labels Jul 19, 2024
@RyanJDick RyanJDick marked this pull request as ready for review July 19, 2024 15:09
@hipsterusername hipsterusername force-pushed the ryan/multidiffusion-seam-blending branch from 3303d8d to 6d09751 Compare July 19, 2024 16:17
@hipsterusername hipsterusername merged commit e16faa6 into main Jul 19, 2024
14 checks passed
@hipsterusername hipsterusername deleted the ryan/multidiffusion-seam-blending branch July 19, 2024 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files invocations PRs that change invocations python PRs that change python files
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants