Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tiling to SpandrelImageToImageInvocation #6594

Merged
merged 5 commits into from
Jul 16, 2024

Conversation

RyanJDick
Copy link
Collaborator

@RyanJDick RyanJDick commented Jul 10, 2024

Summary

Add tiling to the SpandrelImageToImageInvocation node so that it can process large images.

Tiling enables this node to run on effectively any input image dimension. Of course, the computation time increases quadratically with the image dimension.

Some profiling results on an RTX4090:

  • Input 1024x1024, 4x upscale, 4x UltraSharp ESRGAN: 13 secs, <4 GB VRAM
  • Input 4096x4096, 4x upscale, 4x UltraSharop ESRGAN: 46 secs, <4 GB VRAM
  • Input 4096x4096, 2x upscale, SwinIR: 165 secs, <5 GB VRAM

A lot of the time is spent PNG encoding the final image:

  • PNG encoding of a 16384x16384 image takes 83secs @ pil_compress_level=7, 24secs @ pil_compress_level=1

Callout: If we want to start building workflows that pass large images between nodes, we are going to have to find a way to avoid the PNG encode/decode roundtrip that we are currently doing. As is, we will be incurring a huge penalty for every node that receives/produces a large image.

QA Instructions

  • Tested with tiling up to 4096x4096 -> 16384x16384.
  • Test on images with an alpha channel (the alpha channel is dropped).
  • Test on images with odd dimension.
  • Test no tiling (tile_size=0)

Merge Plan

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)

@brandonrising
Copy link
Collaborator

What do you think of adding something like this to the tile loop?

--- a/invokeai/app/invocations/spandrel_image_to_image.py
+++ b/invokeai/app/invocations/spandrel_image_to_image.py
@@ -14,6 +14,7 @@ from invokeai.app.invocations.fields import (
 )
 from invokeai.app.invocations.model import ModelIdentifierField
 from invokeai.app.invocations.primitives import ImageOutput
+from invokeai.app.services.session_processor.session_processor_common import CanceledException
 from invokeai.app.services.shared.invocation_context import InvocationContext
 from invokeai.backend.spandrel_image_to_image_model import SpandrelImageToImageModel
 from invokeai.backend.tiles.tiles import calc_tiles_min_overlap
@@ -104,6 +105,9 @@ class SpandrelImageToImageInvocation(BaseInvocation, WithMetadata, WithBoard):
             image_tensor = image_tensor.to(device=spandrel_model.device, dtype=spandrel_model.dtype)

             for tile, scaled_tile in tqdm(list(zip(tiles, scaled_tiles, strict=True)), desc="Upscaling Tiles"):
+                if context.util.is_canceled():
+                    raise CanceledException
+
                 # Extract the current tile from the input tensor.
                 input_tile = image_tensor[
                     :, :, tile.coords.top : tile.coords.bottom, tile.coords.left : tile.coords.right

Base automatically changed from ryan/spandrel-upscale to main July 16, 2024 19:37
@RyanJDick RyanJDick enabled auto-merge July 16, 2024 19:45
@RyanJDick RyanJDick merged commit 95e9f53 into main Jul 16, 2024
14 checks passed
@RyanJDick RyanJDick deleted the ryan/spandrel-upscale-tiling branch July 16, 2024 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files DO NOT MERGE invocations PRs that change invocations python PRs that change python files
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants