-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add tiling to SpandrelImageToImageInvocation #6594
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RyanJDick
requested review from
lstein,
blessedcoolant,
brandonrising,
hipsterusername and
psychedelicious
as code owners
July 10, 2024 17:01
github-actions
bot
added
python
PRs that change python files
invocations
PRs that change invocations
backend
PRs that change backend files
labels
Jul 10, 2024
Merged
10 tasks
RyanJDick
force-pushed
the
ryan/spandrel-upscale-tiling
branch
from
July 10, 2024 18:25
e8df07d
to
d0d2955
Compare
1 task
What do you think of adding something like this to the tile loop? --- a/invokeai/app/invocations/spandrel_image_to_image.py
+++ b/invokeai/app/invocations/spandrel_image_to_image.py
@@ -14,6 +14,7 @@ from invokeai.app.invocations.fields import (
)
from invokeai.app.invocations.model import ModelIdentifierField
from invokeai.app.invocations.primitives import ImageOutput
+from invokeai.app.services.session_processor.session_processor_common import CanceledException
from invokeai.app.services.shared.invocation_context import InvocationContext
from invokeai.backend.spandrel_image_to_image_model import SpandrelImageToImageModel
from invokeai.backend.tiles.tiles import calc_tiles_min_overlap
@@ -104,6 +105,9 @@ class SpandrelImageToImageInvocation(BaseInvocation, WithMetadata, WithBoard):
image_tensor = image_tensor.to(device=spandrel_model.device, dtype=spandrel_model.dtype)
for tile, scaled_tile in tqdm(list(zip(tiles, scaled_tiles, strict=True)), desc="Upscaling Tiles"):
+ if context.util.is_canceled():
+ raise CanceledException
+
# Extract the current tile from the input tensor.
input_tile = image_tensor[
:, :, tile.coords.top : tile.coords.bottom, tile.coords.left : tile.coords.right |
brandonrising
approved these changes
Jul 11, 2024
hipsterusername
approved these changes
Jul 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
backend
PRs that change backend files
DO NOT MERGE
invocations
PRs that change invocations
python
PRs that change python files
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add tiling to the
SpandrelImageToImageInvocation
node so that it can process large images.Tiling enables this node to run on effectively any input image dimension. Of course, the computation time increases quadratically with the image dimension.
Some profiling results on an RTX4090:
13 secs
,<4 GB VRAM
46 secs
,<4 GB VRAM
165 secs
,<5 GB VRAM
A lot of the time is spent PNG encoding the final image:
83secs @ pil_compress_level=7
,24secs @ pil_compress_level=1
Callout: If we want to start building workflows that pass large images between nodes, we are going to have to find a way to avoid the PNG encode/decode roundtrip that we are currently doing. As is, we will be incurring a huge penalty for every node that receives/produces a large image.
QA Instructions
tile_size=0
)Merge Plan
main
.Checklist