Modular backend - inpaint #6643

StAlKeR7779 · 2024-07-21T19:21:15Z

Summary

Code for inpainting and inpaint models handling from #6577.
Separated in 2 extensions as discussed briefly before, so wait for discussion about such implementation.

Related Issues / Discussions

#6606
https://invokeai.notion.site/Modular-Stable-Diffusion-Backend-Design-Document-e8952daab5d5472faecdc4a72d377b0d

QA Instructions

Run with and without set USE_MODULAR_DENOISE environment.
Try and compare outputs between backends in cases:

Normal generation on inpaint model
Inpainting on inpaint model
Inpainting on normal model

Merge Plan

Nope.
If you think that there should be some kind of tests - feel free to add.

Checklist

The PR has a short but descriptive title, suitable for a changelog
Tests added / updated (if applicable)
Documentation added / updated (if applicable)

…aint model without source image Co-Authored-By: Ryan Dick <[email protected]>

RyanJDick

I tried to test, but ran into a device mismatch error. I expect that it will be easy to reproduce, but if not let me know and I can provide more details.

Also, in addition to the test cases that you have already listed, we should make sure and test both gradient-mask and non-gradient-mask inpainting.

invokeai/backend/stable_diffusion/extensions/inpaint.py

invokeai/backend/stable_diffusion/extensions/inpaint_model.py

invokeai/backend/stable_diffusion/extensions/inpaint.py

invokeai/backend/stable_diffusion/extensions/inpaint_model.py

Co-Authored-By: Ryan Dick <[email protected]>

lstein

// invokeai/app/invocations/denoise_latents.py 
 def invoke(self, context: InvocationContext) -> LatentsOutput:
        if os.environ.get("USE_MODULAR_DENOISE", False):
            return self._new_invoke(context)
        else:
            return self._old_invoke(context)

During the transition period, why not make USE_MODULAR_DENOISE into an app configuration variable? This will then automatically set a config parameter based on an environment variable named INVOKEAI_USE_MODULAR_DENOISE and be consistent with how environment variables are handled elsewhere.

lstein · 2024-07-23T22:28:38Z

Ran into an apparent bug. When inpainting with stable-diffusion-xl-1.0-inpainting-0.1:

$ USE_MODULAR_DENOISE="1" invokeai-web
... multiple log lines omitted...
  File "/home/lstein/Projects/InvokeAI/invokeai/app/services/session_processor/session_processor_default.py", line 129, in run_node
    output = invocation.invoke_internal(context=context, services=self._services)
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/baseinvocation.py", line 289, in invoke_internal
    output = self.invoke(context)
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/denoise_latents.py", line 722, in invoke
    return self._new_invoke(context)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/denoise_latents.py", line 818, in _new_invoke
    result_latents = sd_backend.latents_from_embeddings(denoise_ctx, ext_manager)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/stable_diffusion/diffusion_backend.py", line 45, in latents_from_embeddings
    ext_manager.run_callback(ExtensionCallbackType.PRE_DENOISE_LOOP, ctx)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/stable_diffusion/extensions_manager.py", line 52, in run_callback
    cb.function(ctx)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/stable_diffusion/extensions/inpaint.py", line 77, in init_tensors
    raise ValueError("InpaintExt should be used only on normal models!")
ValueError: InpaintExt should be used only on normal models!

No such issue when USE_MODULAR_DENOISE is false. I checked the config.json for the unet, and it is 9 channels as expected for an inpainting model.

StAlKeR7779 · 2024-07-23T22:33:44Z

Ran into an apparent bug. When inpainting with stable-diffusion-xl-1.0-inpainting-0.1:

$ USE_MODULAR_DENOISE="1" invokeai-web
... multiple log lines omitted...
  File "/home/lstein/Projects/InvokeAI/invokeai/app/services/session_processor/session_processor_default.py", line 129, in run_node
    output = invocation.invoke_internal(context=context, services=self._services)
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/baseinvocation.py", line 289, in invoke_internal
    output = self.invoke(context)
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/denoise_latents.py", line 722, in invoke
    return self._new_invoke(context)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/denoise_latents.py", line 818, in _new_invoke
    result_latents = sd_backend.latents_from_embeddings(denoise_ctx, ext_manager)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/stable_diffusion/diffusion_backend.py", line 45, in latents_from_embeddings
    ext_manager.run_callback(ExtensionCallbackType.PRE_DENOISE_LOOP, ctx)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/stable_diffusion/extensions_manager.py", line 52, in run_callback
    cb.function(ctx)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/stable_diffusion/extensions/inpaint.py", line 77, in init_tensors
    raise ValueError("InpaintExt should be used only on normal models!")
ValueError: InpaintExt should be used only on normal models!

No such issue when USE_MODULAR_DENOISE is false. I checked the config.json for the unet, and it is 9 channels as expected for an inpainting model.

Could you debug if unet_config.variant correct on this model?
Because currently in code InpaintExt/InpaintModelExt selects based on unet_config.variant, while internally code checks for in_channels(and error from this check you see).

lstein · 2024-07-23T22:43:53Z

Could you debug if unet_config.variant correct on this model?
Because currently in code InpaintExt/InpaintModelExt selects based on unet_config.variant, while internally code checks for in_channels(and error from this check you see).

You called it correctly. The model was configured as "normal". Changing its variant to "inpaint" fixed the issue.

Now I have to figure out why the variant was set to normal - maybe an issue with the prober.
[edit] Reinstalling from HuggingFace autoprobed with "inpaint" correctly. Probably an issue with a old install.

RyanJDick

It might make more sense to split the CreateGradientMaskInvocation bug fix out into its own PR. What do you think? This would simplify the testing scope a bit. And will certainly be nice if a rollback is necessary. I'll leave it up to you. If we keep the bug fix in this PR, can you update the PR description to reflect that it also fixes a bug in CreateGradientMaskInvocation - which will have a small impact on behavior.

invokeai/app/invocations/denoise_latents.py

invokeai/app/invocations/create_gradient_mask.py

invokeai/app/invocations/denoise_latents.py

RyanJDick · 2024-07-26T15:06:37Z

RyanJDick · 2024-07-26T15:09:04Z

During the transition period, why not make USE_MODULAR_DENOISE into an app configuration variable? This will then automatically set a config parameter based on an environment variable named INVOKEAI_USE_MODULAR_DENOISE and be consistent with how environment variables are handled elsewhere.

I don't feel strongly either way. But, I didn't suggest it earlier since it's intended to be short-lived and for developers only, so I didn't want to have to worry about handling config migrations.

This reverts commit 9d1fcba.

Co-Authored-By: Ryan Dick <[email protected]>

StAlKeR7779 · 2024-07-26T19:06:47Z

invokeai/backend/stable_diffusion/extensions/inpaint_model.py

Co-Authored-By: Ryan Dick <[email protected]>

@dunkeroni

## Summary Gradient mask node outputs mask tensor with values in range [-1, 1], which unexpected range for mask. It handled in denoise node the way it translates to [0, 2] mask, which looks even more wrongly) From discussion with @dunkeroni I understand him as he thought that negative values will be treated same as 0, so clamping values not change intended node logic. ## Related Issues / Discussions #6643 ## QA Instructions \- ## Merge Plan \- ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Tests added / updated (if applicable)_ - [ ] _Documentation added / updated (if applicable)_

StAlKeR7779 added 2 commits July 21, 2024 20:45

Handle inpaint models

9e7b470

Handle inpainting on normal models

58f3072

StAlKeR7779 requested review from lstein, blessedcoolant, brandonrising, RyanJDick, hipsterusername and psychedelicious as code owners July 21, 2024 19:21

github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files labels Jul 21, 2024

StAlKeR7779 and others added 2 commits July 22, 2024 23:47

Same changes as in other PRs, add check for running inpainting on inp…

5003e5d

…aint model without source image Co-Authored-By: Ryan Dick <[email protected]>

Revert debug change

87eb018

RyanJDick requested changes Jul 23, 2024

View reviewed changes

StAlKeR7779 and others added 3 commits July 23, 2024 23:29

Fix create gradient mask node output

9d1fcba

Suggested changes

c323a76

Co-Authored-By: Ryan Dick <[email protected]>

Use non-inverted mask generally(except inpaint model handling)

19c0024

lstein reviewed Jul 23, 2024

View reviewed changes

Ruff format

416d29f

RyanJDick reviewed Jul 26, 2024

View reviewed changes

invokeai/app/invocations/denoise_latents.py Show resolved Hide resolved

invokeai/app/invocations/create_gradient_mask.py Outdated Show resolved Hide resolved

invokeai/app/invocations/denoise_latents.py Show resolved Hide resolved

Revert "Fix create gradient mask node output"

bd8890b

This reverts commit 9d1fcba.

StAlKeR7779 mentioned this pull request Jul 26, 2024

Fix gradient mask values range #6688

Merged

3 tasks

Suggested changes

5810cee

Co-Authored-By: Ryan Dick <[email protected]>

dunkeroni reviewed Jul 26, 2024

View reviewed changes

invokeai/backend/stable_diffusion/extensions/inpaint_model.py Show resolved Hide resolved

StAlKeR7779 and others added 2 commits July 27, 2024 13:18

Suggested changes

ed0174f

Co-Authored-By: Ryan Dick <[email protected]>

Revert wrong comment copy

84d0288

StAlKeR7779 requested a review from RyanJDick July 28, 2024 15:55

StAlKeR7779 mentioned this pull request Jul 28, 2024

Modular backend - overrides #6692

Open

3 tasks

RyanJDick approved these changes Jul 29, 2024

View reviewed changes

hipsterusername approved these changes Jul 29, 2024

View reviewed changes

Merge branch 'main' into stalker-modular_inpaint-2

693a3ea

RyanJDick enabled auto-merge July 29, 2024 14:15

RyanJDick merged commit 2ad13ac into invoke-ai:main Jul 29, 2024
14 checks passed

StAlKeR7779 deleted the stalker7779/modular_inpaint branch July 29, 2024 14:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modular backend - inpaint #6643

Modular backend - inpaint #6643

StAlKeR7779 commented Jul 21, 2024 •

edited

Loading

RyanJDick left a comment

lstein left a comment

lstein commented Jul 23, 2024 •

edited

Loading

StAlKeR7779 commented Jul 23, 2024

lstein commented Jul 23, 2024 •

edited

Loading

RyanJDick left a comment

RyanJDick commented Jul 26, 2024

RyanJDick commented Jul 26, 2024 •

edited

Loading

StAlKeR7779 commented Jul 26, 2024 •

edited

Loading

Modular backend - inpaint #6643

Modular backend - inpaint #6643

Conversation

StAlKeR7779 commented Jul 21, 2024 • edited Loading

Summary

Related Issues / Discussions

QA Instructions

Merge Plan

Checklist

RyanJDick left a comment

Choose a reason for hiding this comment

lstein left a comment

Choose a reason for hiding this comment

lstein commented Jul 23, 2024 • edited Loading

StAlKeR7779 commented Jul 23, 2024

lstein commented Jul 23, 2024 • edited Loading

RyanJDick left a comment

Choose a reason for hiding this comment

RyanJDick commented Jul 26, 2024

RyanJDick commented Jul 26, 2024 • edited Loading

StAlKeR7779 commented Jul 26, 2024 • edited Loading

StAlKeR7779 commented Jul 21, 2024 •

edited

Loading

lstein commented Jul 23, 2024 •

edited

Loading

lstein commented Jul 23, 2024 •

edited

Loading

RyanJDick commented Jul 26, 2024 •

edited

Loading

StAlKeR7779 commented Jul 26, 2024 •

edited

Loading