Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add control_mode parameter to ControlNet #3535

Merged
merged 18 commits into from
Jun 25, 2023

Conversation

GreggHelt2
Copy link
Contributor

@GreggHelt2 GreggHelt2 commented Jun 14, 2023

This PR adds the "control_mode" option to ControlNet implementation.
Possible control_mode options are:

  • balanced -- this is the default, same as previous implementation without control_mode
  • more_prompt -- pays more attention to the prompt
  • more _control -- pays more attention to the ControlNet (in earlier implementations this was called "guess_mode")
  • unbalanced -- pays even more attention to the ControlNet

balanced, more_prompt, and more_control should be nearly identical to the equivalent options in the auto1111 sd-webui-controlnet extension

The changes to enable balanced, more_prompt, and more_control are managed deeper in the code by two booleans, "soft_injection" and "cfg_injection". The three control mode options in sd-webui-controlnet map to these booleans like:

!soft_injection && !cfg_injection ⇒ BALANCED
soft_injection && cfg_injection ⇒ MORE_CONTROL
soft_injection && !cfg_injection ⇒ MORE_PROMPT

The "unbalanced" option simply exposes the fourth possible combination of these two booleans:
!soft_injection && cfg_injection ⇒ UNBALANCED

With "unbalanced" mode it is very easy to overdrive the controlnet inputs. It's recommended to use a cfg_scale between 2 and 4 to mitigate this, along with lowering controlnet weight and possibly lowering "end step percent". With those caveats, "unbalanced" can yield interesting results.

Example of all four modes using Canny edge detection ControlNet with prompt "old man", identical params except for control_mode:

Screenshot from 2023-06-11 23-53-00
Top middle: BALANCED
Top right: MORE_CONTROL
Bottom middle: MORE_PROMPT
Bottom right : UNBALANCED

I kind of chose this seed because it shows pretty rough results with BALANCED (the default), but in my opinion better results with both MORE_CONTROL and MORE_PROMPT. And you can definitely see how MORE_PROMPT pays more attention to the prompt, and MORE_CONTROL pays more attention to the control image. And shows that UNBALANCED with default cfg_scale etc is unusable.

But here are four examples from same series (same seed etc), all have control_mode = UNBALANCED but now cfg_scale is set to 3.
Screenshot from 2023-06-11 23-48-44
And param differences are:
Top middle: prompt="old man", control_weight=0.3, end_step_percent=0.5
Top right: prompt="old man", control_weight=0.4, end_step_percent=1.0
Bottom middle: prompt=None, control_weight=0.3, end_step_percent=0.5
Bottom right: prompt=None, control_weight=0.4, end_step_percent=1.0

So with the right settings UNBALANCED seems useful.

Copy link
Collaborator

@psychedelicious psychedelicious left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two issues:

  • There is no default control mode
  • The UI placement of the control mode selector doesn't feel correct, this should probably be in the expanded view only

image

@blessedcoolant
Copy link
Collaborator

blessedcoolant commented Jun 14, 2023

Two issues:

  • There is no default control mode
  • The UI placement of the control mode selector doesn't feel correct, this should probably be in the expanded view only

image

There is .. but your Cnet must have been added prior to this PR so it doesnt pick it. Delete it and readd it. The default will show up.

As for the placement, I tried to put it in line with the two others .. but that just meant increasing the length of the image preview..which looked really weird. So opted for this.

Also this should be a primary option. But I dont mind if it goes into advanced. Either works. Your call on that.

@psychedelicious
Copy link
Collaborator

psychedelicious commented Jun 15, 2023

Yep I needed to create a new cnet.

The mode makes the mini view a lot less mini, and it's a big ui element for a single setting. Reluctant to make the mini view any taller, every extra item in param panel makes it less usable :/ just not an easy thing to design. I'd prefer to have it under the expanded view bc otherwise the mini view is a bit too big and cluttered.

@blessedcoolant
Copy link
Collaborator

@psychedelicious It's a primary setting though. If you wanna push it down, feel free. I don't mind. But the bottom part was all processor stuff. And this one isn't. So it needs a new category along with any other future options that might be added.

@psychedelicious
Copy link
Collaborator

I attempted to resolve conflicts but rebasing isn't doing what I expected. I'll defer to @GreggHelt2 to resolve the python side, maybe it should wait until the model manager stuff is in...

…lnet-control-modes

Only "real" conflicts were in:
     invokeai/frontend/web/src/features/controlNet/components/ControlNet.tsx
     invokeai/frontend/web/src/features/controlNet/store/controlNetSlice.ts
@GreggHelt2
Copy link
Contributor Author

Okay I've got current main merged into this branch, and control_mode is working in both node UI and linear UI.
Couldn't pull off a rebase, so setting up for a squash and merge once PR is approved.

Copy link
Collaborator

@psychedelicious psychedelicious left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed a couple bits post-api change, looks good to go!

@blessedcoolant blessedcoolant merged commit 922468b into main Jun 25, 2023
1 of 7 checks passed
@blessedcoolant blessedcoolant deleted the feat/controlnet-control-modes branch June 25, 2023 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants