Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Regional Prompting #5868

Closed
wants to merge 49 commits into from
Closed

WIP: Regional Prompting #5868

wants to merge 49 commits into from

Conversation

RyanJDick
Copy link
Collaborator

What type of PR is this? (check all applicable)

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update
  • Community Node Submission

Have you discussed this change with the InvokeAI team?

  • Yes
  • No, because:

Have you updated all relevant documentation?

TODO: Update docs

  • Yes
  • No

Description

This branch is a WIP implementation of regional prompting.
TODO: Add description...

QA Instructions, Screenshots, Recordings

Test cases to consider:

  • CUDA, ROCm, mps, cpu
    • Check performance and memory utilization.
  • All possible attention settings(xformers, sliced, etc.)
    • Will xformers still work?
  • Compatibility with ControlNet, IP-Adapter, T2I-Adapter.
  • SD and SDXL
  • Combining prompts with / without regions.
  • Region prompts for both positive and negative conditioning.
  • ...

Merge Plan

DO NOT MERGE. This is a WIP.

Added/updated tests?

  • Yes
  • No

@RyanJDick
Fix avoid storing extra conditioning info in two places.
16e5748
@RyanJDick
Remove use of **kwargs in do_unet_step(...), where full parameter lis…
e866e3b 
…t is known and supported.
@RyanJDick
Remove outdated comments related to T2I-Adapters and ControlNets.
bf72cee
@RyanJDick
Merge sequential conditioning and cac conditioning logic to eliminate…
ee3abc1 
… a bunch of duplication.
@RyanJDick
Remove unused constructor declared with typo in name: __int__.
382fa57
@RyanJDick
Add a mask to the ConditioningField primitive type.
58277c6
@RyanJDick
Add support for a list of ConditioningFields in DenoiseLatents.
f590b39
@RyanJDick
Delete unused functions from shared_invokeai_diffusion.py.
7b0326d
@RyanJDick
Remove unused code for attention map saving.
ef51005
@RyanJDick
Initialize a RegionalPromptAttnProcessor2_0 class by copying AttnProc…
ba47880 
…essor2_0 from diffusers.
@RyanJDick
Fix a minor bug in the logic of the IPAttnProcessor2_0. The change wo…
38248b9 
…n't have any functional effect, since this attention implementation was only being used for cross-attention, but the logic should be correct now in case we wanted to use it for self-attention.
@RyanJDick
Add concatenation of multiple text conditioning tensors, and patching…
caa690e 
… of RegionalPromptAttnProcessor2_0 into the UNet.
@RyanJDick
Add RectangleMaskInvocation.
878bbc3
@RyanJDick
Route masks into the RegionalPromptAttnProcessor2_0 processors.
2d5d370
@RyanJDick
Get RegionalPromptAttnProcessor2_0 working with a ton of hacks.
d132fb4
@RyanJDick
Tidy invocation interfaces for RectangleMaskInvocation and AddConditi…
b0fcbe5 
…oningMaskInvocation.
@RyanJDick
Handle conditioned and unconditioned text conditioning in the same wa…
2966c8d 
…y for regional prompt attention.
@RyanJDick
Removed unused function: _prepare_text_embeddings(...)
cfba51a
@RyanJDick
Add symmetric support for regional negative text prompts.
54971af
@RyanJDick
Update various comments related to regional prompting, and delete dup…
845c4e9 
…licate _preprocess_regional_prompt_mask(...) function.
@RyanJDick
Remove dead code related to an old symmetry feature.
cad3e5d
@RyanJDick
Remove scheduler_args from ConditioningData structure.
e7ec13f
@RyanJDick
Split ip_adapter_conditioning out from ConditioningData.
ee1b315
@RyanJDick
Rename ConditioningData to TextConditioningData.
53ebca5
@RyanJDick
Move regional prompt concatenation further up the stack. This solves …
5f49e7a 
…a number of issues.
@RyanJDick
Make regional prompting work with sequential conditioning.
e132afb
@RyanJDick
Raise a clear error message if prompt-to-prompt cross-attention contr…
e7f7ae6 
…ol is triggered when using multiple prompts.
@RyanJDick
Improve the logic for selecting SDXL pooled embeds when handling mult…
bdf3691 
…i-region prompts.
@RyanJDick
Fixup logic around compatibility of prompt-to-prompt, IP-Adapter, reg…
1bbd4f7 
…ional prompting.
@RyanJDick
Add CustomAttnProcessor2_0 class with simultaneous support for IP-Ada…
f44d3da 
…pter and regional prompting.
@RyanJDick
Get multi-prompt attention working simultaneously with IP-adapter.
8989a6c
@RyanJDick
Fix avoid storing extra conditioning info in two places.
4a1acd4
@RyanJDick
Remove use of **kwargs in do_unet_step(...), where full parameter lis…
7d96710 
…t is known and supported.
@RyanJDick
Remove outdated comments related to T2I-Adapters and ControlNets.
d87ff3a
@RyanJDick
Merge sequential conditioning and cac conditioning logic to eliminate…
8721926 
… a bunch of duplication.
@RyanJDick
Remove unused constructor declared with typo in name: __int__.
3e14bd6
@RyanJDick
Delete unused functions from shared_invokeai_diffusion.py.
a5c94fb
@RyanJDick
Remove unused code for attention map saving.
5b3adf0
@RyanJDick
Merge branch 'ryan/remove-attention-map-saving' into ryan/regional-co…
ffc4ebb 
…nditioning
@RyanJDick
Implement (very slow) self-attention regional masking.
942efa0
@RyanJDick
Very experimentation with various regional prompting tuning params.
ad18429
@RyanJDick
Add ability to control regional prompt region weights.
5fad379
@RyanJDick
Merge branch 'main' into ryan/regional-conditioning-tuning
271f8f2
@RyanJDick
Remove AddConditioningMaskInvocaton.
d313e5e
@RyanJDick
Add positive_self_attn_mask_score and self_attn_adjustment_end_step_p…
a665f20 
…ercent to the prompt nodes.
@RyanJDick
(minor) Remove commented code.
bcfb43e
@RyanJDick
Use the correct device / dtype for RegionalPromptData calculations.
41e1a9f
@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files python-tests PRs that change python tests labels Mar 5, 2024
@RyanJDick
Copy link
Collaborator Author

Superseded by #5916

@RyanJDick RyanJDick closed this Mar 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lstein lstein Awaiting requested review from lstein lstein will be requested when the pull request is marked ready for review lstein is a code owner

@blessedcoolant blessedcoolant Awaiting requested review from blessedcoolant blessedcoolant will be requested when the pull request is marked ready for review blessedcoolant is a code owner

@brandonrising brandonrising Awaiting requested review from brandonrising brandonrising will be requested when the pull request is marked ready for review brandonrising is a code owner

@hipsterusername hipsterusername Awaiting requested review from hipsterusername hipsterusername will be requested when the pull request is marked ready for review hipsterusername is a code owner

@psychedelicious psychedelicious Awaiting requested review from psychedelicious psychedelicious will be requested when the pull request is marked ready for review psychedelicious is a code owner

Assignees
No one assigned
Labels
backend PRs that change backend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@RyanJDick