feat: canvas v2

[psychedelicious]

Summary

Canvas V2.

Broad strokes of changes

Single generation tab

There is a single generation tab with two modes:

• Generate: Generated images go directly to the gallery. This replaces the Generation tab workflow, aka yeet mode.
• Compose: Generated images are staged to be added to the canvas. This replaces the Canvas tab workflow.

Layers, Masks, Filters, Tools

Everything is a "layer" or "mask" now.

Layers

Layers can be drawn on, transformed, and filtered. A layer may be created from an image. Two kinds of layers:

1. Raster Layers are roughly equivalent to PS layers. When generating, all raster layers are virtually flattened and use as the initial image.
2. Control Layers are like raster layers, except instead of being flattened down to a single layer, they are individually used for ControlNet/T2IAdapter.

Masks

Masks can be drawn on and transformed. A future enhancement will allow using images as masks. No filters for masks - maybe in the future. Two kinds of masks:

1. Inpaint Masks are what you expect. When generating, like Raster Layers, all inpaint masks are virtually flattened into a single inpaint mask.
2. Regional Guidance Masks work the same as in the current release. When generating, like Control Layers, region masks are used individually along with their prompts and ip adapters.

Filters

All ControlNet "preprocessors" are now exposed as filters. Filters run on the server. Technically, if you can nodeify some kind of image processing, it could be a filter.

Tools

• Brush
• Eraser
• Rectangle
• Move
• View
• Bbox
• Color Picker

Full rewrite

Almost all internal generation logic for the linear UI is new, including graphs and rendering. The control layers implementation was rough draft for Canvas V2, but almost none of it remains.

Canvas rendering implementation

• In Canvas V1, we used konva's react bindings.
• In Control Layers, we used native konva and relied on react to trigger renders.
• In Canvas V2, we use native konva and skip react/redux almost entirely. All critical interactions are direct between user input and konva. Redux is still the source of truth, but we do far fewer round trips to it.

Revised app layout component composition

While the app overall won't look different, they way components are rendered is a bit different - hopefully providing a much snappier UI at the cost of a bit more up-front loading.

Hotkeys

Quite a few of these have changed, and I haven't updated the hotkeys list yet. Sorry.

QA Instructions

🚨 This PR includes an unstable database migration. 🚨

Make sure to set use_memory_db: true in your invokeai.yaml before testing.

Dev builds

I'll be publishing regular dev builds to pypi for testing. I'll list each build as I publish it here.

To test a build, activate your venv and run the listed command.

Build list

1. pip install InvokeAI==4.2.9.dev20240823

   • First dev build.

2. pip install InvokeAI==4.2.9.dev20240824

   • Fix: Rectangles don't draw on first render.
   • Feat: Colored mask preview images.
   • Feat: Improved color picker.
   • Fix: Entities are selected when added.
   • Feat: In Generate mode, inpainted and outpainted images are pasted back on to the source image. In Compose mode, inpainted and outpainted images are not pasted back on to the source image; they are transparent outside the generated region.
   • Feat: Add duplicate entity.

3. pip install InvokeAI==4.2.9.dev3

   • Realized using the date for dev version specifiers isn't going to work if I want to do two in a single day 😅 Using an incrementing number instead.
   • Fix: Error when using ControlNet.
   • Feat: Better logging when enqueuing via Results.

4. pip install InvokeAI==4.2.9.dev4

   • Fix: VAE precision not respected
   • Feat: Canvas entity list has context menu
   • Fix: Aspect ratio not respected when using auto scale
   • Fix: Color picker resetting brush opacity
   • Feat: undo/redo (WIP, lots of organization and cleanup included to support this)

5. pip install InvokeAI==4.2.9.dev5:

   • Rebased on main (so it includes current FLUX implementation, though I don't think inpainting or img2img work...)
   • Feat: Undo/redo. Involved a bit more shuffling of state. I needed to change a ton of selectors to accommodate the new state structure, and did a lot of optimizing along the way.
   • Feat: Collapsible entity groups.
   • Feat: Lockable entities - similar UI to affinity.
   • Feat: Updated action bar -> opacity, mask fill (if a mask layer is selected), add layer button and delete button. Hold shift to delete all.
   • Feat: Spruced up layer styles.

6. pip install InvokeAI==4.2.9.dev6:

   • Fix randomize seed
   • Fix control layer transparency effect not updating
   • Fix preview images sometimes not updating
   • Fix issue w/ queue menu button badge that cost the app a few vertical pixels
   • Layout/styling cleanup
   • Perf issue when opening modals
   • Add reset button to transform
   • Flawed implementation of flip transform (currently commented out pending big brain math)
   • Fix openapi schema generation issue on server that resulted in node pack not being added correctly to schema
   • Migrate add node popover to cmdk (test run for whole-app command palette/omnibar)

7. pip install InvokeAI==4.2.9.dev7:

   • Fix node autoconnect (missed a spot)

8. pip install InvokeAI==4.2.9.dev8

   • Fix: Add node popover interference with view tool
   • Fix: Queue count badge positioning, related layout issues
   • Feat: Remove undo/redo buttons - just hotkeys
   • Fix: Longstanding issue with invoke button loading state when parameters panel is collapsed
   • Feat: Do not select new layer on staging acceptance
   • Feat: Add + buttons to each entity category header
   • Feat: Add delete button to layer header component
   • Feat: Restore context menu for entity list (to add entities)
   • Feat: Hold Alt to temp switch to color picker
   • Feat: Add destination column to session_queue table ❗ This is a breaking change, you'll need to manually fix your DB or just create a new one. You were using a memory db as indicated in the PR, though, right? In case you weren't, here's how to fix it from the sqlite3 CLI, then the app should start up.

     SQLite version 3.37.2 2022-01-06 13:25:41
     Enter ".help" for usage hints.
     Connected to a transient in-memory database.
     Use ".open FILENAME" to reopen on a persistent database.
     sqlite> .open invokeai.db
     sqlite> delete from migrations where version=15;
     sqlite> alter table session_queue drop column origin;
     sqlite>
     

   • Feat: Add toggle for send images to gallery vs canvas. Internal handling for this functionality. Removed queue menu button to make space - its functions (quick access to pause, resume) will go into the cmdk omnibar once that is built.
   • Feat: Restore minimal HUD on canvas. Just shows bbox sizes for now.
   • Fix: Staging hotkeys active at wrong times

9. pip install InvokeAI==4.2.9.dev10 (build 9 had some issues):

   • Fix: Context menu wonkiness. Had to remove one of the right click menus (the one that showed options to add each layer type), couldn't figure out how to make nested menu triggers place nicely, something related to event bubbling.
   • Fix: Issues with modals staying open (like workflow editor settings modal)
   • Fix: Fit view always included the point (0,0) - utility function to get union of rects was incorrect. Fixed and added tests
   • Feat: Save to gallery button. Saves composite of all raster layers. Hold shift to save just the bbox region.
   • Feat: Merge visible. Works for inpaint masks and raster layers. Don't think it makes sense for this to be on the others, bc they have additional state like a prompt or controlnet, and there's no obvious way to merge those. Could be convinced otherwise - maybe merge the layer without the prompt/cnet/etc? Future enhancement to add "merge down".

10. pip install InvokeAI==4.2.9.dev11:

    • Internal: Improved canvas module base class, add docstrings, organise files a bit
    • Fix: Remove object count from entity title, was using it for debugging
    • Feat: Use default IP adapter model when creating Global IP Adapter
    • Feat: [ and ] hotkeys for brush/eraser size
    • Feat: Use improved number input w/ slider popup for brush & eraser width
    • Feat: Move bbox hotkey to c
    • Feat: Add alt+[ and alt+] hotkeys to cycle thru layers
    • Feat: Add q hotkey for quick switch between last two selected layers
    • Feat: One layer can be bookmarked as the dedicated quick switch layer, for example bookmark your inpaint mask and get back to it with q no matter what layer is selected
    • Tweak: "IP Adapter" -> "Global IP Adapter"
    • Tweak: "Control Layers" -> "Layers"
    • Feat: Render brush preview for mask layers at 50% opacity
    • Fix: Disable merge visible when layer type has 1 or 0 entities
    • Feat: Add fit to bbox button in transformer popup, this is useful to snap a control image to the bbox
    • Internal: Clean up CanvasEntityTransformer, move its state to nanostores atoms for reactivity
    • Internal: Clean up and document CanvasStateApi
    • Fix: Rasterization (via transform) baking transparency effect into images

11. pip install InvokeAI==4.2.9.dev12:

    • Internal: Refactored/simplified redux -> canvas state flow, rely more on redux store subscriptions & leverage reselect for memoization
    • Internal: ABCs for various classes
    • Internal: Updated docstrings
    • Internal: Removed a lot of extraneous state that had accumulated
    • Perf: Fix a few places where we canvas was drawn more often than necessary
    • Fix: Model-dependent size settings lost when resetting canvas
    • Fix: Deleting layer while transforming/filtering does not cancel the transform/filter
    • Fix: Save to Gallery doesn't use auto-add board
    • Feat: WIP interaction restrictions during transform, filter, staging
    • Feat: A few minor styling tweaks

12. pip install InvokeAI==5.0.0.dev13:

    • Chore: rebased on main
    • Feat: Improved transparency effect, reducing visual artifacts
    • Feat: Overlay all control layer images on a black bg when exporting them, eliminating undesired edges when the control image doesn't fully fit the bbox
    • Feat: Use cancel by destination instead of origin when canceling staging or resetting canvas, which allows send-to-gallery queue items to continue processing
    • Feat: Invert shift behavior when transforming layer to match photoshop & affinity
    • Feat: Restore aspect ratio preview component in image size area
    • Feat: Reworked image context menu - fix send to canvas, remove send to img2img, add open in viewer
    • Feat: Fit layers to stage on canvas reset
    • Feat: Revised entity action bars - move denoise to global bar, add selected entity bar w/ opacity/filter/transform/duplicate
    • Feat: Add fit bbox to layers button
    • Feat: Move seed out of advanced
    • Fix: Filter preview accidentally committed to layer
    • Fix: When transparency effect is enabled, filters ran on the layer w/ the effect applied
    • Fix: Really fix the bbox scaled size when resetting canvas
    • Fix: Ignore transparency effect when calculating layer bboxes
    • Fix: Transformer rect/handles rendered behind layer
    • Fix: Do not save filter previews to gallery
    • Fix: Invalid stage scale/size borking canvas, most common when loading the app w/ a non-canvas tab selected
    • Fix: Do not allow transform/filter/merge while staging
    • Fix: Do not allow invoke when transforming/filtering
    • Tidy: Temp hide HRF settings while the feature isn't enabled

UI broken?

It is possible that the UI will break when you first open it due to conflicts with persisted state. If the UI totally fails to load, you can run this snippet from the browser's JS console to wipe the problematic persisted state:

// Util to delete a db from the browser (will _not_ delete your SQLite server db!)
function deleteIndexedDB(dbName) {
  const request = indexedDB.deleteDatabase(dbName);
  request.onsuccess = () => { console.log(`Database '${dbName}' deleted successfully.`) };
  request.onerror = (e) => { console.error(`Error deleting database '${dbName}':`, e.target.error) };
  request.onblocked = () => { console.warn(`Database deletion is blocked, likely due to an open connection.`) };
}
// Delete the jacked up invoke db
deleteIndexedDB('invoke');

Non-exhaustive list of known issues/TODOs

Major

• Poor UX w/ generate and compose modes
• No undo/redo
• Metadata recall broken / in flux
• Bbox is currently a "tool". Probably shouldn't be.
• No way to save the whole canvas/selected region
• Canvas session persistence
• A record of user actions taken that result in the current canvas state, which may "flattened" into a single image

Minor

• Layer previews for region and inpaint masks are black shapes
• Hotkeys are scoped, but the scopes may not activate exactly when you expect
• New rectangles don't show up until you draw with the brush
• Color picker preview isn't very usable
• While staging, you can do a lot of stuff that you shouldn't be able to do, which will probably break stuff
• IP Adapters are in the layers area - maybe they should be elsewhere?
• A lot of hotkeys have changed from the current release, maybe in unexpected ways
• There are a number of actions which need hotkeys but don't have them yet
• Opacity slider is awkwardly placed

TODO

• Duplicate layer
• Merge layers
• Drag and drop to rearrange layers
• Collapsible layer groups
• Floating badges for active layers
• HUD / canvas status text
• Pressure sensitivity / tablet mode
• Eraser preview makes layer to be erased transparent

Wishlist

• Eraser brush opacity

Merge Plan

Do not push that button!

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)

[psychedelicious]

Re: eraser brush opacity.

This works fine! We can simply set the opacity of eraser lines to the desired opacity.

Screen.Recording.2024-08-25.at.12.48.40.am.mov

Unfortunately there is a bug when using partial opacity for eraser lines, where pixels that should be transparent (0,0,0,0 - black with alpha 0) end up as (255,255,255,1). That's almost transparent, but not quite. As a reminder, these are 8 bit values, so fully opaque alpha is 255.

The cause is related to how browsers/GPUs store and manipulate alpha channel data. In short, there is a subtle difference between an Image (like a PNG-encoded blob, or whatever the raw data on the GPU is) and ImageData (i.e. an array of 8 bit integers representing the pixels). Internally, the canvas has an Image, but when we want to access its pixels, it outputs them as a UInt8Array.

Here's what happens, admitting some gaps in my understanding:

• The eraser lines render with globalCompositeOperation: 'destination-out', which essentially erases whatever was drawn before it, proportional to the opacity of the eraser stroke.
• You set the eraser opacity to 10% and make a bunch of eraser strokes. Each stroke multiples the opacity of its pixels by 0.1. After however many strokes, the erased region appears to be fully transparent. In actuality, the opacity has an asymptote at 0 - it is not 0. Canvas holds this alpha data with a high precision.
• When we call getImageData to get the pixels and render them, the alpha is quantized to an integer from 0 to 255, ending up as 1. This is how the alpha turns from 0 to 1.
• My brain is mush from face-to-keyboard-ing over this and I don't fully grasp the math, but as described here this alpha value results in the r, g and b values being pushed to 255.

The logic that determines which graph to run is erroneously sees no transparent pixels (they all have alpha 1 isntead of 0), triggering the img2img graph. Not sure how to solve this. Apparently you can use WebGL to get the correct pixel data. Or maybe we could simplify our graphs and merge img2img, inpaint and outpaint into the same graph?

Another lovely effect of this problem is that different browsers can return slightly different values. Chrome gives you (255,255,255,2) instead of (255,255,255,1)... Joy...

---

Oh, and why isn't htis a problem with the existing eraser tool? Well, it is "opaque", so thanks to the compositing, the alpha channel of erased regions is set directly to 0. There's no quantization, so this problem doesn't occur.

[psychedelicious]

Dev build 4 - pip install InvokeAI==4.2.9.dev4:

• Fix: VAE precision not respected
• Feat: Canvas entity list has context menu
• Fix: Aspect ratio not respected when using auto scale
• Fix: Color picker resetting brush opacity
• Feat: undo/redo (WIP, lots of organization and cleanup included to support this)

[psychedelicious]

Dev build 5 - pip install InvokeAI==4.2.9.dev5:

• Rebased on main (so it includes current FLUX implementation, though I don't think inpainting or img2img work...)
• Feat: Undo/redo. Involved a bit more shuffling of state. I needed to change a ton of selectors to accommodate the new state structure, and did a lot of optimizing along the way.
• Feat: Collapsible entity groups.
• Feat: Lockable entities - similar UI to affinity.
• Feat: Updated action bar -> opacity, mask fill (if a mask layer is selected), add layer button and delete button. Hold shift to delete all.
• Feat: Spruced up layer styles.

Screen.Recording.2024-08-27.at.8.04.52.pm.mov

[psychedelicious]

• Dev build 6 - pip install InvokeAI==4.2.9.dev6:
  • Fix randomize seed
  • Fix control layer transparency effect not updating
  • Fix preview images sometimes not updating
  • Fix issue w/ queue menu button badge that cost the app a few vertical pixels
  • Layout/styling cleanup
  • Perf issue when opening modals
  • Add reset button to transform
  • Flawed implementation of flip transform (currently commented out pending big brain math)
  • Fix openapi schema generation issue on server that resulted in node pack not being added correctly to schema
  • Migrate add node popover to cmdk (test run for whole-app command palette/omnibar)

[psychedelicious]

Dev build 13 - pip install InvokeAI==5.0.0.dev13:

• Chore: rebased on main
• Feat: Improved transparency effect, reducing visual artifacts
• Feat: Overlay all control layer images on a black bg when exporting them, eliminating undesired edges when the control image doesn't fully fit the bbox
• Feat: Use cancel by destination instead of origin when canceling staging or resetting canvas, which allows send-to-gallery queue items to continue processing
• Feat: Invert shift behavior when transforming layer to match photoshop & affinity
• Feat: Restore aspect ratio preview component in image size area
• Feat: Reworked image context menu - fix send to canvas, remove send to img2img, add open in viewer
• Feat: Fit layers to stage on canvas reset
• Feat: Revised entity action bars - move denoise to global bar, add selected entity bar w/ opacity/filter/transform/duplicate
• Feat: Add fit bbox to layers button
• Feat: Move seed out of advanced
• Fix: Filter preview accidentally committed to layer
• Fix: When transparency effect is enabled, filters ran on the layer w/ the effect applied
• Fix: Really fix the bbox scaled size when resetting canvas
• Fix: Ignore transparency effect when calculating layer bboxes
• Fix: Transformer rect/handles rendered behind layer
• Fix: Do not save filter previews to gallery
• Fix: Invalid stage scale/size borking canvas, most common when loading the app w/ a non-canvas tab selected
• Fix: Do not allow transform/filter/merge while staging
• Fix: Do not allow invoke when transforming/filtering
• Tidy: Temp hide HRF settings while the feature isn't enabled