ONNX Support #3562

StAlKeR7779 · 2023-06-20T23:19:00Z

Note: this branch based on #3548, not on main

While find out what needs to be done to implement onnx, found that I can do draft of it pretty quickly, so... here it is)
Supports LoRA and TI.
As example - cat with sadcatmeme lora:

Update the text to imaeg and image to image graphs to work with the new model loader. Currently only supports 1.x models. Will update this soon to make it work with all models.

Co-Authored-By: StAlKeR7779 <[email protected]>

Basically updated all slices to be more descriptive in their names. Did so in order to make sure theres good naming scheme available for secondary models.

This reverts commit e0c105f.

So the long names do not get cut off.

…ms to model schema(as they not not referenced in case of Literal definition)

RyanJDick · 2023-07-27T19:58:11Z

Was that yoda image done on an olive model per chance? Olive models don't support applying loras I believe. If it's just a standard onnx model, could you send me the model and lora you used?

Correct. It was an olive-optimized model.

lstein · 2023-07-30T01:11:20Z

Comments:

I had to pip install onnxruntime.
Ran invokeai-model-install --add axodoxian/stable_diffusion_onnx from the command line -- worked
Confirmed that model appears in the Web Model Manager. It doesn't appear in the model manager TUI however; I can help with this.
Tried the text2image linear workflow and got this:

[2023-07-29 20:51:55,415]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/home/lstein/Projects/InvokeAI/invokeai/app/services/processor.py", line 86, in __process
    outputs = invocation.invoke(
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/onnx.py", line 261, in invoke
    unet.create_session(h, w)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 625, in create_session
    raise e
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 623, in create_session
    self.session = InferenceSession(self.proto.SerializeToString(), providers=providers, sess_options=sess)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 383, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 435, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

[2023-07-29 20:51:55,419]::[InvokeAI]::ERROR --> Error while invoking:
[ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

Searched the web and found onnxruntime-gpu. Installed that.
Text2Image works, getting ~9 it/s for a 512x512 image. Stock SD-1.5 is giving ~8 it/s at same resolution.
Img2Img not working for me. The model doesn't appear in the model selection menu.
Tried the node editor. Graphs made with ONNXSD1ModelLoader did not work (validation errors), but Onnx Model Loader program loaded and rendered an image.

Overall very positive experience. Just need a better step-by-step guide for people like me who are coming to the ONNX architecture from basically zero.

lstein

Needs img2img and canvas support and a better step-by-step guide. Overall, though, looks great.

brandonrising · 2023-07-31T13:26:39Z

Comments:

I had to pip install onnxruntime.

Ran invokeai-model-install --add axodoxian/stable_diffusion_onnx from the command line -- worked

Confirmed that model appears in the Web Model Manager. It doesn't appear in the model manager TUI however; I can help with this.

Tried the text2image linear workflow and got this:
[2023-07-29 20:51:55,415]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/home/lstein/Projects/InvokeAI/invokeai/app/services/processor.py", line 86, in __process
    outputs = invocation.invoke(
  File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/onnx.py", line 261, in invoke
    unet.create_session(h, w)
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 625, in create_session
    raise e
  File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 623, in create_session
    self.session = InferenceSession(self.proto.SerializeToString(), providers=providers, sess_options=sess)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 383, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 435, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

[2023-07-29 20:51:55,419]::[InvokeAI]::ERROR --> Error while invoking:
[ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found
Searched the web and found onnxruntime-gpu. Installed that.

Text2Image works, getting ~9 it/s for a 512x512 image. Stock SD-1.5 is giving ~8 it/s at same resolution.

Img2Img not working for me. The model doesn't appear in the model selection menu.

Tried the node editor. Graphs made with ONNXSD1ModelLoader did not work (validation errors), but Onnx Model Loader program loaded and rendered an image.

Overall very positive experience. Just need a better step-by-step guide for people like me who are coming to the ONNX architecture from basically zero.

Thanks for checking it out! I updated the installer to install onnxruntime/onnxruntime-gpu/onnxruntime-directml through optional dependencies based on their elections for different combinations of GPUs and Driver installations. Do you think updating the Readme would suffice telling people they need to install the correct onnxruntime for their environment?

Also yeah, I currently have it setup to only allow onnxruntime for linear text to image and node-editor. It's disabled on all other screens. I was thinking it would be good to go ahead and get it out with minimal functionality and slowly roll out more features as we move forward in separate PRs rather than maintaining this as it gets bigger. Definitely open to a discussion around this though.

I'll definitely work on better documentation for how to use ONNX in workflows, definitely a weak spot in this PR!

…aller

hipsterusername · 2023-07-31T21:34:44Z

Merged after discussion w/ LStein & Brandon in discord.

psychedelicious

There are a number of issues that need to be fixed.

Also, the PR was merged with 9 errors on the frontend check, there are a number of typing problems. I've added comments for each problem.

psychedelicious · 2023-07-31T23:16:25Z

pyproject.toml

@@ -61,6 +61,7 @@ dependencies = [
  "numpy",
  "npyscreen",
  "omegaconf",
+  "onnx",


I had to install this after updating to main in order for the app to run

then I also had to install onnxruntime manually, I guess because onnx was installed, otherwise the app did not start up

I did not ask for onnx. So I'm not sure the optional dependencies are working as expected.

...d/web/src/features/ui/components/tabs/ModelManager/subpanels/ModelManagerPanel/ModelList.tsx

psychedelicious · 2023-07-31T23:19:19Z

invokeai/frontend/web/src/features/parameters/types/parameterSchemas.ts

 export const zMainModel = z.object({
  model_name: z.string().min(1),
  base_model: zBaseModel,
+  model_type: zModelType,
 });

 /**
 * Type alias for model parameter, inferred from its zod schema
 */
 export type MainModelParam = z.infer<typeof zMainModel>;
+
+/**
+ * Type alias for model parameter, inferred from its zod schema
+ */
+export type OnnxModelParam = z.infer<typeof zMainModel>;


Schemas with single valid values for a property should just have the literal there.

If the main model schema needs model_type now, it should have model_type: z.literal("main"). Same for onnx model schema. Otherwise the client will accept an onnx or sd main model in either situation.

invokeai/frontend/web/src/services/api/endpoints/models.ts

invokeai/frontend/web/src/features/nodes/util/graphBuilders/buildLinearTextToImageGraph.ts

invokeai/frontend/web/src/services/api/types.d.ts

...d/web/src/features/ui/components/tabs/ModelManager/subpanels/ModelManagerPanel/ModelList.tsx

invokeai/frontend/web/src/features/nodes/util/graphBuilders/buildCanvasTextToImageGraph.ts

invokeai/frontend/web/src/features/nodes/util/graphBuilders/buildLinearTextToImageGraph.ts

lalith-mcw · 2023-08-02T08:13:39Z

[2023-07-29 20:51:55,415]::[InvokeAI]::ERROR --> Traceback (most recent call last):
File "/home/lstein/Projects/InvokeAI/invokeai/app/services/processor.py", line 86, in __process
outputs = invocation.invoke(
File "/home/lstein/Projects/InvokeAI/invokeai/app/invocations/onnx.py", line 261, in invoke
unet.create_session(h, w)
File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 625, in create_session
raise e
File "/home/lstein/Projects/InvokeAI/invokeai/backend/model_management/models/base.py", line 623, in create_session
self.session = InferenceSession(self.proto.SerializeToString(), providers=providers, sess_options=sess)
File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 383, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/home/lstein/invokeai-main/.venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 435, in _create_inference_session
sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

[2023-07-29 20:51:55,419]::[InvokeAI]::ERROR --> Error while invoking:
[ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Failed to find kernel for GroupNorm(1) (node GroupNorm_0). Kernel not found

This error shouldn't be there if I am trying to run the CPU version. When I try to install the CPU specific toolsets still the error is seen

blessedcoolant and others added 30 commits June 17, 2023 21:28

chore: Update model config type names

67d05d2

chore: Rebuild API

76dd749

fix: Update missing name types to new names

c8dfa49

wip: Update Linear UI Txt2Img and Img2Img Graphs

7a66856

Update the text to imaeg and image to image graphs to work with the new model loader. Currently only supports 1.x models. Will update this soon to make it work with all models.

Generate config names for openapi

16dc78f

chore: Rebuild API with new Model API names

0f3b7d2

wip: Add 2.x Models to the Model List

ce4110b

Add name, base_mode, type fields to model info

dc669d1

chore: Rebuild API - base_model and type added

24673fd

fix: 2.1 models breaking generation

bf0577c

Co-Authored-By: StAlKeR7779 <[email protected]>

feat: Enable 2.x Model Generation in Linear UI

61c426f

wip: Move Model Selector to own file

4133d77

cleanup: Updated model slice names to be more descriptive

28373db

Basically updated all slices to be more descriptive in their names. Did so in order to make sure theres good naming scheme available for secondary models.

Merge branch 'main' into model-manager-ui-30

f0bf32c

feat: Port Schedulers to Mantine

e0c105f

revert: getModels to receivedModels

9634c96

fix: Unserialization key issue

7c9a939

fix: Remove type from Model type name

809ec71

Revert "feat: Port Schedulers to Mantine"

9fda21c

This reverts commit e0c105f.

Merge branch 'main' into model-manager-ui-30

91016d8

fix: merge conflicts

17e2a35

Merge branch 'main' into model-manager-ui-30

d493152

Merge branch 'main' into model-manager-ui-30

b0c4451

fix: Adjust the Schedular select width

cfe81b5

So the long names do not get cut off.

tweal: UI colors

85b4b35

Merge branch 'main' into model-manager-ui-30

7df7a95

Remove default model logic

82b73c5

Rename format to model_format(still named format when work with config)

4cefe37

Update model format field to use enums

46dc751

Set model type to const value in openapi schema, add model format enu…

92c86fd

…ms to model schema(as they not not referenced in case of Literal definition)

brandonrising added 8 commits July 28, 2023 09:32

Just install onnxruntime by default

dc11481

Run python black

2b7b3dd

Merge branch 'main' into feat/onnx

da751da

Run Python black

a2aa66f

Fix issues caused by merge

8935ae0

Fix onnx installer

390ce9f

Remove onnxruntime

d3f6c7f

Update installer

1bbf2f2

brandonrising requested a review from ebr as a code owner July 29, 2023 01:02

Merge branch 'main' into feat/onnx

6ca0c38

lstein requested changes Jul 30, 2023

View reviewed changes

brandonrising changed the title ~~[WIP] ONNX Support~~ ONNX Support Jul 31, 2023

brandonrising and others added 6 commits July 31, 2023 10:58

Merge branch 'main' into feat/onnx

f5ac73b

Regen schema and rebuild frontend after merging main

1bafbaf

Some cleanup after the merge

f784e84

Merge branch 'main' into feat/onnx

af4fd32

Run python black, point out that onnx is an alpha feature in the inst…

aeac557

…aller

Merge branch 'main' into feat/onnx

746afcd

hipsterusername merged commit 81654da into main Jul 31, 2023
7 of 8 checks passed

hipsterusername deleted the feat/onnx branch July 31, 2023 21:34

psychedelicious reviewed Jul 31, 2023

View reviewed changes

hipsterusername restored the feat/onnx branch August 1, 2023 02:57

hipsterusername mentioned this pull request Aug 1, 2023

Revert "ONNX Support" #4104

Closed

psychedelicious mentioned this pull request Aug 1, 2023

fix(ui): post-onnx fixes #4105

Merged

12 tasks

psychedelicious mentioned this pull request Aug 5, 2023

Onnx Pipeline: Inference for text to image conversion #3380

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ONNX Support #3562

ONNX Support #3562

StAlKeR7779 commented Jun 20, 2023 •

edited

Loading

RyanJDick commented Jul 27, 2023

lstein commented Jul 30, 2023

lstein left a comment

brandonrising commented Jul 31, 2023

hipsterusername commented Jul 31, 2023

psychedelicious left a comment

psychedelicious Jul 31, 2023

psychedelicious Jul 31, 2023

lalith-mcw commented Aug 2, 2023

ONNX Support #3562

ONNX Support #3562

Conversation

StAlKeR7779 commented Jun 20, 2023 • edited Loading

RyanJDick commented Jul 27, 2023

lstein commented Jul 30, 2023

lstein left a comment

Choose a reason for hiding this comment

brandonrising commented Jul 31, 2023

hipsterusername commented Jul 31, 2023

psychedelicious left a comment

Choose a reason for hiding this comment

psychedelicious Jul 31, 2023

Choose a reason for hiding this comment

psychedelicious Jul 31, 2023

Choose a reason for hiding this comment

lalith-mcw commented Aug 2, 2023

StAlKeR7779 commented Jun 20, 2023 •

edited

Loading