We updated the PowerPiant model

[zhuang2002]

We have recently developed a new version of PowerPaint, drawing inspiration from Brushnet. The Online Demo has been updated accordingly. In the coming days, we intend to make the model weights available as open source.

Feel free to try it out and give it a star!

The PowerPaint model possesses the ability to carry out diverse inpainting tasks, such as object insertion, object removal, shape-guided object insertion, and outpainting. Notably, we have retained the cross-attention layer that BrushNet had removed, which is essential for task prompt input.

	Object insertion	Object Removal	Shape-guided Object Insertion	Outpainting
Original Image
Output

[Sanster]

Hi, thank you for releasing the new model! I have a question: Can the brushnet weights and text_encoder weights from PowerPaint_v2 be used with any SD1.5 model? For example, runwayml/stable-diffusion-v1-5 or Lykon/DreamShaper

[zhuang2002]

Hi, thank you for releasing the new model! I have a question: Can the brushnet weights and text_encoder weights from PowerPaint_v2 be used with any SD1.5 model? For example, runwayml/stable-diffusion-v1-5 or Lykon/DreamShaper

Yes, it should be feasible. In fact, we use SD 1.5 as the base model for training.

[FlightsMonitor]

fwiw, the old PowerPaint model is not working on hugging face any more. Here's the error I'm getting:

Expand for more Details

===== Application Startup at 2024-04-11 23:38:18 =====

2024-04-12 01:41:46.038 | INFO     | iopaint.runtime:setup_model_dir:85 - Create model directory: /home/user/.cache
- Platform: Linux-5.10.192-183.736.amzn2.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.14
- torch: 2.2.2
- torchvision: 0.17.2
- Pillow: 9.5.0
- diffusers: 0.25.1
- transformers: 4.39.3
- opencv-python: 4.9.0.80
- accelerate: 0.29.2
- iopaint: 1.0.0b10
- rembg: N/A
- realesrgan: N/A
- gfpgan: N/A

The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.


0it [00:00, ?it/s]
0it [00:00, ?it/s]
2024-04-12 01:41:49.196 | INFO     | iopaint.cli:start:162 - Fantasy-Studio/Paint-by-Example not found in /home/user/.cache, try to downloading
2024-04-12 01:41:49.196 | INFO     | iopaint.download:cli_download_model:33 - Downloading model from Huggingface: Fantasy-Studio/Paint-by-Example
2024-04-12 01:41:49.297 | INFO     | iopaint.model.utils:handle_from_pretrained_exceptions:985 - variant=fp16 not found, try revision=fp16
Couldn't connect to the Hub: 404 Client Error. (Request ID: Root=1-6618753d-110cabeb693eb3193b468c96;9a5da8fe-535e-45c7-b654-1854afd869ff)

Revision Not Found for url: https://huggingface.co/api/models/Fantasy-Studio/Paint-by-Example/revision/fp16.
Invalid rev id: fp16.
Will try to load from local cache.
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/iopaint/mode │
│ l/utils.py:982 in handle_from_pretrained_exceptions                          │
│                                                                              │
│    979                                                                       │
│    980 def handle_from_pretrained_exceptions(func, **kwargs):                │
│    981 │   try:                                                              │
│ ❱  982 │   │   return func(**kwargs)                                         │
│    983 │   except ValueError as e:                                           │
│    984 │   │   if "You are trying to load the model files of the `variant=fp │
│    985 │   │   │   logger.info("variant=fp16 not found, try revision=fp16")  │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/huggingface_ │
│ hub/utils/_validators.py:119 in _inner_fn                                    │
│                                                                              │
│   116 │   │   if check_use_auth_token:                                       │
│   117 │   │   │   kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__na │
│   118 │   │                                                                  │
│ ❱ 119 │   │   return fn(*args, **kwargs)                                     │
│   120 │                                                                      │
│   121 │   return _inner_fn  # type: ignore                                   │
│   122                                                                        │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/diffusers/pi │
│ pelines/pipeline_utils.py:1711 in download                                   │
│                                                                              │
│   1708 │   │   │   │   │   "if such variant modeling files are not available │
│   1709 │   │   │   │   │   "modeling files is deprecated."                   │
│   1710 │   │   │   │   )                                                     │
│ ❱ 1711 │   │   │   │   deprecate("no variant default", "0.24.0", deprecation │
│   1712 │   │   │                                                             │
│   1713 │   │   │   # remove ignored filenames                                │
│   1714 │   │   │   model_filenames = set(model_filenames) - set(ignore_filen │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/diffusers/ut │
│ ils/deprecation_utils.py:18 in deprecate                                     │
│                                                                              │
│   15 │                                                                       │
│   16 │   for attribute, version_name, message in args:                       │
│   17 │   │   if version.parse(version.parse(__version__).base_version) >= ve │
│ ❱ 18 │   │   │   raise ValueError(                                           │
│   19 │   │   │   │   f"The deprecation tuple {(attribute, version_name, mess │
│   20 │   │   │   │   f" version {__version__} is >= {version_name}"          │
│   21 │   │   │   )                                                           │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: The deprecation tuple ('no variant default', '0.24.0', "You are 
trying to load the model files of the `variant=fp16`, but no such modeling files
are available.The default model files: {'image_encoder/pytorch_model.bin', 
'vae/diffusion_pytorch_model.bin', 'unet/diffusion_pytorch_model.bin'} will be 
loaded instead. Make sure to not load from `variant=fp16`if such variant 
modeling files are not available. Doing so will lead to an error in v0.24.0 as 
defaulting to non-variantmodeling files is deprecated.") should be removed since
diffusers' version 0.25.1 is >= 0.24.0

During handling of the above exception, another exception occurred:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/huggingface_ │
│ hub/utils/_errors.py:304 in hf_raise_for_status                              │
│                                                                              │
│   301 │   </Tip>                                                             │
│   302 │   """                                                                │
│   303 │   try:                                                               │
│ ❱ 304 │   │   response.raise_for_status()                                    │
│   305 │   except HTTPError as e:                                             │
│   306 │   │   error_code = response.headers.get("X-Error-Code")              │
│   307 │   │   error_message = response.headers.get("X-Error-Message")        │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/requests/mod │
│ els.py:1021 in raise_for_status                                              │
│                                                                              │
│   1018 │   │   │   )                                                         │
│   1019 │   │                                                                 │
│   1020 │   │   if http_error_msg:                                            │
│ ❱ 1021 │   │   │   raise HTTPError(http_error_msg, response=self)            │
│   1022 │                                                                     │
│   1023 │   def close(self):                                                  │
│   1024 │   │   """Releases the connection back to the pool. Once this method │
╰──────────────────────────────────────────────────────────────────────────────╯
HTTPError: 404 Client Error: Not Found for url: 
https://huggingface.co/api/models/Fantasy-Studio/Paint-by-Example/revision/fp16

The above exception was the direct cause of the following exception:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/diffusers/pi │
│ pelines/pipeline_utils.py:1656 in download                                   │
│                                                                              │
│   1653 │   │   model_info_call_error: Optional[Exception] = None             │
│   1654 │   │   if not local_files_only:                                      │
│   1655 │   │   │   try:                                                      │
│ ❱ 1656 │   │   │   │   info = model_info(pretrained_model_name, token=token, │
│   1657 │   │   │   except (HTTPError, OfflineModeIsEnabled, requests.Connect │
│   1658 │   │   │   │   logger.warn(f"Couldn't connect to the Hub: {e}.\nWill │
│   1659 │   │   │   │   local_files_only = True                               │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/huggingface_ │
│ hub/utils/_validators.py:119 in _inner_fn                                    │
│                                                                              │
│   116 │   │   if check_use_auth_token:                                       │
│   117 │   │   │   kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__na │
│   118 │   │                                                                  │
│ ❱ 119 │   │   return fn(*args, **kwargs)                                     │
│   120 │                                                                      │
│   121 │   return _inner_fn  # type: ignore                                   │
│   122                                                                        │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/huggingface_ │
│ hub/hf_api.py:2228 in model_info                                             │
│                                                                              │
│   2225 │   │   if files_metadata:                                            │
│   2226 │   │   │   params["blobs"] = True                                    │
│   2227 │   │   r = get_session().get(path, headers=headers, timeout=timeout, │
│ ❱ 2228 │   │   hf_raise_for_status(r)                                        │
│   2229 │   │   data = r.json()                                               │
│   2230 │   │   return ModelInfo(**data)                                      │
│   2231                                                                       │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/huggingface_ │
│ hub/utils/_errors.py:311 in hf_raise_for_status                              │
│                                                                              │
│   308 │   │                                                                  │
│   309 │   │   if error_code == "RevisionNotFound":                           │
│   310 │   │   │   message = f"{response.status_code} Client Error." + "\n\n" │
│ ❱ 311 │   │   │   raise RevisionNotFoundError(message, response) from e      │
│   312 │   │                                                                  │
│   313 │   │   elif error_code == "EntryNotFound":                            │
│   314 │   │   │   message = f"{response.status_code} Client Error." + "\n\n" │
╰──────────────────────────────────────────────────────────────────────────────╯
RevisionNotFoundError: 404 Client Error. (Request ID: 
Root=1-6618753d-110cabeb693eb3193b468c96;9a5da8fe-535e-45c7-b654-1854afd869ff)

Revision Not Found for url: 
https://huggingface.co/api/models/Fantasy-Studio/Paint-by-Example/revision/fp16.
Invalid rev id: fp16

The above exception was the direct cause of the following exception:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/iopaint/cli. │
│ py:163 in start                                                              │
│                                                                              │
│   160 │   scanned_models = scan_models()                                     │
│   161 │   if model not in [it.name for it in scanned_models]:                │
│   162 │   │   logger.info(f"{model} not found in {model_dir}, try to downloa │
│ ❱ 163 │   │   cli_download_model(model)                                      │
│   164 │                                                                      │
│   165 │   from iopaint.api import Api                                        │
│   166 │   from iopaint.schema import ApiConfig                               │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/iopaint/down │
│ load.py:36 in cli_download_model                                             │
│                                                                              │
│    33 │   │   logger.info(f"Downloading model from Huggingface: {model}")    │
│    34 │   │   from diffusers import DiffusionPipeline                        │
│    35 │   │                                                                  │
│ ❱  36 │   │   downloaded_path = handle_from_pretrained_exceptions(           │
│    37 │   │   │   DiffusionPipeline.download,                                │
│    38 │   │   │   pretrained_model_name=model,                               │
│    39 │   │   │   variant="fp16",                                            │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/iopaint/mode │
│ l/utils.py:986 in handle_from_pretrained_exceptions                          │
│                                                                              │
│    983 │   except ValueError as e:                                           │
│    984 │   │   if "You are trying to load the model files of the `variant=fp │
│    985 │   │   │   logger.info("variant=fp16 not found, try revision=fp16")  │
│ ❱  986 │   │   │   return func(**{**kwargs, "variant": None, "revision": "fp │
│    987 │   │   raise e                                                       │
│    988 │   except OSError as e:                                              │
│    989 │   │   previous_traceback = traceback.format_exc()                   │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/huggingface_ │
│ hub/utils/_validators.py:119 in _inner_fn                                    │
│                                                                              │
│   116 │   │   if check_use_auth_token:                                       │
│   117 │   │   │   kwargs = smoothly_deprecate_use_auth_token(fn_name=fn.__na │
│   118 │   │                                                                  │
│ ❱ 119 │   │   return fn(*args, **kwargs)                                     │
│   120 │                                                                      │
│   121 │   return _inner_fn  # type: ignore                                   │
│   122                                                                        │
│                                                                              │
│ /home/user/.pyenv/versions/3.10.14/lib/python3.10/site-packages/diffusers/pi │
│ pelines/pipeline_utils.py:1905 in download                                   │
│                                                                              │
│   1902 │   │   │   │   raise                                                 │
│   1903 │   │   │   else:                                                     │
│   1904 │   │   │   │   # 2. we forced `local_files_only=True` when `model_in │
│ ❱ 1905 │   │   │   │   raise EnvironmentError(                               │
│   1906 │   │   │   │   │   f"Cannot load model {pretrained_model_name}: mode │
│   1907 │   │   │   │   │   " while trying to fetch metadata from the Hub. Pl │
│   1908 │   │   │   │   │   " above."                                         │
╰──────────────────────────────────────────────────────────────────────────────╯
OSError: Cannot load model Fantasy-Studio/Paint-by-Example: model is not cached 
locally and an error occured while trying to fetch metadata from the Hub. Please
check out the root cause in the stacktrace above.

[Sanster]

@FlightsMonitor Your log shows that you are using Fantasy-Studio/Paint-by-Example, not the powerpaint_v1 model.

Starting the Fantasy-Studio/Paint-by-Example model with iopaint==1.0.0b10 indeed resulted in an error. It can be successfully started with iopaint==1.2.3.

[jryebread]

@Sanster any instructions for downloading and using the V2 PowerPaint model? excited to try it here!