[bug]: Error when using Select Object - Model type should be one of SamHQConfig

[MisterChief95]

Is there an existing issue for this problem?

• I have searched the existing issues

Operating system

Windows

GPU vendor

Nvidia (CUDA)

GPU model

RTX 4090

GPU VRAM

24GB

Version number

5.12.0rc1

Browser

Microsoft Edge 136.0.3240.76 (64-bit)

Python dependencies

Didn't happen during image gen, but just in case:

{
    "version": "5.12.0rc1", 
    "dependencies": {
        "accelerate"  : "1.7.0"       , 
        "compel"      : "2.0.2"       , 
        "cuda"        : "12.8"        , 
        "diffusers"   : "0.33.0"      , 
        "numpy"       : "1.26.3"      , 
        "opencv"      : "4.9.0.80"    , 
        "onnx"        : "1.16.1"      , 
        "pillow"      : "11.0.0"      , 
        "python"      : "3.12.9"      , 
        "torch"       : "2.7.0+cu128" , 
        "torchvision" : "0.22.0+cu128", 
        "transformers": "4.52.0"      , 
        "xformers"    : null            
    }, 
    "config": {
        "schema_version": "4.0.2", 
        "legacy_models_yaml_path": null, 
        "host": "127.0.0.1", 
        "port": 9090, 
        "allow_origins": [], 
        "allow_credentials": true, 
        "allow_methods": ["*"], 
        "allow_headers": ["*"], 
        "ssl_certfile": null, 
        "ssl_keyfile": null, 
        "log_tokenization": false, 
        "patchmatch": true, 
        "models_dir": "models", 
        "convert_cache_dir": "models\\.convert_cache", 
        "download_cache_dir": "models\\.download_cache", 
        "legacy_conf_dir": "configs", 
        "db_dir": "databases", 
        "outputs_dir": "outputs", 
        "custom_nodes_dir": "nodes", 
        "style_presets_dir": "style_presets", 
        "workflow_thumbnails_dir": "workflow_thumbnails", 
        "log_handlers": ["console"], 
        "log_format": "color", 
        "log_level": "info", 
        "log_sql": false, 
        "log_level_network": "warning", 
        "use_memory_db": false, 
        "dev_reload": false, 
        "profile_graphs": false, 
        "profile_prefix": null, 
        "profiles_dir": "profiles", 
        "max_cache_ram_gb": null, 
        "max_cache_vram_gb": null, 
        "log_memory_usage": false, 
        "device_working_mem_gb": 3, 
        "enable_partial_loading": false, 
        "keep_ram_copy_of_weights": true, 
        "ram": null, 
        "vram": null, 
        "lazy_offload": true, 
        "pytorch_cuda_alloc_conf": null, 
        "device": "auto", 
        "precision": "auto", 
        "sequential_guidance": false, 
        "attention_type": "auto", 
        "attention_slice_size": "auto", 
        "force_tiled_decode": false, 
        "pil_compress_level": 1, 
        "max_queue_size": 10000, 
        "clear_queue_on_startup": false, 
        "allow_nodes": null, 
        "deny_nodes": null, 
        "node_cache_size": 512, 
        "hashing_algorithm": "blake3_single", 
        "remote_api_tokens": null, 
        "scan_models_on_startup": false
    }, 
    "set_config_fields": ["legacy_models_yaml_path"]
}

What happened

Error occurs when attempting to do anything segmentation-related.

[2025-05-20 14:16:02,806]::[InvokeAI]::ERROR --> Error while invoking session 4bf5fd8f-1e13-4818-9c92-529dadc82072, invocation bac2e0e8-3b6a-4e9e-b11d-7bc5d89b9f6e (segment_anything): Unrecognized configuration class <class 'transformers.models.sam.configuration_sam.SamConfig'> for this kind of AutoModel: AutoModelForMaskGeneration.
Model type should be one of SamHQConfig.
[2025-05-20 14:16:02,806]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "G:\AI\InvokeAI\.venv\Lib\site-packages\invokeai\app\services\session_processor\session_processor_default.py", line 129, in run_node
    output = invocation.invoke_internal(context=context, services=self._services)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "G:\AI\InvokeAI\.venv\Lib\site-packages\invokeai\app\invocations\baseinvocation.py", line 241, in invoke_internal
    output = self.invoke(context)
             ^^^^^^^^^^^^^^^^^^^^
  File "G:\AI\InvokeAI\.venv\Lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "G:\AI\InvokeAI\.venv\Lib\site-packages\invokeai\app\invocations\segment_anything.py", line 93, in invoke
    masks = self._segment(context=context, image=image_pil)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "G:\AI\InvokeAI\.venv\Lib\site-packages\invokeai\app\invocations\segment_anything.py", line 129, in _segment
    context.models.load_remote_model(
  File "G:\AI\InvokeAI\.venv\Lib\site-packages\invokeai\app\services\shared\invocation_context.py", line 545, in load_remote_model
    return self._services.model_manager.load.load_model_from_path(model_path=model_path, loader=loader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "G:\AI\InvokeAI\.venv\Lib\site-packages\invokeai\app\services\model_load\model_load_default.py", line 114, in load_model_from_path
    raw_model = loader(model_path)
                ^^^^^^^^^^^^^^^^^^
  File "G:\AI\InvokeAI\.venv\Lib\site-packages\invokeai\app\invocations\segment_anything.py", line 107, in _load_sam_model
    sam_model = AutoModelForMaskGeneration.from_pretrained(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "G:\AI\InvokeAI\.venv\Lib\site-packages\transformers\models\auto\auto_factory.py", line 574, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.sam.configuration_sam.SamConfig'> for this kind of AutoModel: AutoModelForMaskGeneration.
Model type should be one of SamHQConfig.

What you expected to happen

Expectation is that the selected raster layer would be segmented as normal.

How to reproduce the problem

From the Canvas

1. Start app
2. Generate new layer (model/prompt does not matter)
3. Duplicate raster layer
4. Crop layer to any smaller BBox size
5. Click Select Object
6. Add Selection Points
7. Click Process
8. Error

From a Workflow

1. Create a new workflow
2. Add a Segment Anything node (settings not important)
3. Attempt to segment an image with the workflow

Additional context

This error occurred immediately after updating to prelease 5.12.0rc1. Did not have the issue with 5.11.x

Discord username

No response

[jennner]

I'm also experiencing this error on Linux (Ubuntu 25.04) with 5.12.0rc2. Rolling back to 5.11 didn't help. Now I'm experiencing the error with 5.11 as well.

[knop3se]

I have the same problem, I cannot use the select tool. I´m on Windows. Latest stable version but also tried latest RC, same problem.

I get this error:

ValueError: Unrecognized configuration class <class 'transformers.models.sam.configuration_sam.SamConfig'> for this kind of AutoModel: AutoModelForMaskGeneration. Model type should be one of SamHQConfig.

I´m not a programmer but it looks like the tool is trying to use a new version of something called Sam (SamHQ) but it's not available or something...

[DKEQ]

Same

[1109507252]

I asked gemini and find a way to fix it.but i don't recommend you guys modify any file.
Okay, here is the consolidated solution based on the final state of your segment_anything.py file. This focuses only on the changes needed for the file to work with SAM-HQ models.

---

Solution to Enable SAM-HQ Models in segment_anything.py for InvokeAI

This solution modifies the segment_anything.py file within your InvokeAI installation to correctly load and use SAM-HQ (High Quality) models, addressing various errors encountered with the default configuration.

File to Modify:
[Your InvokeAI installation path]\.venv\Lib\site-packages\invokeai\app\invocations\segment_anything.py
(e.g., D:\Invoke\.venv\Lib\site-packages\invokeai\app\invocations\segment_anything.py)

Modifications:

1. Update Model Identifiers:
   Locate the SEGMENT_ANYTHING_MODEL_IDS dictionary and change its values to point to the syscv-community SAM-HQ model repositories:

   SEGMENT_ANYTHING_MODEL_IDS: dict[SegmentAnythingModelKey, str] = {
       "segment-anything-base": "syscv-community/sam-hq-vit-base",
       "segment-anything-large": "syscv-community/sam-hq-vit-large",
       "segment-anything-huge": "syscv-community/sam-hq-vit-huge",
   }

2. Update Import Statements:
   Ensure the following import statements related to transformers are present and correct at the top of the file. Specifically, add the imports for SamHQModel and SamHQProcessor:

   from transformers import AutoModelForMaskGeneration, AutoProcessor
   from transformers.models.sam import SamModel # Or your environment's original import path for SamModel
   from transformers.models.sam.processing_sam import SamProcessor # Or your environment's original import path for SamProcessor
   from transformers.models.sam_hq.modeling_sam_hq import SamHQModel
   from transformers.models.sam_hq import SamHQProcessor # Adjusted import path

   Note: The final working import path for SamHQProcessor was determined to be from transformers.models.sam_hq import SamHQProcessor. If this specific path causes a ModuleNotFoundError in other environments, from transformers.models.sam_hq.processing_sam_hq import SamHQProcessor might be an alternative to try, depending on the transformers library version.

3. Modify Type Assertions in _load_sam_model method:
   Locate the _load_sam_model static method. Update the assert isinstance(...) lines to accept both standard SAM and SAM-HQ classes:

   • For sam_model:

     assert isinstance(sam_model, (SamModel, SamHQModel))

   • For sam_processor:

     assert isinstance(sam_processor, (SamProcessor, SamHQProcessor))

Final Steps:

1. Save the modified segment_anything.py file.
2. Restart the InvokeAI application.

• Backup: Always back up the original segment_anything.py file before making modifications.

This modification has been tested to work on my computer, but I'm not sure there is any compatibility

[knop3se]

That worked great! Thanks!!!

[MisterChief95]

I think 5.13 resolved the issue? Upgraded and its working ok now.