Inference results on CPU and GPU are different

[nakayamarusu]

Problem

Using Padim to learn images of MVTec bottles. And after exporting it to a file that can be used in Openvino, I did the inference in Openvino. I can get the correct heatmap if I infer on the CPU, but it doesn't work on the GPU. The GPU uses Intel iRIS Xew, and we have confirmed that OpenVino is compatible.

detail

Train

The learning process was as follows.
anomalib fit -c configs/model/padim.yaml --data configs/folder_bottle.yaml

▼ padim.yaml

model:
  class_path: anomalib.models.Padim
  init_args:
    layers:
      - layer1
      - layer2
      - layer3
    #input_size: [256, 256]
    backbone: resnet18
    pre_trained: true
    n_features: null

metrics:
  pixel: AUROC

▼ folder_bottle_yaml

class_path: anomalib.data.Folder
init_args:
  name: bottle
  root: "dataset/bottle"
  normal_dir: "train/good"
  abnormal_dir: "test/broken_large"
  normal_test_dir: "test/good"
  mask_dir: "ground_truth/broken_large"
  normal_split_ratio: 0
  extensions: [".png"]
  image_size: [256, 256]
  #center_crop: null
  #normalization: imagenet
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 8
  task: CLASSIFICATION
  #transform_config_train: null
  #transform_config_eval: null
  test_split_mode: NONE
  test_split_ratio: 0.2
  val_split_mode: same_as_test
  val_split_ratio: 0.5
  seed: null

Export

The export was performed as follows.
anomalib export --model Padim --export_type OPENVINO --ckpt_path results/Padim/bottle/latest/weights/lightning/model.ckpt

Inference

The inference was made by creating python code as follows.

from anomalib.deploy.inferencers import OpenVINOInferencer
import cv2
import numpy as np
import os

inferencer = OpenVINOInferencer(
    path=r"C:\anomalib_v1\results\weights\openvino\model.onnx",
    metadata=r"C:\anomalib_v1\results\weights\openvino\metadata.json",
    device="GPU",
)

img_path = r"C:\anomalib_v1\dataset\bottle\test\broken_small"

for i, file in enumerate(os.listdir(img_path)):
    
    image = cv2.imread(img_path + "\\" + file)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    input_img = image.astype(np.float32) / 255.0 
    result = inferencer.predict(image = input_img)
    print(i, result.pred_score)
    cv2.imshow("heatmap", cv2.cvtColor(result.heat_map, cv2.COLOR_RGB2BGR))
    cv2.waitKey()

Inference result

Heat map when the device used for inference is CPU as shown below

inferencer = OpenVINOInferencer(
    path=r"C:\anomalib_v1\results\weights\openvino\model.onnx",
    metadata=r"C:\anomalib_v1\results\weights\openvino\metadata.json",
    device="CPU",
)

▲ pred_score : 0.4836

▲ pred_score : 0.5605

Heat map when the device used for inference is GPU as shown below

inferencer = OpenVINOInferencer(
    path=r"C:\anomalib_v1\results\weights\openvino\model.onnx",
    metadata=r"C:\anomalib_v1\results\weights\openvino\metadata.json",
    device="GPU",
)

▲ pred_score : 0.0

▲ pred_score : 0.0

Both CPU and GPU inference use the same model.
Also, only the device = "CPU" and device = "GPU" parts have been changed.
I changed the OpenVino version from 2024.0.0 to 2023.2.0 and tried inference, but it doesn't work

Question

If I just switch the device used from CPU to GPU, I cannot get correct inference results. How can we improve it to enable inference on GPU?

[blaz-r]

Hi, this seems quite unfortunate, especially as it seems to work on CPU. We'll need to ask someone who better understands OpenVINO gpu inference @adrianboguszewski do you maybe know how to proceed here?

[nakayamarusu]

Thank you for your response.
@adrianboguszewski I would greatly appreciate your support in taking care of this task.

[adrianboguszewski]

Hey @nakayamarusu. Thanks for reporting this. It looks like a bug if it works on CPU. I'm out of office this week, so could you please report it here: https://github.com/openvinotoolkit/openvino/issues?

[nakayamarusu]

@adrianboguszewski I’m grateful for your response.
We will report to the above location.