ONNX and Torch inference outputs are not the same

[nhanerc]

Hi,

I run the script below, and the outputs from both models are not the same.

# Text recognition
import torch
from PIL import Image
from torchvision import transforms as T
import onnxruntime as rt

parseq = torch.hub.load("baudm/parseq", "parseq", pretrained=True).eval()
img_transform = T.Compose(
    [T.Resize(parseq.hparams.img_size, T.InterpolationMode.BICUBIC), T.ToTensor(), T.Normalize(0.5, 0.5)]
)

x = torch.rand(1, 3, *parseq.hparams.img_size)
parseq.to_onnx("/tmp/parseq.onnx", x, opset_version=14, do_constant_folding=True)

options = rt.SessionOptions()
options.enable_profiling = False
rec_session = rt.InferenceSession("/tmp/parseq.onnx", options=options)

inp = Image.open("cute-184.jpg").convert("RGB")
inp = img_transform(inp).unsqueeze(0)

logits = rec_session.run(None, {rec_session.get_inputs()[0].name: inp.numpy()})[0]
logits = torch.tensor(logits)
pred = logits.softmax(-1)
label, confidence = parseq.tokenizer.decode(pred)
print("onnx:", label)

logits = parseq(inp)
pred = logits.softmax(-1)
label, confidence = parseq.tokenizer.decode(pred)
print("torch:", label)

The output and torch/onnxruntime version:

I have no idea why it happened.
I also tried this as others suggested, but it also failed to produce the same output.

parseq = torch.hub.load("baudm/parseq", "parseq", pretrained=True, refine_iters=False).eval()

[YenYunn]

My code is almost the same as yours. After converting it to ONNX, did you encounter this error during ONNX inference?
Traceback (most recent call last):
File "D:\PyCharmProject\parseq-main\inference.py", line 65, in
session = onnxruntime.InferenceSession('parseq.onnx',
File "D:\PyCharmProject\parseq-main.venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 383, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "D:\PyCharmProject\parseq-main.venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 435, in _create_inference_session
sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for Where(9) node with name '/Where_19'

[felixdittrich92]

CC @baudm
We have had the same issue: the break in the AR decoding loop isn't translated correctly with ONNX
-> only with AR results in a shape mismatch for max_length dimension
-> with AR + RI everything looks fine but the logits differs +- 0.5

Not breaking the loop in AR fixes the issue (only required if you want to export):
https://github.com/mindee/doctr/pull/1585/files

Time for base model: 0.04333
Time for onnx model: 0.02047
parseq --> mean diff: 1.8520279354561353e-06

This can certainly be solved in a nicer way, but that should solve the problem first.

[Dordor333]

My code is almost the same as yours. After converting it to ONNX, did you encounter this error during ONNX inference? Traceback (most recent call last): File "D:\PyCharmProject\parseq-main\inference.py", line 65, in session = onnxruntime.InferenceSession('parseq.onnx', File "D:\PyCharmProject\parseq-main.venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 383, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "D:\PyCharmProject\parseq-main.venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 435, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for Where(9) node with name '/Where_19'

How did you convert the model to onnx doesn't it raise the error of Exporting the operator 'aten::scaled_dot_product_attention' to ONNX opset version 14 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues.??

[GIN-zhao]

My code is almost the same as yours. After converting it to ONNX, did you encounter this error during ONNX inference? Traceback (most recent call last): File "D:\PyCharmProject\parseq-main\inference.py", line 65, in session = onnxruntime.InferenceSession('parseq.onnx', File "D:\PyCharmProject\parseq-main.venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 383, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "D:\PyCharmProject\parseq-main.venv\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 435, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for Where(9) node with name '/Where_19'

lowering the version of toch,at the meaning time,you need update the parseq/model.py line-117 tensor.bool to tensor.float