Cannot create tensorrt with custom onnx model, while Chainner geretates engine

[zelenooki87]

Script evaluation failed:
Python exception: operator (): output dimensions must be 1 or 3, or enable "flexible_output"
Traceback (most recent call last):
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 203, in run_animejanai
    clip = run_animejanai_upscale(clip, backend, model_conf, trt_settings, num_streams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 256, in run_animejanai_upscale
    return upscale2x(clip, backend, model_conf['name'], num_streams, trt_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 153, in upscale2x
    return upscale2x_trt(clip, engine_name, num_streams, trt_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 165, in upscale2x_trt
    return core.trt.Model(
           ^^^^^^^^^^^^^^^
  File "src\\cython\\vapoursynth.pyx", line 3110, in vapoursynth.Function.__call__
vapoursynth.Error: operator (): output dimensions must be 1 or 3, or enable "flexible_output"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "src\\cython\\vapoursynth.pyx", line 3374, in vapoursynth._vpy_evaluate
  File "src\\cython\\vapoursynth.pyx", line 3375, in vapoursynth._vpy_evaluate
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_encode.vpy", line 20, in <module>
    animejanai_core.run_animejanai_with_keybinding(clip, clip.fps, slot)
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 289, in run_animejanai_with_keybinding
    run_animejanai(clip, container_fps, chain_conf, config['global']['backend'])
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 213, in run_animejanai
    clip = run_animejanai_upscale(clip, backend, model_conf, trt_settings, num_streams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 256, in run_animejanai_upscale
    return upscale2x(clip, backend, model_conf['name'], num_streams, trt_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 153, in upscale2x
    return upscale2x_trt(clip, engine_name, num_streams, trt_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 165, in upscale2x_trt
    return core.trt.Model(
           ^^^^^^^^^^^^^^^
  File "src\\cython\\vapoursynth.pyx", line 3110, in vapoursynth.Function.__call__
vapoursynth.Error: operator (): output dimensions must be 1 or 3, or enable "flexible_output"
Upscaling with command: python\VSPipe.exe -c y4m --arg "slot=1" --arg "video_path=C:\Users\Miki\Videos\PRINCRIMA.AVI" "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_encode.vpy" - | "C:\Users\Miki\AppData\Roaming\VideoJaNai\ffmpeg\ffmpeg.exe"  -i pipe: -i "C:\Users\Miki\Videos\PRINCRIMA.AVI" -map 0:v -c:v ffv1 -max_interleave_delta 0 -max_interleave_delta 0 -map 1:t? -map 1:a?  -map 1:s? -c:t copy -c:a copy -c:s copy "C:\Users\Miki\Videos\davor\PRINCRIMA-animejanai.mkv"
ffmpeg version 2024-12-26-git-fe04b93afa-essentials_build-www.gyan.dev Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 14.2.0 (Rev1, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-zlib --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-sdl2 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libaom --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-dxva2 --enable-d3d11va --enable-d3d12va --enable-ffnvcodec --enable-libvpl --enable-nvdec --enable-nvenc --enable-vaapi --enable-libgme --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libtheora --enable-libvo-amrwbenc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-librubberband
  libavutil      59. 53.100 / 59. 53.100
  libavcodec     61. 28.100 / 61. 28.100
  libavformat    61.  9.101 / 61.  9.101
  libavdevice    61.  4.100 / 61.  4.100
  libavfilter    10.  6.101 / 10.  6.101
  libswscale      8. 13.100 /  8. 13.100
  libswresample   5.  4.100 /  5.  4.100
  libpostproc    58.  4.100 / 58.  4.100
run_animejanai slot 1 chain_1
chain_conf {'min_px': 0.0, 'max_px': inf, 'min_resolution': '0x0', 'max_resolution': '0x0', 'min_fps': 0.0, 'max_fps': inf, 'models': [{'resize_factor_before_upscale': 100.0, 'resize_height_before_upscale': 0.0, 'name': 'upscaler-v1-4x-fp16-00-00-00'}], 'rife': False, 'rife_factor_numerator': 2, 'rife_factor_denominator': 1, 'rife_model': 422, 'rife_ensemble': False, 'rife_scene_detect_threshold': 0.15, 'final_resize_height': 0.0, 'final_resize_factor': 100.0, 'tensorrt_engine_settings': '--fp16'}
upscale2x: scaling 2x from 320x240 with engine=upscaler-v1-4x-fp16-00-00-00; num_streams=4
trt_settings --fp16
upscale2x: scaling 2x from 320x240 with engine=upscaler-v1-4x-fp16-00-00-00; num_streams=4
trt_settings --fp16
Script evaluation failed:
Python exception: operator (): output dimensions must be 1 or 3, or enable "flexible_output"
Traceback (most recent call last):
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 203, in run_animejanai
    clip = run_animejanai_upscale(clip, backend, model_conf, trt_settings, num_streams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 256, in run_animejanai_upscale
    return upscale2x(clip, backend, model_conf['name'], num_streams, trt_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 153, in upscale2x
    return upscale2x_trt(clip, engine_name, num_streams, trt_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 165, in upscale2x_trt
    return core.trt.Model(
           ^^^^^^^^^^^^^^^
  File "src\\cython\\vapoursynth.pyx", line 3110, in vapoursynth.Function.__call__
vapoursynth.Error: operator (): output dimensions must be 1 or 3, or enable "flexible_output"
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "src\\cython\\vapoursynth.pyx", line 3374, in vapoursynth._vpy_evaluate
  File "src\\cython\\vapoursynth.pyx", line 3375, in vapoursynth._vpy_evaluate
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_encode.vpy", line 20, in <module>
    animejanai_core.run_animejanai_with_keybinding(clip, clip.fps, slot)
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 289, in run_animejanai_with_keybinding
    run_animejanai(clip, container_fps, chain_conf, config['global']['backend'])
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 213, in run_animejanai
    clip = run_animejanai_upscale(clip, backend, model_conf, trt_settings, num_streams)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 256, in run_animejanai_upscale
    return upscale2x(clip, backend, model_conf['name'], num_streams, trt_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 153, in upscale2x
    return upscale2x_trt(clip, engine_name, num_streams, trt_settings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Miki\Downloads\Compressed\VideoJaNai-win-Portable\current\backend\animejanai\core\animejanai_core.py", line 165, in upscale2x_trt
    return core.trt.Model(
           ^^^^^^^^^^^^^^^
  File "src\\cython\\vapoursynth.pyx", line 3110, in vapoursynth.Function.__call__
vapoursynth.Error: operator (): output dimensions must be 1 or 3, or enable "flexible_output"
[in#0 @ 000001d2f6b8f400] Error opening input: Invalid data found when processing input
Error opening input file pipe:.
Error opening input files: Invalid data found when processing input

These ase input tensors and output:
Input Tensors:
Name: input, Shape: [1, 192, 192, 3]

Output Tensors:
Name: netOutput, Shape: ['?', 768, 768, 3]

Could you please explain me how to enable "flexible_output" option?
Thank you very much in advance!

[zelenooki87]

Please, some advice?
which tensorrt command to use?
my model is in BHWC format instead BCHW

[zelenooki87]

The problem was that the ONNX model was in BHWC format instead of BCHW. VideoJaNai is excellent, but now the tiling lines on the upscaled video image bother me a bit. Is it possible to disable tiling? Thank you very much.

[JossCamp]

Since version 1.0.0 it has been very buggy, the same thing has happened to me, I recommend you install version 0.0.9 which is much more stable and does not have these problems, if the problem persists try converting the model using "data type" to fp32 and "opset" to 14.