ERROR while exporting iassd_kitti network.

[adrianaryaputra]

Hi there,

I met an error while exporting IA-SSD kitti network downloaded from the IA-SSD model docs tutorial.

here is the error message :

(paddle_env) PS D:\Backed-up Data\Finale\program\Paddle3D> python tools/export.py --config configs/iassd/iassd_kitti.yaml --model models\iassd_kitti.pdparams --save_dir exported_model
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\setuptools\sandbox.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\pkg_resources\__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\pkg_resources\__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
ortools not installed, install it by "pip install ortools==9.1.9490" if you run BEVLaneDet model
W0126 18:15:32.532241 30040 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.2, Runtime API Version: 11.7
W0126 18:15:32.541234 30040 gpu_resources.cc:91] device: 0, cuDNN Version: 8.4.
2024-01-26 18:15:45,522 -     INFO - There are 185/185 variables loaded into IASSD.
2024-01-26 18:15:45,933 -  WARNING - No custom op pointnet2_ops found, try JIT build
Compiling user custom op, it will cost a few seconds.....
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\setuptools\sandbox.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\pkg_resources\__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\pkg_resources\__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\utils\cpp_extension\extension_utils.py:1206: UserWarning: Failed to check compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn('Failed to check compiler version for {}: {}'.format(
2024-01-26 18:15:47,998 -     INFO - pointnet2_ops builded success!
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\tensor\creation.py:1805: UserWarning: paddle.assign doesn't support float64 input now due to current platform protobuf data limitation, we convert it to float32
  warnings.warn(
2024-01-26 18:15:49,873 -  WARNING - No custom op iou3d_nms found, try JIT build
Compiling user custom op, it will cost a few seconds.....
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\setuptools\sandbox.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\pkg_resources\__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\pkg_resources\__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\utils\cpp_extension\extension_utils.py:1206: UserWarning: Failed to check compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn('Failed to check compiler version for {}: {}'.format(
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.35.32215\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
2024-01-26 18:15:54,314 -    ERROR - iou3d_nms builded fail!
Traceback (most recent call last):
  File "tools/export.py", line 168, in <module>
    main(args, rest_args)
  File "tools/export.py", line 156, in main
    model.export(args.save_dir, name=args.save_name, **kwargs)
  File "d:\backed-up data\finale\program\paddle3d\paddle3d\models\detection\iassd\iassd.py", line 261, in export
    paddle.jit.save(self, save_path, input_spec=input_spec)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\wrapped_decorator.py", line 26, in __impl__
    return wrapped_func(*args, **kwargs)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\jit.py", line 649, in wrapper
    func(layer, path, input_spec, **configs)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\wrapped_decorator.py", line 26, in __impl__
    return wrapped_func(*args, **kwargs)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\base.py", line 67, in __impl__
    return func(*args, **kwargs)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\jit.py", line 927, in save
    concrete_program = static_func.concrete_program_specify_input_spec(
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 577, in concrete_program_specify_input_spec
    concrete_program, _ = self.get_concrete_program(
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 485, in get_concrete_program
    concrete_program, partial_program_layer = self._program_cache[cache_key]
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 955, in __getitem__
    self._caches[item_id] = self._build_once(item)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 939, in _build_once
    concrete_program = ConcreteProgram.from_func_spec(
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\wrapped_decorator.py", line 26, in __impl__
    return wrapped_func(*args, **kwargs)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\base.py", line 67, in __impl__
    return func(*args, **kwargs)
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\program_translator.py", line 895, in from_func_spec
    error_data.raise_new_exception()
  File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\fluid\dygraph\dygraph_to_static\error.py", line 350, in raise_new_exception
    six.exec_("raise new_exception from None")
  File "<string>", line 1, in <module>
RuntimeError: In transformed code:

    File "d:\backed-up data\finale\program\paddle3d\paddle3d\models\detection\iassd\iassd.py", line 71, in forward
        if self.training:
    File "d:\backed-up data\finale\program\paddle3d\paddle3d\models\detection\iassd\iassd.py", line 75, in forward
        if getattr(self, "export_model", False):
    File "d:\backed-up data\finale\program\paddle3d\paddle3d\models\detection\iassd\iassd.py", line 76, in forward
        return self.post_process(batch_dict)
    File "d:\backed-up data\finale\program\paddle3d\paddle3d\models\detection\iassd\iassd.py", line 140, in post_process
        for index in range(batch_size):
    File "d:\backed-up data\finale\program\paddle3d\paddle3d\models\detection\iassd\iassd.py", line 163, in post_process
        selected_score, selected_label, selected_box = self.class_agnostic_nms(
    File "d:\backed-up data\finale\program\paddle3d\paddle3d\models\detection\iassd\iassd.py", line 188, in class_agnostic_nms
        if paddle.shape(scores_mask)[0] == 0:
    File "d:\backed-up data\finale\program\paddle3d\paddle3d\models\detection\iassd\iassd.py", line 202, in class_agnostic_nms
        # When order is one-value tensor,
        # boxes[order] loses a dimension, so we add a reshape
        keep, num_out = iou3d_nms.nms_gpu(box_preds,
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
                                          nms_config["nms_thresh"])
        if num_out.cast("int64") == 0:

    File "d:\backed-up data\finale\program\paddle3d\paddle3d\ops\__init__.py", line 186, in __getattr__
        module = self._load_module()
    File "d:\backed-up data\finale\program\paddle3d\paddle3d\ops\__init__.py", line 172, in _load_module
        self.module = self.jit_build()
    File "d:\backed-up data\finale\program\paddle3d\paddle3d\ops\__init__.py", line 159, in jit_build
        return paddle_jit_load(
    File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\utils\cpp_extension\cpp_extension.py", line 944, in load
        _jit_compile(file_path, verbose)
    File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\utils\cpp_extension\extension_utils.py", line 1113, in _jit_compile
        run_cmd(compile_cmd, verbose)
    File "C:\Users\reido\miniconda3\envs\paddle_env\lib\site-packages\paddle\utils\cpp_extension\extension_utils.py", line 1155, in run_cmd
        raise RuntimeError("Failed to run command: {}, errors: {}".format(

    RuntimeError: Failed to run command: <built-in function compile>, errors: Command 'cd /d C:\Users\reido\.cache\paddle_extensions && C:\Users\reido\miniconda3\envs\paddle_env\python.exe iou3d_nms_setup.py build' returned non-zero exit status 1.

any suggestion would be helpful 👍

[Birdylx]

@adrianaryaputra seems like the custom op nms_gpu failed in compilation, can you provide your paddle version?
I will try to reproduce the problem.