camera_detection_bev crashes when using 7 camera channels

[xudooo]

System information

• OS Platform and Distribution: Linux Ubuntu 22.04
• Apollo installed from: source
• Apollo version: 9.0
• Output of apollo.sh config if on master branch:

[tyjt@in-dev-docker:/apollo]$ ./apollo.sh config
[INFO] Apollo Environment Settings:
[INFO]     APOLLO_ROOT_DIR:   /apollo
[INFO]     APOLLO_CACHE_DIR:  /apollo/.cache
[INFO]     APOLLO_IN_DOCKER:  true
[INFO]     APOLLO_VERSION:    master-2025-01-21-a8695d9c5f
[INFO]     DOCKER_IMG:        dev-x86_64-18.04-20240620_1444
[INFO]     APOLLO_ENV:        STAGE=dev USE_ESD_CAN=false
[INFO]     USE_GPU_HOST:      1
[INFO]     USE_GPU_TARGET:    1
[INFO]     GPU_PLATFORM:      NVIDIA
[INFO] Configure .apollo.bazelrc in non-interactive mode
[INFO] You have bazel 5.2.0 installed.
[INFO] Found possible Python library paths:
       /usr/local/lib/python3.6/dist-packages
       /usr/lib/python3/dist-packages
[INFO] Found CUDA 11.1 in:
       /usr/local/cuda-11.1/targets/x86_64-linux/lib
       /usr/local/cuda-11.1/targets/x86_64-linux/include
[INFO] Found cuDNN 8 in:
       /usr/lib/x86_64-linux-gnu
       /usr/include
[INFO] Found TensorRT 7 in:
       /usr/lib/x86_64-linux-gnu
       /usr/include/x86_64-linux-gnu
[ OK ] Successfully configured .apollo.bazelrc in non-interactive mode.
[INFO] Usage: /apollo/scripts/apollo_config.sh [Options]
[INFO] Options:
[INFO]     -i|--interactive      Run in interactive mode
[INFO]     -n|--noninteractive   Run in non-interactive mode
[INFO]     -h|--help             Show this message and exit

Steps to reproduce the issue:

• Please use bullet points and include as much details as possible:

1. use 7 input camera channels in modules/perception/camera_detection_bev/conf/camera_detection_bev_config.pb.txt
2. gdb mainboard
3. mainboard -d modules/perception/camera_detection_bev/dag/camera_detection_bev.dag

Supporting materials (screenshots, command lines, code/script snippets):

1. My GPU is 3060, I tried on a PC with 4080s and encountered the same question;
2. This error is not encountered when using 6 channels of data, is there some configuration that limits certain GPU resources?
3. logs:

I0123 17:32:37.731253 3237779 data_provider.cc:188] [mainboard]Done! (1)
W0123 17:32:37.740542 3237775 channel_buffer.h:70] [mainboard]channel[/tf] read buffer overflow, drop_message[1] pre_index[2] current_index[3] 
W0123 17:32:37.740749 3237775 channel_buffer.h:70] [mainboard]channel[/tf] read buffer overflow, drop_message[8] pre_index[4] current_index[12] 
I0123 17:32:37.767822 3237780 data_provider.cc:148] [mainboard]Fill in GPU mode ...
I0123 17:32:37.768622 3237780 data_provider.cc:188] [mainboard]Done! (1)
I0123 17:32:37.768646 3237780 data_provider.cc:148] [mainboard]Fill in GPU mode ...
I0123 17:32:37.769452 3237780 data_provider.cc:188] [mainboard]Done! (1)
I0123 17:32:37.769470 3237780 data_provider.cc:148] [mainboard]Fill in GPU mode ...
I0123 17:32:37.770526 3237780 data_provider.cc:188] [mainboard]Done! (1)
I0123 17:32:37.770553 3237780 data_provider.cc:148] [mainboard]Fill in GPU mode ...
I0123 17:32:37.771417 3237780 data_provider.cc:188] [mainboard]Done! (1)
I0123 17:32:37.771440 3237780 data_provider.cc:148] [mainboard]Fill in GPU mode ...
I0123 17:32:37.772377 3237780 data_provider.cc:188] [mainboard]Done! (1)
I0123 17:32:37.772400 3237780 data_provider.cc:148] [mainboard]Fill in GPU mode ...
I0123 17:32:37.773506 3237780 data_provider.cc:188] [mainboard]Done! (1)
I0123 17:32:37.773540 3237780 data_provider.cc:148] [mainboard]Fill in GPU mode ...

Thread 11 "mainboard" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe2ca0700 (LWP 3237780)]
0x00007fffdcf4d366 in apollo::perception::base::Image8U::rows (this=0x7ffd9b400000)
    at ./modules/perception/common/base/image_8u.h:120
120	  int rows() const { return rows_; }
(gdb) bt
#0  0x00007fffdcf4d366 in apollo::perception::base::Image8U::rows (this=0x7ffd9b400000)
    at ./modules/perception/common/base/image_8u.h:120
#1  0x00007fff06aab192 in apollo::perception::camera::DataProvider::FillImageData (this=0x5556171f6c00, 
    rows=1080, cols=1920, 
    data=0x55565445e000 "#\033/#\033/!\034/\"\036.#\037.\"\036-!\034.\"\035.#\037/!\033\060\037\035.!\035,#\034+#\036*$ *% *& *%\"*$#+$ ,$\037-'#-*'-*%-+$/,&1-$3/'41(62+73.6604933:71;<3>;5A<9D==H@@JDCMGGPMJRRNUWQW]X[`^_ecbfgfjlhopjuvmw|p|\200t\177\204x\204\212{\205\217\177\207\224\203\215\231\207\224\235\213\225\241\220\233\247\222\236\254\224\243\260\230\251\263\233\255\267\240\261\272\244\266"..., encoding="rgb8")
    at modules/perception/common/camera/common/data_provider.cc:156
#2  0x00007fffdd7a4cc2 in apollo::perception::camera::CameraDetectionBevComponent::OnReceiveImage (
    this=0x555558a03260, 
    msg=std::shared_ptr<apollo::drivers::Image> (use count 4, weak count 0) = {...})
    at modules/perception/camera_detection_bev/camera_detection_bev_component.cc:205
#3  0x00007fffdd7a3e50 in apollo::perception::camera::CameraDetectionBevComponent::<lambda(const std::shared_ptr<apollo::drivers::Image>&)>::operator()(const std::shared_ptr<apollo::drivers::Image> &) const (
    __closure=0x555639a8d560, 
    msg=std::shared_ptr<apollo::drivers::Image> (use count 4, weak count 0) = {...})
    at modules/perception/camera_detection_bev/camera_detection_bev_component.cc:90
#4  0x00007fffdd7a52ca in std::_Function_handler<void(const std::shared_ptr<apollo::drivers::Image>&), apollo::perception::camera::CameraDetectionBevComponent::InitListeners(const apollo::perception::camera::CameraDetectionBEV&)::<lambda(const std::shared_ptr<apollo::drivers::Image>&)> >::_M_invoke(const std::_Any_data &, const std::shared_ptr<apollo::drivers::Image> &) (__functor=..., 
    __args#0=std::shared_ptr<apollo::drivers::Image> (use count 4, weak count 0) = {...})
    at /usr/include/c++/7/bits/std_function.h:316

[daohu527]

The camera's data structure has very strict requirements on the size of the input, such as the length, width and number of images.

Can you explain in detail how to input the 7 images?

[xudooo]

@daohu527
Sorry for the late reply, I used opencv in the camera driver to convert the image to rgb8 encoding, published it using the apollo::drivers::Image structure, and subscribing to those channels in the traffic light recognition works fine. Subscribing to any six channels in camera_detection_bev also works fine, but adding one more channel to seven gives the above error.

camera_detection_bev_config.pb.txt:

camera_name: "pylon_camera_1_rear"
timestamp_offset: 0.0
plugin_param {
  name: "BEVObstacleDetector"
  config_path: "perception/camera_detection_bev/data"
  config_file: "petr.pb.txt"
}
enable_undistortion: false
gpu_id: 0
channel {
  input_camera_channel_name: "/apollo/sensor/camera/cam1_rear/image"
  input_camera_channel_name: "/apollo/sensor/camera/cam2_rear_left/image"
  input_camera_channel_name: "/apollo/sensor/camera/cam3_center_left/image"
  input_camera_channel_name: "/apollo/sensor/camera/cam4_front_left/image"
  input_camera_channel_name: "/apollo/sensor/camera/cam5_front_right/image"
  input_camera_channel_name: "/apollo/sensor/camera/cam6_center_right/image"
  input_camera_channel_name: "/apollo/sensor/camera/cam7_rear_right/image"
  output_obstacles_channel_name: "/apollo/perception/obstacles"
  main_input_camera_channel_name: "/apollo/sensor/camera/cam1_rear/image"
}
frame_id: "pylon_camera_1_rear"