Spinnaker: GenICam::AccessException= Node is not writable

[Benefe]

System Description
Camera Name and Model: Blackfly S USB 3 BFLY-U3-23S6C-C
Operating System: Ubuntu 18.04 LTS
Spinnaker Version: 1.27.0.48
ROS Version: melodic

Computer details
Processor: ARMv8 64-bit CPU
RAM: 8GB

Multiple Cameras Only
Single Camera

Do the cameras work with SpinView?
Yes

Describe the bug
While launching the acquisition node or nodelet the driver try to write to locked node of the camera.
Every time tried to write a locked node this error occurs. Not only AutoExposureTargetGreyValueAuto, but also BinningHorizontal and similar.

As described in (Feature Locking in Spinnaker API) there are states in which nodes are dependent on another and therefore locked.

Would be great to check whether the node ist locked or not before trying to write to.

Error Messages
mue@mue-tx2:~/catkin_ws$ roslaunch spinnaker_sdk_camera_driver acquisition_node.launch
... logging to /home/mue/.ros/log/88957556-8301-11ea-9be7-00044bf67fa2/roslaunch-mue-tx2-9241.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://mue-tx2:36783/

SUMMARY

PARAMETERS

• /acquisition_node/binning: 1
• /acquisition_node/cam_aliases: ['cam0']
• /acquisition_node/cam_ids: [20086912]
• /acquisition_node/color: False
• /acquisition_node/delay: 1.0
• /acquisition_node/distortion_coeffs: [[-0.021141875266...
• /acquisition_node/distortion_model: plumb_bob
• /acquisition_node/exposure_time: 0
• /acquisition_node/frames: 3400
• /acquisition_node/image_height: 1536
• /acquisition_node/image_width: 2048
• /acquisition_node/intrinsic_coeffs: [[1886.9232141485...
• /acquisition_node/live: False
• /acquisition_node/live_grid: False
• /acquisition_node/master_cam: 20086912
• /acquisition_node/max_rate_save: False
• /acquisition_node/projection_coeffs: [[913.700317, 0.0...
• /acquisition_node/rectification_coeffs: [[1.0, 0.0, 0.0, ...
• /acquisition_node/save: False
• /acquisition_node/save_path: ~
• /acquisition_node/save_type: bmp
• /acquisition_node/skip: 20
• /acquisition_node/soft_framerate: 20
• /acquisition_node/target_grey_value: 0
• /acquisition_node/time: False
• /acquisition_node/to_ros: True
• /acquisition_node/utstamps: False
• /rosdistro: melodic
• /rosversion: 1.14.5

NODES
/
acquisition_node (spinnaker_sdk_camera_driver/acquisition_node)

auto-starting new master
process[master]: started with pid [9251]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 88957556-8301-11ea-9be7-00044bf67fa2
process[rosout-1]: started with pid [9262]
started core service [/rosout]
process[acquisition_node-2]: started with pid [9269]
[ INFO] [1587385323.966521931]: [ OK ] USB memory: 1000 MB
[ INFO] [1587385323.971143543]: *** PARAMETER SETTINGS ***
[ INFO] [1587385323.971262522]: ** Date = 20200420
[ INFO] [1587385323.974745899]: Save path set via parameter to: /home/mue
[ INFO] [1587385323.974890671]: Camera IDs:
[ INFO] [1587385323.976844956]: 20086912
[ INFO] [1587385323.979509019]: Camera Aliases:
[ INFO] [1587385323.979602173]: 20086912 >> cam0
[ INFO] [1587385323.982472320]: Unique time stamps for each camera: false
[ INFO] [1587385323.984481999]: color set to: false
[ INFO] [1587385323.985964306]: Exporting images to ROS: true
[ INFO] [1587385323.987357746]: Showing live images setting: false
[ INFO] [1587385323.989271327]: Showing grid-style live images setting: false
[ INFO] [1587385323.991431762]: Max Rate Save Mode: false
[ INFO] [1587385323.993438881]: Displaying timing details: false
[ INFO] [1587385323.994868450]: No. of images to skip set to: 20
[ INFO] [1587385323.996258467]: Init sleep delays set to : 1.00 sec
[ WARN] [1587385323.997582786]: 'fps' Parameter not set, using default behavior: fps=20.00
[ INFO] [1587385323.999158630]: 'exposure_time'=0, Setting autoexposure
[ INFO] [1587385324.000729387]: 'target_grey_value'=0, Setting AutoExposureTargetGreyValueAuto to Continuous/ auto
[ INFO] [1587385324.002276175]: Binning set to: 1
[ INFO] [1587385324.004456002]: Using Software rate control, rate set to: 20
[ INFO] [1587385324.005551260]: Saving images set to: 0
[ INFO] [1587385324.007456264]: Camera Intrinsic Paramters:
[ INFO] [1587385324.008027509]: 1886.923214 0.000000 604.721488 0.000000 1886.666877 493.477267 0.000000 0.000000 1.000000
[ INFO] [1587385324.009729341]: Camera Distortion Paramters:
[ INFO] [1587385324.009921794]: -0.021142 -0.373387 2.385983 3.282457
[ INFO] [1587385324.011236672]: Camera Rectification Paramters:
[ INFO] [1587385324.011449061]: 1.000000 0.000000 0.000000 0.000000 1.000000 0.000000 0.000000 0.000000 1.000000
[ INFO] [1587385324.012874278]: Camera Projection Paramters:
[ INFO] [1587385324.013090859]: 913.700317 0.000000 953.448302 0.000000 0.000000 1063.296631 777.871993 0.000000 0.000000 0.000000 1.000000 0.000000
[ INFO] [1587385324.013162829]: Camera coeffs provided, camera info messges will be published.
[ INFO] [1587385324.013304336]: Creating system instance...
[ INFO] [1587385324.019511393]: Retreiving list of cameras...
[ INFO] [1587385325.741552831]: Numer of cameras found: 1
[ INFO] [1587385325.744269950]: Cameras connected: 1
[ INFO] [1587385325.744604038]: -20086912
[ INFO] [1587385325.788993771]: Dynamic Reconfigure: Level : 4294967295
[ INFO] [1587385325.794768177]: *** FLUSH SEQUENCE ***
[ INFO] [1587385325.794912020]: Initializing cameras...
[ INFO] [1587385326.235905354]: Deinitializing cameras...
[ INFO] [1587385326.278255771]: All cameras deinitialized.
[ INFO] [1587385328.279726905]: Initializing cameras...
[FATAL] [1587385328.534330325]: Unable to set AutoExposureTargetGreyValueAuto to Continuous (enum retrieval). Aborting...
[FATAL] [1587385328.534636700]: Error: Spinnaker: LogicalErrorException NULL pointer dereferenced [-2005]
[ WARN] [1587385328.534728286]: Most likely cause for this error is if your camera can't support color and your are trying to set it to color mode
terminate called after throwing an instance of 'Spinnaker::Exception'
what(): Spinnaker: GenICam::AccessException= Node is not writable. : AccessException thrown in node 'TriggerSoftware' while calling 'TriggerSoftware.Execute()' (file 'CommandT.h', line 61) [-2006]
[acquisition_node-2] process has died [pid 9269, exit code -6, cmd /home/mue/catkin_ws/devel/lib/spinnaker_sdk_camera_driver/acquisition_node __name:=acquisition_node __log:=/home/mue/.ros/log/88957556-8301-11ea-9be7-00044bf67fa2/acquisition_node-2.log].
log file: /home/mue/.ros/log/88957556-8301-11ea-9be7-00044bf67fa2/acquisition_node-2*.log

Screenshots
If applicable, add screenshots to help explain your problem.

[ghost]

Have you tried commenting out those lines in the code?

[Benefe]

Yes I just work my way through, but maybe a future task is to check for writability before write to those nodes or get to know why those nodes are not writeable at this moment.

[ghost]

We already do that, see

spinnaker_sdk_camera_driver/src/camera.cpp

Line 129 in 289f37c

void acquisition::Camera::setEnumValue(string setting, string value) {

[Benefe]

I understand, sorry. I thought this would be a native Exception from Spinnaker SDK.

Is there any problem with my camera or wrong setting?

I wasn't able to set:

• BinningHorizontal
• BinningVertical
• AutoExposureTargetGreyValueAuto
• AutoExposureTargetGreyValue
• PixelFormat
• LineSelector
• LineMode
• AcquisitionFrameRate
• LineSource

[ghost]

you can try opening the camera in SpinView and see what you are able to change / set. What version of the spinnaker library are you using?

[Benefe]

I'm using Spinnaker 1.27.0.48.

Some of these are unlocked, so they should be writable.

[ghost]

I have only tested our code with version 1.24 can you try with that?

[Benefe]

1.24 is only available for Ubuntu 16.04.
I'm using 18.04 LTS
My Issue so far with commenting out writing to locked nodes are

• full frame rate not available
• only Mono not color video

[ghost]

That could be because your platform might not be able to support the full framrate at the full resolution. I think the SDK evaluates the connections limits and then decides on allowable rates. Does
the code work if you set it to Mono, with binning =2 and frame rate as 10 fps ?

[Benefe]

I got a Jetson TX2, it should have enough resources. I also set 1000MB of memory for video buffer. In SpinView I get the full frame rate at full resolution in color. I'll will try these settings with ROS tomorrow.

[ghost]

I wouldn't trust spinview's claim of getting the full rate. Do you see a lag after running it for a couple mins in SpinView?

[Benefe]

In my opinion possible performance limitations have nothing to do with the error occurring when trying to write locked nodes of the camera.

[Benefe]

I wouldn't trust spinview's claim of getting the full rate. Do you see a lag after running it for a couple mins in SpinView?

There is no lag when running it at least for 20min at a stable Framerate of 30 fps.

[ghost]

Not sure what is going on. We have used our code on TX2 extensively with blackfly S and not run into this issue. All our tests are done on 16.04, so maybe the operating system / spinnaker version is somehow causing incompatibilities.

[Benefe]

Switched to another Camera driver, which works on Ubuntu 18.04 and Spinnaker 1.27

[soulslicer]

Switched to another Camera driver, which works on Ubuntu 18.04 and Spinnaker 1.27

What driver do you use?

[Benefe]

For now this Driver works for me. But I will look into to get neufields_robitcs driver to work with Ubuntu 18.04 and the current Spinnaker.

[nicholasadr]

I came across the same issue when running roslaunch spinnaker_sdk_camera_driver node_acquisition.launch:

[FATAL] [1616058135.454513489]: Unable to execute trigger. Aborting...
terminate called after throwing an instance of 'Spinnaker::Exception'
  what():  Spinnaker: GenICam::AccessException= Node is not writable. : AccessException thrown in node 'TriggerSoftware' while calling 'TriggerSoftware.Execute()' (file 'CommandT.h', line 61) [-2006]

I realized that upon checking on spinview, the Trigger Source is somehow set to Line3 which locks the Trigger Software:

Could this be the source of the error? Any idea how to fix this?

My system:
Ubuntu 18.04, ROS Melodic
Spinnaker 2.0.0.147
BFS-U3-120S4C

[ghost]

@nicholasadr we use software triggering in our drive to ensure the most stable results. This is done at

spinnaker_sdk_camera_driver/src/capture.cpp

Line 750 in f6ab290

cams[i].setEnumValue("TriggerSource", "Software");

I am not sure why it is not setting triggering. Can you try to configure SpinView with the same parameters as the driver. Also please post the full output so we can see if there is something else going on.

[nicholasadr]

Thank you for the pointer. I gave a closer look and I've found that the issue stems from anomaly in ROS_ASSERT_MSG as rightly mentioned in #8.

I could run roslaunch spinnaker_sdk_camera_driver node_acquisition.launch succesfully by replacing all instances of ROS_ASSERT_MSG for cam_ids and master_cam:

//ROS_ASSERT_MSG(nh_pvt_.getParam("cam_ids", cam_id_vec),"If cam_aliases are provided, they should be the same number as cam_ids and should correspond in order!");
nh_pvt_.getParam("cam_ids", cam_id_vec);
...
//ROS_ASSERT_MSG(nh_pvt_.getParam("master_cam", mcam_int),"master_cam is required!");
nh_pvt_.getParam("master_cam", mcam_int);

This could be the reason behind issues related to reading from parameter servers such as in #107, #108? Let me know if you need more information.

Note:
Tried to define ROS_ASSERT_ENABLED as mentioned here but it didn't help.

[ghost]

Sounds like this issue occurs in Melodic and for reading cam_ids which is provided as a 'yaml' list that is read in as a vector. Possibly there is some API change between Kinetic and Melodic that causes this.

[nicholasadr]

I've found the root cause. I built my workspace with the cmake argument -DCMAKE_BUILD_TYPE=Release which disabled the ROS_ASSERT macro as mentioned here.

ROS_ASSERT_MSG() works once I removed the argument or replaced with -DCMAKE_BUILD_TYPE=Debug.

[ghost]

So in Release, does it simply ignore the statement and behave as if it is true ? Or does the execution stop there?

Are the parameters still being read correctly to the parameter server?

[nicholasadr]

In Release, you get the same behavior as mentioned in #8, i.e. the ROS_ASSERT_MSG is ignored. Thus, the parameters which are supposed to be read within the ROS_ASSERT_MSG macros (e.g. cam_ids and master_cam) are not read.

[ghost]

Understood, I didn't realize we were reading parameters inside ROS_ASSERT. We will get it fixed in a future release. Thanks for your help finding the root cause.