UR Client Library fails to read from stream

[Payday02]

Affected ROS2 Driver version(s)

humble

Used ROS distribution.

Humble

Which combination of platform is the ROS driver running on.

Linux in a virtual machine

How is the UR ROS2 Driver installed.

Build the driver from source and using the UR Client Library from binary

Which robot platform is the driver connected to.

UR E-series robot

Robot SW / URSim version(s)

latest

How is the ROS driver used.

Through the robot teach pendant using External Control URCap

Issue details

Summary

Though I can ping my robot at the IP address I specify, the ros drivers are unable to correctly read the stream from the robot and therefore will not let me send movement commands from moveit2.

Issue details

My UR3e is connected to the computer through a ethernet to usb-c dongle, which is then routed into my VM running ubuntu 22.04. I am able to launch the drivers with URSim and successfully plan movements with moveit2, but run into this issue when I try to connect the drivers with my physical arm. I have tested and I am able to ping the arm at various IPs set in the teach pendant though the robot always says it is Not connected to network even though the static address settings work. I have ensured that all utilized ports are not in use by anything else and moveit is receiving robot position data so some communication is happening. I am also able to see the ready to receive commands message when I start the URCap external control program on the teach pendant but it throws the following when I try to then send a command:

[ur_ros2_control_node-1] [INFO] [1719421979.084573597] [scaled_joint_trajectory_controller]: Received new action goal

[ur_ros2_control_node-1] [ERROR] [1719421979.085979051] [scaled_joint_trajectory_controller]: Can't accept new action goals. Controller is not running

Steps to Reproduce

Connect robot to computer as described above
ros2 launch ur_robot_driver ur_control.launch.py ur_type:=ur3e robot_ip:=192.168.100.101 launch_rviz:=false

Expected Behavior

Normal driver activation

Actual Behavior

Failed to read from stream, reconnecting in X seconds... messages as seen in log output

Relevant log output

[INFO] [launch]: All log files can be found below /home/pdavis/.ros/log/2024-06-26-12-04-46-040930-FMTenv-5313

[INFO] [launch]: Default logging verbosity is set to INFO

[INFO] [dashboard_client-2]: process started with pid [5318]

[INFO] [controller_stopper_node-3]: process started with pid [5320]

[INFO] [ur_ros2_control_node-1]: process started with pid [5316]

[INFO] [urscript_interface-4]: process started with pid [5322]

[INFO] [robot_state_publisher-5]: process started with pid [5324]

[INFO] [spawner-6]: process started with pid [5326]

[INFO] [spawner-7]: process started with pid [5328]

[INFO] [spawner-8]: process started with pid [5332]

[robot_state_publisher-5] [INFO] [1719421486.714028842] [robot_state_publisher]: got segment base

[robot_state_publisher-5] [INFO] [1719421486.714130615] [robot_state_publisher]: got segment base_link

[robot_state_publisher-5] [INFO] [1719421486.714147026] [robot_state_publisher]: got segment base_link_inertia

[robot_state_publisher-5] [INFO] [1719421486.714159041] [robot_state_publisher]: got segment flange

[robot_state_publisher-5] [INFO] [1719421486.714170182] [robot_state_publisher]: got segment forearm_link

[robot_state_publisher-5] [INFO] [1719421486.714180874] [robot_state_publisher]: got segment ft_frame

[robot_state_publisher-5] [INFO] [1719421486.714191921] [robot_state_publisher]: got segment shoulder_link

[robot_state_publisher-5] [INFO] [1719421486.714202670] [robot_state_publisher]: got segment tool0

[robot_state_publisher-5] [INFO] [1719421486.714213340] [robot_state_publisher]: got segment upper_arm_link

[robot_state_publisher-5] [INFO] [1719421486.714223928] [robot_state_publisher]: got segment world

[robot_state_publisher-5] [INFO] [1719421486.714234307] [robot_state_publisher]: got segment wrist_1_link

[robot_state_publisher-5] [INFO] [1719421486.714244514] [robot_state_publisher]: got segment wrist_2_link

[robot_state_publisher-5] [INFO] [1719421486.714254751] [robot_state_publisher]: got segment wrist_3_link

[ur_ros2_control_node-1] [WARN] [1719421486.716478122] [controller_manager]: [Deprecated] Passing the robot description parameter directly to the control_manager node is deprecated. Use '~/robot_description' topic from 'robot_state_publisher' instead.

[ur_ros2_control_node-1] text not specified in the tf_prefix tag

[ur_ros2_control_node-1] [INFO] [1719421486.716788552] [resource_manager]: Loading hardware 'ur3e' 

[ur_ros2_control_node-1] [INFO] [1719421486.724878949] [resource_manager]: Initialize hardware 'ur3e' 

[ur_ros2_control_node-1] [INFO] [1719421486.724938452] [resource_manager]: Successful initialization of hardware 'ur3e'

[ur_ros2_control_node-1] [INFO] [1719421486.725087631] [resource_manager]: 'configure' hardware 'ur3e' 

[ur_ros2_control_node-1] [INFO] [1719421486.725094619] [URPositionHardwareInterface]: Starting ...please wait...

[ur_ros2_control_node-1] [INFO] [1719421486.725103910] [URPositionHardwareInterface]: Initializing driver...

[controller_stopper_node-3] [INFO] [1719421486.774106121] [Controller stopper]: Waiting for switch controller service to come up on controller_manager/switch_controller

[dashboard_client-2] [INFO] [1719421486.792559293] [UR_Client_Library:]: Connected: Universal Robots Dashboard Server

[dashboard_client-2] 

[ur_ros2_control_node-1] [WARN] [1719421486.819470490] [UR_Client_Library:]: Your system/user seems not to be setup for FIFO scheduling. We recommend using a lowlatency kernel with FIFO scheduling. See https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/real_time.md for details.

[ur_ros2_control_node-1] [INFO] [1719421487.049327926] [UR_Client_Library:]: Negotiated RTDE protocol version to 2.

[ur_ros2_control_node-1] [INFO] [1719421487.055817688] [UR_Client_Library:]: Setting up RTDE communication with frequency 500.000000

[ur_ros2_control_node-1] [INFO] [1719421488.116907014] [URPositionHardwareInterface]: Calibration checksum: 'calib_16756443741236045476'.

[ur_ros2_control_node-1] [ERROR] [1719421489.263515746] [URPositionHardwareInterface]: The calibration parameters of the connected robot don't match the ones from the given kinematics config file. Please be aware that this can lead to critical inaccuracies of tcp positions. Use the ur_calibration tool to extract the correct calibration from the robot and pass that into the description. See [https://github.com/UniversalRobots/Universal_Robots_ROS2_Driver/blob/main/ur_calibration/README.md] for details.

[ur_ros2_control_node-1] [INFO] [1719421489.263703231] [URPositionHardwareInterface]: System successfully started!

[ur_ros2_control_node-1] [INFO] [1719421489.263728774] [resource_manager]: Successful 'configure' of hardware 'ur3e'

[ur_ros2_control_node-1] [INFO] [1719421489.263768615] [resource_manager]: 'activate' hardware 'ur3e' 

[ur_ros2_control_node-1] [INFO] [1719421489.263773340] [URPositionHardwareInterface]: Activating HW interface

[ur_ros2_control_node-1] [INFO] [1719421489.263777846] [resource_manager]: Successful 'activate' of hardware 'ur3e'

[spawner-7] [INFO] [1719421489.287926690] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available

[controller_stopper_node-3] [INFO] [1719421489.487901097] [Controller stopper]: Service available

[controller_stopper_node-3] [INFO] [1719421489.487960798] [Controller stopper]: Waiting for list controllers service to come up on controller_manager/list_controllers

[controller_stopper_node-3] [INFO] [1719421489.487980527] [Controller stopper]: Service available

[ur_ros2_control_node-1] [WARN] [1719421489.508565458] [controller_manager]: Could not enable FIFO RT scheduling policy

[ur_ros2_control_node-1] [WARN] [1719421489.522071204] [UR_Client_Library:]: Your system/user seems not to be setup for FIFO scheduling. We recommend using a lowlatency kernel with FIFO scheduling. See https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/real_time.md for details.

[ur_ros2_control_node-1] [ERROR] [1719421489.830091715] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830114702] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830119586] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830123640] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830127777] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830132398] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830136389] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830140336] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830144241] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830148322] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830152328] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830156358] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830160431] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830164592] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830190558] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421489.830194786] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [ERROR] [1719421490.124246883] [UR_Client_Library:]: Pipeline producer overflowed! <RTDE Data Pipeline>

[ur_ros2_control_node-1] [INFO] [1719421490.146240867] [controller_manager]: Loading controller 'scaled_joint_trajectory_controller'

[ur_ros2_control_node-1] [WARN] [1719421490.581451977] [scaled_joint_trajectory_controller]: [Deprecated]: "allow_nonzero_velocity_at_trajectory_end" is set to true. The default behavior will change to false.

[spawner-6] [INFO] [1719421490.583813515] [spawner_scaled_joint_trajectory_controller]: Loaded scaled_joint_trajectory_controller

[ur_ros2_control_node-1] [INFO] [1719421490.593900104] [controller_manager]: Loading controller 'joint_state_broadcaster'

[ur_ros2_control_node-1] [WARN] [1719421490.610093073] [UR_Client_Library:]: Failed to read from stream, reconnecting in 1 seconds...

[spawner-7] [INFO] [1719421490.990157221] [spawner_joint_state_broadcaster]: Loaded joint_state_broadcaster

[ur_ros2_control_node-1] [INFO] [1719421491.006432772] [controller_manager]: Loading controller 'forward_position_controller'

[spawner-8] [INFO] [1719421491.111308623] [spawner_forward_position_controller]: Loaded forward_position_controller

[ur_ros2_control_node-1] [INFO] [1719421491.171023455] [controller_manager]: Configuring controller 'scaled_joint_trajectory_controller'

[ur_ros2_control_node-1] [INFO] [1719421491.174690008] [scaled_joint_trajectory_controller]: No specific joint names are used for command interfaces. Using 'joints' parameter.

[ur_ros2_control_node-1] [INFO] [1719421491.174710760] [scaled_joint_trajectory_controller]: Command interfaces are [position] and state interfaces are [position velocity].

[ur_ros2_control_node-1] [INFO] [1719421491.174719619] [scaled_joint_trajectory_controller]: Using 'splines' interpolation method.

[ur_ros2_control_node-1] [INFO] [1719421491.269623020] [scaled_joint_trajectory_controller]: Controller state will be published at 100.00 Hz.

[ur_ros2_control_node-1] [INFO] [1719421491.382785326] [scaled_joint_trajectory_controller]: Action status changes will be monitored at 20.00 Hz.

[ur_ros2_control_node-1] [INFO] [1719421491.494109458] [controller_manager]: Configuring controller 'joint_state_broadcaster'

[ur_ros2_control_node-1] [INFO] [1719421491.494179442] [joint_state_broadcaster]: 'joints' or 'interfaces' parameter is empty. All available state interfaces will be published

[ur_ros2_control_node-1] [INFO] [1719421491.587487073] [controller_manager]: Configuring controller 'forward_position_controller'

[ur_ros2_control_node-1] [INFO] [1719421491.626054624] [forward_position_controller]: configure successful

[spawner-6] [INFO] [1719421491.671876074] [spawner_scaled_joint_trajectory_controller]: Configured and activated scaled_joint_trajectory_controller

[spawner-7] [INFO] [1719421491.730725552] [spawner_joint_state_broadcaster]: Configured and activated joint_state_broadcaster

[ur_ros2_control_node-1] [INFO] [1719421492.177151202] [controller_manager]: Loading controller 'io_and_status_controller'

[ur_ros2_control_node-1] [WARN] [1719421492.693518232] [UR_Client_Library:]: Failed to read from stream, reconnecting in 1 seconds...

[spawner-7] [INFO] [1719421493.110171202] [spawner_joint_state_broadcaster]: Loaded io_and_status_controller

[ur_ros2_control_node-1] [INFO] [1719421493.209022273] [controller_manager]: Configuring controller 'io_and_status_controller'

[INFO] [spawner-8]: process has finished cleanly [pid 5332]

[INFO] [spawner-6]: process has finished cleanly [pid 5326]

[spawner-7] [INFO] [1719421493.820724657] [spawner_joint_state_broadcaster]: Configured and activated io_and_status_controller

[ur_ros2_control_node-1] [INFO] [1719421493.983355844] [controller_manager]: Loading controller 'speed_scaling_state_broadcaster'

[ur_ros2_control_node-1] [INFO] [1719421494.326861075] [speed_scaling_state_broadcaster]: Loading UR SpeedScalingStateBroadcaster with tf_prefix: 

[spawner-7] [INFO] [1719421494.355890525] [spawner_joint_state_broadcaster]: Loaded speed_scaling_state_broadcaster

[ur_ros2_control_node-1] [INFO] [1719421494.593874531] [controller_manager]: Configuring controller 'speed_scaling_state_broadcaster'

[ur_ros2_control_node-1] [INFO] [1719421494.593936340] [speed_scaling_state_broadcaster]: Publisher rate set to : 100.0 Hz

[ur_ros2_control_node-1] [WARN] [1719421494.749606715] [UR_Client_Library:]: Failed to read from stream, reconnecting in 2 seconds...

[spawner-7] [INFO] [1719421494.854362853] [spawner_joint_state_broadcaster]: Configured and activated speed_scaling_state_broadcaster

[ur_ros2_control_node-1] [INFO] [1719421495.085332625] [controller_manager]: Loading controller 'force_torque_sensor_broadcaster'

[spawner-7] [INFO] [1719421495.469109334] [spawner_joint_state_broadcaster]: Loaded force_torque_sensor_broadcaster

[ur_ros2_control_node-1] [INFO] [1719421495.623403010] [controller_manager]: Configuring controller 'force_torque_sensor_broadcaster'

[spawner-7] [INFO] [1719421495.797414056] [spawner_joint_state_broadcaster]: Configured and activated force_torque_sensor_broadcaster

[INFO] [spawner-7]: process has finished cleanly [pid 5328]

[ur_ros2_control_node-1] [WARN] [1719421497.774811592] [UR_Client_Library:]: Failed to read from stream, reconnecting in 4 seconds...

[ur_ros2_control_node-1] [WARN] [1719421502.802582755] [UR_Client_Library:]: Failed to read from stream, reconnecting in 8 seconds...

[ur_ros2_control_node-1] [WARN] [1719421511.828092935] [UR_Client_Library:]: Failed to read from stream, reconnecting in 16 seconds...

Accept Public visibility

• I agree to make this context public

[fmauch]

Running the driver in a VM can significantly slow down things / increase delays. I'm not saying, it is not possible to get this somehow reliable, but I expect that this would need some tweaking. It will also be very important how the network adapter is routed to the VM. If possible, I would suggest making the physical NIC available in the VM to reduce possible delays at this stage.