"timeout detected" error messages

[scastle-green]

We are receiving timeout messages from ros_kortex after submitting trajectory plans via moveit.

[ INFO] [1675259731.281967856]: Execution request received
[ INFO] [1675259731.307206096]: New goal received.
[ INFO] [1675259731.307270138]: Joint Trajectory Goal is accepted.
[ERROR] [1675259734.308322063]: Runtime exception detected while sending the trajectory
[ERROR] [1675259734.308413631]: timeout detected: BaseClient::ValidateWaypointList

[ INFO] [1675259734.308713274]: Controller 'gen3_lite_joint_trajectory_controller' successfully finished
[ WARN] [1675259734.308825200]: Controller handle gen3_lite_joint_trajectory_controller reports status ABORTED
[ INFO] [1675259734.308873719]: Completed trajectory execution with status ABORTED ...
[ INFO] [1675259734.309034276]: Execution completed: ABORTED
[ERROR] [1675259739.389072433]: Runtime exception detected while sending the trajectory
[ERROR] [1675259739.389123619]: timeout detected: BaseClient::ExecuteWaypointTrajectory

Version

ROS distribution : noetic
ros_kortex: noetic_devel
commit: db34e0c

Expected to playback the specified trajectory. Have tried with a number of different trajectories and all give the same error.

Any thoughts as to what might be causing this and/or how to overcome it? Trajectories playback ok in simulation but just not on the physical arm.

Thanks for any advice!

[felixmaisonneuve]

Hi @scastle-green,

Your log says the ValidateWaypointList call times out. It is not an error in your trajectory, but more in the communication qith the arm. Can you make sure your arm is correctly connected and you can communicate with it?

Looking at your logs, I feel you might be able to communicate correctly. I am thinking your arm might have an older firmware kit (SWU2.2.0). ValidateWaypointList was added with firmware kit 2.3.0, so if your arm has an older firmware kit, it won't respond to the call and it will eventually timeout.

In you arm's webapp, go the Systems->Upgrade page and validate your firmware bundle is at least 2.3.0.
You can find the most recent firmware bundle (SWU) on the Kinova website

Best,
Felix

[scastle-green]

Hi @felixmaisonneuve ,

Thanks for your response.

Firmware bundle is at 2.3.2-r.3

Cheers in advance,

Simon

[felixmaisonneuve]

Ok, your firmware bundle is good.

Can you run the example "example_full_arm_movement"?

It uses waypoints and ValidateWaypointList as well and we will be able to see if the function is working correctly.
If it does work, the problem might be coming from the trajectory. It times out after 3 seconds. Maybe if the trajectory is super long it takes more than 3s to validate it. I doubt it, but increasing the timeout delay might be an option

[scastle-green]

@felixmaisonneuve

Hey, so I ran the example and it seems to go through fine. I suspect you might be right about the complexity of the trajectories however, even the basic ones seem to timeout as well.

Can you point me to where I might be able to find/adjust the timeout for the validation?

[felixmaisonneuve]

I think you could use the api_rpc_timeout_ms parameter when launching your driver : https://github.com/Kinovarobotics/ros_kortex/tree/noetic-devel/kortex_driver#usage

The default value is 2000ms. You could change it to 5000ms and see and if it changes the time it takes before it times out.

[scastle-green]

Hi @felixmaisonneuve

Sorry for the late response on this. I've just tried increasing this parameter (up to 30000 ms) but it still fails with the same message. Also, it fails quickly (e.g. well before 30 seconds has passed), so I get the feeling that either this parameter is not being received properly or not affecting this function timeout.

Any thoughts?

[felixmaisonneuve]

Can you try adding ROS_INFO("Timeout for api call : %i\n", m_api_options.timeout_ms) in "kortex_driver/src/generated/robot/base_services.cpp" inside the ValidateWaypointList function :

ros_kortex/kortex_driver/src/generated/robot/base_services.cpp

Lines 5274 to 5277 in 6f9a445

	try
	{
	output = m_base->ValidateWaypointList(input, m_current_device_id, m_api_options);
	}

It should look like this :

try
{
    ROS_INFO("Timeout for api call : %i\n", m_api_options.timeout_ms)
    output = m_base->ValidateWaypointList(input, m_current_device_id, m_api_options);
}

You will have to catkin_make after, but it will allow us to see if the timeout value is the expected one when you call ValidateWaypointList.

[c-oechsner]

I am experiencing the same error when sending a trajectory via MoveGroupCommander.execute(). My trajectory has 594 waypoints and has been computed prior by MoveGroupCommander.plan(). I can see the trajectory in my planning scene in Rviz, however I keep getting the mentioned timout error. I am starting kortex_driver with the parameter api_rpc_timeout_ms:=20000 and added the line from @felixmaisonneuve 's comment and rebuilt. However, the output does not show in the console. Does that mean ValidateWaypointList doesn't even get called?

[lfycan666]

I also encountered the same problem,my trajectory has 629 waypoints and has been computed prior by arm_group.compute_cartesian_path().In addition, I have a question about the parameter api_rpc_timeout_ms,the explanation I saw for it is: The default X-axis position of the robot in Gazebo. The default value is 0.0. And I execute the following code on the command line:roslaunch kortex_driver kortex_driver.launch api_rpc_timeout_ms:=5000,the problem still exists：

[ERROR] [1700907318.615351855]: Runtime exception detected while sending the trajectory
[ERROR] [1700907318.615421365]: timeout detected: BaseClient::ValidateWaypointList