Staubli tx90xl not achieving movements close to maximum velocities

[Shoghairg]

We are using moveitcpp to generate trajectories for our staubli tx90xl arm and using the ROS-Industrial Staubli val3 driver to interface with our CS8 controller (both the physical controller and emulated one in SRS). The controller setup in moveit is FollowJointTrajectory. The resultant trajectories are executed at a very slow speed on the physical arm and SRS. One case of a tested motion is Joint 1 being rotated 90 degrees. The time taken initially was 5.3 seconds to complete the path, averaging a speed of only ~0.5 rad/s while the joint is capable of moving 6.9rad/s. This simulated motion can be seen below.

initial.mp4

We adjusted the joint_limits.yaml file to set a high max acceleration value and set the "max acceleration limit" to false so moveit does not default to 1rad/s^2. This increased the speed somewhat to about~0.8rad/s.

We’ve started troubleshooting by looking at the software levels individually. In our gazebo simulations, we can achieve max speed of the arm. At the MoveIt! level we also tested adjusting the number of waypoints generated and how often collision checking is conducted in order to give the arm enough time to accelerate to a higher speed, this increased the speed of the arm further. However, due to the lower number of waypoints, it resulted in choppier movements.

We’ve also tried using the trajectory execution manager or time parameterization in moveit to scale velocities to higher values, but we often reach a ""max velocity exceed" error. Although we are nowhere close to max velocities of the joints, the controller seems to produce the error and will not execute a trajectory. We have tried to check the velocities through the joint_path_command topic produced by the industrial robot client which show velocities to reach 5.8rad/s, however the topics seem to show an incorrect velocity or the arm controller adjusts it as we are not getting close to those values when monitoring feedback from the physical arm. The topic /joint_states also does not seem to reflect correct velocity values for the joints, as shown in the attached plots along with the video of the motion executed in gazebo (max acceleration value was set to 2 rad/s^2).

plotted_motion.mp4

Are there other parameters and values that should be adjusted in the val3 to make the arm move faster?

On the MCP of the arm, we have set the motion descriptor (mDesc) for acceleration, deceleration, and velocity percentages to higher than 100%, this also did not result in any changes in speed. We attempted to edit the VAL3 application, ros_server, pgx files, particularly pushMotion.pgx. Not being too familiar with VAL3, we tried to overwrite the l_mTrajPt.mDesc.decel/accel/vel values, this did increase the speed a bit as well (simulated on SRS). We tried several values for velocity increasing over 100 to up to 900 and up to 9999 for accel and decel and the maximum achieved joint speed (with waypoint changes + acceleration + velocity/accel overwrite) for joint 1 was ~2.2 rad/s.

spedup.mp4

However all attempts to increase the arm’s speed plateau and we do not reach even close to the maximum (or nominal) speed of the joints. Any help or additional troubleshooting ideas would be greatly appreciated! @gavanderhoorn @marshallpowell97

[gavanderhoorn]

just a quick comment on this:

We adjusted the joint_limits.yaml file to set a high max acceleration value and set the "max acceleration limit" to false so moveit does not default to 1rad/s^2. This increased the speed somewhat to about~0.8rad/s.

"max acceleration limit" is not something I recognise. There is max_acceleration and has_acceleration_limits in the joint_limits.yaml file of your MoveIt configuration package.

max_acceleration should be a scalar, a double, and should be equal to the maximum acceleration (in rad/s^2) which the joint is capable of.

has_acceleration_limits should also be a scalar, a boolean, indicating whether there is a valid acceleration limit provided.

Note: setting has_acceleration_limits to false means there is no known acceleration limit. This will make MoveIt use a default of 1.0 rad/s^2.

From your description it's unclear to me what exactly you have done, but this would be the first thing to check.

To expedite things, I would suggest, instead of describing all sorts of settings and attempts, to show code, link to repositories and include verbatim copies of configuration settings and files. Be sure to format things properly by using a Github code block.

---

Edit:

In our gazebo simulations, we can achieve max speed of the arm.

this is almost meaningless, as a Gazebo simulation has very little in common with a real robot. It's an almost completely different set of nodes.

The topic /joint_states also does not seem to reflect correct velocity values for the joints, as shown in the attached plots along with the video of the motion executed in gazebo (max acceleration value was set to 2 rad/s^2).

I don't understand this.

You first describe observations for the industrial_robot_client, and then say something about /joint_states and Gazebo.

IRC is not used with Gazebo (see also my previous comment on that).

[Shoghairg]

@gavanderhoorn Thanks for the reply. Yes, I meant to say the has_acceleration_limits has been set to true and max_acceleration has been set, we could not find documentation for the maximum acceleration (in rad/s^2) which the joint is capable of. We varied it from 2 rad/s^2 to even trying 9999 rad/s^2. The speed increase eventually plateaus as it was assumed the discretization of the path allows only so much time to accelerate.
We therefore tried to change the longest_valid_segment_fraction: in ompl_planning.yaml . The current one is set to longest_valid_segment_fraction: 0.009 , altered from the default of longest_valid_segment_fraction: 0.005 .
We have seen the most changes from these two alterations as well as overwriting the values in pushMotion.pgx adding the following lines:

  l_mTrajPt.mDesc.vel =200
  l_mTrajPt.mDesc.accel = 200
  l_mTrajPt.mDesc.decel = 200

I don't understand this.

You first describe observations for the industrial_robot_client, and then say something about /joint_states and Gazebo.

IRC is not used with Gazebo (see also my previous comment on that).

That's fair, that was a pretty random note I had added in to mention that with either topic of 'joint_path_command' or 'joint_states' were not being reliable enough for us to make concrete conclusions on the behavior of the joints.

[gavanderhoorn]

There are a few other issues on the tracker which are similar I believe.

See #2 fi. Specifically #2 (comment).

We haven't addressed ros-industrial/staubli#35, so it could be that you're running into that issue.

[gavanderhoorn]

I would also recommend to create some Wireshark captures of the traffic between the ROS PC and the Staubli controller (or SRS).

That should give us some insight into what gets transferred to the controller.

[Shoghairg]

@gavanderhoorn In reagrds to #2 (comment), we were not able to find nominal values anywhere in the tx90xl documentation. However, we did try setting it to 50-80% of the maximum values and did not have an improvement or change of response.

ros-industrial/staubli#35 could be the issue. Although we have not seen the arm achieve values close to the max or even 50% of the max, with even just one joint moving.

I created some wireshark captures and dissected the packets, This was for the x, y motion pattern once again, in which joint 1 is the technically the only one moving to complete the motion.

The velocity value transferred to the controller goes over 0.5, however, the motion we observe does not seem to reflect that, considering the maximum of joint 1 is 6.9rad/s. However, following #2 (comment) should we be considering the nominal velocity to be our maximum (obtained at vel of 1) and considering nominal at 50% of the maximum? Meaning the max we would generally see of a joint is rather 3.45rad/s?

Is it feasible to set the velocity over 100? as mentioned before being done by modifying pushMotion.pgx and overwriting velocity to over 100 l_mTrajPt.mDesc.vel =200. This definitely helped with our speed but, again, not close to a 6.9rad/s value. With the overwrite the best speed seen is still about 2.2rad/s (estimated from distance joint rotates and time taken).

[gavanderhoorn]

I created some wireshark captures

did you intend to link to the captures? I don't see a link in the comment.

and dissected the packets,

I hope you found ros-industrial/packet-simplemessage.

As to what you observe, I cannot provide any guidance right now. It's rather difficult to help diagnose these kinds of problems without access to the hw itself.

If you can provide the wireshark captures and the JointTrajectorys which 'caused' that traffic I might be able to check some things.

[Shoghairg]

I hope you found ros-industrial/packet-simplemessage.

Yes, I did. I've linked the captures in a zip file here: simple_message.zip

[gavanderhoorn]

And the trajectories?

[Shoghairg]

And the trajectories?

simple_message.txt

[gavanderhoorn]

And the trajectories?

simple_message.txt

I meant the JointTrajectory messages which get sent to the driver.

Perhaps as part of the FollowJointTrajectory action goal.

[gavanderhoorn]

Taking a quick look at the capture:

• the first point in the trajectory gets sent in packet 855
• the last traj point is sent in pkt 1236
• dt between these two is approx 1.37 seconds
• the robot starts to really move in pkt 892 (although the STATUS pkt claims it's moving in 874 (almost immediately after receiving the first traj pt)
• the robot stops moving (ie: reaches the destination for J1) in pkt 1355
• dt between pkts 892 and 1355 is approx 1.71 seconds
• total motion for J1 is from 1.5905 rad to 0.0015 rad (or approx 1.589 rad or approx 91 degrees)
• 1.589 rad in 1.71 seconds is approx 0,929 rad/sec, which is about 53 degrees per second

This is obviously not very fast compared to the max joint velocity limits which appears to be about 400 deg/sec for J1.

[Shoghairg]

I meant the JointTrajectory messages which get sent to the driver.
Perhaps as part of the FollowJointTrajectory action goal.

Sorry, I'm not too familiar with wireshark and not sure what you mean. Aren't these the JointTrajectory messages? They are being sent to the driver, of message type Joint Tracjectory Point. That was the only message found to be sent to the driver when using wireshark Lua dissector.

[gavanderhoorn]

"the driver" is the ROS node, in this case industrial_robot_client.

JointTrajectory is the ROS message type which is used to communicate joint trajectories between ROS nodes.

In the case of industrial_robot_client, there is also a joint_trajectory_action server which exposes a FollowJointTrajectory action server which is what MoveIt communicates with.

The industrial_robot_client converts JointTrajectory messages into SimpleMessage messages and sends it to the Staubli controller.

The data you've captured contains the SimpleMessage side.

We're currently lacking the ROS side to all of this, so we cannot compare what the industrial_robot_client received to what is sent to the controller.

[Shoghairg]

* 1.589 rad in 1.71 seconds is approx 0,929 rad/sec, which is about 53 degrees per second

This is obviously not very fast compared to the max joint velocity limits which appears to be about 400 deg/sec for J1.

What about the velocity values in the message, the maximum value is 0.521296 in pkt 2673. If the controller received that value, should we not be expecting faster speeds?

The data you've captured contains the SimpleMessage side.
We're currently lacking the ROS side to all of this, so we cannot compare what the industrial_robot_client received to what it sent to the controller.

Is this what you meant? I created another capture and did a rosbag for the topic /arm_controller/follow_joint_trajectory/goal follow_joint_trajectory.zip as well as joint_path_command
joint_path_command.zip which I think may be what the industrial_robot_client is converting into SimpleMessage.

[gavanderhoorn]

What about the velocity values in the message, the maximum value is 0.521296 in pkt 2673. If the controller received that value, should we not be expecting faster speeds?

I did not make any statements about whether or not things should go faster. It was simply a summary of the network traffic captured while the driver was transmitting the first trajectory.

Is this what you meant? I created another capture and did a rosbag for the topic /arm_controller/follow_joint_trajectory/goal follow_joint_trajectory.zip as well as joint_path_command
joint_path_command.zip which I think may be what the industrial_robot_client is converting into SimpleMessage.

yes, this is what we were missing.

[Shoghairg]

I did not make any statements about whether or not things should go faster. It was simply a summary of the network traffic captured while the driver was transmitting the first trajectory.

Yes, I was just wondering if you did have any thoughts on it?

yes, this is what we were missing.

Looking through these, from the rostopics, the maximum velocity achieved is 3.62 rad/s. The average velocity is around 2.45rad/s.
Considering the packets- motion starting at pkt216 (time: 0.900294597) (J0: 1.591900229) and terminating at pkt501 (time:1.8974699) (J0: 0.002294761), the max speed seen is 0.518 (around 3.57rad/s), on average around 2.4rad/s once again. Calculating considering the time take for Joint 1 to travel, it is ~1.6rad/s)

The execution of the arm does not seem to correspond to the speeds it is receiving.
Increasing the acceleration limits set in the yaml file does increase the speed values produced by the rostopic and simplemessage but still the arm does not execute like the speeds these values read.

[gavanderhoorn]

The execution of the arm does not seem to correspond to the speeds it is receiving.

yes, I agree with you.

Increasing the acceleration limits set in the yaml file does increase the speed values produced by the rostopic and simplemessage but still the arm does not execute like the speeds these values read.

changing the acceleration settings should not be necessary, so that's at best a work-around and not something I would be happy with.

At the moment the problem appears to be on the execution side (ie: inside the VAL3 application), but I don't have any suggestions as to where to look for potential problems in the implementation right now.

And just to make sure: other VAL3 programs which execute the same motion can achieve the programmed velocities? Also on the real robot?

[Shoghairg]

And just to make sure: other VAL3 programs which execute the same motion can achieve the programmed velocities? Also on the real robot?

Sorry on the delayed response, did not have a chance to test on the arm for a while.
We had not tested out other VAL3 programs but have done so now.
We tested out for motions for x to y path as well as shorter ones covering a square. Videos below- left is rosserver, right is our other val 3 application.

split_x_y.mp4

split_sqaure.mp4

I don't know of a way to record data to get the average speed through just running the val3 application, welcome any ideas here.
Seems like similar speeds are achieved using the other val3 application and rosserver when going from point to point.
For the square motion, latency seems to be introduced from moveit when going to the calculating trajectory for next goal point/ going to the next goal point instead of the cohesive blending seem with the other val3 application. The continuous accel/deccel for moveit having to come to a stop before executing next goal point may be slowing things down, would you possibly have any workaround implementations to this like moveit servo or other?
In regard to speed, since they appear to be the same, we are thinking this may just be because 100% velocity is not infact the maximum velocity we have been comparing to in the datasheets, instead it is a nominal speed (we have not been able to find a specific value for nominal documented in tx90xl). If that is the case, then clipping to 100% is not infact clipping to maximum. Could this be an explanation as to why we do not appear to be traveling at the velocities that the arm receives?

[gavanderhoorn]

Could it be you've interpreted maximum velocity attainable as the velocity at which motions are always executed?

Motions must be planned to be smooth, so velocity profiles always include ramp up and ramp down phases. Without sufficient distance to travel, no robot will get near the maximum specced velocity limits.

[Shoghairg]

Motions must be planned to be smooth, so velocity profiles always include ramp up and ramp down phases. Without sufficient distance to travel, no robot will get near the maximum specced velocity limits.

Is there another layer of smoothing occurring besides the trajectories generated by moveit? Aren't the values sent to the controller factored in for that?
Our generated velocities do ramp up and down, but looking through the SimpleMessage topics and the our arm execution. The arm does not appear to travel at those speeds, that is if we consider that 100% is infact the maximum velocity.

We have attempted to ramp up the acceleration limit to 9999 for test and that does not seem to near a maximum either, which theoretically should allow reaching the maximum velocity even if the distance is tool long.
The manual does not list a maximum accel/deccel. I guess it would be possible that the maximum accel attainable by the physical arm would not be sufficient to reach maximum in a short duration.

I had missed it before but there are the nominal values listed in the manual as well and I'm just wondering if that's what we should consider is the achievable point (even though we don't have a load at this point when running tests), and more so wondering if the controller views the nominal as it's 100%.

[gavanderhoorn]

Is there another layer of smoothing occurring besides the trajectories generated by moveit?

My comment was a general one, not one specific to your usage with ROS or plain VAL3 programs.

All robot controllers filter user-programmed trajectories. It doesn't matter whether you've already taken that into account -- of course, if your trajectory is already sufficiently smooth, the filtering won't change that much, but it's still performed.

I had missed it before but there are the nominal values listed in the manual as well and I'm just wondering if that's what we should consider is the achievable point

you should certainly take the nominal values as nominal (ie: what would be 100%).

We have attempted to ramp up the acceleration limit to 9999 for test

yes, please do not do this.

and that does not seem to near a maximum either, which theoretically should allow reaching the maximum velocity even if the distance is tool long.

it all depends on distance, and what the physical capabilities are of the robot you're controlling. If the motors cannot reach the acceleration you specify, it just won't happen.

The acceleration limit you can configure is still several steps removed from the actual motion controller limits, and then there are still safety systems and other filtering mechanisms.

[gavanderhoorn]

But with your latest comments about native performance with VAL3, it starts to look like the driver reaches similar performance.

I'm not claiming here everything is OK and it can't be improved, but from the videos I don't get the impression the driver is significantly slower -- at least not with the motions you've tested.

[marshallpowell97]

I had missed it before but there are the nominal values listed in the manual as well and I'm just wondering if that's what we should consider is the achievable point (even though we don't have a load at this point when running tests), and more so wondering if the controller views the nominal as it's 100%.

When we are talking about the mdesc, 100% is the nominal, not the maximum. For most applications, going above 100% is ok, and I have even seen 200% or more do just fine with low/no payload.