EGM frequency

[lIlIlI1]

Hi @jontje , I have a question about EGM frequency.

So, I'm using this library with IRB 4600, and I tried to print out the time it took to get new data from the robot both using real robot and RobotStudio. Since I used your sample codes, I'll just put the codes that are related to this issue here:

while(ros::ok())
{
ros::Time start_time =  ros::Time::now();
    if(egm_interface.waitForMessage(500))
    {
        ros::Time time_now = ros::Time::now();
        std::cout<<time_now-start_time<<std::endl;
    }
}

It turns out in RobotStudio, it takes about 0.016s, and 0.025s when working with the real robot. My setup is following:

• using EGM_ControllerInterface
• SampleRate in RAPID code is set as 4 ms
• RobotWare 6.08
• Protobuf 3.6.1
• ubuntu 14.04.

I noticed that you mentioned control lag in issue #15 and I checked ABB documentation. As you also mentioned in issue #25 that you're using 4ms primarily, so the questions are:

• do you also have this much lag when using joint velocity control?
• is there a way to reduce the lag?
• the fastest 4ms is just for the robot to take new control references, which is open loop, and closed loop usually has 10-20 ms delay, is that correct?

Appreciate any feedback. Thanks.

[jontje]

Hello @lIlIlI1,

The EGMControllerInterface assumes that the user always wants to send a command back to the robot, after a message has been received. This means that the inner loop is waiting for the user to call the write(…) method, and it is likely that this is interfering with your measurements since it seems like you don’t do that.

I am planning to change this so that the interface class can be configured to skip the waiting.

In the meanwhile, you can try again after commenting out the following lines in your local clone:
https://github.com/ros-industrial/abb_libegm/blob/master/src/egm_controller_interface.cpp#L82-L85

Your questions:

• do you also have this much lag when using joint velocity control?

I have not made any measurements myself, and I haven't noticed any problems due to control lag in the applications I have worked with. However, those applications haven't had strict requirements on the motions.

• is there a way to reduce the lag?

You can try and turn off the EGM filter that is used by default inside the robot controller. You need to update the system configurations and you can do this according to:

RobotStudio --> Controller tab --> Configuration --> Motion --> External Motion Interface Data --> Update the instance you are using (or add a new instance) to use Level = Raw instead of Filtering

You need to restart the robot controller afterwards.

• the fastest 4ms is just for the robot to take new control references, which is open loop, and closed loop usually has 10-20 ms delay, is that correct?

Those are the values from the manual and I haven't verified them myself. One thing to note is that the robot controller sends out messages every 4 ms, and then it's up to the external side to reply with new EGM references within the next 4 ms. When new references have been received, then there is the delay until they actually affect the robot's motion.

I hope this helps!

[gavanderhoorn]

Closing due to inactivity.

[jbeck28]

I'd like to ask this here since it seems related to the conversation. I'm also working on reducing the controller lag, and implementing a ros_control based driver, and I've noticed two things.

1. The controller lag is giving me jerky starts to motions, since it causes the controller to overreact. I will admit I may have been a bit zealous in tuning the PID, but that was in an attempt to reduce the control lag. I see now that this is likely more a matter of the robot controller than the ros_control loop that I've implemented.

2. I've noticed control loop jitter on the order of 500 microseconds, sometimes a bit more. Does anyone know how much of this is due to the robot controller? It's totally possible that the code I'm using (which was lifted largely from a KTH YUMI package) isn't the most real-time safe, and since I'm using a realtime kernel, this might be the cause... But before I start digging too far into that, I'm curious what is reasonable to expect from the robot controller itself?

[jontje]

Hi @jbeck28,

Have you tried any of the suggestions I mentioned in the comment above? And what RobotWare version are you using?

1. The controller lag is giving me jerky starts to motions, since it causes the controller to overreact. I will admit I may have been a bit zealous in tuning the PID, but that was in an attempt to reduce the control lag. I see now that this is likely more a matter of the robot controller than the ros_control loop that I've implemented.

Do you mean that this occurs when you tell the robot controller to start up the EGM communication? Or when you already have an EGM channel established, and then activate your ros_control based driver's controller?

EGM has a configurable ramp in time that should make the initial motion smoother, but this is only active when the EGM channel is starting up.

2. I've noticed control loop jitter on the order of 500 microseconds, sometimes a bit more. Does anyone know how much of this is due to the robot controller? It's totally possible that the code I'm using (which was lifted largely from a KTH YUMI package) isn't the most real-time safe, and since I'm using a realtime kernel, this might be the cause... But before I start digging too far into that, I'm curious what is reasonable to expect from the robot controller itself?

I haven't made any measurements myself, so I unfortunately don't know. Is your driver's loop driven by data (i.e. loops when EGM messages are available), or do you for example use something like ros::Duration(<time>).sleep()?

[jbeck28]

@jontje thanks for your reply, and sorry for my delay. Time got away from me.

I did try everything listed in that comment, and I will say the control lag seemed to improve, I now see lag on the order of 40 to 50 ms, rather than 70 to 80.

Do you mean that this occurs when you tell the robot controller to start up the EGM communication? Or when you already have an EGM channel established, and then activate your ros_control based driver's controller?

The jerky start of motion actually typically takes place well after the EGM communication has been started. Typically, it occurs after execution of a long trajectory. I'll run a trajectory through moveit which takes roughly 20 minutes (often 100,000 points on the trajectory msg), then once the robot reaches the end pose of the trajectory, I'll wait a bit for it to settle, and then issue a move command to get it out of the way... at which point there's a control lag, and the controller wildly overshoots leading to the "dynamic load" error on the controller. Often the robot won't even try to move as the commanded velocity is so large.

Come to think of it, this might even be unrelated, as it seems only to really happen after execution of trajectories. But it really did seem caused by controller lag when I commented.

I haven't made any measurements myself, so I unfortunately don't know. Is your driver's loop driven by data (i.e. loops when EGM messages are available), or do you for example use something like ros::Duration().sleep()?

The loop at first would loop only when EGM messages are available, using the waitForMessage() method. I will add a timeout of less than 4ms, and I suppose if this doesn't fix the jitter, the issue likely isn't with EGM..?

[stephanie-eng]

I've also encountered what I think is the same issue:
I'm using abb_egm_rws_managers, so let me know if you'd rather I post this issue on that repo instead. When trying to understand the latency, I put a timer around waitForMessage. The time difference is generally small, but cyclically spikes as high as 15 - 20 ms.

  auto t1 = std::chrono::high_resolution_clock::now();
  egm_manager_->waitForMessage(0);
  auto t2 = std::chrono::high_resolution_clock::now();

I've tried to read through the code to investigate, and this is my understanding of the architecture this library:

The clock indicates which methods that block with a condition variable. Since I'm using the abb_egm_rws_managers, waitForMessage blocks using the p_new_message_cv condition variable, but it also seems that the waitForMessage from abb_libegm blocks using a (different) condition variable. Is it possible for there to be a deadlock due to the blocking condition variables in waitForMessage and readOutputs, and if so, would that be related to the latency I'm seeing?

[marctuscher]

Hi,

I actually encountered the same problem. I wasn't yet able to track it down completely, however, I fixed the control lag by switching from EGMManager to EGMControllerInterface. Maybe that also helps you.

[pucciland95]

Hi all,

I am also looking into this problem. The ABB documentation about EGM states:

The time between writing a new position until that given position starts to affect the actual robot position, is usually around 20 ms

Do you think there is an additional delay using this repo, or am I misunderstanding something?