Yaw error in hardware test

[MartinSegaert]

Hello,

We have recently started to do hardware tests with a T650 UAV.

We started to test the control of the UAV for small trajectory steps of 1m in x, then y and then z by using default gains. It was able to work well. Later we changed to higher gains. With these new gains, the UAV starts to rotate close to 160° at the moment we ask it to go to the start of its trajectory. The step asked at that moment is very small (around 20-30 cm in x,y,z) as the UAV is close to the start of its trajectory already. No error is printed in the controller tab, except " Activating emergency land: Yaw error 160/90".

We are wondering why an increase of the gains produces a yaw rotation when asking to go to the trajectory start. No errors or warnings that could help us understand what happens are printed.

Would you have any clue what the issue could be?

[klaxalk]

Hey, this is really strange. Could you provide a rosbag? Alternatively, could you provide the tmux script logs, which should be logged automatically by our tmux sessions?

[MartinSegaert]

Hello @klaxalk,
Sorry for the late response. I had only acces to these files today. Here below you can find the log files. As bag files are not supported on github. I made a Wetransfer link containing the bag file of the test as well as the corresponding mat file. Thanks for you help.

https://we.tl/t-7pFfvz4bhW

1_Roscore.log
2_Rosbag.log
3_Sensors.log
4_Status.log
5_Control.log
6_AutoStart.log
7_slow_odom.log
8_odom_diag.log
9_mavros_diag.log
10_rtk_serial.log
11_load_trajectory_mini.log
12_goto_trajectory_start.log
13_start_trajectory_tracking.log
14_Eland.log
15_kernel_log.log

[klaxalk]

Dear Martin, I checked the bag. It appears that the jump in yaw error occurred due to a jump in the magnetometer heading provided by PixHawk:

The best response of the system in this scenario is the emergency landing, which I hope is what you got :-).
The problem has nothing to do with you changing the gains or anything you do on the side of the MRS System.

Have you calibrated the magnetometer in Pixhawk properly? This (and the other sensors' calibration) must be done after moving the UAV to a new location. When calibrating it, ensure you are far from high-voltage powerlines and follow the instructions in the QGroundControl properly. The UAV needs to be rotated around its center of gravity (not around the person holding it), and the direction of rotation matters (CCW).

[MartinSegaert]

Dear @klaxalk ,
Thanks a lot for your help. I calibrated the compass, gyroscope, accelerometer and level horizon on the location where the test was done. I will calibrate it again to see if it solves the issue.

However, it is still weird that the problem occured 3 times in a row right when the "goto_trajectory_start" command is pressed. This is why we believed it is not an odometry issue. Is there a link between goto_trajectory_start and the jump in magnetometer heading ?

[MartinSegaert]

Hi @klaxalk,

Today I did hardware tests again. As discussed above, I calibrated again the pixhawk of the UAV to hopefully resolve the issue. Unfortunately, it was not resolved.

1. I first tested the same test as 3 weeks ago. Again, the UAV started to rotate shortly after the go to trajectory start was asked. The only error in the control.log is again "[ControlManager] Activating emergency land: Yaw error ..." and there is also a jump in the compass heading provided by the pixhawk as you pointed out in the previous test 3 weeks ago.

2. Then, I tested the same test with lower gains that worked some time ago. The same problem occurred. This time not at the go to trajectory, but at the third reference step ( a +0.5 step in z-direction). Again, the same error can be seen in the control.log and there is a jump in the compass heading.

3. I then tested another UAV that I also calibrated on site today. When first testing the test with lower gains, it worked fine.

4. To verify that this UAV was working well, I tried the test it with the higher gains. There the problem occurred at the third reference step (-1m in x, -1m in y and -0.5m in z).

Here are the plotted result of the last test:

On the left, it shows the x,y,z position, the thrust and the angular body rates. On the right is the mavros/global_position/compass_hdg.

It is at the moment of the jump in the compass heading measurement that the desired angular body rate is saturating and that the UAV starts to rotate. The error in the control.log is also giving an yaw error of 125, which is the value of the jump in the compass heading measurement.

We now know that 2 different UAVs that were calibrated today on site have the same problem.

My two main questions are:

• What do you think could be the reason of such systematic jumps in the compass heading measurement and how could I solve it ?
• Is the mavros/global_position/compass_hdg a raw data from the pixhawk or is it something processed (karman filter,...) ?

Thoughts and information that could be useful to understand the problem:

a. The firmware on both pixhawks is 1.13.2. We upgraded from 1.11.2 @spurnvoj said that the old firmware is not compatible with the latest CTU system (#149 (reply in thread)). Is firmware 1.13.2 the good one and compatible with the CTU system of February 2023 ?

b. Could it be due to an interference from cables on the UAV itself ? Altought, cabling on the T650 seems to be similar to the one on the f450 where it worked fine.

c. Before the first UAV had this issue, tests worked all right. Later, I recalibrated the sensors and ESCs for the first time on site. It is possibly just after this that the issue arose. What could there be wrong with the calibration on site? There is no high-voltage power line. Only some screens, computers and an extension cord. I calibrated the compass, the gyroscope, the accelerometer and the horizon level. For the compass calibration, I rotated the UAV CCW around its center of mass, with the same initial orientation as in the tests.

d. There are several solar panels at around 20 m from the take-off zone and from where the calibration was done. Is it possible this would give an issue?

e. To be sure the pixhawk doesn't move during tests, we added little strips of duct tape on the sides of the pixhawk. Could this be a problem?

f. Are we sure it is only a problem of compass measurement? The cause of the problem is clearly the jump in heading measurement, but it is weird that it happens just during changes of reference steps and not at any moment. Also, the test with less aggressive gains seems to be more resilient to this issue.

g. On our T650 UAVs, we have a pixhawk 4, a GPS module (Pixhawk 4 GPS module: which has an integrated compass) and a RTK system. Is the magnetometer used by the pixhawk the one in the pixhawk itself or the one of the GPS module? If it is the one of the pixhawk, using the one of the GPS could solve interference problems. Was this already used by CTU ?
In "ctu-mrs/mrs_uav_odometry/config/uav/rtk.yaml" it is specified: "heading_estimator: "PIXHAWK" ". Is that the one inside the pixhawk? There is also the option to use "GYRO" instead of "PIXHAWK". Could that be a good alternative?

Here is wetransfer link for a ZIP file containing the Rosbag file and log files of each test. (numbered from 1 to 4 like explained above)
https://we.tl/t-baPbzaIy5Z

and a smaller ZIP only containing the data of the last test:
https://we.tl/t-66UK8sZe4M

Thanks a lot for for all your help.

Have a nice day,

Martin

[MartinSegaert]

Hi @klaxalk

I did some searching on how to the external compass of the GPS module that is further away from interferences.

I found that the pixhawk was already using two compasses. An internal one and an external one. The use priority of these are respectively medium and high. With this configuration, when changing the orientation of only the GPS module and not the UAV, the heading stays the same in QGroundControl. Only If the internal compass is disabled, we see that the heading is provided by the external compass.

We would like to try hardware tests with the internal compass disabled and using only the external compass of the GPS module as heading measurement. Would you recommend that ? Or is disabling the internal compass completely something to avoid ? Ideally we would like to still use both, but it seems that the only way that the measurement of the external compass is used, is by disabling the internal one.

Thanks again for helping us,

Martin

[MartinSegaert]

The issue has been resolved by disabling the internal compass in QGroundControl. (CAL_MAG0_PRIO on disabled) and using only the external compass in the GPS module. (CAL_MAG1_PRIO on high). The default configuartion were CAL_MAG0_PRIO = medium and CAL_MAG1_PRIO on high.

[petrapa6]

Confirming the solution. @stibipet has observed the same issue and solved it the same way by disabling the internal compass.