Airspeed: Terrible problems again.

[tubeme]

Hello,

We've had a peaceful times without airspeed problems but the dreadful problems arose upon us again. :)

1. Five month ago we reported that we cannot arm the PX4 (the command arming) if the airspeed is flat 0 m/s. We had to blow in the pitot while arming in order to work. This is happening with EagleTree sensor which is much calmer than the original sensor and when on the ground it just stays at 0 m/s. In version 1.5.0 it is the same story. In order to arm we have to blow in the pitot...

2. In version 1.5.2 and 1.5.3 we could not perform a single successful airspeed calibration. Both versions are tested on two setups - AUAV + Original Sensor and Pixhawk + EagleTree sensor. In two different configs - VTOL and Airplane.

[dagar]

Could you provide more detail? Any error messages or screenshots?
At what point does it fail? After you're instructed to blow?

@AndreasAntener

[AndreasAntener]

@tubeme as Daniel suggested, please provide some information on how it fails, which messages you get when (before, between, after blowing, etc.).

[tubeme]

@AndreasAntener @dagar I will provide more info when I get to the machine at our workshop. By memory, it gets error at 2/3 in to the procedure while calibrating and the progress bar moving. Way before it asks to blow in the pitot. Cant remember the messages. Will post them later.

[tubeme]

The first problem is way more interesting. It is silly when one of us have to stay on all fours and blow in the pitot while the other have to arm... We did not try this alone. It is very interesting how a single person can arm this way :)

[tubeme]

It does not want to even begin the airspeed calibration with the machine I'm using. It is AUAV X2 + original airspeed. Here you go. v1.5.3

[LorenzMeier]

Please share a picture of the front and back of the sensor. You have bad hardware or a too long I2C cable.

[pkocmoud]

There are airspeed testing features included on the NSH command line driver. Calibration in not necessary. Have you tried this?

[tubeme]

@LorenzMeier Neither bad hardware neither long i2c cables. On all three systems I'm trying right now v1.5.0 is flying killing well, it has no problems with airspeed calibration. The problem started when we installed 1.5.2. We use different FC and airspeed sensors for testing. The problem persists. I see that the v1.5.2 release states "Improves the calibration handling on Pixhawk 2.1 boards." Could this change introduced some kind of problem? We did not try 1.5.1. But it seems that 1.5.1 introduced a lot of changes and improvements as well. So it might be somewhere there.

[tubeme]

@pkocmoud We did not try this utils. What are they? In general we don't need to test the sensor. The sensors are working, that is for sure.

[dagar]

This is failing because the airspeed sensor driver is reporting a non zero error count.
https://github.com/PX4/Firmware/blob/master/src/modules/commander/airspeed_calibration.cpp#L146

I added this after trying to help someone that had a long i2c run. Airspeed still seemed like it was responding on the ground, but their logs were filled with huge spikes and dropouts.

It's possible that requiring an error count of 0 for airspeed calibration is too strict. Maybe a small number of errors are tolerable?

1. Change the message to be clear about it failing due to an error count and to check the sensor, reset the system, and try again.
2. Allow a very small number of errors.
3. Only fail if the error count changes between calibration start and finish.

@AndreasAntener @LorenzMeier thoughts?

[dagar]

@tubeme once you see that error message you'll need to power cycle the vehicle to try again. If you check perf on the command line (or the perf files on sdcard) it'll show the error count.

airspeed_comms_errors: 0 events

[LorenzMeier]

Who say's that @tubeme doesn't have a bad hardware setup? The fact that it worked before doesn't say anything about it being ok. Plus @tubeme had a lot of obscure airspeed related issues in the past that we didn't experience on other vehicles.

Until proven otherwise (photographs of the sensor and wiring and the output of the test command of the sensor) I'd say this is a proper check which is failing.

[dagar]

I agree, bad hardware setup is still possible.
At a minimum we should add a more descriptive message for the calibration failing with next steps.

[pkocmoud]

These errors are exactly what we found as part of pixhawk/Hardware#55 .

@tubeme Please run on the NSH terminal: meas_airspeed info
Note the number of errors. If you have one of my AUAV i2c hubs, this would help this problem.

[dagar]

Slightly better calibration failure error message here - #6324

[chief51]

Not sure if this helps...but I had problems calibrating the digital airspeed sensor I purchased from Jordi MRobotics.

The only way I could get it to calibrate was to ensure to stop air from going in the pitot tube prior to powering up my pixracer. It has worked every time I needed to calibrate it. If not, I would get a message of either it missing or cal would fail.

My question and not sure if this happening prior to the master being updated with this fix, that at times I needed to calibrate it again because it wouldn't arm at times.

Ben

[tubeme]

On the first system:

aspd_com_err: 0 events
aspd_read: 18502 events, 0 overruns, 9823195us elapsed, 530us avg, min 383us max 3439us 363.380us rms
aspd_buf_of: 2 events

I will post images from the sensor later when I get to the workshop.

[tubeme]

System 2:

aspd_com_err: 0 events
aspd_read: 5738 events, 0 overruns, 4750114us elapsed, 827us avg, min 497us max 13284us 658.601us rms
aspd_buf_of: 2 events

[LorenzMeier]

Is that the Hobbyking sensor? If yes, then it might very well have read issues. Can you try the calibration again, then read the perf counter?

[tubeme]

And the Third system:

aspd_com_err: 1 events
aspd_read: 9737 events, 0 overruns, 9957957us elapsed, 1022us avg, min 497us max 9854us 984.300us rms
aspd_buf_of: 2 events

I will post images tomorrow.

[tubeme]

The sensor from the first system is Eagle Tree v3 airspeed sensor. The second system has HK sensor. The third system has the original 3DR sensor. With the 1.5.3 and 1.5.2 I cannot perform the calibration at all on all three systems with the errors i sent on the screen. The sensor from the first system is Eagle Tree v3 airspeed sensor.

All the three systems work perfect with the 1.5.0. We perform airspeed calibration we fly the systems. the Airspeed readings are adequate in flight and in the logs. BTW after the problems with 1.5.2 and 1.5.3 we downgraded to 1.5.0 to continue our work.

[LorenzMeier]

@tubeme The check you run into was designed in to guard against a known reliability issue with the 3DR and the HK airspeed sensors. You need a level shifter for them to operate reliably. The same is likely true for the Eagle Tree sensor. So the fact that it seemed to work for you doesn't mean your hardware setup is correct.

To the contrary, all indication are that all three of your systems require changes to your airspeed sensing setup and the new check catched it.

[LorenzMeier]

@tubeme I get an error count of zero on my setup after a long runtime and performing the calibration:

aspd_com_err: 0 events
aspd_read: 259258 events, 0 overruns, 197698293us elapsed, 762us avg, min 493us max 3406us 484.759us rms
aspd_buf_of: 2 events

[LorenzMeier]

Level shifter:
http://www.auav.co/product-p/i2c-hub.htm

Sensor (@pkocmoud should be able to confirm this is the fixed design)
http://www.auav.co/product-p/i2c-as.htm

[tubeme]

@LorenzMeier OK so I understand that there is a new check past v1.5.2 that makes 100% of our speed sensors obsolete?... Two of the configurations have about 500 hours on them with this setup and the one with Eagle tree sensor has good 50 hours of flight. All the flying was airspeed dependent Position, Altitude, Mission and all that was done on faulty hardware... I still doubt that.

The Eagle Tree v3 specs in the Users Manual say it accepts 3v-16v which means that with this sensor there is no voltage problems. Also this sensor is very good indeed and very reliable. We have three of these and they all work very good, never gave any problems at all !!! Compared to HK and 3DR sensors.

Also there is another detail I want to share with you. For HKPilot32 i measured the i2c with 5.1v. Given the sensor is 5v then there should have been no problems at all. For the AUAV X2 i measured 4.96v on the i2c bus. I don't see a problem as well.

Lorenz I will buy the AUAV shifters for all the configs. But is it really hardware problem?? And if yes give us direction where to get airspeed sensors up to specs?

[tubeme]

Some images.

[LorenzMeier]

@tubeme The system detects and reports an issue. It really doesn't matter if you survived this so far. Make sure that the bus voltages are correct and that you don't have too long cabling - what is the total wire length of your external I2C bus?

Do you have the post-flight perf file of one of your flights? What is the I2C error count it reports?

[tubeme]

@LorenzMeier We are very careful with the i2c wiring because it is susceptible to rf. We keep the wiring as short as possible, the original cabling usually but not much longer if needed in the different setups.

The errors in the post flight logs of the machines are 95% 0 errors and very seldom 1 error and on couple of occasions 3 errors. This is for the past two months for two of the machines.

As I explained the voltages are OK in all the setups.

And also we did not survived, but rather thrived an all weather situations without a hitch with the same sensors in flight

There is no obvious reason why the calibration does not start. We will continue looking for some sort of general hardware problem.

[LorenzMeier]

You indicated that you can't calibrate because of non-zero error counts and you made the implicit argument that this is on three different systems. So what exactly are you seeing as issue? If the error count is almost always zero, why can't you calibrate then?

Do you have other failure cases of the calibration routine?

[tubeme]

I never mentioned non-zero error counts! Look at my initial issue 1. if Airspeed = 0 cannot arm 2. Cannot begin airspeed calibration with v1.5.2 and 1.5.3 with the screen provided above.

I will do another re-flash with the 1.5.3 and try again...

The calibration routine does not start at all...

With v1.5.0 everything OK. We can perform calibration without any problems on all three systems.

Never had airspeed calibration issues up to now.

[dagar]

Ok, so two different things going on here.

1. if airspeed is 0 you can't arm
   Do you get any error message or anything on screen? Does it have to be non zero during the request to arm or does it have to have any airspeed reading before arming will work?

2. error count non zero causes calibration to fail

[tubeme]

Yes two thing totaly different.

1. No we have no error message for this. It happens only with eagle tree sensor, because the sensor does not measure speed up to 4 m/s so on the ground the airspeed measurement is flat 0 m/s (not so for the HK and 3DR sensors). it arms with the button but then on command it gives the tino tino tino alarm without any indication in QGC of what might be the reason. We found the solution by mistake!!! Now we have to blow in the pitot tube to have some airspeed reading different than 0 and next with arm command the plane arms perfect, ie the airspeed reading should be prior the request for arming. After that everything is perfect but it is unprofessional and inconvenient.

2. Probably the non zero causes calibration to fail. So I understand that the non zero check did not exist in 1.5.0? And if I use the AUAV i2c splitter with level shifter I will solve all the problems we have and we will be able to calibrate, is that correct? Are you all fly with level shifters??

[dagar]

1. Ok, we'll need to get someone with an eagletree sensor to investigate. I don't have one.

2. It's conservative, but better safe than sorry.
   With several different newer 3.3V sensors I tried multiple times and couldn't get a single error. After losing planes to stupid airspeed sensor problems I'd rather be conservative here. If you're confident there's no issue despite the driver telling you otherwise I can show where to change the code for your own usage...

[tubeme]

@dagar I can send you one eagle tree sensor? Give me s-mail address...

[dagar]

How do you find it compares to the MS4525DO sensor? I could do a quick comparison, especially wrt temperature drift. E-mail me? [email protected]

[tubeme]

Works with all sensors available to us! Thanks to all.

There are some minor glitches with the Eagle Tree. But tomorrow I will send a spare sensor to Daniel

@LorenzMeier @dagar