Tools: Add RealFlight Autotest

[robertlong13]

On the dev call, we were discussing testing #27839 in RealFlight. I thought it would be nice to have a framework for running optional automated tests in RealFlight. I ran the test before and after the PR and loaded both logs into the PID Review Tool and compared. They look the same (though @IamPete1, it would be really nice to be able to load two logs in here for this).

Draft PR because I'd like to get some advice on the right way to do add this. Do we do this by adding disabled tests to things like quadplane.py? Do we add separate files like quadplane_realflight.py? Not sure. This is my a-bit-hacky placeholder that I used to test the aformentioned PR (though I'm very late as it's been merged for a week).

(This PR is currently based on top of #28282, so only the last two commits in this will be here after the other is merged)

[peterbarker]

Not a great fan of needing to pass in the realflight address.

Ideally we:

• integrate this correctly with vehicleinfo.py
• get rid of "external: True" from vehicleinfo.py, instead specify which external simulator is involved
• factor out the code from FlyEachFrame which sets things up from the vehicleinfo definition
• read configuration for external simulator from a .json file - same code as we use for callisto.json but pointing out how to use the external simulator
• extreme stretch goal allow use of external simulator from sim-on-hardware
• use the same information from vehicleinfo.py in sim_vehicle.py to make it easy for it to use the same definitions.

[peterbarker]

Yeah, very much not the right way to do things :-)

[robertlong13]

Not a great fan of needing to pass in the realflight address.

Unfortunately I see no good way around this. The address will be different for every person, unless they are crazy and running autotests with Cygwin. If they are using WSL (like me), I can have a shell command I get it automatically, but if someone uses separate machines (which I believe is more common in the core dev group), or uses a VM, we need to give them the option to override.

If we want to force this to only work in WSL and Cygwin, I can remove that.

[robustini]

Yes, self-testing could also be interesting.
I have already tested the reset position on startup and it works well.

[robertlong13]

Big update after long discussion with Peter on voice chat. I'm really only addressing one of his bullet points above, but he seemed generally in favor of this when we spoke.

I've eliminated the dirty hack for how I made CI skip this test (and replaced it with something that still isn't perfect but hopefully tolerable). Now, unless you have REALFLIGHT_IPADDR set or pass the --realflight-address argument, this test automatically passes. Again, the idea here is to be able to manually run some tests locally and watch/analyze to make sure everything works correctly, while still getting to work within our extensive autotest framework. This does not handle automatically launching RealFlight, selecting the aircraft/airport (though I have some ideas for a tool that could do that in the future).

Added the ability for SITL to look for an environment variable for the RealFlight IP address, which saves a little typing vs passing it in the frame name. On WSL, I use this in my bashrc (because frustratingly, the windows host IP is not static):

# IP Address of the Windows host running RealFlight for ArduPilot SITL
export REALFLIGHT_IPADDR=$(host $(hostname).local | grep "has address" | awk 'NR==1 {print $4}')

For those of you running RealFlight on a separate windows box, you can just make that constant.

This is now integrated with vehicleinfo.py (now that I understand that file's purpose). You can launch with sim_vehicle.py -v ArduPlane -D -f realflight-titan-cobra (and I'd like to add a few more from the SITL models repo). When launching with sim_vehicle, it defaults to using RealFlight's built-in joystick handling, but if you want to use MAVProxy's RC, you can disable the FDM with the RC_PROTOCOLS parameter and launch with the extra --mavproxy-args --sitl=127.0.0.1:5501.

There's one last sneaky issue that I'm hoping @tridge might be able to shed some light on. When I run with the autotest, I get occasional socket timeouts between SITL and RealFlight. When I fly manually, I basically never get these. With autotest, I get them randomly, on average maybe every 10 minutes. With the current 1s timeout, I'd get a large disturbances when it happens. I changed it to timeout after a max of three frames and now when a socket timeout happens, it handles it gracefully after 25ms (how I'm getting 10,000 errors in 25ms doesn't make sense; the update function isn't called nearly that fast) with no noticeable symptoms when they occur.