Combining simulation with CFclient

[knmcguire]

A continuation of the discussion we had here https://github.com/orgs/bitcraze/discussions/796

@simondlevy is doing very awesome work with the simulator and connecting it with the cfclient (see Show&Tell). So this discussion is to coordinate the PRs necessary to achieve this and to coordinate the efforts!

My thoughts or some current events that are important:

1. Connecting directly to the cfclient is nice, but how about doing it through the crazyflie python library (CFLIB) instead, and then connecting it through the CFclient? The cfclient is more 'front end' and the cflib is more backend, and I feel this connection is more 'back end'
2. There are currently still a lot of issues with thefirmware bindings that the output is not exactly the same. It might be saver to first focus on the 'simple controllers' that have been made in python first, although those only support abs. height and relative velocity which might be limiting, but then we can leave the
3. Currently I was busy with revamping the simulation repository by adding proper documentation instead of just the readme and changing the folder structure. That was on hold for a short while due to an intermediate deadline but this is very close to completion. It would be better to wait with PRs like this after I've merged it (hopefully this week or the next).

@simondlevy let me know what your thoughts are!

sorry that means that the previous work you put in the PRs will be need to redone, but I do believe that if we plan this correctly, it will be a better integration and better future maintenance possible. I am genuinely excited for this feature so I'm more than happy to help out!

[simondlevy]

Thanks for your thoughts and suggestions, @knmcguire. Having never worked on a large, complex project like Crazyflie, I probably didn't appreciate the difficulty and downsides of doing a large PR like this. I'm still eager to contribute to Crazyflie in any way I can, and I have no problem going through several iterations of ideas before they're ready for public use. The core hack I wrote (threaded socket communication of the stick demands / telemetry) will still be usable in whatever form the new features take, I think, so the time certainly wasn't wasted.

As for your point 2 (using simpler ad-hoc PID controller instead of firmware PID bindings), that was a modification that I made yesterday and attempted as part of my second PR. As for using the cflib instead of the client directly, that is also something that you mentioned earlier that I should have looked at (I keep mixing up the cflib and the cfclient!)

Like you I'm working against a deadline this month (the Neuromorphic Vision Deck I've posted about) and am also teaching an intensive AI course for the next six weeks. Hence I agree that it makes sense to hold off on these modifications for a while. That would also give me time to look into the cflib, and possibly your toolbelt suggestion as well.

For the time being, since you mentioned strategizing, I'm wondering what you think about a gradualist approach. For example, when I started this effort, I first simply added a "webots" item to the client's Interfaces pulldown, with a stubbed callback method, and of course tested that the actual connection to my Crazyflie2.1 still worked. With this approach there's a minimum likelihood of merge conflicts, but of course it creates more individual PRs and presents the user with unimplemented front-end features. I'll be eager to know your thoughts, and I look forward to returning to this when we both have time.

[knmcguire]

Ah no worries! Then probably it is better to take increment steps and to not rush anything. Besides you might have changed your approach after the deadline and everything :). I'll make a new thread with my idea of the strategy

[knmcguire]

So this is what I have in mind for the PR strategy:

1- A simple socket example stand-alone in simulation repo. the simulation starts with the controller type initializing the socket connection, which sends back telemetry. So we have one pythons script to be executed separately, that listens to the socket and maybe send a simple commands. So both the standalone socket connector python script and the new python controller for webots are both in the PR for crazyflie-simulation.
2- Socket communication by the cflib. This is where the cflib instead of sending the package structures through crazyradio, sends it through the socket connection instead. This can also be a stand alone PR for the crazyflie-python-library as that is a significant change
3- Connecting simulation with the cflib. Now that both things are in place, then we can start connecting simulation and cflib. That means that the example scripts for cflib should work for simulation. This will probably require more unpacking magic on the crazyflie simulation side as ideally we want to send the exact same content as is done through the Crazyradio (so only a simulation PR is necessary)
4- Adding simulation option to the CFclient Once the simulation is started, a socket opens, and when pressing the scan button on the cfclient, it shouldn't not only see all real crazyflies but all simulated crazyflies as well! If everything runs well on the crazyflie python library, this hopefully will only require a simple change in the cfclient side.

That is what I had in mind, but perhaps some things will change along the way!

I think that step 1 is very easy to implement, so depending on your time, you can add that socket listening script. And there are already aspects that you have implemented already, but we just have to find the right spots for it.

What do you think?

[knmcguire]

So adding the simple socket script here I mean: bitcraze/crazyflie-simulation#43. I can just merge it and integrate it with the latest changes I did here bitcraze/crazyflie-simulation#44 (shouldn't be too difficult)

And for the other steps, we will need to be a bit more careful and I've also got to discuss some things with my colleagues in what sense they think is wise. perhaps I can already work on step 2 while you are busy with your upcoming deadlines.

[simondlevy]

Thanks, this sounds like a good plan. At this point it sounds like there is nothing for me to do immediately, but if you reach a point where you would like my input, just let me know.

[knmcguire]

I've specified it in the first simulation PR as I think we can merge it with a small adjustment. Let me know if you need me to help out with this. The other planned PRs let's save them for quieter times.

[simondlevy]

Yes, I could use some help with this!

[knmcguire]

I've made a standalone example in the PR such that your commits are still preserved in history :) let me know what you think.

[simondlevy]

Thanks, that looks great! I'm looking forward to being and official contributor 8^) I appreciate the tip about where to do PRs, too.

[knmcguire]

cool! I'll just go ahead and merge it then. I'll link you to any upcoming PRs in case I'd like your input, like adding sockets to the cflib. I'd like to say I'll be working on this next week but I'm always overly optimistic about how much time I have at the end of the day 😅 so it might also that I won't have time for it until in a month or 2 ( I'm currently on partial sickleave )

[knmcguire]

by the way, you are already an official contributor to the crazyflie-firmware remember ;) so your name will be on the next release change notes!