Port to ESP32 on Arduino code base

[Darren777]

Arduino code base is really lacking any form of MQTT5.0 client libraries, I think this could extremely useful not just to me but too many other people running this software on a host of the latest boards.

[X-Ryl669]

I don't know if there is a standard IP stack on Arduino. If there is one, porting the code to this stack should be possible.
I've written 2 different port (for BSD's standard socket, in eMQTT5 repository) and LWIP socket code (here), it's not too hard to do. The very specific requirement are that those options are available:

• TCP_NODELAY (not absolutely required, but improve performance a lot)
• fcntl/O_NONBLOCK
• SO_SNDTIMEO, SO_RCVTIMEO

Else, it'll require a per-platform adaptation code, which is a lot less fun.

[Darren777]

Thanks for your time and feedback. I have been building a passion project over the past few years and have run into the absolute need for MQTT5.0 to run on my client side ESP32 devices which run on Arduino code base. I specialize in front and backend programming but my experience with embedded systems is extremely low, I've managed to get by learning C++ and using great libraries however I think what you speak of is beyond my current knowledge. If it is something you would be interested in doing and helping creating a library for please let me know, I think it can be extremely useful as microcontrollers like the ESP32 have received great widespread public adoption recently and many people are using the Arduino code base due to the vast resources.

[X-Ryl669]

I've looked at the Arduino network status and they have decided to over simplify the TCP/IP stack to remove all notion of layers and sockets. Said differently, their implementation lacks the primitives required to build a compliant MQTT 5.0 client.
That's because a compliant MQTT 5.0 client require the client to be able to receive packets asynchronously (you can have multiple packets transiting on the network at the same time), send without NAGLE algorithm (so that the client does not accumulate multiple small packet in a bigger packet), have a timeout for some operations etc...
The generic "Ethernet" code (or "WIFIEthernet") does not support this, so it would mean a very limited feature set of MQTT 5.0.

So, it'll be very hard to port to generic Arduino.

If however, we only need to target Espressif CPU (ESP8266 and ESP32), then they have exposed a way to disable NAGLE algorithm, and, if I understand correctly, they have written non blocking socket code, so it might work for the very usual, simple case of a client connecting a broker to get/send status & commands. I was wondering, did you try to build the code as-is on your Arduino IDE ? Because if you are using an ESP32, it's very likely that you can directly include "lwip" since it's the default IP code on ESP32. And in that case it'll work (follow the eMQTT5 repository's client for an example code or the test code in this repository).

[Darren777]

Thanks for the reply and time man, I am going to look into what you said and play around with it. Will share my results, hopefully positive ! haha

[Darren777]

Hey man, I appreciate your in depth explanation and have attempted to implement it but I have not succeeded. All the other parts of my system are exactly where they need to be, except the MQTT5 capability for my ESP32 devices that are running on Arduino code base. I do not have the resources to tackle this and would be happy to remunerate you for doing some private work on this and building a library which I and many others could benefit from. Let me know what you think, thanks

[X-Ryl669]

Sorry for late answer, I completely missed this message.

I'm not using Arduino framework for my projects, so I can't estimate the effort required to port this. I've looked at competing options like espMqttClient and they seem to use lwip as I said earlier, but with some Arduino's wrapper on it (see here). This project is a MQTT v3.1.1 client.

I'm sorry but I don't have enough time to complete an Arduino port (and I probably don't have a hardware that's compatible with Arduino's boards). I don't like Arduino's IDE mainly because they lack a debugger and the build system is hidden, so it make developing stuff very hard. I don't think it would make sense to build a hacky ESP32-only Arduino port that's using esp-idf underneath because that's how arduino is built on ESP32.

Maybe you could try to port your code to PlatformIO using Arduino-ESP32 component (so your Arduino code is building just fine on ESP32), since it's running on esp-idf underneath where you can include this library. I've found this post that's explaining quite well the implication of this.

I can probably support you for any question and you can document how you did it so all the users will profit from this.