New Controller: ESP-Now

[happytm]

I wanted to bring to your attention following links on this subject:

https://www.youtube.com/watch?v=6NsBN42B80Q

https://github.com/SensorsIot/ESP-Now-Tests/blob/master/EspNow_Slave_ESP8266/EspNow_Slave_ESP8266.ino

https://www.youtube.com/watch?v=ILnc9KQjHEQ

https://www.instructables.com/id/ESP-NOW-Home-Automation-Esp8266-Raspberry-Pi-MQTT/

https://github.com/HarringayMakerSpace/ESP-Now

https://github.com/HarringayMakerSpace/sonoff-adhoc

Also please keep eye on this project:

https://hackaday.io/project/161896-linux-espnow

It is interesting because with bunch of battery powered cheap esp8266 there is a possibility of creating low power home automation network without using high power consuming wifi or expensive lora modules. Andreas show in above video the power consumption is very low. If esp-now receiver (as a main controller powered by mains power) can be implemented in espeasy that will be lot better than their power hungry P2P network using UDP protocol.

Thanks

[enesbcs]

Interesting project, as i understand ESPNOW is a connectionless protocol which looks like just UDP. And also it works in a lower logical level of the network, which is currently handled by the Linux operating system. (vendor specific action frame of 802.11)
It can only work through a serially attached ESP-01 with linux. It can be integrated into Controllers, but i have no further idea about new device plugins, as ESPNOW is only a raw protocol.

[TD-er]

Well I have plans to add support for ESPnow to ESPeasy as fallback when there is no (known) network in reach.
So I want to transport ESPeasy p2p protocol over ESPnow and also help setup new nodes.

One thing that makes ESPnow really interesting is that it is much faster in delivering "the first byte" compared to WiFi.
So ideal for battery powered devices.

[happytm]

I think it should be other way around to make sensor truly low power. Instead of espnow as backup on remote low power sender sensor it should be primary sender protocol and if in case espnow fails on remote sender it can send MQTT message to public or private broker directly as shown below. All the remote sensor unit does not need espeasy software but some custom sketch to do the above.Only Master Unit (receiver for espnow messages from remote sensors) need to have espeasy software. The master unit with espeasy will behave like MQTT broker or it is like Mysensor Gateway.

For low power normal mode if espnow is ok & working properly it will be like this.

Link to code for sender & receiver (serial gateway connected to raspberry pi or as a MQTT gateway) units is at the link below.

https://github.com/Wim3d/ESP-NOW/blob/master/Master_sender.ino

Now only thing to do is add some receiver code in espeasy. Below is examples of receiver code.One is serial gateway like Mysensor protocol and one is MQTT gateway.

https://github.com/Wim3d/ESP-NOW/blob/master/Slave_receiver.ino

https://github.com/SensorsIot/ESP-Now-Tests/blob/master/EspNow_Slave_ESP8266/EspNow_Slave_ESP8266.ino

Thanks

[TD-er]

I agree that your suggestion is using the least power, but there's a rather big but here.
You need to pair nodes (based on MAC address) for using ESPnow.
Also there is a limited number of pre-configured connections (pairings) between nodes.
If I remember correctly, it is 10 or 20.
So you cannot have 1 node to serve more than 10 or 20 other nodes via ESPnow.

But still it is a great solution for having a battery powered node to communicate with one that's always powered and let that one act as some kind of gateway.

[happytm]

In normal home automation there are some sensors which needs to be outdoor (for example , 1 outdoor device for weather station and few indoors device for windows and doors or device measuring oil or water tank levels where sometime mains power is not available ). So most of the time ratio of wired to battery powered devices are less than 1:5.By implementing espnow protocol not only it helps with outdoor battery powered devices conserve power but it also help extend the range by creating small mesh network. I agree with you that this protocol is not suitable for larger network of devices.My plan is to use it as a farm sensors network using soil moisture & light sensors where mains power to sensors will be impractical at each sensor.

[TD-er]

Well for the more longer range and larger networks, I am working on something here :)
And it will also be with power conserving in mind.

[happytm]

Eagerly waiting for it.

Thanks

[happytm]

Here is an attempt to connect ESP device (sender) using ESP-Now protocol to linux machine(receiver):

https://hackaday.io/project/161896-linux-espnow
https://github.com/thomasfla/Linux-ESPNOW

Thanks

[enesbcs]

I am afraid in the RPI world ESPNOW through linux kernel wireless driver is a very-very early alpha stage thing, i guess that it will be always easier to attach an ESP-01 with serial to the RPI and join to the ESPNOW across it.

Altough it is very interesting, but I read a few sentences in the hackaday ESPNOW description, which did not really win my liking:

• ... Since some wireless card support monitoring and injection of raw packets... (emphasis on "some")

• "the ESP-NOW protocol is built to wait for an ACK frame after every packet sent. It ensures the reception. And, while the ACK is not received, the ESP-NOW library tries to send the same packet over and over again." A workaround for this to use broadcasting virtually everything.. (really?)

• "Setting the ESP to send over broadcast to avoid ACKs is only available on ESP32. "

[happytm]

I was just thinking out loud. You are right it is very alpha stage. Hopefully one day RPIEasy will serve as one controller receiving all data packets from all low battery powered espnow devices.You are right the best option right now is second esp device connected to pi via serial collecting all data from remote sensors (good thing is it's $ 2 extra).But if we figure out adhoc network for pi it will be simpler solution. The first issue you raised is solved somewhere else.According to link below Raspberry Pi's built in wifi can be put in monitor mode:

HarringayMakerSpace/sonoff-adhoc#1

He is using fake MAC ID itself to send data between devices.Very cool hack.
https://github.com/HarringayMakerSpace/sonoff-adhoc

For small range indoor network there is another way:

https://forum.arduino.cc/index.php?topic=10555.0

https://forum.arduino.cc/index.php?topic=228312.0

We don't need IR protocol for this one on software side (just serial ) and I have to work on increasing the IR range to make it useful.This could be used for monitor windows and door sensors indoor with button cell batteries.The key is to design IR led with very narrow beam to focus on IR receiver at Raspberry PI to extend the range.I think 15 to 20 feet could be achievable and that would be enough for most rooms for home automation.

Thanks

[enesbcs]

It sounds good, we will se.
Another possibility for low-range direct communication between an RPI Zero W and an ESP32 is their integrated Bluetooth.

[happytm]

We discussed receiving IR signal over serial but following is sending IR from RPIEasy to some IR switches scattered around in the house. In order to test indoor IR network I have ordered these actuator devices:

They are $3 each for quantity of 3 or more.

https://www.ebay.com/itm/AC-Power-Energy-Saving-IR-Infrared-Remote-Control-Wireless-Outlet-Switch-Socket/263189804948?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2057872.m2749.l2649

bigclivedotcom had teardown & review of these devices a while back and they come with unnamed microcontroller and 5V power supply.It looks like they are hackable for any other uses (maybe replacing controller with ESP-8266 or Atmega328 with .96" Oled and whatever sensors required) :

https://www.youtube.com/watch?v=RdqzdMUO6QE

[happytm]

The microcontroller chip looks like Attiny84 and if that is the case then it is even better to upload custom sketch to it although the default program looks quite good to turn any old mains device into remote controlled one without any programming knowledge.

[happytm]

I have just posted following proposal on OpenMQTTGateway:

1technophile/OpenMQTTGateway#421

1technophile/OpenMQTTGateway#196

Let's see if they respond.

Thanks

[happytm]

My attempt to use esp-now and adhoc network. It works ok. I am no coder so I had to bring together snippets of code from many sources. Please someone good at coding can improve it. I like the adhoc network better because data from remote sender unit can be received by any hardware like raspberry pi zero and it is faster than espnow. We can put raspberry pi in monitor mode as shown here:

https://github.com/HarringayMakerSpace/sonoff-adhoc

Thanks

[enesbcs]

About this "adhoc" networking:
Using MAC address field to transfer data from one endpoint to another is an interesting scenario but it is NOT a legitimate usage of the Data Link Layer. See ARP poisoning:
https://www.tutorialspoint.com/ethical_hacking/ethical_hacking_arp_poisoning.htm
For this reason i do not recommend and will not support this method.

On the other side ESP-NOW is based on IEEE802.11 Action Vendor frame standard and is a legit protocol, seems nice, i think that it has to be implemented.

[enesbcs]

With RPIEasy i plan to use an ESP8266 through serial as a receiver, and another ESP8266 as a sender. I am trying to setup a generic sketch now.

On the ESPEasy the ESPNow implementation will be tricky as:
espressif/arduino-esp32#878

[happytm]

Very good news.Espeasy is not important here as ESP8266 connected via serial could serve as just transparent gateway collecting data and sending data to other ESPNow or Lora devices and that way there is less of complexities associated with Espeasy's wifi. If implemented with ESPNow we can have mesh network of sensor or actuator devices working together doing their own job and interacting with RPIEasy via ESP8266 gateway with ESPNow & Lora protocol.This also solves problem of hardware complexities associated with other protocols like RF and zigby. Option for Gateway software replacing ESPEasy could be ESPCoreRules firmware which is strip-down version of ESPEasy.He just published new big update and I have been testing it for couple of days by adding ESPNow to it and it is working fine. We can further strip it down by removing all wifi and mqtt code and we still have rules engine like ESPEasy which is main feature of ESPEasy. Link is below:

SmartNodeRules/ESPCoreRules@8b2ab73

I am eagerly waiting for ESPNow.

Thank you for all your efforts.

[happytm]

Another nice feature on gateway could be this which we are lacking right now in RPIEasy:

https://github.com/1technophile/OpenMQTTGateway/blob/development/main/ZgatewayIR.ino

Thanks

[enesbcs]

#24

[enesbcs]

First experimental version of the ESPNow node firmware is available here: (defaults to Sender/receiver can be switched to serial gateway mode, 1 firmware for all task)
https://github.com/enesbcs/EasyESPNow

[enesbcs]

Commit e51b007 introduces first approach for the RPIEasy serial-ESPNow controller

[happytm]

Thank you.

You were probably first in publishing major IOT framework on Raspbberry PI and now you are again first one to implement ESPNow protocol (beyond simple example) on ESP8266 as a practical ready to use IOT firmware.I hope this grow to be the best IOT platform. For me this will be fun for next week to test this firmware and report back. I will need some information on settings in global settings file. I am sure you will add some instructions in readme file.

Thank you again.

[enesbcs]

Thanx! As i said i am not a real programmer, but a problem solver. So i grabbed the old and simple ESPEasy R147 (the amazing work of team letscontrolit), then purged the WiFi and Webserver codes from it, and added a new Controller for ESPNow communication. I had to hack some core functions to make it work. /it's more hacking than programming :) / There are not so many changeable settings, i will write them to the README. I hope it will work with larger number of nodes, currently i've tested with one GW and one SenderAndReceiver endpoint. I know that some display problems may appear on the RPIEasy side in the ESPNow Management page, when replies did not arrives in time, error handling has to be much more sophisticated, consider it as an early beta. (in case -1 appears in fields, try to refresh the page)
I read several articles about ESPNow with ESP8266 and broadcasting messages which mentions that it is not working. I've used Core 2.5.2 for compilation and broadcasting just works fine. There are precompiled binaries on the github, the default WifiChannel is 1, the default unit number is 254 which can be changed by serial commands, or remotely with RPIEasy.

[happytm]

What is the minimum setup required? As far as I can read into code there is 1 ESP8266 (Serial Gateway?) which is connected to RPIEasy via serial and it is unit 0. All other ESP8266 devices are sending or receiving or both.Their unit number could be 1 to 8 (I want to setup 8 remote sensors).My question is how unit # 1 to 8 know serial gateway's MAC address to connect to it? I loaded all devices with stock 1 mb binary you provided but I do not see any activity on serial gateway's serial console except the setup information.I setup mode 3 on all remote devices and mode 1 on serial gateway.

Am I doing something wrong here?

I assume setup is like this:

https://user-images.githubusercontent.com/36576353/60380551-9a95a580-9a15-11e9-8cc8-d736ffac79b1.png

Please help me.

Thanks

[enesbcs]

Yes the setup is the same as the picture, you need one serial GW with a node ID 1-254, 0 will never be a good choice as 0 means "send to all units". The other units node number needs to be unique in the range 1-254.
There are no need for knowing the MAC address as broadcasting is used, and it is working.
I've tested it with two D1 Mini, (untested with 1mb puya flash) one is connected to my PC with USB and one is connected to an USB wall adapter for power only.
For simple testing please try the following:

1. upload binary to 2 pieces of ESP826 with FULL FLASH ERASE! (and after first start it may need 1-2 reboots per device to init with factory reset settings..)
2. connect ESP8266 Support Pro Mini Extender #1 to a Raspberry PI through serial OR an Ubuntu PC with USB (and of course with RPIEasy)
3. set Unit number to 1 at "Config" menu, add ESPNow at "Controller" menu and select correct serial port, set baudrate to 115200 default, try to set default destination node to 0 (altough this is not a serious setting for GW) than you have to see: "Serial ESPNow GW initialized" at the console. If not, than your serial communication is failing, try to fix before going deeper.
4. connect ESP8266 P2P Node list #2 to an 5VDC wall adapter, and refresh the main page of RPIEasy, a new node with default number 254 has to be appear (without any setting)
5. On RPIEasy go to the "ESPNow endpoint management" page, select node 254 and "Manage selected node" - Select Send&Receive, unit number 2, destination node 1, and set wifi channel to the same as the local ESP.
   (the local ESP also can be managed on the same page named "local node")

[happytm]

What gpio pins on RPI and ESP8266 should I use to connect ESP8266 with RPI? I never used serial port of raspberry pi in the past.

Thanks

[enesbcs]

I recommend first to try it with a PC and USB <-> D1 Mini (or with an RPI and OTG USB cable) as enabling serial in a RPI Zero W needs a lot more actions and excercise...
First you have to disable onboard Bluetooth, to release the default serial line and of course you have to enable UART. (Hardware/Pinout&Ports)

Than use command line 'raspi-config' tool: 5- Interfacing Options-P6 Serial- (Kernel logging disabled + serial port hardware enabled)

For connection see:
https://www.instructables.com/id/Connect-an-ESP8266-to-your-RaspberryPi/

[enesbcs]

I setup mode 3 on all remote devices and mode 1 on serial gateway.

I've just noticed that you write mode 1, which is for "SENDING only mode". Gateway is Mode 0. But do not worry about it, RPIEasy ESPNow controller enforces Mode0 at Init stage on every startup through serial (if serial works of course). :)

[happytm]

I think serial is working for me now but I get following message:

21:11:00 Event: Clock#Time=Fri,21:11
21:11:10 Try to init serial /dev/ttyAMA0 speed 115200
21:11:10 Serial connected /dev/ttyAMA0
21:11:15 Serial ESPNow GW adapter is not compatible!
21:11:20 Try to init serial /dev/ttyAMA0 speed 115200
21:11:20 Serial connected /dev/ttyAMA0
21:11:25 Serial ESPNow GW adapter is not compatible!
21:11:44 Try to init serial /dev/ttyAMA0 speed 115200
21:11:44 Serial connected /dev/ttyAMA0
21:11:49 Serial ESPNow GW adapter is not compatible!
21:12:00 Event: Clock#Time=Fri,21:12
21:12:14 CMD: serialcommand,espnow,mode
21:12:14 CMD: serialcommand,espnow,mode
21:12:15 CMD: serialcommand,espnow,mode
21:12:42 Try to init serial /dev/ttyAMA0 speed 115200
21:12:42 Serial connected /dev/ttyAMA0
21:12:47 Serial ESPNow GW adapter is not compatible!
21:13:01 Event: Clock#Time=Fri,21:13
21:13:09 CMD: serialcommand,i2cscanner
21:14:00 Event: Clock#Time=Fri,21:14

Getting closer.

Thanks

[enesbcs]

If you see "Serial ESPNow GW adapter is not compatible!" than no answer arrived on serial.
If you enable Debug log setting at Tools->Advanced, than a working serial looks like:

17:45:46: Try to init serial /dev/ttyUSB0 speed 115200
17:45:46: Serial connected /dev/ttyUSB0
17:45:50: SERIAL: >espnow,mode,0
17:45:50: SERIAL: 0
17:45:50: SERIAL: Ok
17:45:51: Serial ESPNow GW initialized
17:45:51: SERIAL: >unit,1
17:45:51: SERIAL: 1
17:45:51: SERIAL: Ok
17:45:51: SERIAL: >espnow,dest,0
17:45:51: SERIAL: 0
17:45:51: SERIAL: Ok
17:45:51: SERIAL: >espnow,chan,1
17:45:51: SERIAL: 1
17:45:51: SERIAL: Ok
17:45:51: SERIAL: >settings
17:45:51: SERIAL: System Info
17:45:51: SERIAL: Build         : 148
17:45:51: SERIAL: Unit          : 1
17:45:51: SERIAL: Free mem      : 42952
17:45:51: SERIAL: MAC           : CC:50:E3:5D:3E:EE
17:45:51: SERIAL: Ok

[enesbcs]

I've found a possible problem, if the logging level is not set to Debug, the communication is not handled, working on it.

[enesbcs]

Yep, i just tested it on a Zero W, with Enable UART: Enabled, Internal Bluetooth: Disabled and after rebooting:

18:41:20: Pi Zero W 40 pins
18:41:20: Try to init serial /dev/ttyAMA0 speed 115200
18:41:20: Serial connected /dev/ttyAMA0
18:41:24: SERIAL: >espnow,mode,0
18:41:24: SERIAL: 0
18:41:24: SERIAL: Ok
18:41:25: Serial ESPNow GW initialized
...
18:41:25: SERIAL: >settings
18:41:25: SERIAL: System Info
18:41:25: SERIAL: Build         : 148
18:41:25: SERIAL: Unit          : 1
18:41:25: SERIAL: Free mem      : 42952
18:41:25: SERIAL: MAC           : CC:50:E3:5D:3E:EE
18:41:25: SERIAL: Ok

[happytm]

I was never able to make it work after many efforts. For me the process is still not clear. I had both esp8266 devices flashed with bin files you provide.The receiver esp device is connected to serial port of raspberry pi as you shown earlier. I guess I am not doing something right in RPIEasy setup.

Anyway , how many milliseconds or microseconds a sender unit (esp8266) will take to send a simple 6 byte message & go to deep sleep in your estimate?

Thanks.

[enesbcs]

Please test the Raspberry serial first with another device to see that it works. At least with an USB CP2102 or FTDI from the Raspberry serial to your PC...
You have to Disable kernel logging at Interfacing options/Serial and Enable the serial port hardware. If after rebooting Raspberry at RPIEasy/Hardware menu the "Disable Serial port usage by kernel" button appears than something went wrong.
Or simply the ESP01 is unable to deal with the 115200 baud serial speed?
Make sure that bluetooth is disabled, when bluetooth still enabled than both ttyS0 and ttyAMA0 devices may appear which is not good.

Otherwise as i said before, with a proper Wemos D1 Mini, you can even connect it directly with a microUSB-USB cable to a Linux computer or directly to the Raspberry Pi with an OTG. So please put your ESP-01 away a little bit, until you find out where is the problem.

The process can not be even simpler, if the serial connection works, RPIEasy sets the serially connected ESP gateway settings automatically, and the precompiled (remote) binary target node is "0" so it will broadcast to every unit when powered on.

It still works to me, I've just flashed another 2 nodemcu, and even an ESP32 3 days ago. There were some compatibility issues with ESP32 but now it is solved, an ESP8266 gateway is now able to receive from an ESP32 without problem through ESPNow.

[happytm]

Thank you. My mistake. My kernel logging was on and I disabled it and everything working fine now.

Is there any test code to measure time it needs to send message on sender unit so we can estimate power consumption estimate?

Thanks again.

[enesbcs]

I'm very happy that it works.

IEEE802.11 Action Vendor frame is at least 30 bytes without data + EasyESPNow sensor data is 14 bytes for a one value Task, such as a Switch Device.
So you can count with 44 - 56 bytes for a package size.

I know how LoRa data transmission is counted, but i have no clue about ESPNow, sorry. Maybe i can add some code to measure the ellapsed time and print it to serial before going to sleep.
I am sure that it is faster than basic AP-STA based communication several times.

Be prepared that you will loose remote control when you enable deepsleep. :) Deepsleep loop can be cancelled by pulling the pin GPIO16(D0) to GND.

[TD-er]

As far as I know, the payload of ESP-now could be up-to 250 bytes, which kind of suggests it only has a MAC address as header.

Transmission speed is hardly adding up to the transmission time, but initializing the radio might be.
It doesn't have to connect to an AP, so there you're saving a lot of time.
But it may need to wait for the receiving end to wake up (if it needs to be woken up)
Typical such an interval of waking up on the receiver side may be 102.4 msec.
So I guess it may take up-to about 200 msec to send an ESP-now packet. (100 msec on average)

In comparison, connecting to a WiFi AP does take about 3.5 seconds if it has to scan all channels for the known SSID. If it still knows the last known channel and BSSID (MAC address of the AP), then connection time is about 900 - 1000 msec.
And receiving an IP-address may add another 30 - 3000 msec, which is also not needed for ESP-now.

[happytm]

Thanks @enesbcs & @TD-er.

[enesbcs]

I measured the sending of 14 bytes through ESPNow. It takes 80-120milliseconds, however it does not contain the wake up and init process, just the sending. So the 200msec may be correct.
For fastest results ALIVE_PERIOD has to be changed to 0 in the sketch.

[happytm]

Please look at following post. He is using linux machine on receiving side. He claims lot faster communication using ESPNow.

https://hackaday.io/project/161896-linux-espnow/log/162377-more-testing-with-rt-preemptb

https://hackaday.io/project/161896-linux-espnow/log/168678-1khz-closed-loop-control-of-up-to-16-motors-over-wifi

Thanks.

[TD-er]

It can be faster, if the receiving side has the WiFi radio set to no sleep.
The delay is due to the beacon interval.
Not sure if possible with ESPnow active, but normally when you have AP mode active it will keep the radio on receiving mode continuously. So you could test with AP enabled.

[happytm]

Thank you for your explanation. I wonder this could be implemented on raspberry pi as a receiver device?

Thanks.

[enesbcs]

In theory, everything is possible. But i do not think it will be a good solution, so I'm sure I won't be the one to develop this. For me, the serial gateway is the best and only acceptable solution for a Raspberry, as the other one:

1. needs full RT kernel compilation for Raspberry, and of course someone who will maintain it (dislike)
2. setting integrated WiFi card to ESPNow-receiving mode will kill normal TCP/IP connectivity (big dislike)
3. as i am using broadcast sending mode, there are no importance if anyone listening on the other side.. the serial ESPNow gateway is always on. (it is in COMBO mode in fact)

[happytm]

I did not think about all implications behind this so now I understand your concerns.

Thank you.