Sniffer mode to capture firmware updates #1650
Comments
|
Promiscuous mode would be possible as you can set both DTU and inverter Serial ID on the current menus. Those could be added both/all to the maximum five NRF pipes. Difficulty would be to guess the channels used by both Hoymiles DTU and inverter. But switching between the five NRF channels might be sufficient as the messages/packets are usually repeated 15 times/transmitted 16 times. So switching channel after 2-4ms may be sufficient, especially if configurable. Payload parsing may not be needed in the first place. Though that and a pass-through repeater mode may be added at a later stage ;) |
|
I don't like the idea of channel hopping for captures. |
|
The typical WR <-> DTU messages are repeated by the NRF radio 15 times, i.e. 16 packets as you can see in the following pictures. One packet with 15 repeats takes about ~35 ms to be sent, i.e. a single packet will take about 2,5-3 ms with a ~1 ms gap between repeated packets.
So switching the five channels after 2-5 ms listening on each should allow you to listen on each channel 2-3 times during the complete transmission period for a single packet with all its retransmits. And also the inverter is changing the channel from time to time, which has been known as channel hopping. Please note that it may be easier (and has been done initially with the Hoymiles DTU Lite too) to simply hook two serial2usb converters onto the RX/TX pins of an original Hoymiles DTU: BAUD 125000, 8N1 should be the correct settings for the UART communication between the main cpu and the RF chips. See #1540 (comment) |
|
@stefan123t OK, since I have an original DTU, it might be easier to make both signals available from the outside to capture data. As long I don't have to open the lid of the transceiver, all is fine :) |
|
Here you have it, my new HMDUSI - HoyMiles Dual USB Sniffer Interface
Now I need to do some software to capture data with exact time stamps. Do you happen to know something which can be used? |
|
I think capturing the data should be very straightforward with a simple python script. If you run that twice in two separate terminal windows, I think you're good. Make sure pyserial is installed: import serial
import datetime
# Replace baud_rate with the actual Hoymiles baud rate
def read_serial(port, baud_rate=9600, log_location='log.txt'):
try:
ser = serial.Serial(port, baud_rate, timeout=1)
print(f"Connected to {port}")
try:
with open(log_location, "a") as file:
while True:
if ser.in_waiting > 0:
data = ser.readline().decode('utf-8').rstrip()
timestamp = datetime.datetime.now()
log_entry = f"{timestamp}: {data}\n"
print(log_entry)
file.write(log_entry)
except KeyboardInterrupt:
print("Interrupted by user, exiting.")
except Exception as e:
print(f"Error: {e}")
except serial.SerialException as e:
print(f"Serial error: {e}")
finally:
try:
ser.close()
print("Serial port closed.")
except:
print("Error closing serial port.")
# Replace with your port, e.g., 'COM3' for Windows, '/dev/ttyUSB0' for Linux/macOS
read_serial('/dev/tty.usbserial-0001', 9600, "log_1.txt")Use |
|
According to other users info the BAUD rate is 125000 and 8N1 for parity, start/stop bits. The Grid Profile update has been traced #900 (comment) by now, but you can still contribute the firmware update if you want. Eventuall they both use the 0x0A DOWN_DAT main command, though that ist just a quick guess. To get a firmware dump you would need access to the JTAG/SWD port on the inverter. Some other users do have that at the moment and have contributed a firmware dump already. |
|
I modified the code a bit to provide a more or less usable output. Before doing any more steps with the DTU and updates, can you please take a look to the dumps and check if this is usable at all? Thanks! #!/usr/bin/python3
import serial
import datetime
import binascii
# Replace baud_rate with the actual Hoymiles baud rate
def read_serial(port, baud_rate=9600, log_location='log.txt'):
try:
ser = serial.Serial(port, baud_rate, timeout=1)
print(f"Connected to {port}")
try:
with open(log_location, "a") as file:
while True:
if ser.in_waiting > 0:
data = ser.read(ser.in_waiting)
h = binascii.hexlify(data)
timestamp = datetime.datetime.now()
log_entry = f"A: {timestamp}: {h}\n"
print(log_entry)
file.write(log_entry)
file.flush()
except KeyboardInterrupt:
print("Interrupted by user, exiting.")
except Exception as e:
print(f"Error: {e}")
except serial.SerialException as e:
print(f"Serial error: {e}")
finally:
try:
ser.close()
print("Serial port closed.")
except:
print("Error closing serial port.")
# Replace with your port, e.g., 'COM3' for Windows, '/dev/ttyUSB0' for Linux/macOS
read_serial('/dev/ttyUSB0', 125000, "log_1.txt") |
|
Nice that you got it working! I'm glad I could contribute this small bit. PS: ```py |
|
Thanks for your hint, slways something new to learn ;-) I will keep the logging running for a while. |
|
Here we go, these logs were collected during 5 to 6 hours until night came to send the inverters asleep. I can't tell which of these logs are RX or TX but following log has above combined and sorted by time: This should make analysis easier. I'll detach the DTU and use OpenDTU instead now. |
|
Any feedback about the logs? Are the captures usable? I plan to perform firmware updates on all inverters on friday. |
|
Hi @broth-itk, I did not have time to look into the logs in detail. I just looked into the combined output again and found that after several minutes of the Given the above communication B: are the TX messages from the DTU to the inverter (e.g. Main command |
|
Thanks for your feedback!
But as time is precious, if the collected dumps are complete, postprocessing can always be done. |
|
Here you go, these logs include the firmware update of two -4T inverters to 1.0.27 (Firmware Build Date 2023-06-05 10:24:00): |
|
@broth-itk I have downloaded them but I will not be able to analyse them in the coming weeks. |
|
@stefan123t Thanks! The logs contain all data since startup of the DTU. Maybe there is something interesting - indeed. |
|
Huch ich sollte mal die Logs vom Firmware Update von @broth-itk auswerten und wie versprochen dokumentieren. @broth-itk ich habe das log_1.txt ein wenig aufgeräumt, die Kommandos auf dem UART / der Seriellen Schnittstelle zwischen MCU und NRF / CMT Modul beginnen immer mit |
|
@stefan123t ist das tatsächlich so, dass die Frames stets mit 7e beginnen und mit 7f enden? Ist sichergestellt, dass ein 7e innerhalb eines Pakets nicht auftritt ;-) Das Script kann ich gerne anpassen und demnächst einen neuen Trace (wegen Power Distribution Logic) ziehen. |
|
Hallo Bernhard @broth-itk ja die 7e/7f werden durch 7d5e/7d5f escaped, 7d selbst durch 7d5d. Ich habe es hier detailliert beschrieben: #1421 (comment) |
|
Danke @stefan123t, habe das Script angepasst und werde das morgen tagsüber ausprobieren. |
|
@broth-itk Danke die Traces passen prinzipiell schon so wie sie jetzt sind. |
|
Ich habe das Script so erweitert, dass nun alles in einer Datei (Timestamp, Hex-Dump), nach Frames getrennt und mit Prefix RX/TX versehen ist. |
|
So sieht das nun aus, denke damit kann man etwas besser arbeiten:
Die Frames werden einzeln aus dem Datenstrom herausgesucht und ins Log geschrieben. Ich lasse die DTU morgen mal den ganzen Tag online und beobachte. |
|
@broth-itk ja sieht gut aus die Richtung passt. Z.B. das MainCmd 0x15 REQ_ARW_DAT_ALL mit dem SubCmd RealTimeRunData_Debug [sic] 0x0B ist die Anfrage nach den aktuellen Werten. Die geht von der DTU zum WR >>. Die Antwort wäre dann 0x95 (immer MainCmd | 0x80) vom WR zur DTU <<. Die DTU stellt aber nebenher auch ganz viele 0x56 Anfragen und bekommt 0xD6 Antworten zurück. Weiß jemand zufällig wie das 0x56 genau aufgebaut ist ? Siehe auch hier die Dokumentation im Source Code: OpenDTU/lib/Hoymiles/src/commands/ChannelChangeCommand.cpp Lines 9 to 18 in dc5eb96 Die Diskussion zum ChannelChangeCommand sollten wir m.E. hier weiterführen: #2137 (comment) |
|
Anbei ein großer Trace von heute morgen.
|
|
@stefan123t Update nachdem ich beide Geräte einmal vom Strom getrennt habe und sie wieder angeschlossen habe hat es nach einer knappen halben Stunde wieder funktioniert sie haben sich synchronisiert und jetzt läuft es wieder |
|
@AlexJacu bitte ein eigenes Issue aufmachen oder Deinen Kommentar löschen. Hier geht es nur um die Implementierung / Dokumentation des Sniffer Modus. Danke. |
Ich schaue mir gerade alles an um zu verstehen wie die Daten fließen. Danke. |
|
@DanielR92 schau mal hier sind die beiden Firmware Updates von @broth-itk Die sind für die beiden Inverter mit den Serial IDs Er hatte hier und weiter oben in diesem issue seine Inverter genauer beschrieben:
|
|
Ach ja und die letzten Logs habe ich mir noch nicht genauer angesehen. Das müsste mW die Version mit der „Power Distribution Logic“ aus #2362 sein |
Is your feature request related to a problem? Please describe.
OpenDTU team want captures of firmware upgrades of the inverters.
Describe the solution you'd like
Switch OpenDTU in "sniffer" mode to listen, receive and store all communication from Hoymiles DTU to the inverters.
The captured data should be send over the network to syslog server for further analysis.
Wishlist:
Once we have (unencrypted?) firmware dumps of the inverters, we can start disassembling ;-)
Comments are welcome. Thanks!
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: