Serial2MqttGateway

The Serial2MqttGateway functions as a gateway between MQTT and serial devices, which can each be adressed and interfered with via unique topics based on the device's respective device ID.
The application is based on the SerialPortGateway.

Features:

• All of SerialPortGateway's features:
  • Auto discovery of new serial devices
  • Processing of incoming messages from the devices
    • Easy to use message format for communicating with serial devices
  • Sending messages/commands to specific devices or all devices at once
  • Automatic handling of IO Errors
  • Logging of all actions
  • Designed as a base class for other applications to be build upon
    • Callbacks for easily handling different events (New messages, devices added/deleted )
    • Easy to use interface for interfering with the gateway
  • Parallelization of all time critical or potentially blocking tasks
  • Quick configuration of the gateway via easy config files
  • Can be used either dockerized or standalone
• Communication via MQTT with the serial devices
  • Messages coming from the devices are send to an MQTT broker.
    • Incoming messages are organized hierarchically in topics, based on their device ID and message type.
  • Commands can be delivered over MQTT to each device connected.
• Gateway-Commands, which can control the Serial2MqttGateway over MQTT.

Table of Contens

1. Files and Folder structure
2. Dependencies
3. Installation
4. Including and compiling Serial2MqttGateway in a project
5. Code Documentation
6. Requirements for communicating between the Serial2MqttGateway and serial devices
7. MQTT topic hierarchy
8. Usage
   1. Configuration files
      1. Main Config file
      2. Hardware ID Whitelist
      3. Serial Port Blacklist
   2. Starting the application in a Docker container
   3. Communicating with the gateway
   4. Communicating with the devices
9. Notes
10. License

Files and Folder structure

• config contains the configuration files
  • Sample Config file
  • Sample hardware ID whitelist
  • Sample serial port blacklist
  • cert contains a sample cert (from Let's Encrypt, in case you have a service which uses a cert issued by them)
• dependencies is the place where all dependencies get downloaded to (See Installation for further details)
• src contains the source code
  • Serial2MqttGateway class
  • serial2mqtt-gateway main application runtime
• .env is an environment file for Docker
• .gitmodules contains references to the dependencies
• build.sh is a script for building the application
• clearMqttTopics.sh is a script for clearing all retained MQTT topics
• docker-compose.yml is a docker-compose for easy deployment
• Dockerfile is the Dockerfile which can be used for building an image with all needed dependencies
• Makefile is for compiling the application

Dependencies

If you follow the Installation instructions, some dependencies will be cloned directly into the dependencies folder:

• SerialPortGateway
  • Exception
  • Logger
  • Config

In case you're going to use the Docker image, you're good and don't require anything other than Docker in order to deploy/compile/test the Serial2MqttGateway or any enhancements done by yourself.

In case you want to use the Serial2MqttGateway without Docker, you need the following few things installed:

• gcc, g++, libc-dev, make, curl, wget, git.
  • g++ must be version 5 or higher.
  • git must be version 2 or higher.
• wjwwood's serial library. This also requires:
  • cmake, python, python-pip, catkin
• paho.mqtt.c
  • must be version 1.3.9 or higher
• paho.mqtt.cpp
  • must be version 1.2.0 or higher

If there's anything not working, try to sync your configuration with that from the Dockerfile (Or drop me an issue if nothing helps).

Installation

1. Clone the repo
2. Change the directory into the repo
3. Execute git submodule update --init --recursive --remote to download all dependencies needed for using/compiling/executing the gateway
4. Building:
   1. Build the Docker image: make buildDockerImage or docker-compose build
   2. Build the application without Docker: ./build.sh

Including and compiling Serial2MqttGateway in a project

C++14 is required for compilation.

In order to use Serial2MqttGateway, <path>/Serial2MqttGateway/src/Serial2MqttGateway.hpp needs to be included.
In order to compile Serial2MqttGateway, following files need to be compiled and linked:

• <path>/Serial2MqttGateway/dependencies/SerialPortGateway/dependencies/Logger/src/Logger.cpp
• <path>/Serial2MqttGateway/dependencies/SerialPortGateway/dependencies/Logger/src/LogType.cpp
• <path>/Serial2MqttGateway/dependencies/SerialPortGateway/dependencies/Exception/src/Exception.cpp
• <path>/Serial2MqttGateway/dependencies/SerialPortGateway/dependencies/Config/src/ConfigExceptions.cpp
• <path>/Serial2MqttGateway/dependencies/SerialPortGateway/dependencies/Config/src/Config.cpp
• <path>/Serial2MqttGateway/dependencies/SerialPortGateway/src/SerialDevice.cpp
• <path>/Serial2MqttGateway/dependencies/SerialPortGateway/src/SerialMessage.cpp
• <path>/Serial2MqttGateway/dependencies/SerialPortGateway/src/SerialPortGateway.cpp

(Take a look at the Makefile.)

Code Documentation

Besides this document, you can find detailed documentation in the SerialPortGateway.
The sources are also documented where necessary/useful, but everything that's not documented in the SerialPortGateway should be pretty self explanatory.

Requirements for communicating between the Serial2MqttGateway and serial devices

• Messages from devices need to follow the SerialMessageFormat.
• The command declared via the config key COMMAND_GETID needs to be understood by every device connected and result in a validly formatted message from the device to the gateway directly after. This message needs to correspond to the message type declared with the config key MESSAGE_TYPE_ID and needs to contain the device's ID.
• The message delimiter declared with the config key MESSAGE_DELIMITER needs to be in sync with the message delimiter the connected devices are using.
• The baud rate declared with the config key BAUD_RATE needs also to be used by every device connected for sending and receiving data.

MQTT topic hierarchy

In order to communicate with the MQTT broker in a consistent way, a topic hierarchy had to be defined. This hierarchy allows to access a fixed structure through which all necessary information can be exchanged.
In the following examples and tables there's also defined whether the topic is/must be retained, and on which QoS-Level information should be exchanged.
This is crucial for the gateway working correctly.

The hierarchy generally starts with the gateway ID, and can optionally be supplemented by a prefix (The gateway ID will only be preceeded by a slash (/) if a prefix is given):
<optionalPrefix>/<gatewayId>

The gateway ID is then followed by a subtopic: <optionalPrefix>/<gatewayId>/<subtopic>

Those subtopics can be one of the following:

Topic	Purpose	QoS-Level	Retained?
connected	Shows whether the gateway is connected or not (0 or 1)	2	true
devices	Contains all device information & (sub-)topics	2	true
command	Endpoint on which the gateway listens on commands Messages send to this endpoint should NOT be retained, to prevent unwanted execution of commands	2	false
response	Topic where possible responses get send to after a gateway command has been invoked	2	true
devicelist	Topic where the device list gets send to, if requested via gateway command	2	true

In the case of the devices topic, the topic has a subtopic for each device ID which is or was connected:
<optionalPrefix>/<gatewayId>/devices/<deviceId>

And then again, this topic has a subtopic for each message type the specific device has send a message with:
<optionalPrefix>/<gatewayId>/devices/<deviceId>/<messageType>

Apart from those subtopics based on message types, there are two special subtopics a device can have:

Topic	Purpose	QoS-Level	Retained?
connected	Shows whether the device is connected or not (0 or 1)	2	true
command	Endpoint on which the gateway listens on commands for serial devices Messages send to this endpoint should NOT be retained, to prevent unwanted execution of commands.	2	false

These two subtopics should never be used as a message type by any of the connected devices.

Every topic can be listened on, in order to -for instance- log data to a database or something similar.
This could be easily done with a wildcard subscription: <optionalPrefix>/<gatewayId>/#
Refer to MQTT's documentation in order to get more information on how to subscribe to topics in different ways.

A full topic hierarchy of a gateway could look like this:

gateways/
  |--->gateway1/
  |      |--->connected
  |      |--->command
  |      |--->response
  |      |--->devicelist
  |      \--->devices/
  |             |--->device1/
  |             |      |--->connected
  |             |      |--->command
  |             |      |--->id
  |             |      |--->active
  |             |      |--->info
  |             |      \--->status
  |             \--->device2/
  |                    |--->connected
  |                    |--->command
  |                    |--->id
  |                    |--->active
  |                    |--->info
  |                    \--->status
  |--->gateway2/
  ...

Usage

Configuration files

The Serial2MqttGateway can be easily configured with three different config files.

Main Config file

This is the basic config of the gateway. It uses the Config class.
On top of the mandatory configuration keys defined here, the gateway needs following additional keys defined:

Key	Purpose	Value Description	Default
GATEWAY_ID	- ID of the gateway - Needed to use and distinguish multiple gateways on the same topic level from one another - Is also required to be able to receive gateway commands	String	testgateway1
MQTT_PROTOCOL	Specifies the protocol to use for communication with a broker	String - tcp for unencrypted MQTT/TCP communication - ssl for encrypted MQTT/TCP communication - ws for unencrypted websocket communication - wss for encrypted websocket communication	tcp
MQTT_CERTIFICATE_AUTHORITY_PATH	Specifies the location of all trusted CA files, in case you want to use encrypted communication for MQTT	String - none if no CA trust store is used - <path/to/ca/files> if a CA trust store is used	/etc/ssl/certs/
MQTT_SERVER_CERTIFICATE_FILE	Configuration on whether to use a server certificate or not, in case you want to use encrypted communication for MQTT	String - none if no server certificate is used - <path/to/serverCertificateFile> if a server certificate is used	none
MQTT_CLIENT_CERTIFICATE_FILE	Configuration on whether to use a client certificate or not, in case you want to use encrypted communication for MQTT	String - none if no client certificate is used - <path/to/clientCertificateFile> if a client certificate is used	none
MQTT_CLIENT_KEY_FILE	Configuration on whether to use a client certificate or not, in case you want to use encrypted communication for MQTT	String - none if no client certificate is used - <path/to/clientKeyFile> if a client certificate is used	none
MQTT_HOST	MQTT host, where the broker is running	String Hostname or IP	test.mosquitto.org
MQTT_PORT	MQTT port, where the broker is listening on	Port number	1883
MQTT_WAIT_UNTIL_RECONNECT	Time to wait until the next reconnect attempt in ms	Integer	1000
MQTT_USERNAME	MQTT username to use when logging in	String Can be empty
MQTT_PASSWORD	MQTT password to use when logging in	String Can be empty
MQTT_TOPIC_PREFIX	MQTT topic prefix which should be used Is set before every topic the Serial2MqttGateway is publishing on	String Can be empty	test/gateways

Hardware ID Whitelist

The hardware ID whitelist lists all allowed hardware IDs;

• If the list is empty, no whitelist-checks will be performed.
• If the list has one or more entries, only devices matching one of those hardware IDs can/will be added to the gateway.

The hardware IDs are formatted in the following way: <VendorID>:<ProductID>
The hardware IDs are separated from each other by a line break. A hardware ID whitelist can look like the following:

1a86:7523
0403:6001
2341:0042

These IDs can be retrieved with Linux tools such as "lsusb" or "udevadm".

Serial Port Blacklist

The serial port blacklist lists all forbidden serial port paths that should be ignored when scanning for or adding new devices. The port paths are separated from each other by a line break. A port blacklist can look like the following:

/dev/ttyS0
/dev/ttyS1
/dev/ttyS2
/dev/ttyS3

Starting the application in a Docker container

• The container needs to run in priviledged mode in order to gain access to the serial ports.
• In case you want to use the container just as your development environment, do one of the following things:
  • docker-compose up -d devenv; docker attach serial2mqttgateway_daemon_<number>;
  • docker run --privileged -it serial2mqttgateway /bin/bash
• For starting the application in a Docker container, simply use the docker-compose.yml-File included;
  Run docker-compose up -d daemon if you want to have the serial2mqtt-gateway run in the background.\
• In case you didn't build the image already, docker-compose will do this automatically for you.

Take a look at the Notes for some restrictions when using Docker as container runtime in conjunction with serial ports.

Communicating with the gateway

In order to send commands to the gateway, you must send a message with the corresponding content to this topic:
<optionalPrefix>/<gatewayId>/command

If the command requires an argument, the argument needs to be separated by a space character:
<command> <argument>

There are following commands currently implemented:

Command	Purpose	Arguments
devicelist	Publishs a list of all connected devices on <optionalPrefix>/<gatewayId>/devicelist
serialportlist	Publishs a list of the system's available serial ports on <optionalPrefix>/<gatewayId>/response
mappinglist	Publishs a list of all mappings between connected devices by their ID & their corresponding serial port on <optionalPrefix>/<gatewayId>/response
broadcast	Broadcasts a message/command to all connected serial devices	<message>
add	Tries to add a serial device on a specific serial port	<path/to/serial/port>
addnew	Tries to add all new serial devices, which are not already registered/added in the gateway
delete	Tries to delete a serial device with a specific device ID	<deviceId>
deleteall	Tries to delete all currently registered serial devices from the gateway

Communicating with the devices

In order to send commands to a specific device, you must send a message with the corresponding content to this topic:
<optionalPrefix>/<gatewayId>/devices/<deviceId>/command

Notes

• In case you're using Arduinos with the Serial2MqttGateway, you can use the ArduinoStreamCommander.
  But this doesn't mean you have to, as long as you follow the requirements for communication.
• Docker can -without doing anything else- only provide devices or ports to the container which are already connected when the container is started. In this case this means that all serial devices to be connected must be initially available at least at the start of the container, so that the port numbers are registered in the container. Dis- and reconnects that happen afterwards -and thus the exchange of devices to other ports- are unproblematic, but to register additional devices a reboot is required.
  Meaning: If more ports should be used than when the container was initially started, the container must be restarted.

License

   Copyright 2019-2022 Jan-Eric Schober

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.