PSM Data Feed Support, DockerHub Docs, Commsignia Log Offloading Support

[dmccoystephenson]

PR Details

Description

PSM Data Feed Support

This PR introduces support for PSM message processing to the ODE. Support for both required and optional fields is present, however a sample message that covers any of the optional fields is not available for testing. All required fields have been tested with sample PSM messages generated from the NJDOT deployment.

This pipeline takes in PSM messages on the new UDP port (44940) and outputs processed JSON PSM messages on the topic: topic.OdePsmJson. As with other data feeds, multiple other topics are used for internal processing/management: topic.OdeRawEncodedPSMJson and topic.OdePsmTxPojo.

DockerHub Documentation

Documentation for using the image has been added to docs/dockerhub.md.

Commsignia Log Offloading Support

This PR adds support for processing log offloading from Commsignia devices. The primary change here is supporting the new line characters that are included in these log messages, as well as parsing off the WSMP message header in the payload.

Additional Changes

• Deprecated some test messages
• Added new logging variables for PPM/ACM to docker-compose.yml & sample.env files

Related Issue

Commsignia Log Offloading Support

To process offloaded logs in both CDOT/WYDOT deployment. In preparation of the anticipated firmware release that supports log offloading.

Motivation and Context

PSM Data Feed Support

Required for processing Personal Safety Messages through the ODE.

How Has This Been Tested?

PSM Data Feed Support

The convention from previous data feeds was followed for unit testing and verified message processing with sample messages from the NJDOT deployment.

Commsignia Log Offloading Support

Added new unit tests to verify log processing of old messages (without newline characters).
Tested the log offloading of all .gz example log files in the data directory. To run this test, run the ODE through docker, at the minimum run the zookeeper, kafka, ODE, AEM, and ADM services. Copy all of the .gz files from the jpo-ode/data folder into the jpo-ode/uploads/bsmlog folder. This folder is monitored by the ODE and the files are ingested into the appropriate topics. If the processing succeeds the files are forwarded to the jpo-ode/uploads/backup folder and if they fail it goes to the jpo-ode/uploads/failed directory.

Types of changes

• Defect fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that cause existing functionality to change)
• New Documentation

Checklist:

• I have added any new packages to the sonar-scanner.properties file
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
  ODE Contributing Guide
• I have added tests to cover my changes.
• All new and existing tests passed.

[dan-du-car]

git clone --recurse-submodules https://github.com/CDOT-CV/jpo-ode
cd jpo-ode
mvn test

[dmccoystephenson]

git clone --recurse-submodules https://github.com/CDOT-CV/jpo-ode
cd jpo-ode
mvn test

The proposed changes are on the 'dev' branch. I recommend using the following commands to test:

git clone --recurse-submodules https://github.com/CDOT-CV/jpo-ode
cd jpo-ode
git checkout origin/dev
mvn test

[dan-du-car]

git clone --recurse-submodules https://github.com/CDOT-CV/jpo-ode
cd jpo-ode
mvn test

The proposed changes are on the 'dev' branch. I recommend using the following commands to test:

git clone --recurse-submodules https://github.com/CDOT-CV/jpo-ode
cd jpo-ode
git checkout origin/dev
mvn test

Yes, it was in dev.

[dmccoystephenson]

What output do you get when running mvn --version? @dan-du-car

Do you experience the same behavior when using the dev container for the project?

@dmccoystephenson The dev container works. Thanks!
I was running it in host machine ubuntu 22.04 Jammy.

mvn --version
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /home/dev/Downloads/apache-maven-3.9.6
Java version: 21.0.2, vendor: Oracle Corporation, runtime: /usr/lib/jvm/jdk-21-oracle-x64

[Michael7371]

git clone --recurse-submodules https://github.com/CDOT-CV/jpo-ode
cd jpo-ode
mvn test

The proposed changes are on the 'dev' branch. I recommend using the following commands to test:

git clone --recurse-submodules https://github.com/CDOT-CV/jpo-ode
cd jpo-ode
git checkout origin/dev
mvn test

Yes, it was in dev.

Have you followed the "Downloading the source code from a non-default branch" section of the root readme? It may help with getting the submodule references setup the same way as the dev branch does.

[dmccoystephenson]

This PR is ready for re-review.

[dan-du-car]

Looks good. Thank you!