Firmware Manager Upgrade Scheduler and Runner

docker-compose-addons.yml

@@ -123,11 +123,11 @@ services:
         max-size: '10m'
         max-file: '5'
 
-  firmware_manager:
+  firmware_manager_upgrade_scheduler:
     build:
       context: services
-      dockerfile: Dockerfile.firmware_manager
-    image: jpo_firmware_manager:latest
+      dockerfile: Dockerfile.fmus
+    image: jpo_firmware_manager_upgrade_scheduler:latest
     restart: on-failure:3
 
     ports:
@@ -138,6 +138,27 @@ services:
       PG_DB_USER: ${PG_DB_USER}
       PG_DB_PASS: ${PG_DB_PASS}
 
+      UPGRADE_RUNNER_ENDPOINT: ${FIRMWARE_MANAGER_UPGRADE_RUNNER_ENDPOINT}
+
+      LOGGING_LEVEL: ${FIRMWARE_MANAGER_LOGGING_LEVEL}
+    volumes:
+      - ${GOOGLE_APPLICATION_CREDENTIALS}:/google/gcp_credentials.json
+      - ${HOST_BLOB_STORAGE_DIRECTORY}:/mnt/blob_storage
+    logging:
+      options:
+        max-size: '10m'
+        max-file: '5'
+
+  firmware_manager_upgrade_runner:
+    build:
+      context: services
+      dockerfile: Dockerfile.fmur
+    image: jpo_firmware_manager_upgrade_runner:latest
+    restart: on-failure:3
+
+    ports:
+      - '8090:8080'
+    environment:
       BLOB_STORAGE_PROVIDER: ${BLOB_STORAGE_PROVIDER}
       BLOB_STORAGE_BUCKET: ${BLOB_STORAGE_BUCKET}
 

sample.env

@@ -8,6 +8,7 @@ KC_HOST_IP=${DOCKER_HOST_IP}
 
 # Firmware Manager connectivity in the format 'http://endpoint:port'
 FIRMWARE_MANAGER_ENDPOINT=http://${DOCKER_HOST_IP}:8089
+FIRMWARE_MANAGER_UPGRADE_RUNNER_ENDPOINT=http://${DOCKER_HOST_IP}:8090
 
 # Allowed CORS domain for accessing the CV Manager API from (set to the web application hostname)
 # Make sure to include http:// or https://

services/Dockerfile.firmware_manager → services/Dockerfile.fmur

@@ -4,13 +4,13 @@ WORKDIR /home
 
 ADD addons/images/firmware_manager/requirements.txt .
 ADD addons/images/firmware_manager/resources/xfer_yunex.jar ./tools/
-ADD addons/images/firmware_manager/*.py .
+ADD addons/images/firmware_manager/upgrade_runner/*.py .
 ADD common/*.py ./common/
 
 RUN pip3 install -r requirements.txt
 RUN apt-get update
 RUN apt-get install -y default-jdk
 RUN apt-get install -y iputils-ping
 
-CMD ["/home/firmware_manager.py"]
+CMD ["/home/upgrade_runner.py"]
 ENTRYPOINT ["python3"]

services/Dockerfile.fmus

@@ -0,0 +1,13 @@
+FROM python:3.12.2-slim
+
+WORKDIR /home
+
+ADD addons/images/firmware_manager/requirements.txt .
+ADD addons/images/firmware_manager/upgrade_scheduler/*.py .
+ADD common/*.py ./common/
+
+RUN pip3 install -r requirements.txt
+RUN apt-get update
+
+CMD ["/home/upgrade_scheduler.py"]
+ENTRYPOINT ["python3"]

services/addons/images/firmware_manager/README.md

@@ -12,20 +12,20 @@
 
 ## About <a name = "about"></a>
 
-This directory contains a microservice that runs within the CV Manager GKE Cluster. The firmware manager monitors the CV Manager PostgreSQL database to determine if there are any RSUs that are targeted for a firmware upgrade. This monitoring is a once-per-hour, scheduled occurrence. Alternatively, this micro-service hosts a REST API for directly initiating firmware upgrades - this is used by the CV Manager API. Firmware upgrades are then run in parallel and tracked until completion.
+This directory contains two microservices that run within the CV Manager GKE Cluster. The firmware manager upgrade scheduler monitors the CV Manager PostgreSQL database to determine if there are any RSUs that are targeted for a firmware upgrade. This monitoring is a once-per-hour, scheduled occurrence. Alternatively, this micro-service hosts a REST API for directly initiating firmware upgrades - this is used by the CV Manager API. Firmware upgrades then schedule off tasks to the firmware manager upgrade runner that is initiated through an HTTP request. This allows for better scaling for more parallel upgrades.
 
 An RSU is determined to be ready for upgrade if its entry in the "rsus" table in PostgreSQL has its "target_firmware_version" set to be different than its "firmware_version". The Firmware Manager will ignore all devices with incompatible firmware upgrades set as their target firmware based on the "firmware_upgrade_rules" table. The CV Manager API will only offer CV Manager webapp users compatible options so this generally is a precaution.
 
 Hosting firmware files is recommended to be done via the cloud. GCP cloud storage is the currently supported method, but a directory mounted as a docker volume can also be used. Alternative cloud support can be added via the [download_blob.py](download_blob.py) script. Firmware storage must be organized by: `vendor/rsu-model/firmware-version/install_package`.
 
-Firmware upgrades have unique procedures based on RSU vendor/manufacturer. To avoid requiring a unique bash script for every single firmware upgrade, the Firmware Manager has been written to use vendor based upgrade scripts that have been thoroughly tested. An interface-like abstract class, [base_upgrader.py](base_upgrader.py), has been made for helping create upgrade scripts for vendors not yet supported. The Firmware Manager selects the script to use based off the RSU's "model" column in the "rsus" table. These scripts report back to the Firmware Manager on completion with a status of whether the upgrade was a success or failure. Regardless, the Firmware Manager will remove the process from its tracking and update the PostgreSQL database accordingly.
+Firmware upgrades have unique procedures based on RSU vendor/manufacturer. To avoid requiring a unique bash script for every single firmware upgrade, the firmware manager upgrade runner has been written to use vendor based upgrade scripts that have been thoroughly tested. An interface-like abstract class, [base_upgrader.py](base_upgrader.py), has been made for helping create upgrade scripts for vendors not yet supported. The firmware manager upgrade runner selects the script to use based off the RSU's "model" column in the "rsus" table. These scripts report back to the firmware manager upgrade scheduler on completion with a status of whether the upgrade was a success or failure. Regardless, the Firmware Manager will remove the process from its tracking and update the PostgreSQL database accordingly.
 
 List of currently supported vendors:
 
 - Commsignia
 - Yunex
 
-Available REST endpoints:
+Available Firmware Manager Upgrade Scheduler REST endpoints:
 
 - /init_firmware_upgrade [ **POST** ] `{ "rsu_ip": "" }`
   - `rsu_ip` is the target RSU being upgraded (The target firmware is separately updated in PostgreSQL, this is just to get the Firmware Manager to immediately go look)
@@ -36,6 +36,10 @@ Available REST endpoints:
   - Used to list all active upgrades in the form:
     `{"active_upgrades": {"1.1.1.1": {"manufacturer": "Commsignia", "model": "ITS-RS4-M", "target_firmware_id": 2, "target_firmware_version": "y20.39.0", "install_package": "blob.blob"}}}`
 
+Available Firmware Manager Upgrade Runner REST endpoints:
+
+- /run_firmware_upgrade [ **POST** ] `{ "ipv4_address": "", "manufacturer": "", "model": "", "ssh_username": "", "ssh_password": "","target_firmware_id": "",  "target_firmware_version": "", "install_package": ""}`
+
 ## Requirements <a name = "requirements"></a>
 
 To properly run the firmware_manager microservice the following services are also required:

services/addons/images/firmware_manager/requirements.txt

@@ -1,7 +1,8 @@
 APScheduler==3.10.4
 google-cloud-storage==2.14.0
 flask==3.0.0
-paramiko==3.3.1
+marshmallow==3.20.1
+paramiko==3.5.0
 pg8000==1.30.2
 requests==2.31.0
 scp==0.14.5