Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code Cleanup, Submodule Replacement, Error Handling Improvements & Deployment Enhancements #555

Merged
merged 70 commits into from
Sep 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
0a994a8
Remove stale websocket code
mwodahl May 29, 2024
1fcb7a5
Testing update
mwodahl May 29, 2024
d5a7353
Updated TimDeleteController to interpret snmp error code 10 more accu…
dmccoystephenson Jun 4, 2024
19adfb4
Updated RsuDepositor to interpret snmp error code 10 separately from …
dmccoystephenson Jun 4, 2024
ffd6066
Merge pull request #86 from CDOT-CV/snmp/interpret-error-code-10
payneBrandon Jun 13, 2024
b4e6ee0
Merge branch 'dev' into Update/remove-websocket-code
mwodahl Jun 13, 2024
041d22d
Expanded project reference update documentation in `release_process.md`
dmccoystephenson Jun 14, 2024
ce5e57d
Updated formatting in section 3 of `release_process.md` file
dmccoystephenson Jun 14, 2024
b2209b8
Added steps to update versions of each project rather than just their…
dmccoystephenson Jun 14, 2024
02cbb8e
Removed duplicate instruction to update jpo-ode pom.xml versions in `…
dmccoystephenson Jun 14, 2024
16e2129
Updated mapfile reference in ppm*.properties files
dmccoystephenson Jun 18, 2024
c834498
Merge branch 'usdot-jpo-ode:develop' into dev
dmccoystephenson Jun 19, 2024
d5f2916
Printed stack trace upon receiving bad encoded data from ACM only if …
dmccoystephenson Jun 19, 2024
92b00cd
Add a startup script to the Dockerfile entry point to support a SSH/S…
drewjj Jun 21, 2024
d25b957
Removed 'ppm_tim' service from docker compose files
dmccoystephenson Jun 24, 2024
471d98a
Removed `ppmTim.properties` file
dmccoystephenson Jun 24, 2024
87c8ff0
Removed `rde_tim_depositor` service from docker compose files
dmccoystephenson Jun 24, 2024
2940ea6
Updated TIM data flow diagram to reflect removal of `ppm_tim` service
dmccoystephenson Jun 25, 2024
7f60880
Added .gitignore for data-flow-diagrams directory
dmccoystephenson Jun 25, 2024
09c2fd3
Added `ppm_data/*` to root .gitignore
dmccoystephenson Jun 25, 2024
5c102cd
Merge pull request #83 from CDOT-CV/Update/remove-websocket-code
payneBrandon Jul 8, 2024
3e957b5
Merge pull request #91 from CDOT-CV/errors/handle-bad-data-from-acm
payneBrandon Jul 8, 2024
ff5bc02
Merge pull request #93 from CDOT-CV/docker/remove-ppm-tim-service
payneBrandon Jul 8, 2024
febaa8e
Merge branch 'dev' into config/update-mapfile-reference-for-ppm
dmccoystephenson Jul 10, 2024
54f8a7e
Update .github/workflows/dockerhub.yml
dmccoystephenson Jul 10, 2024
5ee2239
Merge pull request #90 from CDOT-CV/config/update-mapfile-reference-f…
payneBrandon Jul 11, 2024
88251dd
Added documentation on when to use the SSH server and how to.
drewjj Jul 12, 2024
3d4f103
Moved the script to the scripts directory
drewjj Jul 12, 2024
6917a0f
test artifact publishing
Michael7371 Jul 15, 2024
d26a23e
fixed the upload endpoint
Michael7371 Jul 15, 2024
531172b
changing trigger to only run "publish java package" on tag creation
Michael7371 Jul 15, 2024
76c99dd
Merge pull request #92 from CDOT-CV/startup_script
drewjj Jul 18, 2024
92e7aec
adding documentation for github artifacts usage
Michael7371 Jul 29, 2024
12097fe
Merge pull request #89 from CDOT-CV/docs/expand-project-reference-upd…
payneBrandon Jul 30, 2024
9fc1a30
Changed USER_UID argument in dev container dockerfile to fix build fa…
dmccoystephenson Jul 31, 2024
74f3b2e
fixes to readme
Michael7371 Aug 2, 2024
1a10412
Merge pull request #95 from CDOT-CV/fix/fix-dev-container-build-failure
drewjj Aug 2, 2024
240b1f0
initial commit
Michael7371 Aug 12, 2024
82f27a8
adding javadoc of strip method and a unit test
Michael7371 Aug 12, 2024
2373765
fixing documentation with updated schema
Michael7371 Aug 12, 2024
ab87864
fix failed unit tests
Michael7371 Aug 13, 2024
e712156
Update UperUtilTest.java
Michael7371 Aug 13, 2024
f643e45
Merge pull request #94 from CDOT-CV/github-actions-artifact-publishing
payneBrandon Aug 14, 2024
efa2385
updates to major version, schemas, and unit tests
Michael7371 Aug 14, 2024
9647040
update schema files with new version number and updated asn1 field to…
Michael7371 Aug 15, 2024
39beed6
changed asn1 field to be nullable
Michael7371 Aug 16, 2024
75b469c
Merge pull request #96 from CDOT-CV/asn1-preservation
payneBrandon Aug 16, 2024
f550a69
removal of jpo-s3-deposit and addition of jpo-utils repo
Michael7371 Aug 28, 2024
f2d1da6
updates to use jpo-utils repo
Michael7371 Aug 28, 2024
30658df
updates to use separate env files for ode and utils repo.
Michael7371 Aug 28, 2024
00be9ef
removing hard coded jpo-utils profiles
Michael7371 Aug 28, 2024
aa218c6
removing jpo-s3 from sonar-scan and document references
Michael7371 Aug 28, 2024
fbfa3e8
fixe make build not having env files as well as adding jpo-utils to u…
Michael7371 Aug 29, 2024
883fa1d
adding reference to jpo-utils readme for COMPOSE_PROFILE documentation
Michael7371 Aug 29, 2024
2de9b6a
update jpo-utils ref, adding docs for git submodule issues, fixed jpo…
Michael7371 Sep 3, 2024
95a9037
adding health checks to all resources
Michael7371 Sep 3, 2024
762a714
Merge pull request #97 from CDOT-CV/s3-deposit-removal
Michael7371 Sep 6, 2024
da7107b
Added release notes for version 3.0.0
dmccoystephenson Sep 8, 2024
d3cc678
Merge pull request #98 from CDOT-CV/q3-release/release-notes
drewjj Sep 9, 2024
3037e8e
Modified default sdw subscription topic to 'topic.SDWDepositorInput'
dmccoystephenson Sep 10, 2024
7eb3c3d
Merge pull request #99 from CDOT-CV/docker/modify-default-sdw-subscri…
dmccoystephenson Sep 10, 2024
f9b5bde
fixing srm schema to allow strings and nulls for the "recordGenerated…
Michael7371 Sep 13, 2024
b1728dd
updates to "recordGeneratedBy" schema definition
Michael7371 Sep 17, 2024
2917a57
Merge pull request #101 from CDOT-CV/schema-fixes
mwodahl Sep 17, 2024
d19a562
commit to publish jar file without springboot dependencies
Michael7371 Sep 20, 2024
384911c
testing artifact-publish workflow
Michael7371 Sep 20, 2024
e458147
testing putting profile in ode-svcs instead of parent pom
Michael7371 Sep 20, 2024
3b0329b
removing test from artifact-publish workflow
Michael7371 Sep 20, 2024
a2c30c6
removing trigger on pull request
Michael7371 Sep 23, 2024
0f66b7d
Merge pull request #102 from CDOT-CV/fix/reduce-publish-jar-size
drewjj Sep 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,25 +10,25 @@ FROM eclipse-temurin:21-jdk
# will be updated to match your local UID/GID (when using the dockerFile property).
# See https://aka.ms/vscode-remote/containers/non-root-user for details.
ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_UID=1234
ARG USER_GID=$USER_UID

# Configure apt
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get -y install --no-install-recommends apt-utils dialog 2>&1 \
#
# Create a non-root user to use if preferred - see https://aka.ms/vscode-remote/containers/non-root-user.
&& groupadd --gid $USER_GID $USERNAME \
&& useradd -s /bin/bash --uid $USER_UID --gid $USER_GID -m $USERNAME \
# [Optional] Add sudo support for the non-root user
&& apt-get install -y sudo \
&& echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME\
&& chmod 0440 /etc/sudoers.d/$USERNAME \
#
# Verify git, needed tools installed
&& apt-get -y install git openssh-client less iproute2 procps curl lsb-release zip unzip sed kafkacat telnet
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get -y install --no-install-recommends apt-utils dialog 2>&1

# Create a non-root user to use if preferred - see https://aka.ms/vscode-remote/containers/non-root-user.
RUN groupadd --gid $USER_GID $USERNAME
RUN useradd -s /bin/bash --uid $USER_UID --gid $USER_GID -m $USERNAME
# [Optional] Add sudo support for the non-root user
RUN apt-get install -y sudo
RUN echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME
RUN chmod 0440 /etc/sudoers.d/$USERNAME

# Verify git, needed tools installed
RUN apt-get -y install git openssh-client less iproute2 procps curl lsb-release zip unzip sed kafkacat telnet

#-------------------Install SDKMan----------------------------------
RUN curl -s https://get.sdkman.io | bash
Expand Down
31 changes: 31 additions & 0 deletions .github/workflows/artifact-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Publish Java Package
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What will be the approx. artifact size and where we are using this ? As our ODE GitHub org is on free plan we have artifact storage limit of 500mb.

Copy link
Contributor

@iyourshaw iyourshaw Sep 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approx:
jpo-ode-common.jar - 38 KB
jpo-ode-core.jar - 154 KB
jpo-ode-plugins.jar - 436 KB
jpo-ode-svcs.jar - 134 MB (large because it includes the Spring web stuff. We could probably make it much smaller by only releasing the normal jar instead of the Spring Boot uberjar)

I'm not certain, but this seems to be saying that the storage limits for maven packages don't apply to public repos:
https://docs.github.com/en/billing/managing-billing-for-github-packages/about-billing-for-github-packages

I am fairly certain that there is no time limit on Maven packages (which are not the same as Gitub Action build artifacts)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked, and the normal jar for jpo-ode-svcs is approx 219KB, so if we changed the action to publish that, the total space would be less than 1MB for everything

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iyourshaw Got it, I'm good with this change. This storage limits are applied for only private repositories. Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SaikrishnaBairamoni Here is the relevant PR that addresses this issue. We will work on getting this update merged ASAP.


on:
push:
tags:
- 'jpo-ode-*'

jobs:
build:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up JDK 21
uses: actions/setup-java@v4
with:
java-version: '21'
distribution: 'adopt'

- name: Remove snapshot from version
run: mvn versions:set -DremoveSnapshot

- name: Build with Maven
run: mvn -B clean package --file pom.xml -DskipTests -Ppackage-jar

- name: Publish to GitHub Packages
run: mvn --batch-mode -Dgithub_organization=${{ github.repository_owner }} deploy -DskipTests -Ppackage-jar
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2 changes: 1 addition & 1 deletion .github/workflows/dockerhub.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Replcae Docker tag
- name: Replace Docker tag
id: set_tag
run: echo "TAG=$(echo ${GITHUB_REF##*/} | sed 's/\//-/g')" >> $GITHUB_ENV

Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -129,3 +129,7 @@ settings.xml
/jpo-security-svcs/*.log
/qa/test-harness/*.log
/qa/test-harness/.vscode

#########################
### jpo-cvdp ###
ppm_data/*
7 changes: 3 additions & 4 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,6 @@
[submodule "jpo-cvdp"]
path = jpo-cvdp
url = https://github.com/usdot-jpo-ode/jpo-cvdp.git
[submodule "jpo-s3-deposit"]
path = jpo-s3-deposit
url = https://github.com/usdot-jpo-ode/jpo-s3-deposit.git
branch = master
[submodule "jpo-security-svcs"]
path = jpo-security-svcs
url = https://github.com/usdot-jpo-ode/jpo-security-svcs.git
Expand All @@ -17,3 +13,6 @@
[submodule "qa/test-harness/ode-output-validator-library"]
path = qa/test-harness/ode-output-validator-library
url = https://github.com/usdot-jpo-ode/ode-output-validator-library.git
[submodule "jpo-utils"]
path = jpo-utils
url = https://github.com/usdot-jpo-ode/jpo-utils.git
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SaikrishnaBairamoni Do we need to add github actions to scan this new repo?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dan-du-car we did not setup GitHub actions for jpo-utils repo seems like this is new, I can create a story for this in our backlog. Thanks!

3 changes: 0 additions & 3 deletions .sonarqube/sonar-scanner.properties
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ sonar.modules= asn1_codec, \
jpo-ode-core, \
jpo-ode-plugins, \
jpo-ode-svcs, \
jpo-s3-deposit, \
jpo-sdw-depositor, \
jpo-security-svcs

Expand All @@ -17,7 +16,6 @@ jpo-ode-consumer-example.sonar.projectBaseDir = /home/circleci/project/jpo-ode
jpo-ode-core.sonar.projectBaseDir = /home/circleci/project/jpo-ode-core
jpo-ode-plugins.sonar.projectBaseDir = /home/circleci/project/jpo-ode-plugins
jpo-ode-svcs.sonar.projectBaseDir = /home/circleci/project/jpo-ode-svcs
jpo-s3-deposit.sonar.projectBaseDir = /home/circleci/project/jpo-s3-deposit
jpo-sdw-depositor.sonar.projectBaseDir = /home/circleci/project/jpo-sdw-depositor
jpo-security-svcs.sonar.projectBaseDir = /home/circleci/project/jpo-security-svcs

Expand All @@ -28,7 +26,6 @@ jpo-ode-consumer-example.sonar.sources = src
jpo-ode-core.sonar.sources = src
jpo-ode-plugins.sonar.sources = src
jpo-ode-svcs.sonar.sources = src
jpo-s3-deposit.sonar.sources = src
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are replacing the s3 depositor with jpo-utils, is it possible to add jpo-util to this scanner? @SaikrishnaBairamoni

Copy link
Contributor

@SaikrishnaBairamoni SaikrishnaBairamoni Sep 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can update the sonar properties as this PR changes are from CDOT fork I cannot make changes to this

jpo-sdw-depositor.sonar.sources = src
jpo-security-svcs.sonar.sources = src

Expand Down
17 changes: 6 additions & 11 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,10 @@ WORKDIR /home
COPY --from=builder /home/jpo-ode-svcs/src/main/resources/application.properties /home
COPY --from=builder /home/jpo-ode-svcs/src/main/resources/logback.xml /home
COPY --from=builder /home/jpo-ode-svcs/target/jpo-ode-svcs.jar /home
COPY ./scripts/startup_jpoode.sh /home

ENTRYPOINT ["java", \
"-Djava.rmi.server.hostname=$DOCKER_HOST_IP", \
"-Dcom.sun.management.jmxremote.port=9090", \
"-Dcom.sun.management.jmxremote.rmi.port=9090", \
"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.local.only=true", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
"-Dlogback.configurationFile=/home/logback.xml", \
"-jar", \
"/home/jpo-ode-svcs.jar"]
RUN apk add openssh
RUN apk add openrc
RUN rc-update add sshd

ENTRYPOINT ["sh", "/home/startup_jpoode.sh"]
39 changes: 22 additions & 17 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,33 +1,38 @@
default:
$(info Make target options:)
$(info `make start` to build and run the ODE)
$(info `make start` to run the ODE)
$(info `make build` to build the ODE)
$(info `make stop` to stop the ODE)
$(info `make delete` to destroy the current Docker containers)
$(info `make delete` to stop the ODE and remove the volumes)
$(info `make rebuild` to stop, delete, and then rebuild the containers)
$(info `make clean-build` to rebuild the containers without using the cache)

start:
ifeq ("$(wildcard ./jpo-ode-private)", "")
$(error "ERROR: Directory `jpo-ode-private` not found in ${PWD}")
endif
ifeq ("$(wildcard ./asn1_codec/asn1c_combined/J2735_201603DA.ASN)", "")
cp ./jpo-ode-private/j2735/J2735_201603DA.ASN ./asn1_codec/asn1c_combined/J2735_201603DA.ASN
endif
ifndef DOCKER_HOST_IP
$(error ERROR: Environment variable DOCKER_HOST_IP is not set)
ifeq ("$(wildcard .env)", "")
$(error "ERROR: jpo-ode Environment file `.env` not found in ${PWD}")
endif
ifndef DOCKER_SHARED_VOLUME
$(error ERROR: Environment variable DOCKER_SHARED_VOLUME is not set)
ifeq ("$(wildcard ./jpo-utils/.env)", "")
$(error "ERROR: jpo-utils Environment file `.env` not found in ${PWD}")
endif
docker compose up -d

build:
ifeq ("$(wildcard .env)", "")
$(warning "WARNING: Environment file `.env` not found in ${PWD}")
$(error "ERROR: jpo-ode Environment file `.env` not found in ${PWD}")
endif
ifeq ("$(wildcard ./jpo-utils/.env)", "")
$(error "ERROR: jpo-utils Environment file `.env` not found in ${PWD}")
endif
docker-compose up --build -d
docker compose build

stop:
docker-compose down
docker compose down

delete:
docker-compose rm -fvs
docker compose down -v

rebuild:
$(MAKE) stop delete start
$(MAKE) stop delete build start

clean-build:
docker compose build --no-cache
Loading
Loading