Configurable Kafka Properties, Dev Container Java Update & GCS BLOB Support #59
Conversation
Update Hardcoded Kafka Properties to be Configurable
Update Dev Container Java Version
dmccoystephenson
changed the title
dmccoystephenson
changed the title
…NAPSHOT Changed version to 1.5.0-SNAPSHOT
Release notes for jpo-s3-deposit 1.5.0
| KAFKA_ENABLE_AUTO_COMMIT: ${KAFKA_ENABLE_AUTO_COMMIT} | ||
| KAFKA_AUTO_COMMIT_INTERVAL_MS: ${KAFKA_AUTO_COMMIT_INTERVAL_MS} | ||
| KAFKA_SESSION_TIMEOUT_MS: ${KAFKA_SESSION_TIMEOUT_MS} | ||
| GOOGLE_APPLICATION_CREDENTIALS: '/google/gcp_service_account.json' |
There was a problem hiding this comment.
Is this google credential value: ${GOOGLE_APPLICATION_CREDENTIALS}
There was a problem hiding this comment.
Yes. The value of the environment variable is mounted to /google/gcp_service_account.json and then that file is referenced in the application as GOOGLE_APPLICATION_CREDENTIALS.
| KAFKA_ENABLE_AUTO_COMMIT: ${KAFKA_ENABLE_AUTO_COMMIT} | ||
| KAFKA_AUTO_COMMIT_INTERVAL_MS: ${KAFKA_AUTO_COMMIT_INTERVAL_MS} | ||
| KAFKA_SESSION_TIMEOUT_MS: ${KAFKA_SESSION_TIMEOUT_MS} | ||
| GOOGLE_APPLICATION_CREDENTIALS: '/google/gcp_service_account.json' |
There was a problem hiding this comment.
The value of the environment variable is mounted to /google/gcp_service_account.json and then that file is referenced in the application as GOOGLE_APPLICATION_CREDENTIALS.
| depositToGCS(gcsStorage, bucketName, record); | ||
| } else { | ||
| logger.error("Invalid destination: " + destination); | ||
| System.exit(1); | ||
| } | ||
| } catch (Exception e) { | ||
| int retryTimeout = 5000; |
There was a problem hiding this comment.
logger.error("Error depositing to AWS. Retrying in " + retryTimeout / 1000 + " seconds",
e);
Maybe include depositing to GCS besides AWS in the error message?
There was a problem hiding this comment.
This log statement now references the specified destination rather than 'AWS' being hardcoded.
|
@dmccoystephenson It looks good. I have some minor comments. |
| "CONFLUENT_KEY": "<confluent key>", | ||
| "CONFLUENT_SECRET": "<confluent secret>" | ||
| } | ||
| "envFile": "${workspaceFolder}/.env", |
There was a problem hiding this comment.
A sample.env file is provided. In the README the user is instructed to copy this file and rename the copy to .env
|
I've marked this PR as a draft until CDOT-CV#24 is merged. |
…-2024 Addressing USDOT PR Comments 6/11/2024
|
This PR is ready for re-review @dan-du-car |
Make Kafka Properties Configurable
Problem
Some properties used by Kafka are hardcoded at this time and should be configurable.
Solution
Users can now set the following as environment variables:
Dev Container Java Version Update
Problem
The dev container for the project is not using Java 21 at this time.
Solution
The dev container has been updated to use Java 21.
Additional Changes
The launch.json file now references the .env file.
GCS BLOB Support
Problem
There is no support for depositing into GCS Cloud storage buckets at this time.
Solution
The ability to deposit kafka messages as BLOBs into a GCS Cloud storage bucket has been added.
Testing
This was tested locally with Docker using a service account that had the following permissions: