-
Notifications
You must be signed in to change notification settings - Fork 42
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add sonataflow postgres db migrator tool for data index and jobs service
- Loading branch information
Showing
19 changed files
with
1,401 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
* | ||
!target/*-runner | ||
!target/*-runner.jar | ||
!target/lib/* | ||
!target/quarkus-app/* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
#Maven | ||
target/ | ||
pom.xml.tag | ||
pom.xml.releaseBackup | ||
pom.xml.versionsBackup | ||
release.properties | ||
.flattened-pom.xml | ||
|
||
# Eclipse | ||
.project | ||
.classpath | ||
.settings/ | ||
bin/ | ||
|
||
# IntelliJ | ||
.idea | ||
*.ipr | ||
*.iml | ||
*.iws | ||
|
||
# NetBeans | ||
nb-configuration.xml | ||
|
||
# Visual Studio Code | ||
.vscode | ||
.factorypath | ||
|
||
# OSX | ||
.DS_Store | ||
|
||
# Vim | ||
*.swp | ||
*.swo | ||
|
||
# patch | ||
*.orig | ||
*.rej | ||
|
||
# Local environment | ||
.env | ||
|
||
# Plugin directory | ||
/.quarkus/cli/plugins/ | ||
*.iml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
# sonataflow-db-migrator | ||
|
||
This is a quarkus postgres database migrator application for Sonataflow Data Index and Jobs Service applications. | ||
|
||
## Running the application in dev mode | ||
Though you can run the application locally in dev mode but it is advisable to run this application as a container image as described in the next section. | ||
The primary reason not to run as standalone application in dev mode, is that by default there are no DDL migration files included in the source. | ||
However the DDL files are dynamically included from its respective sources when a container image is created. | ||
|
||
You can run your application in dev mode that enables live coding using: | ||
|
||
```shell script | ||
./mvnw compile quarkus:dev | ||
``` | ||
|
||
## Build and Run container image locally | ||
You can build the cekit container image by using the provided image builder shell script | ||
```shell | ||
./build-container-image.sh | ||
``` | ||
Ensure the script completes without errors. | ||
If you may have a cekit specific Python virtual environment, be sure to activate it, so the script can find the cekit command. | ||
```shell | ||
virtualenv ~/cekit | ||
source ~/cekit/bin/activate; | ||
``` | ||
|
||
Assuming you have Postgres database running locally and e.g. have a di database for data index and js database for jobs service, you can run the image by command as follows, just substitute appropriate values. | ||
```shell | ||
podman run \ | ||
--env MIGRATE_DB_DATAINDEX=true \ | ||
--env QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL=<data-index-db-url e.g. jdbc:postgresql://host.docker.internal:5432/di> \ | ||
--env QUARKUS_DATASOURCE_DATAINDEX_USERNAME=<data-index-db-user> \ | ||
--env QUARKUS_DATASOURCE_DATAINDEX_PASSWORD=<data-index-db-password> \ | ||
--env QUARKUS_FLYWAY_DATAINDEX_SCHEMAS=dataindex \ | ||
--env MIGRATE_DB_JOBSSERVICE=true \ | ||
--env QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL=<jobs-service-db-url e.g. jdbc:postgresql://host.docker.internal:5432/js> \ | ||
--env QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME=<jobs-service-db-user> \ | ||
--env QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD=<jobs-service-db-password> \ | ||
--env QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS=jobsservice \ | ||
docker.io/apache/incubator-kie-kogito-service-db-migration-postgresql:999-SNAPSHOT | ||
``` | ||
|
||
### Environment variables | ||
| NAME | DESCRIPTION | DEFAULT | | ||
|---|---|---| | ||
| MIGRATE_DB_DATAINDEX | Set to true if you want to migrate data index database, set to false otherwise | false | | ||
| QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL | Data index database url e.g. jdbc:postgresql://host.docker.internal:5432/di| jdbc:postgresql://localhost:5432/postgres | | ||
| QUARKUS_DATASOURCE_DATAINDEX_USERNAME | Data index database username| postgres | | ||
| QUARKUS_DATASOURCE_DATAINDEX_PASSWORD | Data index database password| postgres | | ||
| QUARKUS_FLYWAY_DATAINDEX_SCHEMAS | Data index database schema| dataindex | | ||
| MIGRATE_DB_JOBSSERVICE | Set to true if you want to migrate jobs service database, set to false otherwise | false | | ||
| QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL | Jobs service database url e.g. jdbc:postgresql://host.docker.internal:5432/js| jdbc:postgresql://localhost:5432/postgres | | ||
| QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME | Jobs service database username| postgres | | ||
| QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD | Jobs service database password| postgres | | ||
| QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS | Jobs service database schema| jobsservice | |
37 changes: 37 additions & 0 deletions
37
images/tools/sonataflow-db-migrator/build-container-image.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
#!/bin/sh | ||
|
||
# cleanup temporary files | ||
cleanup () { | ||
rm -rf target | ||
rm -rf src/main/resources/postgresql | ||
rm -rf tmp | ||
rm -f src/main/cekit/modules/kogito-postgres-db-migration-deps/sonataflow-db-migrator-1.0-SNAPSHOT-runner.jar | ||
} | ||
|
||
# Start with cleanup | ||
cleanup | ||
|
||
# Get Data Index/ Jobs Service DDL Files | ||
mkdir -p tmp | ||
# Change the variables below, as needed | ||
DDL_VERSION=10.0.999-SNAPSHOT | ||
DDL_FILE=kogito-ddl-10.0.999-20240806.011718-23-db-scripts.zip | ||
DDL_URL=https://repository.apache.org/content/groups/snapshots/org/kie/kogito/kogito-ddl/$DDL_VERSION/$DDL_FILE | ||
wget $DDL_URL | ||
mv $DDL_FILE tmp | ||
cd tmp | ||
unzip $DDL_FILE | ||
mv ./postgresql ../src/main/resources | ||
cd .. | ||
|
||
# Create an Uber jar | ||
mvn package -Dquarkus.package.jar.type=uber-jar | ||
cp target/sonataflow-db-migrator-1.0-SNAPSHOT-runner.jar src/main/cekit/modules/kogito-postgres-db-migration-deps | ||
|
||
# Build the container image | ||
cd src/main/cekit | ||
cekit -v build podman | ||
|
||
# Cleanup | ||
cd ../../.. | ||
cleanup |
Oops, something went wrong.