Skip to content

Commit

Permalink
Add sonataflow postgres db migrator tool for data index and jobs service
Browse files Browse the repository at this point in the history
  • Loading branch information
rhkp committed Aug 9, 2024
1 parent fd1c1e3 commit afe41a1
Show file tree
Hide file tree
Showing 19 changed files with 1,401 additions and 0 deletions.
5 changes: 5 additions & 0 deletions images/tools/sonataflow-db-migrator/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
*
!target/*-runner
!target/*-runner.jar
!target/lib/*
!target/quarkus-app/*
44 changes: 44 additions & 0 deletions images/tools/sonataflow-db-migrator/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#Maven
target/
pom.xml.tag
pom.xml.releaseBackup
pom.xml.versionsBackup
release.properties
.flattened-pom.xml

# Eclipse
.project
.classpath
.settings/
bin/

# IntelliJ
.idea
*.ipr
*.iml
*.iws

# NetBeans
nb-configuration.xml

# Visual Studio Code
.vscode
.factorypath

# OSX
.DS_Store

# Vim
*.swp
*.swo

# patch
*.orig
*.rej

# Local environment
.env

# Plugin directory
/.quarkus/cli/plugins/
*.iml
56 changes: 56 additions & 0 deletions images/tools/sonataflow-db-migrator/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# sonataflow-db-migrator

This is a quarkus postgres database migrator application for Sonataflow Data Index and Jobs Service applications.

## Running the application in dev mode
Though you can run the application locally in dev mode but it is advisable to run this application as a container image as described in the next section.
The primary reason not to run as standalone application in dev mode, is that by default there are no DDL migration files included in the source.
However the DDL files are dynamically included from its respective sources when a container image is created.

You can run your application in dev mode that enables live coding using:

```shell script
./mvnw compile quarkus:dev
```

## Build and Run container image locally
You can build the cekit container image by using the provided image builder shell script
```shell
./build-container-image.sh
```
Ensure the script completes without errors.
If you may have a cekit specific Python virtual environment, be sure to activate it, so the script can find the cekit command.
```shell
virtualenv ~/cekit
source ~/cekit/bin/activate;
```

Assuming you have Postgres database running locally and e.g. have a di database for data index and js database for jobs service, you can run the image by command as follows, just substitute appropriate values.
```shell
podman run \
--env MIGRATE_DB_DATAINDEX=true \
--env QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL=<data-index-db-url e.g. jdbc:postgresql://host.docker.internal:5432/di> \
--env QUARKUS_DATASOURCE_DATAINDEX_USERNAME=<data-index-db-user> \
--env QUARKUS_DATASOURCE_DATAINDEX_PASSWORD=<data-index-db-password> \
--env QUARKUS_FLYWAY_DATAINDEX_SCHEMAS=dataindex \
--env MIGRATE_DB_JOBSSERVICE=true \
--env QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL=<jobs-service-db-url e.g. jdbc:postgresql://host.docker.internal:5432/js> \
--env QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME=<jobs-service-db-user> \
--env QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD=<jobs-service-db-password> \
--env QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS=jobsservice \
docker.io/apache/incubator-kie-kogito-service-db-migration-postgresql:999-SNAPSHOT
```

### Environment variables
| NAME | DESCRIPTION | DEFAULT |
|---|---|---|
| MIGRATE_DB_DATAINDEX | Set to true if you want to migrate data index database, set to false otherwise | false |
| QUARKUS_DATASOURCE_DATAINDEX_JDBC_URL | Data index database url e.g. jdbc:postgresql://host.docker.internal:5432/di| jdbc:postgresql://localhost:5432/postgres |
| QUARKUS_DATASOURCE_DATAINDEX_USERNAME | Data index database username| postgres |
| QUARKUS_DATASOURCE_DATAINDEX_PASSWORD | Data index database password| postgres |
| QUARKUS_FLYWAY_DATAINDEX_SCHEMAS | Data index database schema| dataindex |
| MIGRATE_DB_JOBSSERVICE | Set to true if you want to migrate jobs service database, set to false otherwise | false |
| QUARKUS_DATASOURCE_JOBSSERVICE_JDBC_URL | Jobs service database url e.g. jdbc:postgresql://host.docker.internal:5432/js| jdbc:postgresql://localhost:5432/postgres |
| QUARKUS_DATASOURCE_JOBSSERVICE_USERNAME | Jobs service database username| postgres |
| QUARKUS_DATASOURCE_JOBSSERVICE_PASSWORD | Jobs service database password| postgres |
| QUARKUS_FLYWAY_JOBSSERVICE_SCHEMAS | Jobs service database schema| jobsservice |
37 changes: 37 additions & 0 deletions images/tools/sonataflow-db-migrator/build-container-image.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/bin/sh

# cleanup temporary files
cleanup () {
rm -rf target
rm -rf src/main/resources/postgresql
rm -rf tmp
rm -f src/main/cekit/modules/kogito-postgres-db-migration-deps/sonataflow-db-migrator-1.0-SNAPSHOT-runner.jar
}

# Start with cleanup
cleanup

# Get Data Index/ Jobs Service DDL Files
mkdir -p tmp
# Change the variables below, as needed
DDL_VERSION=10.0.999-SNAPSHOT
DDL_FILE=kogito-ddl-10.0.999-20240806.011718-23-db-scripts.zip
DDL_URL=https://repository.apache.org/content/groups/snapshots/org/kie/kogito/kogito-ddl/$DDL_VERSION/$DDL_FILE
wget $DDL_URL
mv $DDL_FILE tmp
cd tmp
unzip $DDL_FILE
mv ./postgresql ../src/main/resources
cd ..

# Create an Uber jar
mvn package -Dquarkus.package.jar.type=uber-jar
cp target/sonataflow-db-migrator-1.0-SNAPSHOT-runner.jar src/main/cekit/modules/kogito-postgres-db-migration-deps

# Build the container image
cd src/main/cekit
cekit -v build podman

# Cleanup
cd ../../..
cleanup
Loading

0 comments on commit afe41a1

Please sign in to comment.