Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add migrateDBOnStart property under persistence #507

Merged
merged 10 commits into from
Aug 21, 2024
Merged

Add migrateDBOnStart property under persistence #507

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a4eacc1
Updates to persistence flags for DI/JS
rhkp Jul 8, 2024
07a5e8f
DI/JS: add MigrateDBOnStartUp attribute to operator
rhkp Jul 15, 2024
93ff7af
Move MigrateDBOnStartUp flag under persistence
rhkp Jul 16, 2024
16bc3fc
Address review comments - pass 1
rhkp Aug 8, 2024
2f67726
Address review comments - changes to see if e2e tests pass
rhkp Aug 13, 2024
040a234
Address review comments - remove JS image tag
rhkp Aug 14, 2024
45a9f24
Add mandatory migrateDBOnStartUp properties
rhkp Aug 14, 2024
8ac91f5
Minor changes to e2e tests run
rhkp Aug 20, 2024
3b77fd7
Minor change to make unit tests work
rhkp Aug 21, 2024
9afa484
Run make generate-all and commit
rhkp Aug 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ build-4-debug: generate ## Build manager binary with debug options.

.PHONY: run
run: manifests generate ## Run a controller from your host.
go run ./main.go
go run ./main.go -v=2 -controller-cfg-path=$(CURDIR)/config/manager/controllers_cfg.yaml

.PHONY: debug
debug: build-4-debug ## Run a controller from your host from binary
Expand Down Expand Up @@ -338,7 +338,7 @@ generate-all: generate generate-deploy bundle addheaders vet fmt

.PHONY: test-e2e # You will need to have a Minikube/Kind cluster up in running to run this target, and run container-builder before the test
test-e2e:
go test ./test/e2e/* -v -ginkgo.v -ginkgo.no-color -ginkgo.junit-report=./e2e-test-report.xml -timeout 60m
go test ./test/e2e/* -v -ginkgo.v -ginkgo.no-color -ginkgo.junit-report=./e2e-test-report.xml -timeout 120m

.PHONY: before-pr
before-pr: test generate-all
Expand Down
5 changes: 4 additions & 1 deletion api/v1alpha08/sonataflow_persistence_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,14 @@ type PlatformPersistencePostgreSQL struct {
// the operator will add the necessary JDBC properties to in the workflow's application.properties so that it can communicate
// with the persistence service based on the spec provided here.
// +optional
// +kubebuilder:validation:MaxProperties=1
// +kubebuilder:validation:MaxProperties=2
type PersistenceOptionsSpec struct {
// Connect configured services to a postgresql database.
// +optional
PostgreSQL *PersistencePostgreSQL `json:"postgresql,omitempty"`

// Whether to migrate database on service startup?
MigrateDBOnStartUp bool `json:"migrateDBOnStartUp"`
}

// PersistencePostgreSQL configure postgresql connection for service(s).
Expand Down
14 changes: 12 additions & 2 deletions bundle/manifests/sonataflow.org_sonataflowplatforms.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -567,8 +567,11 @@ spec:
persistence:
description: Persists service to a datasource of choice. Ephemeral
by default.
maxProperties: 1
maxProperties: 2
properties:
migrateDBOnStartUp:
description: Whether to migrate database on service startup?
type: boolean
postgresql:
description: Connect configured services to a postgresql
database.
Expand Down Expand Up @@ -625,6 +628,8 @@ spec:
required:
- secretRef
type: object
required:
- migrateDBOnStartUp
type: object
podTemplate:
description: PodTemplate describes the deployment details
Expand Down Expand Up @@ -8439,8 +8444,11 @@ spec:
persistence:
description: Persists service to a datasource of choice. Ephemeral
by default.
maxProperties: 1
maxProperties: 2
properties:
migrateDBOnStartUp:
description: Whether to migrate database on service startup?
type: boolean
postgresql:
description: Connect configured services to a postgresql
database.
Expand Down Expand Up @@ -8497,6 +8505,8 @@ spec:
required:
- secretRef
type: object
required:
- migrateDBOnStartUp
type: object
podTemplate:
description: PodTemplate describes the deployment details
Expand Down
7 changes: 6 additions & 1 deletion bundle/manifests/sonataflow.org_sonataflows.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2105,8 +2105,11 @@ spec:
persistence:
description: Persistence defines the database persistence configuration
for the workflow
maxProperties: 1
maxProperties: 2
properties:
migrateDBOnStartUp:
description: Whether to migrate database on service startup?
type: boolean
postgresql:
description: Connect configured services to a postgresql database.
maxProperties: 2
Expand Down Expand Up @@ -2160,6 +2163,8 @@ spec:
required:
- secretRef
type: object
required:
- migrateDBOnStartUp
type: object
podTemplate:
description: PodTemplate describes the deployment details of this
Expand Down
14 changes: 12 additions & 2 deletions config/crd/bases/sonataflow.org_sonataflowplatforms.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -568,8 +568,11 @@ spec:
persistence:
description: Persists service to a datasource of choice. Ephemeral
by default.
maxProperties: 1
maxProperties: 2
properties:
migrateDBOnStartUp:
description: Whether to migrate database on service startup?
type: boolean
postgresql:
description: Connect configured services to a postgresql
database.
Expand Down Expand Up @@ -626,6 +629,8 @@ spec:
required:
- secretRef
type: object
required:
- migrateDBOnStartUp
type: object
podTemplate:
description: PodTemplate describes the deployment details
Expand Down Expand Up @@ -8440,8 +8445,11 @@ spec:
persistence:
description: Persists service to a datasource of choice. Ephemeral
by default.
maxProperties: 1
maxProperties: 2
properties:
migrateDBOnStartUp:
description: Whether to migrate database on service startup?
type: boolean
postgresql:
description: Connect configured services to a postgresql
database.
Expand Down Expand Up @@ -8498,6 +8506,8 @@ spec:
required:
- secretRef
type: object
required:
- migrateDBOnStartUp
type: object
podTemplate:
description: PodTemplate describes the deployment details
Expand Down
7 changes: 6 additions & 1 deletion config/crd/bases/sonataflow.org_sonataflows.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2106,8 +2106,11 @@ spec:
persistence:
description: Persistence defines the database persistence configuration
for the workflow
maxProperties: 1
maxProperties: 2
properties:
migrateDBOnStartUp:
description: Whether to migrate database on service startup?
type: boolean
postgresql:
description: Connect configured services to a postgresql database.
maxProperties: 2
Expand Down Expand Up @@ -2161,6 +2164,8 @@ spec:
required:
- secretRef
type: object
required:
- migrateDBOnStartUp
type: object
podTemplate:
description: PodTemplate describes the deployment details of this
Expand Down
8 changes: 6 additions & 2 deletions controllers/platform/services/services.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ package services

import (
"fmt"
"strconv"

"github.com/apache/incubator-kie-kogito-serverless-operator/controllers/cfg"
"github.com/apache/incubator-kie-kogito-serverless-operator/utils/kubernetes"
Expand Down Expand Up @@ -224,8 +225,10 @@ func (d DataIndexHandler) ConfigurePersistence(containerSpec *corev1.Container)
c := containerSpec.DeepCopy()
c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(p.PostgreSQL, d.GetServiceName(), d.platform.Namespace)...)
migrateDBOnStart := strconv.FormatBool(d.platform.Spec.Services.DataIndex.Persistence.MigrateDBOnStartUp)
// specific to DataIndex
c.Env = append(c.Env, corev1.EnvVar{Name: quarkusHibernateORMDatabaseGeneration, Value: "update"}, corev1.EnvVar{Name: quarkusFlywayMigrateAtStart, Value: "true"})
c.Env = append(c.Env, corev1.EnvVar{Name: quarkusHibernateORMDatabaseGeneration, Value: "update"}, corev1.EnvVar{Name: quarkusFlywayMigrateAtStart, Value: migrateDBOnStart})

return c
}
return containerSpec
Expand Down Expand Up @@ -389,7 +392,8 @@ func (j JobServiceHandler) ConfigurePersistence(containerSpec *corev1.Container)
p := persistence.RetrievePostgreSQLConfiguration(j.platform.Spec.Services.JobService.Persistence, j.platform.Spec.Persistence, j.GetServiceName())
c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(p.PostgreSQL, j.GetServiceName(), j.platform.Namespace)...)
// Specific to Job Service
c.Env = append(c.Env, corev1.EnvVar{Name: "QUARKUS_FLYWAY_MIGRATE_AT_START", Value: "true"})
migrateDBOnStart := strconv.FormatBool(j.platform.Spec.Services.JobService.Persistence.MigrateDBOnStartUp)
c.Env = append(c.Env, corev1.EnvVar{Name: "QUARKUS_FLYWAY_MIGRATE_AT_START", Value: migrateDBOnStart})
c.Env = append(c.Env, corev1.EnvVar{Name: "KOGITO_JOBS_SERVICE_LOADJOBERRORSTRATEGY", Value: "FAIL_SERVICE"})
return c
}
Expand Down
12 changes: 10 additions & 2 deletions controllers/sonataflowplatform_controller_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,8 +260,16 @@ func TestSonataFlowPlatformController(t *testing.T) {
// Check with persistence set
ksp.Spec = v1alpha08.SonataFlowPlatformSpec{
Services: &v1alpha08.ServicesPlatformSpec{
DataIndex: &v1alpha08.ServiceSpec{},
JobService: &v1alpha08.ServiceSpec{},
DataIndex: &v1alpha08.ServiceSpec{
Persistence: &v1alpha08.PersistenceOptionsSpec{
MigrateDBOnStartUp: false,
},
},
JobService: &v1alpha08.ServiceSpec{
Persistence: &v1alpha08.PersistenceOptionsSpec{
MigrateDBOnStartUp: false,
},
},
},
Persistence: &v1alpha08.PlatformPersistenceOptionsSpec{
PostgreSQL: &v1alpha08.PlatformPersistencePostgreSQL{
Expand Down
21 changes: 18 additions & 3 deletions operator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1059,8 +1059,11 @@ spec:
persistence:
description: Persists service to a datasource of choice. Ephemeral
by default.
maxProperties: 1
maxProperties: 2
properties:
migrateDBOnStartUp:
description: Whether to migrate database on service startup?
type: boolean
postgresql:
description: Connect configured services to a postgresql
database.
Expand Down Expand Up @@ -1117,6 +1120,8 @@ spec:
required:
- secretRef
type: object
required:
- migrateDBOnStartUp
type: object
podTemplate:
description: PodTemplate describes the deployment details
Expand Down Expand Up @@ -8931,8 +8936,11 @@ spec:
persistence:
description: Persists service to a datasource of choice. Ephemeral
by default.
maxProperties: 1
maxProperties: 2
properties:
migrateDBOnStartUp:
description: Whether to migrate database on service startup?
type: boolean
postgresql:
description: Connect configured services to a postgresql
database.
Expand Down Expand Up @@ -8989,6 +8997,8 @@ spec:
required:
- secretRef
type: object
required:
- migrateDBOnStartUp
type: object
podTemplate:
description: PodTemplate describes the deployment details
Expand Down Expand Up @@ -19002,8 +19012,11 @@ spec:
persistence:
description: Persistence defines the database persistence configuration
for the workflow
maxProperties: 1
maxProperties: 2
properties:
migrateDBOnStartUp:
description: Whether to migrate database on service startup?
type: boolean
postgresql:
description: Connect configured services to a postgresql database.
maxProperties: 2
Expand Down Expand Up @@ -19057,6 +19070,8 @@ spec:
required:
- secretRef
type: object
required:
- migrateDBOnStartUp
type: object
podTemplate:
description: PodTemplate describes the deployment details of this
Expand Down
4 changes: 4 additions & 0 deletions test/testdata/platform/persistence/generic_from_platform_cr/02-sonataflow_platform.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ spec:
services:
dataIndex:
enabled: false
persistence:
migrateDBOnStartUp: true
podTemplate:
initContainers:
- name: init-postgres
Expand All @@ -38,6 +40,8 @@ spec:
command: [ 'sh', '-c', 'until (echo 1 > /dev/tcp/postgres.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local/5432) >/dev/null 2>&1; do echo "Waiting for postgres server"; sleep 3; done;' ]
jobService:
enabled: false
persistence:
migrateDBOnStartUp: true
podTemplate:
initContainers:
- name: init-postgres
Expand Down
2 changes: 2 additions & 0 deletions test/testdata/platform/persistence/overwritten_by_services/02-sonataflow_platform.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ spec:
dataIndex:
enabled: false
persistence:
migrateDBOnStartUp: true
postgresql:
jdbcUrl: jdbc:postgresql://postgres:5432/sonataflow?currentSchema=data-index-service
secretRef:
Expand All @@ -50,6 +51,7 @@ spec:
jobService:
enabled: false
persistence:
migrateDBOnStartUp: true
postgresql:
jdbcUrl: jdbc:postgresql://postgres:5432/sonataflow?currentSchema=jobs-service
secretRef:
Expand Down
2 changes: 2 additions & 0 deletions test/testdata/platform/services/dev/postgreSQL/02-sonataflow_platform.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ spec:
dataIndex:
enabled: false
persistence:
migrateDBOnStartUp: true
postgresql:
jdbcUrl: jdbc:postgresql://postgres:5432/sonataflow?currentSchema=data-index-service
secretRef:
Expand All @@ -41,6 +42,7 @@ spec:
jobService:
enabled: false
persistence:
migrateDBOnStartUp: true
postgresql:
jdbcUrl: jdbc:postgresql://postgres:5432/sonataflow?currentSchema=jobs-service
secretRef:
Expand Down
2 changes: 2 additions & 0 deletions test/testdata/platform/services/preview/postgreSQL/02-sonataflow_platform.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ spec:
dataIndex:
enabled: true
persistence:
migrateDBOnStartUp: true
postgresql:
jdbcUrl: jdbc:postgresql://postgres:5432/sonataflow?currentSchema=data-index-service
secretRef:
Expand All @@ -47,6 +48,7 @@ spec:
jobService:
enabled: true
persistence:
migrateDBOnStartUp: true
postgresql:
jdbcUrl: jdbc:postgresql://postgres:5432/sonataflow?currentSchema=jobs-service
secretRef:
Expand Down
1 change: 1 addition & 0 deletions test/testdata/workflow/persistence/by_service/04-sonataflow_callbackstatetimeouts.sw.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ metadata:
sonataflow.org/version: 0.0.1
spec:
persistence:
migrateDBOnStartUp: false
postgresql:
secretRef:
name: postgres-secrets
Expand Down
1 change: 1 addition & 0 deletions ...sistence/from_platform_overwritten_by_service/03-sonataflow_callbackstatetimeouts.sw.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ metadata:
sonataflow.org/version: 0.0.1
spec:
persistence:
migrateDBOnStartUp: false
postgresql:
secretRef:
name: postgres-secrets
Expand Down
4 changes: 4 additions & 0 deletions ...ta/workflow/persistence/from_platform_with_di_and_js_services/02-sonataflow_platform.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ spec:
services:
dataIndex:
enabled: true
persistence:
migrateDBOnStartUp: true
podTemplate:
initContainers:
- name: init-postgres
Expand All @@ -42,6 +44,8 @@ spec:
command: [ 'sh', '-c', 'until (echo 1 > /dev/tcp/postgres.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local/5432) >/dev/null 2>&1; do echo "Waiting for postgres server"; sleep 3; done;' ]
jobService:
enabled: true
persistence:
migrateDBOnStartUp: true
podTemplate:
initContainers:
- name: init-postgres
Expand Down
4 changes: 3 additions & 1 deletion ...m_with_no_persistence_required/03-sonataflow_callbackstatetimeouts-no-persistence.sw.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ metadata:
sonataflow.org/description: Callback State Timeouts Example k8s
sonataflow.org/version: 0.0.1
spec:
persistence: {}
persistence: {
migrateDBOnStartUp: true
}
podTemplate:
container:
env:
Expand Down
Loading