implement migration for adapter and pipeline element configurations #2077

bossenti · 2023-10-23T12:10:34Z

Purpose

see #2045

See detailed explanation

Remarks

PR introduces (a) breaking change(s): no

PR introduces (a) deprecation(s): yes

Deprecations

Deprecate create() methods of ProcessingElementBuilder that do not contain a version
Deprecate create() methods of AdapterConfigurationBuilder that do not contain a version

…ipeline-elements

…es by the app id

…on-for-adapter-and-pipeline-element-configurations # Conflicts: # ui/projects/streampipes/platform-services/src/lib/model/gen/streampipes-model.ts # ui/src/app/add/components/endpoint-item/endpoint-item.component.ts # ui/src/scss/sp/main.scss

…pter-and-pipeline-element-configurations' into 2045-implement-migration-for-adapter-and-pipeline-element-configurations # Conflicts: # streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/migration/AdapterMigrator.java # streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/migration/MigrationResult.java # streampipes-extensions-api/src/main/java/org/apache/streampipes/extensions/api/migration/ModelMigrator.java # streampipes-extensions-management/src/main/java/org/apache/streampipes/extensions/management/model/SpServiceDefinition.java # streampipes-extensions-management/src/main/java/org/apache/streampipes/extensions/management/model/SpServiceDefinitionBuilder.java

…on-for-adapter-and-pipeline-element-configurations

This reverts commit 646e792.

dominikriemer · 2023-10-27T20:33:53Z

...sions-api/src/main/java/org/apache/streampipes/extensions/api/migration/AdapterMigrator.java

+    if (obj instanceof ModelMigrator<?, ?>) {
+      return this.config().equals(((ModelMigrator<?, ?>) obj).config());
+    }
+    return false;


Just out of curiosity, where is this equals method used? Is this the only reason the AdapterMigrator has to be an abstract class instead of an interface?

streampipes-client/src/main/java/org/apache/streampipes/client/api/AdminApi.java

streampipes-model/src/main/java/org/apache/streampipes/model/graph/DataProcessorInvocation.java

streampipes-model/src/main/java/org/apache/streampipes/model/graph/DataSinkInvocation.java

dominikriemer · 2023-10-28T09:01:03Z

@bossenti Is there a reason why we have two endpoints at the core side for managing migrations of adapters and pipeline elements?
Does it have any implications if we remove the filtering logic from ExtensionsModelSubmitter to the core and have a single migration method there for adapters and pipeline elements?
This would ease managing the status of services.

dominikriemer · 2023-10-28T11:29:05Z

I created a PR which breaks down the migration into smaller code units and creates a single request from extensions to core for submitting migration configs: #2101

* Extract MigrationResource logic into smaller units * Use single request for submitting migrations from extensions to core * refactor: remove dead code * style: change formatting --------- Co-authored-by: bossenti <[email protected]>

bossenti · 2023-10-30T06:51:32Z

@dominikriemer you already addressed all your comments in your PR #2101, right?

…on-for-adapter-and-pipeline-element-configurations # Conflicts: # streampipes-model/src/main/java/org/apache/streampipes/model/base/InvocableStreamPipesEntity.java

dominikriemer · 2023-10-30T07:10:12Z

@bossenti I didn't address the comments above - do you know where the equals method is used from the AdapterMigrator or can we change it to an interface?

bossenti · 2023-10-30T07:22:33Z

@dominikriemer Yes it is. We use it to determine if a migration has already been registered at the extensions service so that we can't register the same migration multiple times.

dominikriemer and others added 22 commits October 9, 2023 16:24

feat(#2002): Align adapter registration with other pipeline elements

d61b2bc

Fix checkstyle

dbf4cec

style: remove trailing whitespace

d437edc

Merge branch 'dev' into 2002-harmonize-registration-of-adapters-and-p…

f74ae39

…ipeline-elements

Add initial draft of migration concept

969d2ac

refactor: fix logger configuration

6f8ad83

refactor: extend storage implementations by method to get all instanc…

01ec032

…es by the app id

refactor: update generated typescript model

a23df74

feat: add version to models & builders

ecd20c4

refactor: implement string representation of Notification

646e792

feat: implement data model for migration

0792a01

feat: register migrations at service

6bc8bc1

Merge remote-tracking branch 'origin/dev' into 2045-implement-migrati…

599c83b

…on-for-adapter-and-pipeline-element-configurations

Revert "refactor: implement string representation of Notification"

9f88b9a

This reverts commit 646e792.

refactor: use correct Notification class

879ced4

feat: introduce migrate extensions resource

e65fe82

feat: introduce migrate adapter endpoint

5ae832a

feat: implement adapter migration at the core

5696491

remove data lake migration

20a9ddf

ensure order & uniqueness of migrations

cd0e41d

github-actions bot added dependencies Pull requests that update a dependency file java Pull requests that update Java code ui Anything that affects the UI backend Everything that is related to the StreamPipes backend labels Oct 23, 2023

bossenti added 3 commits October 23, 2023 14:12

remove redundant exception

18c1bc6

remove redundant exception

b31a3ea

add tests

7277ddb

github-actions bot added the testing Relates to any kind of test (unit test, integration, or E2E test). label Oct 23, 2023

bossenti added 2 commits October 27, 2023 09:27

test

3f83ae2

style: fix checkstyle issues

1b1b4c0

bossenti changed the title ~~2045 implement migration for adapter and pipeline element configurations~~ implement migration for adapter and pipeline element configurations Oct 27, 2023

bossenti linked an issue Oct 27, 2023 that may be closed by this pull request

Implement migration for adapter and pipeline element configurations #2045

Closed

refactor: remove legacy type definition

542d632

bossenti added the deprecation Assigned to pull requests or issues that introduce a deprecation. label Oct 27, 2023

bossenti added 2 commits October 27, 2023 10:46

refactor: update generated TS models

798ed3a

fix: add missing license header

d0d28c0

bossenti marked this pull request as ready for review October 27, 2023 08:48

Fix adapter model migration, add OPC adapter migration as sample

672745e

dominikriemer reviewed Oct 27, 2023

View reviewed changes

Fix typo

28b21c6

dominikriemer reviewed Oct 27, 2023

View reviewed changes

streampipes-client/src/main/java/org/apache/streampipes/client/api/AdminApi.java Outdated Show resolved Hide resolved

dominikriemer reviewed Oct 27, 2023

View reviewed changes

streampipes-model/src/main/java/org/apache/streampipes/model/graph/DataProcessorInvocation.java Outdated Show resolved Hide resolved

dominikriemer reviewed Oct 27, 2023

View reviewed changes

streampipes-model/src/main/java/org/apache/streampipes/model/graph/DataSinkInvocation.java Outdated Show resolved Hide resolved

tenthe approved these changes Oct 29, 2023

View reviewed changes

Improvements to extension migration (#2101)

10375f3

* Extract MigrationResource logic into smaller units * Use single request for submitting migrations from extensions to core * refactor: remove dead code * style: change formatting --------- Co-authored-by: bossenti <[email protected]>

Merge remote-tracking branch 'origin/dev' into 2045-implement-migrati…

f662872

…on-for-adapter-and-pipeline-element-configurations # Conflicts: # streampipes-model/src/main/java/org/apache/streampipes/model/base/InvocableStreamPipesEntity.java

bossenti mentioned this pull request Oct 30, 2023

docs: add documentation section for migrations apache/streampipes-website#128

Merged

bossenti added 2 commits October 30, 2023 08:40

refactor: remove redundant assignments

765f8ca

refactor: provide migrators as interface

a61bf62

bossenti merged commit a34a4dd into dev Oct 30, 2023
17 of 18 checks passed

bossenti deleted the 2045-implement-migration-for-adapter-and-pipeline-element-configurations branch October 30, 2023 08:43

bossenti mentioned this pull request Nov 19, 2023

refactor: add value for version parameter in adapter description #2206

Merged

IsaakKrut mentioned this pull request Aug 26, 2024

Handle case for element that is older than all migrations provided #2094

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement migration for adapter and pipeline element configurations #2077

implement migration for adapter and pipeline element configurations #2077

bossenti commented Oct 23, 2023 •

edited

Loading

dominikriemer Oct 27, 2023

dominikriemer commented Oct 28, 2023

dominikriemer commented Oct 28, 2023 •

edited

Loading

bossenti commented Oct 30, 2023

dominikriemer commented Oct 30, 2023

bossenti commented Oct 30, 2023

implement migration for adapter and pipeline element configurations #2077

implement migration for adapter and pipeline element configurations #2077

Conversation

bossenti commented Oct 23, 2023 • edited Loading

Purpose

Remarks

Deprecations

dominikriemer Oct 27, 2023

Choose a reason for hiding this comment

dominikriemer commented Oct 28, 2023

dominikriemer commented Oct 28, 2023 • edited Loading

bossenti commented Oct 30, 2023

dominikriemer commented Oct 30, 2023

bossenti commented Oct 30, 2023

bossenti commented Oct 23, 2023 •

edited

Loading

dominikriemer commented Oct 28, 2023 •

edited

Loading