Support migration of adapter and pipeline element configurations

[dominikriemer]

Body

Currently, no mechanism exists to update the configuration of pipeline elements. For instance, when a new configuration is added to an adapter or data processor, existing processors can stop working since the configuration of installed pipeline elements and pipelines which use these elements is broken.

I will open a discussion for the approach to migrate pipeline elements, this issue collects the subtasks to implement such a concept:

• Harmonize registration of adapters and pipeline elements #2002
• Develop migration concept
• Implement migration for adapter and pipeline element configurations #2045
• Improve lifecycle for managing core and extension initialization actions #2104

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.

[bossenti]

Introduce the ability to migrate existing adapters & PE

Important

The current implementation only allows modifications that affect the configuration of
an element. Changes that affect the event stream are out of scope for this iteration
(see more at next steps).

This can be split into major parts:

• update description for new elements: adapter, processor, sinks
• update descriptions for existing elements: adapter, processor, sinks

Rationale

As of now, a user cannot simply upgrade to a recent StreamPipes version if this version introduces changes in a pipeline element (adapter, processor, or sink). To create new instances of an affected pipeline element the user needs to update them first via the UI.
Secondly, existing instances of an affected pipeline element were required to be recreated manually.
Therefore, we now introduce so-called migrations to update both the descriptions used for new elements and the existing elements automatically in the background.

General Approach

A developer is now enabled to define one or multiple Migration for a pipeline element.
Therefore, one of AdapterMigrator, DataProcessorMigrator, or DataSinkMigrator from streampipes-extensions-api needs to be implemented. In addition, the migration needs to be added to the service definition in the extensions model submitter, e.g., AllExtensionsIIoTInit.

From now on, the migration is fully automatically handled according to the following flow:

Update Description for New Elements

For every migration that is sent from the extensions service to the core, the core does update the description of the corresponding element in the storage (couchDB). This is done by requesting the current configuration from the extensions service and overriding the existing one in the storage. This automatic update of all element descriptions makes the manual update in the UI redundant. Thus we can remove this functionality as an upcoming step.

Update Descriptions for Existing Elements

For every migration that is sent from the extensions service to the core, the core checks if there are any existing elements that are affected by this migration. If so, the migration is requested by the extensions service which then executes the migration on behalf of the core.
In case this is successful, the core can simply update the configuration.
If not, we stop the corresponding pipeline element. For processors & sinks, the corresponding pipeline is even marked with "needs attention".
However, (at least for adapters) this is not yet the ideal solution for handling migration failures.

Important Information

• The description of all pipeline elements is extended by an additional field version which needs to be provided in the future. However, we can make this change step by step since the version always defaults to 0.
• The order in which a migration is registered at the service definition does not affect anything. The order of the migrations is always ensured so that the earliest migrations are always applied first.
• Before a migration is applied to an element, it is checked if this element needs the migration. So we don't need to worry about migrations being applied multiple times to the same element.

Next steps

The following steps need to be handled (at least) to fully support migrations at StreamPipes:

• Handle migration failures properly #2093
• Handle case for element that is older than all migrations provided #2094
• Remove update functionality from UI #2095
• Assets need to be updated as well (only for adapters) #2096
• Enable migrations for scenarios where the event stream is affected #2097
• Handle cases with multiple extension services providing the same pipeline elements (handled by Improve lifecycle for managing core and extension initialization actions #2104)
• Implement first migration for S7 adapter #2098