Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add documentation section for migrations #128

Merged
merged 2 commits into from
Nov 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 179 additions & 0 deletions docs/06_extend-sdk-migrations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
---
id: extend-sdk-migration
title: "SDK Guide: Pipeline Element Migration"
sidebar_label: "SDK: PE Migration"
---

Pipeline element migrations allow you to automatically update and migrate both existing and pipeline elements when a new
version of StreamPipes is installed. This means that whenever you upgrade StreamPipes, all existing and future
pipeline elements will be directly compatible with the new version without any manual interaction. Pipeline elements
include adapters, data processors, and data sinks.

:::info
Migrations will make their debut in StreamPipes version `0.93.0` and will be an integral part of the system going
forward.
However, it's important to note that this feature is not available in any of the previous versions of StreamPipes. To
take full advantage of migrations and their benefits, it is recommended to upgrade to version `0.93.0` or later. This
will
ensure that you have access to the latest enhancements and maintain compatibility with the evolving StreamPipes
platform.
:::

## Define Migrations

Whenever a pipeline element, be it an adapter, data processor, or data sink, undergoes changes that result in
modifications to its configuration options, developers must additionally create a migration procedure. This migration
process should be capable of smoothly transitioning all affected instances from the previous version to the new one.
The migration itself is automatically managed and executed by StreamPipes. Developers are only responsible for two key
aspects:

* **Implementing the concrete migration**: Developers need to craft the specific migration logic that facilitates the
seamless transition of configuration options.
* **Registering the migration**: Developers should register their migration procedures at the extensions service,
allowing StreamPipes to identify and apply the necessary updates to affected instances.

By adhering to these two essential tasks, developers can ensure a hassle-free evolution of pipeline elements while
StreamPipes handles the orchestration of the migration process.

The following gives a concrete example of creating a migration for
the [S7 adapter](./pe/org.apache.streampipes.connect.iiot.adapters.plc4x.s7.md).
Thereby, we assume this adapter has received a new input element which determines whether the connection should be made
authenticated or not.
This is represented by a simple boolean that is visualized as a toggle button in the UI.

### Implementing a Concrete Migration

StreamPipes offers three distinct migration mechanisms tailored to specific types of pipeline
elements: `IAdapterMigrator`, `IDataProcessorMigrator`, and `IDataSinkMigrator`.
These migration mechanisms are presented as interfaces and require the implementation of two fundamental methods:

* `config()`: This method defines the configuration for the migration, encompassing all essential metadata related to
the migration process.
* `migrate()`: Within this method, the actual migration logic is to be implemented. It serves as the operational core
for facilitating the migration for the respective pipeline element.

In accordance with the example described above, we will implement the `Plc4xS7AdapterMigrationV1` in the following.

:::note
Before we begin, it's important to familiarize ourselves with two key conventions that guide our approach to migrations:

* To maintain clarity and organization, all migration classes associated with a specific pipeline element are located
within a dedicated sub-package named `migration`. This sub-package is nested within the package of the respective
pipeline element.
* Migration classes are named according to a specific schema: `<elementClassName>MigrationV<targetVersion>`. For
example, if you are working on a migration for the PLC4x S7 adapter targeting version 1, the migration class would be
named `Plc4xS7AdapterMigrationV1`.
:::

Let's begin with providing the migration's configuration:

```java
@Override
public ModelMigratorConfig config() {
return new ModelMigratorConfig(
"org.apache.streampipes.connect.iiot.adapters.plc4x.s7",
SpServiceTagPrefix.ADAPTER,
0,
1
);
}
```

The migration config consists of the following four parts:

* `targetAppId`: this needs to equal the app id of the targeted element
* `modelType`: the type of the element to be migrated, this can be one
of: `SpServiceTagPrefix.ADAPTER`, `SpServiceTagPrefix.DATA_PROCESSOR`, `SpServiceTagPrefix.DATA_SINK`.
* `fromVersion`: the version of the element that the migration expects as input
* `toVersion`: the version the element has after the migration (needs to be at least `fromVersion + 1`)

The second step is to implement the actual migration logic.
In our example, we need to extend the existing static properties by an additional boolean property.

```java
@Override
public MigrationResult<AdapterDescription> migrate(AdapterDescription element, IStaticPropertyExtractor extractor) throws RuntimeException {

var config = element.getConfig();

var slideToggle = new SlideToggleStaticProperty();
slideToggle.setDefaultValue(false);
slideToggle.setLabel("Authentication required?");
config.add(slideToggle);

element.setConfig(config);
return MigrationResult.success(element);
}
```

We've completed all the necessary steps for our migration. The final task remaining is to register the migration within
the service definition.

### Registering the Migration

Only when the migration is registered at the service definition, the migration is sent to the StreamPipes core service.
Therefore, we need to add the migration to the same service definition as the element to migrate.
In our example this is defined in `ConnectAdapterIiotInit`:

```java jsx {22-24} showLineNumbers
@Override
public SpServiceDefinition provideServiceDefinition() {
return SpServiceDefinitionBuilder.create("connect-adapter-iiot",
"StreamPipes connect worker containing adapters relevant for the IIoT",
"",
8001)
.registerAdapter(new MachineDataSimulatorAdapter())
.registerAdapter(new FileReplayAdapter())
.registerAdapter(new IfmAlMqttAdapter())
.registerAdapter(new RosBridgeAdapter())
.registerAdapter(new OpcUaAdapter())
.registerAdapter(new Plc4xS7Adapter())
.registerAdapter(new Plc4xModbusAdapter())
.registerAdapter(new KafkaProtocol())
.registerAdapter(new MqttProtocol())
.registerAdapter(new NatsProtocol())
.registerAdapter(new HttpStreamProtocol())
.registerAdapter(new PulsarProtocol())
.registerAdapter(new RocketMQProtocol())
.registerAdapter(new HttpServerProtocol())
.registerAdapter(new TubeMQProtocol())
.registerMigrators(
new Plc4xS7AdapterMigrationV1()
)
.build();
```

<br />

## How Migrations are Handled Internally

Migrations are handled by an interplay between the Extension Service, which provides the migrations,
and the StreamPipes Core Service, which manages the migrations, as shown in the figure below:
<img src="/img/06_sdk_migrations/migration-flow.png" alt="Interplay of extensions service and core to handle migrations"/>

When an extensions service is initiated and has successfully registered itself with the core, it proceeds to send a
request to the core. This request includes a comprehensive list of all available migrations that have been registered
for it. Since this collection of migrations may encompass multiple migrations that affect the same pipeline element,
the migrations are first de-duplicated and then sorted based on their version range before being transmitted.

Upon receiving these migrations, the core's actions can be categorized into two distinct parts:

* Update descriptions for new elements
* Update descriptions for existing elements

### Update Descriptions for New Elements

Each migration transmitted from the extensions service to the core triggers the core to update the description of the
corresponding element stored in CouchDB. This is achieved by requesting the current configuration from the extensions
service and subsequently overwriting the existing configuration in the storage.

### Update Descriptions for Existing Elements

For each migration sent from the extensions service to the core, the core conducts a thorough check to determine if any
existing elements are affected by this migration. If such elements are identified, the extensions service is tasked with
requesting and subsequently executing the migration on behalf of the core.

In scenarios where multiple applicable migrations exist for a single pipeline element, they are sequentially applied.
Success in this process allows the core to seamlessly update the configuration. However, if any issues arise, the
corresponding pipeline element is halted. In the case of processors and sinks, the associated pipeline is even marked
with a `needs attention` label, which comes apparent in the UI.
1 change: 1 addition & 0 deletions website-v2/sidebars.json
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@
"extend-client",
"extend-sdk-functions",
"extend-sdk-event-model",
"extend-sdk-migration",
"extend-sdk-stream-requirements",
"extend-sdk-static-properties",
"extend-sdk-output-strategies",
Expand Down
7 changes: 7 additions & 0 deletions website-v2/src/css/customTheme.css
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
--ifm-color-primary-dark: #18125A;
--ifm-color-primary-darker: #171155;
--ifm-color-primary-darkest: #130E46;
--ifm-code-background: var(--color-primary-light);
--ifm-code-padding-horizontal: 0.2rem;
--ifm-code-padding-vertical: 0.12rem;

--color-primary: rgb(27, 20, 100);
--color-accent: rgb(57, 181, 74);
Expand All @@ -17,6 +20,10 @@
--ifm-heading-font-family: 'Roboto';
}

code {
color: var(--color-accent);
}

.navbar__logo {
height: 3rem;
margin-left: 0.25rem;
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading