StreamPipes Structure

[tenthe]

Background

This discussion is about the structure and design of StreamPipes and its APIs.
Recently we have been working on some issues to reformat the code. During this work, I realized that some concepts within the software are not entirely clear and that we also have several legacy modules that can probably be removed. I wasn't quite sure where to start, so I took some time to get an overview of the current implementation.

Goal

I would like to look at the entire project and discuss with you the code structure, APIs, libraries, and third-party services (e.g. databases, message brokers).
The goal is to clean up the code base and create software that is easy to develop, easy to use, and easy to maintain.

Overview

Here is a first overview of the different roles, StreamPipes services and other dependencies.
This by no means claims to be complete. Rather, it is intended to serve as a starting point for discussion.

Next Steps

• I started with an overview of the current Java modules, see post below.
• Additionally, we should create a similar overview for the REST APIs. I think they are a good starting point for restructuring and refactoring.
• The artifacts (e.g., architecture diagrams) and descriptions that emerge from the discussion should be integrated into the documentation to help others quickly understand the design decisions

Contribution

🎯 This discussion should only be the starting point.
👏 I think we have already made great progress in the last few months. Thank you to everyone who has participated!
🗣️ Any kind of feedback and discussion is highly welcome!

[tenthe]

Overview StreamPipes Modules

I have looked at all the modules we have in our Java codebase and tried to summarize their main intent. I also added some discussion points that I noticed.

• client / client-python
  • Clients to interact with the StreamPipes API
• commons
  • Constants, Exceptions, Networking, Parser, Random(Generators), ZIP
• config
  • Configurations that can be changed by users, as well as configuratioins for the core service
• connect-management
  • Contains the adapter management, that is required in the service extensions during runtime
  • DISCUSS Should we rename the classes?
• data-explorer
  • Application logic to query data from the time-series database
  • DISCUSS Should we change the module name?
  • DISCUSS Which classes should be moved to the REST modules
  • DISCUSS In this module we have a strong coupling to the InfluxDB
• data-explorer-commons
  • Configurations, InfluxStore, ImageStore, TimeSeriesStore
  • DISCUSS Can we move the code to module data-explorer
• data-export
  • Responsible to manage the export / import of StreamPipes configurations
  • DISCUSS Think about a different name. Can be confused with export of data explorer data.
• dataformat
  • Those modules contain all the implementation for the data formats
  • (cbor, fst, json, smile)
• extensions
  • Contains all modules with adapters, processors, and sinks
• extensions-api
  • Interfaces for the extensions (See module below)
• extensions-management
  • Application logic relevant for the modules that are implemented in extensions
• integration-tests
  • Contain tests for services that have third party requirements
• logging
  • DISCUSS Do we still need this module or is it legacy code
• mail
  • Logic to send emails via the system
• maven-plugin
  • Maven plugin to extract pipeline element information from code and provide it for the website documentation
• measurement-units
  • logic to deal with different units
• messaging
  • Different implementation for internally used message brokers
  • (jms, kafka, mqtt, nats)
• model
  • Contains all model classes of StreamPipes
  • This model is also used to automatically generate the model for TypeScript
• model-client
  • Model classes that are only required by the UI
  • This model is also used to automatically generate the model for TypeScript
• model-shared
  • Only contains annotations TsIgnore & TsModel
• performance-tests REMOVED
• pipeline-management
  • Contains application logic for the pipeline editor
• platform-service
  • Currently contains Resources for the data lake
  • DISCUSS Move those classes into other modules. How will we structure the REST modules?
• resource-management
  • DISCUSS What is managed in this module and what could be moved to pipeline-management or connect-management
• rest
  • The impl package currenlty contains the most REST endpoints
  • rest-core-base
    • Only contains two classes AbstractAuthGuardedRestResource & AbstactRestResource
  • rest-extensions
    • Rest enpoints of service extensions
    • Currently mainly endpoints to manage connect & pipelines
  • rest-shared
    • (annotation, impl, serializer, util )
• sdk
  • builder, extractor, helpers,
  • Application logic to extend StreamPipes functionalities
• sdk-bundle
  • Contains no classes, is only used to budle several packages
  • DISCUSS Do we need this or can we move this to sdk instead
• security-jwt
  • Logic to handle json web tokens
• serializers-json
  • Contains one class, the JacksonSerializer
  • DISCUSS Do we require a seperate module for this?
• service-base
  • Base module for StreamPipes services (service core and service extension)
• service-core
  • Is executed when StreamPipes is started
  • Contains migration scripts and starts up REST endpooint
• service-discovery
  • Only contains class SpServiceDiscovery
  • service-discovery-api
    • Interface of the service discovery
  • service-discovery-consul
    • Consul implementation of the service discovery interface
• service-extensions
  • Module for extension services containing (adapters, processing elements, sinks, functions) during runtime
  • Contains ExtensionsModelSubmitter that is implemented by all extension services
• sources
  • Abstract classes AbstractAdapterIncludedStream & AbstractAlreadyExistingStream
  • Required for streams that are not created by connect adapters
  • DISCUSS Should we continue to support this or only rely on the connect API?
• storage-api
  • Interfaces of all classes that can be persistet in the database
• storage-couchdb
  • CouchDB implementation of the storage-api
• storage-management
  • Contains two classes StorageDispatcher & StorageManager
• test-utils
  • Contains utility classes to ease the creation of unit tests
• user-management
  • Responsible for user management within StreamPipes
  • Packages: authentication, encryption, jwt, model, service
• vocabulary
  • Contains classes with different vocabularies
• wrapper
  • Base module for the different wrappers
  • wrapper-distributed
    • Base module for the distributed wrappers
    • Contains one abstract class DistributedRuntime
  • wrapper-flink
    • Wrapper for Flink
  • wrapper-kafka-streams
    • Wrapper for Kafka Streams
  • wrapper-python
    • Python wrapper
    • DISCUSS Is this module still relevant for the new Python integration?
  • wrapper-siddhi
    • Wrapper for Siddhi engine
  • wrapper-standalone
    • Module for standalone wrappers that are not distributed

[bossenti]

@tenthe thanks a lot for starting the discussion and preparing it that well 🙏🏼
So where and how do we want to start?
Do we want to have a dedicated disussion for each DISCUSS entry? Or just start a thread for each within this discussion?

I think a great outcome of the conceptual work would we to have the overciew you provided somewhere clearly documented

[tenthe]

I am currently working on replacing the data set adapters as described in #1115.
With this change we will hopefully get rid of some dependencies and simplify the code structure.

Aditionally, my suggestion would be to start with the modules/discussions that I wasn't sure we still needed. Maybe we can delete or restructure some of them to reduce the number of modules. This shouldn't take much time.
What do you think about this?

[bossenti]

Sounds like a solid first step 👍🏼
Do we want to create (a) issue(s) for that?

[tenthe]

Maybe we can work through the items one by one. My suggestion would be to first have a short discussion about it and then make an issue out of it.

What do you think about that?
There should also be a lot of topics that are good first issues once it is clear what to do.

I'm afraid that if we directly make issue out of everything, things often aren't quite clear yet and they are not worked on for a long time.

[bossenti]

I really like your suggestion, let's do it step by step 💪🏼