Merge branch 'dev'

.github/workflows/deploy-website.yml

@@ -13,31 +13,25 @@ jobs:
       - name: Setup Node.js
         uses: actions/setup-node@v1
         with:
-          node-version: 14.1.0
-      - name: Build Website
-        run: |
-          npm install
-          npm run build-website
+          node-version: 18.16.1
       - name: Copy asf yaml
         run: |
-          cp .asf.yaml build/
+          cp .asf.yaml website-v2/build/
       - name: Copy htaccess
         run: |
-          cp .htaccess build/
+          cp .htaccess website-v2/build/
       - name: Build Docs
         run: |
-          mkdir build/docs
-          cd documentation/website
+          cd website-v2
           npm install
           npm run build
-          cp -r build/streampipes-docs/* ../../build/docs
       - name: Copy Python docs
         run: |
-          mkdir -p ./build/docs/docs/python
-          cp -r docs-python/* ./build/docs/docs/python
+          mkdir -p ./website-v2/build/docs/docs/python
+          cp -r docs-python/* ./website-v2/build/docs/docs/python
       - name: Deploy
         uses: peaceiris/[email protected]
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           PUBLISH_BRANCH: asf-site
-          PUBLISH_DIR: ./build
+          PUBLISH_DIR: ./website-v2/build

.github/workflows/pr-validation.yml

@@ -13,15 +13,10 @@ jobs:
       - name: Setup Node.js
         uses: actions/setup-node@v1
         with:
-          node-version: '14.x'
-
-      - name: Build website
-        run: |
-          npm install
-          npm run build-website
+          node-version: '18.16.1'
 
       - name: Build documentation
         run: |
-          cd documentation/website
+          cd website-v2
           npm install
-          npm run build
+          npm run build

.gitignore

@@ -16,3 +16,7 @@ website/i18n/*
 
 build/
 documentation/website/build/
+
+website/.docusaurus
+website-v2/.docusaurus/**
+website-v2/.docusaurus

Dockerfile

@@ -1,8 +1,7 @@
 FROM nginx
 
-COPY documentation/website/build/streampipes-docs/ /usr/share/nginx/html/docs
+COPY website-v2/build/ /usr/share/nginx/html
 COPY docs-python/ /usr/share/nginx/html/docs/docs/python
-COPY build/ /usr/share/nginx/html
 
 COPY nginx_config/nginx.conf /etc/nginx/nginx.conf
 COPY nginx_config/default.conf /etc/nginx/conf.d/default.conf

README.md

@@ -4,28 +4,24 @@
 ## Intro
 This repository includes the website and documentation of Apache StreamPipes on [https://streampipes.apache.org](https://streampipes.apache.org).
 Two different technologies are used to create both sites:
-* The website uses [harp.js](http://harpjs.com/) to create static pages from ejs files.
-* The documentation uses [Docusaurus](https://docusaurus.io/) to create a versioned documentation.
+* The website & documentation uses [Docusaurus](https://docusaurus.io/) to create a versioned documentation.
+* The Python docs are auto-generated from the main code repo using MkDocs.
 
 Both parts are merged to a single directory during the CI process.
 
-Node.js and npm are required to build both sites.
+Node.js and npm are required to build the documentation.
 
-## Build & run website
+## Build Website & Documentation
 
-* Run ``npm install`` from the root directory of this repository.
-* Run ``npm run start-website`` to start the website locally for debugging purposes. The content will be served on port 9000.
-* Run ``npm run build-website`` to build the website. This will create a folder ``build`` that contains the generated static HTML and asset files.
-
-## Build Documentation
-
-* Run ``npm install`` from the ``documentation/website`` directory.
-* Run ``npm run start`` to start the documentation for debugging purposes. The content will be served on ``http://localhost:3000/docs``
-* Run ``npm run build`` to build the documentation This will create a folder ``build/streampipes-docs`` that includes all required static files.
+* Run ``npm install`` from the ``website-v2`` directory.
+* Run ``npm run start`` to start the documentation for debugging purposes. The content will be served on ``http://localhost:3000``
+* Run ``npm run build`` to build the documentation This will create a folder ``website-v2/build`` that includes all required static files.
 * Run ``npm run ds-version NEW_VERSION`` to release a new docs version.
 
 ## Deploy
 
-* First, build the website as explained above.
-* Then, build the documentation as explained above.
-* Copy the content from the generated documentation (content from ``streampipes-docs``) to the ``build`` folder into a subdirectory named ``docs``.
+* First, build the website and documentation as explained above.
+* Copy the content from the `docs-python` to the ``website-v2/build`` folder into a subdirectory named ``python``.
+
+The deployment step is done automatically using the asf.yaml process. 
+Code must be merged into the `master` branch in order to trigger an update of the website.

docs/01_try-installation.md

@@ -0,0 +1,62 @@
+---
+id: try-installation
+title: Installation
+sidebar_label: Installation
+---
+
+import DownloadSection from '@site/src/components/download/DownloadSection.tsx';
+
+The easiest way to install StreamPipes is our Docker-based installation. For production-grade deployments, we also
+recommend looking at our Kubernetes support, which is also part of the installation kit.
+
+## Prerequisites
+
+The Docker-based installation requires **Docker** and **Docker Compose** to be installed on the target machine.
+Installation instructions can be found below.
+
+:::info Install Docker
+Go to https://docs.docker.com/installation/ and follow the instructions to install Docker for your OS. Make sure
+docker can be started as a non-root user (described in the installation manual, don’t forget to log out and in
+again) and check that Docker is installed correctly by executing docker-run hello-world
+:::
+
+### Supported operating systems
+
+The Docker-based installation supports the operating systems **Linux**, **Mac OS X** and **Windows 10**. Older windows
+versions are not fully compatible with Docker. Linux VMs running under Windows might cause network problems with Docker,
+therefore some manual work might be needed to make StreamPipes run properly.
+
+### Web Browser
+
+The StreamPipes application itself will be accessible through a web browser. We recommend a recent version of Chrome (
+best experience), Firefox or Edge.
+
+## Install StreamPipes
+
+<DownloadSection version={'0.93.0'}></DownloadSection>
+
+## Setup StreamPipes
+
+Once you've opened the browser at the URL given above, you should see the StreamPipes application as shown below. At
+initial startup, StreamPipes automatically performs an installation process.
+After the installation has finished, continue by clicking on "Go to login
+page", once all components are successfully configured.
+
+On the login page, enter your credentials, then you should be forwarded to the home page.
+
+Congratulations! You've successfully managed to install StreamPipes. Now we're ready to build our first pipeline!
+
+<div class="my-carousel docs-carousel">
+    <img src="/img/01_try-installation/04_home.png" alt="Home page"/>
+</div>
+
+:::danger Errors during the installation process
+In most cases, errors during the installation are due to an under-powered system.
+If there is a problem with any of the components, please restart the whole system (`docker-compose
+down` and eventually also delete the volumes).
+Please also make sure that you've assigned enough memory available to Docker.
+:::
+
+## Next Steps
+
+That's it! Have a look at the usage guide to learn how to use Apache StreamPipes.

docs/01_try-overview.md

@@ -0,0 +1,128 @@
+---
+id: user-guide-introduction
+title: Apache StreamPipes Documentation
+sidebar_label: Overview
+---
+
+This is the documentation of Apache StreamPipes.
+
+<img class="docs-image docs-image-small docs-image-no-shadow" style={{padding: '10px'}} src="/img/01_try-overview/01_streampipes-overview.png" alt="StreamPipes Overview"/>
+
+
+<div class="container grid col-3">
+    <div class="column">
+        <div class="toc-box">
+            <div class="toc-header border-thin">
+                🚀 Try
+            </div>
+            <div class="toc-content">
+                <div class="toc-section-header"><b>Your first steps with Apache StreamPipes:</b></div>
+               <a href="try-installation.html">Install StreamPipes 🔗</a>
+            </div>
+        </div>
+    </div>
+    <div class="column">
+        <div class="toc-box">
+            <div class="toc-header border-thin">
+                💡 Concepts
+            </div>
+            <div class="toc-content">
+                <div class="toc-section-header"><b>Learn about some general concepts of StreamPipes:</b></div>
+                <a href="concepts-overview.html">Overview 🔗</a>
+            </div>
+        </div>
+    </div>
+    <div class="column">
+        <div class="toc-box">
+            <div class="toc-header border-thin">
+                🎓 Use
+            </div>
+            <div class="toc-content">
+                <div class="toc-section-header"><b>Learn how to use the various modules of StreamPipes:</b></div>
+                <a href="use-connect.html">StreamPipes Connect 🔗</a>, <a href="use-pipeline-editor.html">Pipeline Editor 🔗</a>, 
+                <a href="use-managing-pipelines.html">Managing Pipelines 🔗</a>, 
+                <a href="use-live-dashboard.html">Live Dashboard 🔗</a>, <a href="use-data-explorer.html">Data Explorer 🔗</a>, 
+                <a href="use-notifications.html">Notifications 🔗</a>
+            </div>
+        </div>
+    </div>
+    <div class="column">
+        <div class="toc-box">
+            <div class="toc-header border-thin">
+                📚 Pipeline Elements
+            </div>
+            <div class="toc-content">
+                <div class="toc-section-header"><b>Available pipeline elements in StreamPipes:</b></div>
+                <a href="pe/org.apache.streampipes.connect.protocol.stream.kafka.html">Adapters 🔗</a>, 
+                <a href="pe/org.apache.streampipes.processors.aggregation.flink.aggregation.html">Data Processors 🔗</a>, 
+                <a href="pe/org.apache.streampipes.sinks.databases.jvm.couchdb.html">Data Sinks 🔗</a> 
+            </div>
+        </div>
+    </div>
+    <div class="column">
+        <div class="toc-box">
+            <div class="toc-header border-thin">
+                ⚡ Deploy 
+            </div>
+            <div class="toc-content">
+                <div class="toc-section-header"><b>How to set up StreamPipes in test and production environments:</b></div>
+                <a href="deploy-docker.html">Docker 🔗</a>, <a href="deploy-kubernetes.html">Kubernetes 🔗</a>, <a href="../deploy
+                -use-ssl">Use SSL 🔗</a>
+            </div>
+        </div>
+    </div>
+    <div class="column">
+        <div class="toc-box">
+            <div class="toc-header border-thin">
+                💻 Extend
+            </div>
+            <div class="toc-content">
+                <div class="toc-section-header"><b>Write your own pipeline elements for StreamPipes:</b></div>
+                <a href="extend-setup.html">Development Setup 🔗</a>, <a href="extend-cli.html">CLI 🔗</a>, <a href="../extend
+                -archetypes">Maven Archetypes 🔗</a>,
+                <a href="extend-tutorial-data-sources.html">Tutorial Data Sources 🔗</a>, <a href="../extend-tutorial-data
+                -processors">Tutorial Data Processors 🔗</a>, <a href="extend-tutorial-data-sinks.html">Tutorial Data Sinks 🔗
+                </a>,
+                <a href="extend-sdk-event-model.html">Event Model 🔗</a>, <a href="extend-sdk-stream-requirements.html">Stream
+                 Requirements 🔗</a>, <a href="extend-sdk-static-properties.html">Static Properties 🔗</a>,
+                <a href="extend-sdk-output-strategies.html">Output Strategies 🔗</a>
+            </div>
+        </div>
+    </div>
+    <div class="column">
+        <div class="toc-box">
+            <div class="toc-header border-thin">
+                🔧 Technicals
+            </div>
+            <div class="toc-content">
+                <div class="toc-section-header"><b>Learn about technical concepts behind the curtain:</b></div>
+                <a href="technicals-architecture.html">Architecture 🔗</a>, <a href="technicals-user-guidance.html">User
+                 Guidance 🔗
+                </a>, <a href="technicals-runtime-wrappers.html">Runtime Wrappers 🔗</a>,
+                <a href="technicals-messaging.html">Messaging 🔗</a>, <a href="technicals-configuration.html">Configuration 🔗</a>
+            </div>
+        </div>
+    </div>
+    <div class="column">
+        <div class="toc-box">
+            <div class="toc-header border-thin">
+                👪 Community
+            </div>
+            <div class="toc-content">
+                <div class="toc-section-header"><b>Get support and learn how to contribute to StreamPipes:</b></div>
+                <a href="community-get-help.html">Get Help 🔗</a>, <a href="community-contribute.html">Contribute 🔗</a>
+            </div>
+        </div>
+    </div>
+    <div class="column">
+        <div class="toc-box">
+            <div class="toc-header border-thin">
+                🐍 StreamPipes Python
+            </div>
+            <div class="toc-content">
+                <div class="toc-section-header"><b>Discover what we offer for the Python world:</b></div>
+                <a href="/docs/docs/python/latest">Python Documentation 🔗</a>
+            </div>
+        </div>
+    </div>
+</div>

docs/01_try-tutorial.md

@@ -0,0 +1,20 @@
+---
+id: try-tutorial
+title: Interactive Tutorial
+sidebar_label: Interactive Tutorial
+---
+
+Once you've installed StreamPipes and see the home screen, you'll see a number of modules that are part of the StreamPipes toolbox.
+As a first step, you might be interested in taking the interactive tutorial that helps you create your first pipeline.
+Switch to the **Pipeline Editor** and you will see a dialog that asks you for the start of the interactive tutorial:
+
+<img className="docs-image" src="/img/01_try-tutorial/01_tutorial-welcome.png" alt="Tutorial Welcome Page"/>
+
+Click **Start Tour** to start the tour. In this tour, you'll build a simple pipeline that monitors (simulated) live data from a water tank system.
+Within the tour, perform the actions as recommended and click **Next** to trigger the next steps. Some tour steps won't require to select **Next**, but wait for you to take the recommended action.
+You can cancel the tour anytime by clicking the **Exit Tour** button.
+
+<img className="docs-image" src="/img/01_try-tutorial/02_tutorial-process.png" alt="Tutorial Welcome Page"/>
+
+Now after you've built your first pipeline, you might be interested in reading about some of our core [concepts](02_concepts-overview) 
+

docs/02_concepts-overview.md

@@ -0,0 +1,35 @@
+---
+id: concepts-overview
+title: StreamPipes Concepts
+sidebar_label: Overview
+---
+
+To understand how StreamPipes works, the knowledge of a few core concepts, illustrated below, will be helpful.
+
+<img className="docs-image" src="/img/02_concepts-overview/01_overview.png" alt="Overview of concepts"/>
+
+## Adapter
+An adapter connects to any external data source and forwards received events to the internal StreamPipes system. Within StreamPipes, the output of adapters are available in form of the two primary building blocks **Data Set** and **Data Stream**.
+Adapters can be either created by using StreamPipes Connect, a module to easily connect to new data sources directly from the user interface, or by defining an adapter using the provided Software Development Kit (SDK).
+
+## Data Set / Data Stream
+**Data Streams** and **Data Sets** represent the primary source for working with events in StreamPipes.
+A stream is an ordered sequence of events, where an event typically consists of one or more observation values and additional metadata. The "structure" (or schema) of an event provided by a data stream or set is stored in the internal semantic schema registry of StreamPipes.
+While data streams are typically unbounded, data sets have a fixed end and are internally "replayed" by the system from beginning to end once they are used as part of a pipeline.
+As follows, although when referring to data streams, most concepts also apply for data sets.
+
+## Data Processor
+**Data Processors** in StreamPipes transform one or more input data streams into an output data stream.
+Such transformations can be rather simple, e.g. filtering based on a predefined rule or more complex, e.g. applying rule-based or learning-based algorithms on the data.  
+Data Processors can be applied on any data stream that matches the input requirements of a processor. In addition, most processors can be configured by providing user-defined parameters directly in the user interface.
+Processing elements define stream requirements that are a set of minimum properties an incoming event stream must provide. Data processors can keep state or perform stateless operations.
+At runtime, data streams are processed by using one of the underlying runtime wrappers (see the developer guide for more details).
+
+## Data Sink
+**Data Sinks** consume event streams similar to Data Processors, but do not provide an output data stream. As such, data sinks typically perform some action or trigger a visualization as a result of a stream transformation.
+Similar to data processors, sinks also require for the presence of specific input requirements of any bound data stream and can be customized.
+StreamPipes provides several internal data sinks, e.g., to create notifications, visualize live data or persist historical data of incoming streams. In addition, various data sinks are provided to forward data streams to external systems such as databases.
+
+## Pipeline
+A pipeline in Apache StreamPipes describes the transformation process from a data stream to a data sink. Typically, a pipeline consists of at least one data stream (or data set), zero or more data processors and at least one data sink.
+Pipelines are built by users in a graphical way using the **Pipeline Editor** and can be started and stopped at any time.