DCNiOS is an open-source command-line tool to easily manage the creation of event-driven data processing flows. DCNiOS, Data Connector through Apache NiFi for OSCAR, facilitates the creation of event-driven processes connecting a Storage System like dCache or Amazon S3 to a scalable OSCAR cluster by employing predefined dataflows that are processed by Apache NiFi.
Apache NiFi is a reliable system to process and distribute data through powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
OSCAR is an open-source platform for serverless event-driven data processing of containerized applications across the computing continuum.
The DCNiOS command-line interface allows you to create a NiFi dataflow though a YAML file (see figure).
After creating a dataflow using DCNiOS, when an event is captured (e.g., Server-sent event by dCache), Apache NiFi stores it in an internal queue and then sends it for processing to an OSCAR cluster at a customized rate. In the OSCAR cluster, a user-defined service creates a job to process the stored data.
Therefore, DCNiOS has been made to interact with NiFi and deploy a complete dataflow. It uses HTTP calls to communicate with a Nifi cluster.
Apache NiFi can be automatically deployed using the Infrastructure Manager (IM) on a dynamically provisioned Kubernetes Cluster. Note that a NiFi registry is not required.
This software has received a silver badge according to the Software Quality Baseline criteria defined by the EOSC-Synergy project.
- OSCAR cluster with services deployed.
- NiFi Cluster deployed.
- A Python distribution such as Anaconda.
Create an environment with conda and activate it.
conda create --name dcnios python=3.7.6
conda activate dcniosInstall all the requirements defined in requirements.txt
pip install -r requeriments.txtInstall the minimum requirements for DCNiOS:
pip install pyyaml==6.0 requests==2.28.2 oscar_python==1.0.3- Germán Moltó mailto:[email protected]
- Estíbaliz Parcero mailto:[email protected]
- Sergio Langarita mailto:[email protected]
Instituto de Instrumentación para Imagen Molecular (I3M), Centro Mixto CSIC — Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, España.
There is only one version in maintenance:
- The main branch in the source code repository maintains a working state version of the software component.
- Documentation is updated with the new software versions involving any substantial or minimal change in the application's behavior. Any issue can be reported in the Issues section of the GitHub project.
- Documentation is updated whenever reported as inaccurate or unclear.
DCNiOS is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.
This work was supported by the project “An interdisciplinary Digital Twin Engine for science’’ (interTwin), which has received funding from the European Union’s Horizon Europe Programme under Grant 101058386.
You can find more information in the OSCAR's blog.