Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support output strategies in pipeline code #3312

Merged
merged 5 commits into from
Oct 31, 2024

Conversation

dominikriemer
Copy link
Member

@dominikriemer dominikriemer commented Oct 29, 2024

Purpose

Support custom and user-defined output strategies in the pipeline-as-code model.

Example:

id: test
name: Test
description: ""
pipelineElements:
  - type: stream
    ref: 0_6fSh
    id: sp:spdatastream:Kmneup
  - type: processor
    ref: 1_ScpA
    id: org.apache.streampipes.processors.filters.jvm.project
    connectedTo:
      - 0_6fSh
    configuration: []
    output:
      keep:
        - s0::timestamp
        - s0::volume_flow
  - type: processor
    ref: 2_QGAv
    id: org.apache.streampipes.processors.enricher.jvm.jseval
    connectedTo:
      - 1_ScpA
    configuration:
      - jsFunction: >-
          function process(event) {
              // do processing here.
              // return processed event.
             // Type 'event' and press Ctrl+Space to see available fields.
             // Example: 
              return {timestamp: event.timestamp, tempInCelsius: (event.tempInKelvin - 273.15)};
          }
    output:
      userDefined:
        - fieldName: timestamp
          runtimeType: http://www.w3.org/2001/XMLSchema#long
          semanticType: http://schema.org/DateTime
        - fieldName: volume_flow
          runtimeType: http://www.w3.org/2001/XMLSchema#float
          semanticType: http://schema.org/Number
        - fieldName: volume_flow2
          runtimeType: http://www.w3.org/2001/XMLSchema#float
          semanticType: http://schema.org/Number  
  - type: sink
    ref: 3_9Wl9
    id: org.apache.streampipes.sinks.internal.jvm.datalake
    connectedTo:
      - 2_QGAv
    configuration:
      - timestamp_mapping: s0::timestamp
      - db_measurement: asddd2
      - schema_update: Update schema
      - dimensions_selection: []
      - ignore_duplicates: false
createOptions:
  start: false

Remarks

PR introduces (a) breaking change(s): no

PR introduces (a) deprecation(s): no

@github-actions github-actions bot added java Pull requests that update Java code ui Anything that affects the UI backend Everything that is related to the StreamPipes backend labels Oct 29, 2024
@dominikriemer dominikriemer merged commit 7783d88 into dev Oct 31, 2024
22 checks passed
@dominikriemer dominikriemer deleted the support-output-strategies-in-pipeline-code branch October 31, 2024 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Everything that is related to the StreamPipes backend java Pull requests that update Java code ui Anything that affects the UI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants