Pipeline State Synopsis
The POET Pipeline documents the execution of a pipeline run in an object that is rendered in JSON format. This structured data, known as the "PipelineState", is a machine readable "synopsis" with several uses related to flow, monitoring, and analysis.
During the pipeline run, the data is written into a temporary file space that is referenced by a path stored in an environment variable called PIPELINE_STATE_JSON.
Near the end of the run, the data is sent to the InfluxDB storage for use in dashboards and for historical reference. After that, it's deleted from the temporary file space.
- Use the state of previous steps to influence following steps.
- Any pipeline step may query the current "PipelineState" for data that can be used in the current step.
- Monitor pipeline execution.
- Data extracted from the "PipelineState" can be used to determine health of the execution
- Report pipeline data
- Data can be sent to dashboard, analysis tools, and archived for later reference.
The PipelineState data can be visualized this (with some data elided for clarity):
The pipeline contains a collection of steps, which in turn contains a collection of commands. Each element records when it was executed and for how long.
The full schema may be found at pipeline-state-schema.yml