An event drive, asynchronous job scheduling system that is Kubernetes native.
Original use case is to drive data pipeline execution and on-the-fly dependency resolution (DAGS) via configuration, rather than through static dags that are linked to a particular programming language

TODO:

• Create CLI Kaly

• Refactor to use rabbit MQ

• Include jupyerlab and sample notebooks for people to poke around

• Docs

• Abstract out IDP

• Add mongodb image

• Terraform stack for cloud assets

• Integrate kubernetes with IdP

• Secrets injection and storage mechanism

• Demo pipelines highlighting features

  • Dynamic pipeline generation
  • Flexible declaration of downstream dependencies - safe deletes and updates (via feature toggling)
  • Inline secrets declaration (interpolation by CLI when writing pipelines)
  • Observability and autoscaling
  • Integrated pipeline development
  • Integrated model development

• Addendums for productionalization

• Switch from pipeline to job nomenclature

• Gettings started dev