[WIP] [Newbie] [RFC] Flyte Local Cluster

[Future-Outlier]

Tracking issue

https://github.com/flyteorg/flyte/issues/

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (81ce71e) 58.11% compared to head (2c08a8d) 58.12%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4657   +/-   ##
=======================================
  Coverage   58.11%   58.12%           
=======================================
  Files         626      626           
  Lines       53815    53815           
=======================================
+ Hits        31277    31282    +5     
+ Misses      20036    20033    -3     
+ Partials     2502     2500    -2     

Flag	Coverage Δ
unittests	58.12% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hamersaw]

@Future-Outlier this is great, we have had more than a few internal discussions on this deployment model. Would love to help shepherd this as much as possible. A few things to note here:
(1) Change runtime from Kubernetes to Process - we already have an internal application that does this using a separate Flyte plugin to execute using local processes. Perhaps this is something we can talk about open-sourcing for this model.
(2) A large component (perhaps the largest) that is missing here is the use of etcd to store FlyteWorkflow CRs that track execution. FlyteAdmin creates these, and using the watch API FlytePropeller picks them up and then modifies them to persistently track workflow execution status'. In this scenario, it warrants some discussion about what local execution here means for this construct . For example, are executions persisted across the local cluster restart? etc.

[Future-Outlier]

@Future-Outlier this is great, we have had more than a few internal discussions on this deployment model. Would love to help shepherd this as much as possible. A few things to note here:

(1) Change runtime from Kubernetes to Process - we already have an internal application that does this using a separate Flyte plugin to execute using local processes. Perhaps this is something we can talk about open-sourcing for this model.

(2) A large component (perhaps the largest) that is missing here is the use of etcd to store FlyteWorkflow CRs that track execution. FlyteAdmin creates these, and using the watch API FlytePropeller picks them up and then modifies them to persistently track workflow execution status'. In this scenario, it warrants some discussion about what local execution here means for this construct . For example, are executions persisted across the local cluster restart? etc.

Thank you really much, I will discuss more with @pingsutw and update the rfc here!

[Future-Outlier]

etcd information cache -> local file system