Flink Runner User Interaction

[ranchodeluxe]

Problem

If you're a k8s fan then using pangeo-forge-runner.FlinkOperatorBakery isn't that gross out of the box. But the k8s interface is a hard pill to swallow for those who are less familiar. Plus there is only the terminal UI/UX.

Long-Term Goal

Create a UI/UX experience that approximates what GCP DataFlow does for Apache Beam. Tools such as Argo[ CD || Workflows] should be able to handle most of this (especially the auth and RBAC for workflows) but maybe others have different ideas on how to accommodate this:

Shopping list:

• we need better abstractions and defaults where users will just POST a couple pieces of information (recipe repo urls, branches etc) to an API to kick off a job
• we need a place for users to auth and configure their secrets to use on jobs
• we need better UI/UX about job progression, success, failures
• failures should give users All The Logs 🍾 (including some type of log retention of job managers and task managers for past jobs)
• failures should follow some simple heuristics to infer and tell users why their jobs are failing (think about all the gross JVM memory issues that could happen)
• users should be redirected to their running jobs (possibly only theirs)
• users need authentication which would translate into some form of tagging users to running jobs/pods
• running jobs should provide a way (within some days-long expiry) of allowing users to load the flink job management UI so that can see all the nice things that flink gives feedback about

Short-Term Goal

Just use any CI/CD system such as GH Actions (possibly codify it in GH Actions plugin) and we can skip some of the above requirements and user experience for now

[ranchodeluxe]

The "Short-Term Goal" mentioned above is over here (https://github.com/NASA-IMPACT/veda-pforge-job-runner) and has been an invaluable exercise in understanding user interaction, user needs and where flink/pangeo-forge is feeling flunky