Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try Flink on HPC #128

Open
cisaacstern opened this issue Nov 6, 2023 · 2 comments
Open

Try Flink on HPC #128

cisaacstern opened this issue Nov 6, 2023 · 2 comments

Comments

@cisaacstern
Copy link
Member

With #114, we should be able to run Pangeo Forge on Flink on HPC. This may not be quite as simple as an eventual Dask deployment, but #109 is realistically a longer-term project, so Flink is likely our best option for now.

Tagging @negin513 and @TomNicholas as possibly interested in trying this out at NCAR. If either of you do have interest in this, #124 should be helpful once it goes in.

@yuvipanda
Copy link
Collaborator

Note that I don't think we should try flink on HPC, described as more traditional slurm-like setup.

I remember @negin513 mentioning they may have a kubernetes cluster - so this should be about trying flink on an on prem kubernetes cluster, rather than any HPC system. The on-prem cluster may be run by a HPC center though :)

@cisaacstern
Copy link
Member Author

Important clarification, thanks @yuvipanda!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants