K8s Spark Enhancements

[peridotml]

A couple enhancements to the k8s spark plugin.

Choosing Machine Types

The k8s paradigm of abstracting away machine types adds additional complexity. Why? We need to know the machine types in order to reasoning through the executor and driver cpu / mem configuration. Getting this wrong can be costly too!

It has become much more common to request specific machine types for spark clusters. Users would need to specify a driver type and a worker type. I imagine that we should be able to generate the k8s spark config automatically.

Spark UI

Understanding the performance of a spark cluster is fairly complex. It requires a combination of logs and looking through the spark UI in real time. It would be great to reach parity with other spark offerings here.

[davidmirror-ops]

2023-11-09 Contributor's meetup notes: this idea needs a champion, otherwise it will have to be closed.

[davidmirror-ops]

I think the Spark operator lets users express Driver/Executor resources but not sure about the "machine types". Without further signals from the community, it will be hard to prioritize this one.
If anyone wants to shepherd this feature, feel free to raise an Issue.