Support leader election mechanism for Spark operator HA

[tkanng]

Hi, this PR adds support for leader election mechanism for Spark operator HA by utilizing leaderelection, according to #458.

[tkanng]

I'm not sure this implementation is good enough, for the reason that leaderelection is an alpha API. So please help me check it again. Thanks!

[liyinan926]

@tkanng thanks for working on this! I have the same concern that the leader election API is alpha and may change significantly in the future. I don't think the master branch is a good place for this change. A new branch forked from the master seems a better home for this PR and gives people a way to build and test this.

[michalzxc]

I have high hopes for this. Right now if you will delete spark operator pod, all spark jobs will remain broken until you will edit them and remove "status" section.
Deploying new version, spot instance termination, any operation what makes spark operator pod recreated breaks all sparkapplications

[tkanng]

@liyinan926 Thanks for your comments! I'll close this pr soon. Would it be great to add a section to design.md or developer-guide.md about HA support? If so, I'd like to do it.

[liyinan926]

@tkanng yes, a section in design.md or developer-guide.md sounds great.

[tkanng]

close it, via #522 .
Thank you for your patience!.