Add parallel gmres implementation

[quantumsteve]

Please review the developer documentation
on the wiki of this project that contains help and requirements.

Proposed changes

Describe what this PR changes and why. If it closes an issue, link to it here
with a supported keyword.

Haven't tried with GPUs. I think this will be easiest with CUDA-aware MPI.

Kept existing serial implementation to avoid additional copies. We may want to revisit this soon.

What type(s) of changes does this code introduce?

Put an x in the boxes that apply.

• Bugfix
• New feature
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no api changes)
• Build related changes
• Documentation content changes
• Other (please describe):

Does this introduce a breaking change?

• Yes
• No

What systems has this change been tested on?

Only tested on continuity_1 CPU and 4 processes. Need to try more PDEs with different numbers of processes.

Checklist

Put an x in the boxes that apply. You can also fill these out after creating
the PR. If you're unsure about any of them, don't hesitate to ask. This is
simply a reminder of what we are going to look for before merging your code.

• this PR is up to date with current the current state of 'develop'
• code added or changed in the PR has been clang-formatted
• this PR adds tests to cover any new code, or to catch a bug that is being fixed
• documentation has been added (if appropriate)

[mkstoyanov]

The MPI_Comm_split() call causes global communication, see the algorithm description here:

https://www.mpich.org/static/docs/v3.1.3/www3/MPI_Comm_split.html

It's OK to use (and for prototype purposes), but eventually this step should be done only once. You can have a reducer class, where you initialize an object once and you keep the communicator, then you just call MPI_Allreduce() for every iteration.