Communicating gradients: Necessary?

[inducer]

@majosm shared this view of what's being communicated in a specific computation (help?) by mirgecom:

https://gist.githubusercontent.com/majosm/4e847cd82a5dc9918e6c8f2984cb0c76/raw/87d41902543dde4deee39dcda127b0bd82b030b1/gistfile1.txt

It seems that values of the gradient in all three directions are being communicated, when conceivably just the dot product with the normal might suffice, by making use of the linearity of the fluxes.

[MTCam]

This warrants a more detailed look, but a few of things to keep in mind:

0. Potential for communication savings is there, but we'd need to do quite a bit of surgery to fix up the operators, fluxing routines, and the interfaces/logic of their interactions. To my knowledge, we don't actually have a set of profiling data that indicates that such a change would be a win for us.
1. Currently the gradient is exchanged multiple times unnecessarily. I suggest we eliminate the unnecessary comm before we evaluate how much this suggested change would buy us.
2. Gradient cross-terms appear in the viscous stress tensor - will need to be computed and dotted with the normal before exchange (we don't currently exchanged the viscous stress tensor, we compute it rank-locally).
3. Vector transport fluxes are needed for both (+/-) sides of the interface for some numerical flux treatments. Even though in the end we use only the normal component, the (+/-) vector fluxes are used to form auxiliary quantities.
4. Physical AV smoothness field can (and does) depend on grad(CV). This is needed to compute the physical viscosity when it is turned on - just need to be aware of this requirement.

None of these items are show-stoppers; just things that come to mind when considering this change.