Per PE Quiet interface

[manjugv]

Closes #317

[agrippa]

As a non-implementer, I may need some help with this.

Is there a performance benefit to this routine over maintaining 1 context per PE I communicate with? At the application level that might undesirably lead to memory consumption scaling with # of PEs, so are there tricks in the runtime layer that mean it won't?

Could the options parameter to shmem_ctx_create be used to set a property on contexts (e.g. SHMEM_ONLY_USED_WITH_ONE_PE) that could have a similar effect rather than introducing a new API?

[manjugv]

@agrippa That is definitely one way to do it. As a matter of fact, with that option, one can do more optimizations. However, it is much stronger semantic than the proposal here. The context is creating a communication stream. With your proposed change, you are creating a stream per PE and you have an ability to couple that stream with a thread.

[shamisp]

@manjugv UCX support this, but if you look at the code it comes with the cost. It is not huge but you have to track outstanding messages on the PE. If this semantics is useful for OpenSHMEM users, I'm all for it.

[nspark]

WG feedback: Clarify that the target_pes pointer isn't accessed if npes is zero.

[manjugv]

@jdinan Do you need anything else on this before merging?

[jdinan]

@manjugv I added a few doc editor updates for you to review and apply. I also see that there are unresolved comments on this PR. Please resolve these, e.g. by adding doc edit changes to this PR, filing new issues, new PRs, etc.

[manjugv]

@jdinan I accepted the commit. Thanks.

[jdinan]

@manjugv Could you please resolve the open comments on this ticket before merging? It looks like some of these should have issues created for follow-up before the next spec release.

[manjugv]

@jdinan I have resolved all comments. Thanks.