IAWG: First pass at publish_lookup chapter

[dsolt]

IAWG's first pass at changes to the Publish/Lookup chapter. This is a WIP, but far enough along that people may be interested in seeing what we are working on. Commits are not yet signed-off.

[rhc54]

I haven't been able to look at the figure, but I suspect this opens up a real can of worms. The problem is that the "namespace" range is encompassed within the "session" range, and so now (as you point out later in the text) there is ambiguity as to which value of "tree" is intended. I don't think we want to make publish/lookup that complex.

Keep in mind that this is a rarely used feature. It's intent is not to support detailed database-like operations, but rather to assist in async rendezvous or coordination operations. Thus, we don't need complex store/retrieve rules like you see with SQL.

We could just require that all keys be globally unique, but that could cause some difficulty for app-to-app rendezvous (e.g., "ocean" trying to rendezvous with "air"). Attempting to prefix keys with user or job IDs helps, but it still imposes a requirement that the other app know the information used to prefix the key - and that might not always be doable.

Perhaps the best option is to just require that you specify the range from which the data is to be retrieved. Thus, if you want the key value that was published on the "session" range, you have to ask for the "session" range. If you want the value published on the "namespace" range, then you have to ask for that range. If you ask for the value published on the "namespace" range and it wasn't published there (e.g., it was published only on the "session" range), then you get "not found".

Or something like that - my point being that this sounds too complex and is likely more than what we really need or want to support. We could even just eliminate the ability to publish the same key on different ranges - I have no issue with doing so, and I can't think of anyone currently using this API who would be impacted by it.

[dsolt]

I agree. I was just trying to capture what I thought the document says currently, but I agree it is very complex to understand and to implement and it still has non-determinant cases (results that may vary based on implementation). I'm all for making something that is as simple as possible both to understand and implement, even if that means it is not very rich in features. Like you said, this is just meant to be a simple rendezvous mechanism. I need to think more about what options we have and what we could do without breaking backward compatibility. This is one of those cases where I don't think anyone is invested in the way things "work" now, but not sure how much liberty we should take on changing behavior.

On another note, I would love to see the Publish API take 2 info arrays... one for the things to publish and one for modifiers to the call. As it is now if I call Publish with an array of infos like:

PMIX_RANGE=PMIX_RANGE_NAMESPACE
tree=oak
PMIX_RANGE=PMIX_RANGE_GLOBAL
tree=elm
PMIX_TIMEOUT=10

then we have several problems: If the implementation doesn't handle or recognize PMIX_TIMEOUT then it will actually publish PMIX_TIMEOUT as a key since it doesn't recognize it as a directive. Also, will it publish tree=oak at the namespace range and tree=elm at the Global range or will it publish only one of the values of tree at some range (either namespace or global)? Things get really tricky when the things to publish are combined with the directives.

But deprecating existing API's for new Publish API's seems like a lot of work for something that very few products are using and probably have figured out a way to not get tripped up by these things already. I don't want the pmix-standard to be a burden to implementations or progress, but I would like it to describe how things actually work so someone can develop new clients without getting lost or confused. I'll see if I can think up some way to simplify what is supported without breaking backward compatibility.

[rhc54]

So don't deprecate the API, but add an attribute for specifying modifiers to the call. It would take a pmix_data_array_t containing an array of pmix_info_t that has the modifiers in them. I don't think that would be much of a problem for backwards compatibility.

If you prefer, you could add an attribute for the things to publish, leaving all else in the call to be modifiers. Either way should be fine.

I wouldn't worry too much about changes in behavior at this point. OMPI is the primary user of it (for connect/accept), and I've already outlined a way for them to use PMIx_Group instead - which is a far better and more performant solution for that use-case.

[dsolt]

I like that idea. We could have both approaches for a while, then move towards deprecating the old way after a while.

[rhc54]

Thinking some more, we should probably add the attribute for the things to publish (as opposed to the modifiers) so we remain consistent with the other APIs - whose info arrays all contain modifiers.

[rhc54]

Yeah, I think we have to preclude this possibility - it should be an error.

[naughtont3]

Suggested change

	ranges. Custom ranges are consider different if they have different members.
	ranges. Custom ranges are considered different if they have different members.

[naughtont3]

Suggested change

	function, so the receiver must copy/protect the data prior to returning if it needs to be retain ed.
	function, so the receiver must copy/protect the data prior to returning if it needs to be retained.

[dsolt]

Please use emoji reactions ON THIS COMMENT to indicate your position on this proposal.

You do not need to vote on every proposal
If you have no opinion, don't vote - that is also useful data
If you've already commented on this issue, please still vote so
we know your current thoughts
Not all proposals solve exactly the same problem, so we may end
up accepting proposals that appear to have some overlap
This is not a binding majority-rule vote, but it will be a very
significant input into the corresponding ASC decision.

Here are the meanings for the emojis:

Hooray or Rocket: I support this so strongly that I
want to be an advocate for it
Heart: I think this is an ideal solution
Thumbs up: I'd be happy with this solution
Confused: I'd rather we not do this, but I can tolerate it
Thumbs down: I'd be actively unhappy, and may even consider
other technologies instead
If you want to explain in more detail, feel free to add another
comment, but please also vote on this comment.

[jjhursey]

@dsolt Are all of the comments in this PR resolved? (If so then can you mark them as resolved?)

[abouteiller]

Why not just making this undefined behavior/incorrect code? This doesn't look useful at first glance.

[rhc54]

I'm actually not sure that statement is correct - the range determines who can access it. I'm not sure why the publisher would need to be a member of that group. The publisher always "owns" the data and can "unpublish" it.

[jjhursey]

I remember @dsolt had some slides on this that were presented in the Q2 meeting. From the notes I have the following

publish/lookup supports multiple ranges

validity of publishing same key twice on the same range (WG will look more into it, it is believed to be illegal code with undefined behavior)
Group prefers solution where LOOKUP queries only the range it is given (i.e., to be visible, the PUBLISH must target the same range). No ‘inheritance’ of published items. The same key may be published on different range

So I think this was targeted at addressing those comments, but my memory is failing me on the details. We will discuss this more today.

[dsolt]

Good catch. At one point in our revision process we had determined that the publisher and process calling Lookup both had to be part of the group, but we decided to remove that restriction and allow the publisher to not be part of the group of processes, so this should have been removed when we made that switch.

[abouteiller]

Reading this text I stumbled again on the question of wether ranges are "nested" or not. The new text in PMIx_Lookup below makes it very clear that it is not, but some forewarning here (or a ref to the lookup rules \ref{chap:pub:retrules}.) could help clarify.

[dsolt]

I'll leave this one for discussion during the reading.

[jjhursey]

Per IAWG on Aug. 15, 2022:

• We should make clean that though the ranges are overlapping in definition, they are not nested. So ranges must match.
• We should add the forward reference in the top paragraph of this section to the lookup section where this is more clearly noted.

[abouteiller]

This seems to contradict the rule from #398 (comment)

[dsolt]

Yes, this is correct. The other location is wrong.

[jjhursey]

For the meeting today, attached is a PDF rendering of the current state of this PR.

• pmix-standard-3q-2022-pr-398.pdf

[jjhursey]

Suggested change

	publishing to a \refconst{PMIX_RANGE_CUSTOM}
	Publishing to a \refconst{PMIX_RANGE_CUSTOM}

[dsolt]

now a nop

[rhc54]

I worked thru the chapter and made some recommended changes Ralph

…

On Aug 9, 2022, at 7:31 AM, Josh Hursey ***@***.***> wrote: @jjhursey commented on this pull request. In Chap_API_Publish.tex <#398 (comment)>: > \refconst{PMIX_ERR_DUPLICATE_KEY} error. - +publishing to a \refconst{PMIX_RANGE_CUSTOM} ⬇️ Suggested change -publishing to a \refconst{PMIX_RANGE_CUSTOM} +Publishing to a \refconst{PMIX_RANGE_CUSTOM} — Reply to this email directly, view it on GitHub <#398 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAST5YECWDB5OWW3W4LWDL3VYJTVFANCNFSM5SQEB4SA>. You are receiving this because you commented.

[jjhursey]

Suggested change

	the implementation should
	distinguishing between info array elements that specify keys and directives as follows:
	the implementation should
	distinguish between info array elements that specify keys and directives as follows:

[dsolt]

done

[jjhursey]

Suggested change

	\item The requestor must be a member of the publisher's range.
	\item The requestor must be a member of the range specified by the publisher.

[jjhursey]

Is this required (e.g., requestor specifies 'node' range, and publisher specifies 'global' range)? Or is it enough that the requestor be part of the publisher's range?

[dsolt]

The new model is that the ranges must match... if you publish to 'global', then the requestor must specifically lookup on 'global' range.

[jjhursey]

Suggested change

	should it be assumed to be data to be published.
	should be assumed to be data to be published.

[jjhursey]

Note from 3Q 2022 ASC (Aug. 11, 2022):

• Let's require PMIX_DATA_TO_PUBLISH as the only way to publish data.
  • This has some backward compatibility issues.
    • Open MPI and (maybe) MPICH would be impacted
  • We should add an error if PMIX_DATA_TO_PUBLISH is not provided in the publish call.
  • In v5, mark not using PMIX_DATA_TO_PUBLISH as deprecated.
• Should we just create a new API with an additional required parameter for the data_to_publish (what to name it? PMIx_Publish_data?)
  • Then we just deprecate the old API instead of adding a new attribute to work around it.

pmix_status_t PMIx_Publish_NEW(const pmix_info_t info_to_publish[], size_t ninfos,
                               const pmix_info_t info_directives[], size_t ndirs);

[jjhursey]

Per IAWG on Aug. 15, 2022:

• Add new PMIx_Publish2 and pmix_server_publish2_fn_t with two sets of pmix_info_ts
• Deprecate the PMIx_Publish API and corresponding pmix_server_publish_fn_t API.
• Do not add the PMIX_DATA_TO_PUBLISH attribute. Instead, prefer the new API.
• In the Revision appendix note, PMIx_Publish was deprecated and replaced by PMIx_Publish2 (same with server API)
• No longer an issue on the ordering of info data and directives (this approach avoids this whole issue)

[jjhursey]

Note from 3Q 2022 (Aug. 11, 2022)

• What does it mean for two PMIX_RANGE_CUSTOMs to match? Is it a strict ordering or is a subset fine?
• Is the publisher required to be part of the range?
  • Publisher does need to be a member of the range?
  • "If I publish that data I own that data and can look it up so that I can unpublish"
    • Current description would prevent the lookup but not the unpublish.
    • Owner should always be able to look it up even if not in the range. Maybe need a clause here? What about overlapping keys?
• Publish means "put some data for a set of consumers to lookup". The consumers might not know who published the data. For a custom range, those looking up might not know the full group. Alternatively, the lookup could specify a list including the publisher.

[jjhursey]

Per IAWG on Aug. 15, 2022:

• Does the publisher have to be part of the range it publishes to?
  • The publisher does not need to be a member of the range.
  • The publisher is the owner of the data, so it is the only one able to unpublish the data, but must specify the range where that data was published.
  • The publisher is the owner of the data but cannot lookup the data if it is not part of the range in which it was pushed.
• If I'm looking up from a PMIX_RANGE_CUSTOM does it have to match what the publisher provided?
  • PMIX_RANGE_NAMESPECE (example)
    • Publish: means that the data is to be available to a set of lookup'ers in my namespace.
    • Lookup: means that I want data provided by a publisher in my namespace.
  • PMIX_RANGE_CUSTOM
    • Publish: 'custom' means I want the data to be available to this set of lookup'ers. Might not include the publisher.
    • Lookup: 'custom' means I want data provided by one of these publishers. Does not need to know all of the processes that have access to the data - only who is providing the data.
  • Generally PMIX_RANGE_ means:
    • Publish: Define who can look up the data
    • Lookup: Define who published the data
• Does the range have to match on the publish and lookup?
  • I'm publishing to range X, and I'm looking up on range X.
  • PMIX_RANGE_CUSTOM is the only one that takes an argument list of processes. It's the only one where the publisher and lookup'er might be using the same value associated with the PMIX_RANGE_CUSTOM key.
• What if the publisher publishes the same key (different values) to different custom ranges that both include the client looking up the key? Or two different publishers both publish the same key (different values) that include the lookup client in different PMIX_RANGE_CUSTOMs.
  • This is not allowed.
  • The Server should reject the lookup operation as conflicting entries. The client should then change their custom range to include one
• What If I want to lookup a value published by a process outside of my namespace?
  • Publisher will use PMIX_RANGE_SESSION or PMIX_RANGE_CUSTOM (if they know the lookup'er)
  • Lookup'er will user PMIX_RANGE_CUSTOM with the publisher's name
• Future work: Maybe extend PMIX_RANGE_NAMESPACE with an argument of which namespace. Currently, it is the same as the caller.

[jjhursey]

Suggested change

	process is known in advance, for example, two namespaces that do not share a child-parent relationship.
	process is known in advance (e.g., two namespaces that do not share a child-parent relationship).

[jjhursey]

Note from 3Q 2022 (Aug. 11, 2022)

• Old text was intended. That the published data is intended solely for the host environment not for application/tool processes.
  • Suggest leave the old text
• PMIX_RANGE_GLOBAL : Anyone under that RM instance (basically what the new text is saying)

[jjhursey]

3Q 2022:

• Reading turned up a few items that need further discussion.
• Will follow up in the IAWG to file tune the last few items.
• Intention to bring this back for presentation during the next quarterly meeting.

[dsolt]

Here is the latest draft with the changes from the first reading (i.e. introduce a PMIx_Publish2, get rid of PMIX_DATA_TO_PUBLISH, and specify that the range of lookup is used to determine eligible publishers).

Will update the PR soon after WG has had the chance to review.

pmix-standard.pdf

[jjhursey]

PMIx ASC 4Q 2022 Meeting presented a straw poll regarding:

Advice to the working group on how best to proceed with the Lookup semantics related to the range specified by the lookup and the scenario where there are duplicate keys published in separate ranges.

Three options were presented:

• Option 1: Change the lookup range to specify where the data was published, not by whom it was published.
• Option 2: If no range is specified in the lookup and multiple matches for a key exist in different ranges, then return a specific status code (e.g., PMIX_MULTIPLE_MATCH) and the full set of matching keys with associated range information.
• Option 3: Do not allow publishing on ranges other than the publisher's namespace or global. Lookup would specify one or the other range.
• Other

Sorted by choice preference

Option	First Choice	Second Choice	Third Choice	No	Abstain
Option 2	6	3	2	1	0
Option 1	1	8	1	1	1
Option 3	3	0	4	4	1
Other	3	0	0	1	8

Other comments:

• "I'd like to present a modified form of Option 2 that might provide an acceptable landing spot."
• "Seemed like Option-3 required users to encode into key to make it functional, so that is why I abstained on that item, b/c I was not entirely sure that was satisfactory option."
• "Use the approach given in the current PR which requires the lookup process to know the namespace of the publisher."
• "If options 1 and 2 are compatible, then I think the best case is to do both."

[jjhursey]

PMIx ASC 4Q 2022 day 2 meeting discussed this further and outlined a path forward. See the notes below

• https://github.com/pmix/pmix-standard/wiki/ASC-Q4-2022-Meeting#day-2-oct-27-2022

[jjhursey]

Notes from IAWG meeting Oct. 31, 2022

• Introduce new APIs for Publish/Lookup/Unpublish (suffix with _data / _ds / _datastore)
  • Leave old API as they are with all ambiguities
• The same key can be published by different or the same publisher into the datastore
• Mechanism for lookup
  • Filter by access rules
  • Filter by source rules
  • Filter to find key
  • More than one value may be returned as an array of values (with qualifiers)
• Data store unique tuple
  • <key, value, publisher, access permissions, timestamp, handler id>
• Publish
  • Put a key/value pair into the datastore
  • Access permissions: Can specify the range of recipients that can access it (range)
    • Maybe we have a default?
  • Publisher can publish the same key multiple times - accumulates in the data store (duplicates allowed)
    • If a publisher publishes the same thing 10 times, then the lookup sees all 10 values
    • FUTURE WORK: Could add a 'REPLACE' attribute to replace given the handle?
  • Handed back a handler/id to reference when unpublishing
  • No limitations beyond this
• Lookup
  • Ask for a 'key'
  • Filtered by access rules defined by the publisher
  • Source rules: Can specify the range of publishers from which to get the data
  • Since multiple matches can happen
    • If multiple matches for a 'key,' then you get multiple values back with qualifiers
    • Always get back an array - even if it is an array of one item
    • Qualifiers: publisher, timestamp, access permissions, others?
• Unpublish
  • Only the original publisher can unpublish
  • Pass the handle from publishing to reference the data that was published
    • FUTURE WORK: Maybe add a qualifier to unpublish anything from me with this key (may match more than one)
  • Add 'access permissions'/range to modify the access permissions?

[dsolt]

List of items to deal with:

• Define PMIX_PUBLISH_IDLEN.
• Do we want something like PMIX_PUBLISH_ID_ALL_MINE (like PMIX_PUBLISH_ID_ALL, but only for caller)
• Create API to translate a publish-id to a proc + epoch number
• macro's for creating/freeing pmix_pdsdata_t's
• Discuss how lookup values are released freed (these are arrays of values for each key and they are allocated by the system)
• lookup blocking and non-blocking are so different. Should we separate keys from return values in the blocking call so it looks more like the non-blocking call.
• Define PMIX_PDSDATA type (pmix_data_type_t) to describe the type PMIx_pdsdata_t
• Define PMIX_PUBLISH_ID type (pmix_data_type_t) to describe the type PMIx_publish_id_t

[dsolt]

I added a new commit for:

• lookup blocking and non-blocking are so different. Should we separate keys from return values in the blocking call so it looks more like the non-blocking call.

[jjhursey]

PMIx ASC 1Q 2023 Pleanary Notes (see Notes Day 1 for more details)

• Noted some use cases
  • Rendevous with failed Publisher processes
    • Publisher A publishes 'find_me_key'
    • Publisher A dies
    • Publisher B takes over Publisher A's role
    • Publisher B unpublishes 'find_me_key'
    • Publisher B publishes 'find_me_key'
  • Published key for the first consumer
    • Publisher A publishes 'for_you'
    • Consumer Z looks up 'for_you'
    • Consumer Z unpublishes 'for_you'
      • Make it automatic on first lookup - so only 1 gets it
    • Publisher and Consumer must have the same uid/gid (owner)
  • Lookup should have a sense of consistency
    • Publisher A publishes 'key'
    • Consumer Z lookup 'key'
    • Publisher B publishes 'key'
    • Consumer Y lookup 'key'
    • Consumers must see at least Publisher A's key
• The IAWG will need to have a reference implementation before voting.
  • Once semantics are ironed out then someone will need to work on the implementation
• Is the old publish compatible with the new publish_datastore?
  • No, they would be separate storages and not share information. This is for the simplicity of the RM implementation and the semantics from the client side.
• On lookup, how do you know which scope the value was published in?
  • Add a scope attribute on the lookup to define the scope. There will be a default.
  • You only get values exactly at that specified level.
  • Scopes are not nested. If not published at the 'session' level you won't see it even if it was published at the 'namespace' level.
• Next Step
  • Finish the text for the presentation in the next quarterly
  • Discussion of implementation