Add inclusive and exclusive scan (prefix sum) operations #488

nspark · 2022-02-28T17:36:11Z

This PR adds inclusive and exclusive scan (prefix sum) operations as shmem_sum_inscan and shmem_sum_exscan operations. Some comments:

MPI's inclusive scan is just called MPI_Scan.
MPI_Exscan leaves the contents of the destination buffer on rank 0 undefined. The proposed shmem_sum_exscan explicitly zeros the destination buffer on PE 0. (Rationale: MPI_Exscan supports multiple operators; this PR only supports addition.)
C++ uses the "full" names: std::inclusive_scan, std::exclusive_scan
Inclusive scan seems somewhat more common, though I personally envision using exclusive scan more for indexing in gather-like operations.
The example is a bit weak.
I personally want value-based reductions/scans way more than array-based reductions/scans.

nspark · 2022-03-09T21:51:31Z

Has anyone seen a scan operation that generalizes on the order in which entries are accumulated (e.g., bottom-up, as drafted, vs. top-down)?

Alternatively, can an OpenSHMEM team be created that reverses the PE order? For example, is the following valid...?

shmem_team_t world_reversed;
shmem_team_split_strided(SHMEM_TEAM_WORLD, shmem_n_pes() - 1, -1, shmem_n_pes(), NULL, 0, &world_reversed);

AFAICT, there's nothing precluding the creation of such a team. I'd guess a lot of first implementations of OpenSHMEM 1.5 might break on this, though.

jdinan · 2022-03-10T15:07:39Z

I think the MPI answer to this would be to create a communicator that renumbers the MPI processes. I don't think that a negative stride is forbidden by the OpenSHMEM spec, but I'm also not sure whether it's something that we intended to support. The legacy collectives didn't support a negative stride, since the stride argument was treated as 2^(stride). Can anyone remember better than me? @davidozog or @manjugv?

nspark · 2022-03-23T20:00:15Z

Making a note to consider the behavior of scan operations and NaN values (cf. #467); for example:

static double dst = 0;
static double src = 0;
src = (shmem_my_pe() == shmem_n_pes() / 2) ? NAN : shmem_my_pe();
shmem_sum_inscan(SHMEM_TEAM_WORLD, &dst, &src, 1);
if (shmem_my_pe() >= shmem_n_pes())
  assert(isnan(dst));

wrrobin · 2022-03-25T17:21:31Z

Are we keeping the src single element data type for now and add the support for arrays later? Or, is the example above a specific one? For arrays, do we need to provide the indices similar to what C++ has?

content/shmem_scan.tex

jdinan · 2022-05-27T18:00:58Z

content/shmem_scan.tex

+  The \source{} and \dest{} arguments must either be the same
+  symmetric address, or two different symmetric addresses
+  corresponding to buffers that do not overlap in memory. That is,
+  they must be completely overlapping or completely disjoint.


Should we apply the clarifications from #290 here, as well?

Is #290 the right reference here? I don't see how that applies here.

🤦 It was #490. Please incorporate that (minor) change to the reductions text here.

content/shmem_scan.tex

nspark · 2022-11-17T15:10:52Z

Are we keeping the src single element data type for now and add the support for arrays later? Or, is the example above a specific one? For arrays, do we need to provide the indices similar to what C++ has?

@wrrobin This API accepts arrays. The example just happens to only use a single element.

kwaters4 · 2024-07-11T17:15:23Z

@naveen-rn Add me to this issue

jdinan · 2024-08-15T17:16:30Z

content/shmem_scan.tex

+%% C11
+\begin{C11synopsis}
+int @\FuncDecl{shmem\_sum\_inscan}@(shmem_team_t team, TYPE *dest, const TYPE *source, size_t nreduce);
+int @\FuncDecl{shmem\_sum\_exscan}@(shmem_team_t team, TYPE *dest, const TYPE *source, size_t nreduce);


Should the last parameter be named something like nscan or nelem instead of nreduce?

BKP · 2024-08-15T17:31:00Z

example_code/shmem_scan_example.c

+int collect_at(shmem_team_t team, void *dest, const void *source, size_t nbytes, int who) {
+  static size_t sym_nbytes;
+  sym_nbytes = nbytes;
+  shmem_team_sync(team);


Can we get rid of this sync if the src and dest buffers are different and dest is statically initialized?
If not, we may need to address this case in the section matter above.

BKP · 2024-08-15T17:34:52Z

content/shmem_scan.tex

+  \LibConstRef{SHMEM\_TEAM\_INVALID} or is otherwise invalid, the
+  behavior is undefined.
+
+  Before any \ac{PE} calls a scan routine, the \dest{} array on all


In response to the message in the example section below:
By omission, this is saying that the src arrays on all pes does not need to be ready.

jdinan · 2024-08-15T17:39:18Z

content/shmem_scan.tex

+  }
+  \apiargument{OUT}{dest}{
+    Symmetric address of an array, of length \VAR{nreduce} elements,
+    to receive the result of the scan routines.  The type of


Suggested change

to receive the result of the scan routines. The type of

to receive the result of the scan operations. The type of

jdinan · 2024-08-15T17:39:34Z

content/shmem_scan.tex

+  }
+  \apiargument{IN}{source}{
+    Symmetric address of an array, of length \VAR{nreduce} elements,
+    that contains one element for each separate scan routine.


Suggested change

that contains one element for each separate scan routine.

that contains one element for each separate scan operation.

jdinan · 2024-08-15T17:42:41Z

content/shmem_scan.tex

+
+  Before any \ac{PE} calls a scan routine, the \dest{} array on all
+  \acp{PE} participating in the operation must be ready to accept the
+  results of the operation.  Otherwise, the behavior is undefined.


@davidozog Proposed text for the collectives section committee. We would add it here and to the other collectives:

The \source{} buffer at the local \ac{PE} must be ready to be read by any PE in the team. The application does not need to synchronize to ensure that the \source{} buffer is ready across all \acp{PE} prior to calling this routine.

Please feel free to bikeshed this and improve the text. :)

jdinan · 2024-08-23T19:27:55Z

Collectives section committee -- there are several minor wording changes mentioned on this PR. Please incorporate those changes as section edits.

Initial shmem_scan API

52375b3

davidozog mentioned this pull request Mar 25, 2022

Team split ops do not support negative strides Sandia-OpenSHMEM/SOS#1050

Closed

nspark mentioned this pull request Mar 25, 2022

Should team strides be positive? #491

Closed

jdinan added HadReading NeedsChangeLogEntry labels May 27, 2022

jdinan added this to the OpenSHMEM 1.6 milestone May 27, 2022

jdinan reviewed May 27, 2022

View reviewed changes

content/shmem_scan.tex Show resolved Hide resolved

jdinan reviewed May 27, 2022

View reviewed changes

nspark commented May 27, 2022

View reviewed changes

content/shmem_scan.tex Outdated Show resolved Hide resolved

Fix typo in shmem_scan

8d65b65

jdinan added the blocker label Feb 29, 2024

naveen-rn assigned BKP May 30, 2024

jdinan reviewed Aug 15, 2024

View reviewed changes

BKP reviewed Aug 15, 2024

View reviewed changes

jdinan reviewed Aug 15, 2024

View reviewed changes

jdinan added PendingBallot Ratified and removed PendingBallot blocker labels Aug 23, 2024

jdinan merged commit 1d6f40e into openshmem-org:master Aug 23, 2024

This was referenced Aug 29, 2024

scan: 488 section committee edits (nelems/overlap) #526

Closed

scan: 488 section committee edits (nelems/overlap) davidozog/openshmem-specification#9

Merged

collectives: clarify src buffer entry requirements davidozog/openshmem-specification#10

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add inclusive and exclusive scan (prefix sum) operations #488

Add inclusive and exclusive scan (prefix sum) operations #488

nspark commented Feb 28, 2022 •

edited

Loading

nspark commented Mar 9, 2022

jdinan commented Mar 10, 2022

nspark commented Mar 23, 2022

wrrobin commented Mar 25, 2022

jdinan May 27, 2022

nspark Nov 17, 2022

jdinan Aug 15, 2024

nspark commented Nov 17, 2022

kwaters4 commented Jul 11, 2024

jdinan Aug 15, 2024 •

edited

Loading

BKP Aug 15, 2024 •

edited

Loading

BKP Aug 15, 2024

jdinan Aug 15, 2024

jdinan Aug 15, 2024

jdinan Aug 15, 2024 •

edited

Loading

jdinan Aug 15, 2024

jdinan commented Aug 23, 2024

	to receive the result of the scan routines. The type of
	to receive the result of the scan operations. The type of

	that contains one element for each separate scan routine.
	that contains one element for each separate scan operation.

Add inclusive and exclusive scan (prefix sum) operations #488

Add inclusive and exclusive scan (prefix sum) operations #488

Conversation

nspark commented Feb 28, 2022 • edited Loading

nspark commented Mar 9, 2022

jdinan commented Mar 10, 2022

nspark commented Mar 23, 2022

wrrobin commented Mar 25, 2022

jdinan May 27, 2022

Choose a reason for hiding this comment

nspark Nov 17, 2022

Choose a reason for hiding this comment

jdinan Aug 15, 2024

Choose a reason for hiding this comment

nspark commented Nov 17, 2022

kwaters4 commented Jul 11, 2024

jdinan Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

BKP Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

BKP Aug 15, 2024

Choose a reason for hiding this comment

jdinan Aug 15, 2024

Choose a reason for hiding this comment

jdinan Aug 15, 2024

Choose a reason for hiding this comment

jdinan Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

jdinan Aug 15, 2024

Choose a reason for hiding this comment

jdinan commented Aug 23, 2024

nspark commented Feb 28, 2022 •

edited

Loading

jdinan Aug 15, 2024 •

edited

Loading

BKP Aug 15, 2024 •

edited

Loading

jdinan Aug 15, 2024 •

edited

Loading