Should coforall intent impact a nested 'on' statement?

[mppf]

In Chapel, the coforall + on pattern is common. For example

  coforall loc in Locales {
    on loc {
      // do something
    }
  }

I realized that I wasn't sure what would happen if you use a with (in x) clause on the coforall in that case. Will it create a local copy of x per locale? It would be convenient if it did; although perhaps that is baking in the behavior of merging coforall+on into nonblocking remote tasks.

The following program demonstrates this pattern and below I show the output. The behavior today is inconsistent:

• with (in x) does in fact cause x to be local on each locale
• BUT an array stored within a record is not copied in this case

Contrast with forall over a block domain, where both the x variable and the inner array are localized to the destination locale.

What should it do?

use BlockDist;

proc main() {
  var x: R;
  x.v = 1;

  writeln("coforall+on");
  coforall loc in Locales with (in x) {
    on loc {
      writeln("I'm on locale ", here.id, " with copy ", x,
              " on locale ", x.locale.id,
              " x.elts on locale ", x.elts[0].locale.id);
    }
  }

  const D = blockDist.createDomain(0..<numLocales); 
  writeln("forall over Block with 1 element per locale");
  forall locid in D with (in x) {
    writeln("I'm on locale ", here.id, " with copy ", x,
            " on locale ", x.locale.id,
            " x.elts on locale ", x.elts[0].locale.id);
  }
}

record R {
  var v: int;
  var elts: [0..10] int;
  proc init() {
    writeln("R.init() on ", here.id);
    v = 42;
  }
  proc init=(const ref rhs: R) {
    writeln("R.init=() on ", here.id, " rhs is on ", rhs.locale.id);
  }
  // unrelated issue: uncommenting this leads to compilation error
  /*operator =(ref lhs, const ref rhs) {
    writeln("R.= on ", here.id, " rhs is on ", rhs.locale.id);
  }*/
}

Compile and run

$ chpl aa.chpl  && ./aa -nl 3

R.init() on 0
coforall+on
R.init=() on 0 rhs is on 0
R.init=() on 0 rhs is on 0
I'm on locale 0 with copy (v = 0, elts = 0 0 0 0 0 0 0 0 0 0 0) on locale 0 x.elts on locale 0
R.init=() on 0 rhs is on 0
I'm on locale 1 with copy (v = 0, elts = 0 0 0 0 0 0 0 0 0 0 0) on locale 1 x.elts on locale 0
I'm on locale 2 with copy (v = 0, elts = 0 0 0 0 0 0 0 0 0 0 0) on locale 2 x.elts on locale 0
forall over Block with 1 element per locale
R.init=() on 0 rhs is on 0
R.init=() on 0 rhs is on 0
R.init=() on 0 rhs is on 0
R.init=() on 2 rhs is on 2
R.init=() on 1 rhs is on 1
R.init=() on 0 rhs is on 0
R.init=() on 2 rhs is on 2
R.init=() on 1 rhs is on 1
I'm on locale 0 with copy (v = 0, elts = 0 0 0 0 0 0 0 0 0 0 0) on locale 0 x.elts on locale 0
I'm on locale 1 with copy (v = 0, elts = 0 0 0 0 0 0 0 0 0 0 0) on locale 1 x.elts on locale 1
I'm on locale 2 with copy (v = 0, elts = 0 0 0 0 0 0 0 0 0 0 0) on locale 2 x.elts on locale 2

[bradcray]

Will it create a local copy of x per locale?

My guess is that, by a strict semantic interpretation of the language, it wouldn't and shouldn't (and that we should introduce on intents to guarantee this, as proposed in #26752 and #18585).

But I could imagine rvf firing, depending on what happens to x within and around the loop. For your example, x seems to be read-only within the loop making it seem like it'd plausibly be firing. Have you checked whether it is or tried disabling it to see whether the behavior is different? [note that I think rvf for a record with an array field should forward the array as well, but am not sure what behavior it might have today].

This also gets into the longstanding sticky question of what .locale on an RVF'd variable should return, or whether querying .locale should disable RVF to avoid confusion.

[mppf]

Using --no-remote-value-forwarding does not change the inconsistency. (Neither does --fast, FWIW).

[bradcray]

@mppf: You've made me really curious about the source of the inconsistency given that the forall loop is implemented in terms of coforall and on... 🤔

[mppf]

I have a pretty good guess why it's inconsistent. I think it's because the forall has two coforalls in the iterator, one for locales, (then the on statement), then a coforall for tasks. Both coforalls will get the in x intent behavior (to implement the forall intent). As a result, the inner coforall will copy x to the current locale.

[bradcray]

That's a great guess—sounds very likely to be the case to me, thanks!