Discussion: Broadcasting

[orthoxerox]

It's a different take on the same issue as dotnet/roslyn#8022. While there I tried to use the syntax from Idris, where values are marked to be lifted, here I propose to use the syntax from Julia, where it's operations that are explicitly lifted.

First, some unambiguous syntax that is supposed to work:

int[] a = ...;
int[] b = ...;

var c = a .+ b; //Elementwise addition
//var c = a.Zip(b, (a_, b_) => a_ + b_);

var d = a .* 2; //Elementwise doubling
//var d = a.Select(a_ => a_ * 2);
var e = Math.Sign.(b); //Elementwise static method call
//var e = b.Select(b_ => Math.Sign(b_));

string[] f = ...;

var g = f..Length; //Elementwise property access
//var g = f.Select(f_ => f_.Length);
var h = f.[0]; //Elementwise indexer access
//var h = f.Select(f_ => f_[0]);

Action<int>[] i = ...;

var j = i.(a); //Elementwise delegate call
//var j = i.Zip(a, (i_, a_) => i_(a_));

Open issue 1: method calls with parameters

And now some syntax that I am not so sure about:

var k = f..ToLower();
//var k = f.Select(f_ => f_.ToLower());

var l = f..Substring.(a);
//var l = f.Zip(a, (f_, a_) => f_.Substring(a_));

var m = f..Substring.(a, b);
//var m = f.Zip(a.Zip(b, (a_, b_) => (a_, b_)), (f_, (a_, b_)) => f_.Substring(a_, b_));

var n = f..ToLower().[0];
//var n = f.Select(f_ => f_.ToLower()).Select(f_ => f_[0]);

Invocation of parameterless instance methods looks benign enough and meshes well with property access. But when you add parameters, it starts to looks ungainly. First, there are two broadcasting dots for a single operation. Expressions for l and m look a lot like the expression for n, but have different semantics. I am not sure if I can get rid of one of the dots.

f..Substring(a) could mean "broadcast both the receiver and the arguments", but then it would conflict with the way static methods are broadcast.

On the other hand, f.Substring.(a) could also mean the same thing, but then parameterless methods would also have to look that way, f.ToString.(), making them look different from properties/fields. This would also make supporting IntelliSense impossible.

Open issue 2: how to bind

Julia supports broadcasting only for arrays. I want C# to support it for all types that have Select and Zip, i.e., LINQable types. It's obvious that it won't be possible to do the binding the LINQ way, 100% syntactically. There will have to be a lowering step that has access to the types of the bound subnodes and rewrites them into Select and Zip. But what happens when there are two types that are incompatible?

int[] a = ...;
Option<int> b = ...;

var c = a .+ b;
//The naive approach can result in:
//var c = a.Select(a_ => b.Select(b_ => a_ + b_));
//c is IEnumerable<Option<int>>

One solution would be to forbid such operations. But then this won't work:

Option<int>[] a = ...;
Option<int> b = ...;

var c = a .- b; //Or is it .+ at all?

It's obvious that we want to subtract b from each element of a, so it's supposed to compile into this:

var c = a.Select(a_ => a_.Zip(b, (a__, b_) => a__ - b_));

How can the compiler guess that it has to look two steps down.

Perhaps it will be possible to add as many dots as possible to indicate to the compiler we want to broadcast twice:

Option<int>[] a = ...;
Option<int> b = ...;

var c = a ..- b; //Do it twice!

but again, everything goes out the window with a small change:

Option<int>[] a = ...;
Option<int> b = ...;

var c = b ..- a; //Do it in reverse!

If we start checking left to right, we end up with an Option<int>. IEnumerable is not zippable, so the compiler writes:

var c = b.Select(b_ => b_ .- a);

Then it tries to broadcast subtraction to int and IEnumerable<Option<int>>. int has no Select or Zip, so it applies Select to the other operand:

var c = b.Select(b_ => a.Select(a_ => b_ - a_));

This won't typecheck, since b_ and a_ have incompatible types.

Theoretically, the compiler could try lowering again and again, checking operands (or receiver and arguments) in various order until it would find a version that does typecheck. I am not sure if it's acceptable.

[jveselka]

So this is just a syntax sugar over LINQ?
I believe implicit parameter lambda syntax would be more general while being almost as short. I think there was a proposal somewhere.

int[] a = ...;
int[] b = ...;

var c = a.Zip(b, _1 + _2);
//var c = a .+ b; //Elementwise addition
//var c = a.Zip(b, (a_, b_) => a_ + b_);

var d = a.Select(_ * 2);
//var d = a .* 2; //Elementwise doubling
//var d = a.Select(a_ => a_ * 2);
var e = b.Select(Math.Sign); //you can already do this
//var e = Math.Sign.(b); //Elementwise static method call
//var e = b.Select(b_ => Math.Sign(b_));

string[] f = ...;

var g = f.Select(_.Length);
//var g = f..Length; //Elementwise property access
//var g = f.Select(f_ => f_.Length);
var h = f.Select(_[0]);
//var h = f.[0]; //Elementwise indexer access
//var h = f.Select(f_ => f_[0]);

Action<int>[] i = ...;

var j = i.Zip(a, _1(_2));
//var j = i.(a); //Elementwise delegate call
//var j = i.Zip(a, (i_, a_) => i_(a_));

[orthoxerox]

@zippec Yes, #91. Shorthand lambdas are more explicit, but not so convenient when you are dealing with lots of subexpressions. x .+ y .* z is a handful to write even with shorthand lambdas. Plus when your data source changes and x is no longer an Option<int>, but an int, you have to rewrite the expression yourself.

[bondsbw]

Infix math operators would be nice. If the dot-symbol syntax is a problem, how about this:

var c = a [+] b;
var d = a [*] 2;

[jveselka]

x .+ y .* z

I must say that this seems completely cryptic to me. But maybe its just a matter of practice.

[TonyValenti]

@bondsbw
or:

var c = a[] + b[];
var d = a[] * 2[];

[DavidArno]

As usual, I suspect I'm missing something here, but surely what we really want to be able to write is:

int[] a = ...;
int[] b = ...;
int c = ...;
var d = a + b;
var e = d + c;

This could be achieved via either the shapes or "extension everything" proposals and adding + operators to IEnumerable<T>.

[orthoxerox]

@DavidArno this will solve the issue for one operator and one monad. What about this use case?

var userID = GetUserID(); //returns Either<Exception, int>
var userData = GetUserByID.(userID); //accepts int, returns Either<Exception, User>

Yes, userID.SelectMany(GetUserByID) is short enough, but is really obscure.

[DavidArno]

@orthoxerox,

OK, so I did forget something 😀 Forgot that this proposal isn't just about element-wise operator/method calling, but also about lifting in monads. Back to the drawing board for me, then...

[bondsbw]

@DavidArno The plain + syntax is fine too, but my first thought was to reserve it for array concatenation (a more general operation).