policies at the intersection of algebras, unit conversion and value conversion

[erikerlandson]

Ever since I started this project I have puzzled over what the "best" policies are for coulomb to support out-of-box in terms of numeric operations (algebras) as they interact with unit conversions and value conversions.

In the current ("scala-2") version of coulomb, I "hard-code" this interaction to a certain degree: The convention is:

1. convert RHS value to Rational
2. apply any unit conversion to LHS unit
3. convert rhs (rational) to LHS value type
4. add RHS to LHS (or more generally, apply whatever algebra is in effect for LHS value type)
   (see here for description)

In scala-3 coulomb, I have redesigned the system of context ("implicit") types and functions so that there is no a-priori bias toward LHS value types (or units, even): both the output value type and unit type are dependent types, taking advantage of scala-3's significantly improved support for dependent typing.

For better or worse, this re-opens the question of what out-of-box policies are best to support (or even how to define what "best" is).
In this thread I'm going to try and articulate some possible "tiers" of policies, particularly when generalized to working with typelevel/algebra (AdditiveSemigroup and friends), and the numeric types in typelevel/spire.

cc @cquiroz @armanbilge

[erikerlandson]

One possible tier, call it Tier 0: no operators are applied unless both value-type AND unit-type are identical. If you need to add two values and they are not identical, you have to explicitly convert using q.toValue and/or q.toUnit (In fact, at tier 0, maybe q.to[V,U] is not enabled so you have to make your own choice about which order these happen in)

This is arguably the "safest", or least ambiguous, tier - there are no contextual assumptions about whether or how to convert units or values, or in what order, etc.

One could always support this tier out-of-box.

[erikerlandson]

correct, unit types are NOT closed under * or /

[erikerlandson]

policies for * and / are easier in some respects, because you do not have to convert any units. The operation is always defined - you can't run into the problem of it being illegal to add Meters + Seconds. Meters * Seconds is "(Meter)(Seconds)", etc. It's not closed, but it's also always definable.

[armanbilge]

It seems to me that you have a VectorSpace, where your vectors are Quantity[V, U] and your scalars are V. Vectors can be added/subtracted, but they can only be multiplied/divided by scalars.

[erikerlandson]

mathematically, (V, U) is a group with respect to multiply, however it only works when you allow the set of all possible unit expressions - then it is closed.

Scala's type system is sort of blind to the "true" math here. I can define clever implicits to always yield a correct UO output unit type, but scala itself will never recognize that as "being of the possible set of unit expressions" it just knows that the new UO is different than either UL or UR.

In scala, you could get a bit closer to this idea if you made all unit expression types <: UnitExpr -- however the types themselves would still be different to the compiler, unless you up-cast to UnitExpr, in which case you immediately lose any of the specific information you need to do your unit analysis, so it's unhelpful.

[armanbilge]

I'd like to think more about this. It reminds me of a similar problem I have in armanbilge/schrodinger#74.

[erikerlandson]

Tier 1 is cases where you have Quantity[VL, U] + Quantity[VR, U] - in this case you need to consider two additional bits of context:

1. What is the output value-type VO? In "scala-2 coulomb" that was always just VL, but in my new world it might be either VL, or VR, or even some third type, depending on what policies are defined.
2. Do the necessary conversions VL->VO, VR->VO, exist? For the "big-4" builtin types (Double,Float,Long,Int) the answer is always yes, but for third party types like spire, or other custom types, possibly not.

So, is there an out-of-box policy here?

A possible policy is to not get into this kind of ambiguity. Only support operations when left and right are identical. This is actually a very reasonable policy, and I can see having it be one that the user can import.

Another possible policy is to involve a new kind of typeclass, a ValueResolver, that maps (at type level) (VL, VR) -> VO. For example, a standard resolution in built-in types is (Double, Int) -> Double, or (Int, Long) -> Long, etc. The standard numeric promotions. However, things get more ambiguous for something like spire types. There is no standard consensus on whether Rational is ">=" Algebraic or not, and spire does not provide one. I could perhaps define a few "obvious" ones, or simply say that if someone wants to use this kind of policy, they have to supply their own ValueResolver. One could define the obvious standard ones, like ValueResolver[Double, Int] -> Double, etc.

this kind of policy also raises the issue of which "value space" to do the actual operation in. I believe I have an idea here, which is to always do the operation in the output-type space VO. This policy would be a straightforward generalization of the "big-4" builtin types. For example, Int+ Double is basicaly: Resolve (Int,Double) -> Double (VO), and the semantic is to map Int->Double, and Double->Double (identity) and add the two doubles to get the result.

If you have some non-standard types (e.g. from spire), and you do not have a coherent ValueResolver policy, then this will simply fail to compile, and you will have to do all your operations over identical types, and do any value conversions with toValue, and that is that.

[erikerlandson]

A useful property of this idea is that if you wanted "scala-2 coulomb backward compat policy" you could define context function that just always does ValueResolver[VL, VR] -> VL

[erikerlandson]

Tier 3 - Quantity[VL, UL] + Quantity[VR, UR] - we must consider both value-type conversions and unit-type conversions.

One question we can ask: does one convert values, and then convert units? or does one convert units, then values?

I believe that the "best" answer is to convert values, then units. Why? Assume that we are also using the ValueResolver concept in the section above. In theory almost any crazy Resolver policy is possible, however practically speaking, the semantics of ValueResolver is PROBABLY to up-cast to numeric types that have more resolution, or at least to not throw a significant amount away. Again, the standard types are instructive: we resolve (Int, Double) -> Double. And even for non-standard types a similar property will hold. One is unlikely to Resolve (Int, Double) -> Int, and throw away fractional values, to do these kind of computations. So a reasonable, and likely numerically safe, policy is:

1. Resolve VL -> VO
2. Resolve VR -> VO
3. Convert UR -> UL (in VO type space) (in the case of add/subtract - here I have to choose one or the other, there is no sane configurable policy that i can imagine, since the space of unit types is unbounded)
4. add left and right values (again, both in VO space).
5. Return Quantity[VO, UL]

Note, if one is multiplying or dividing, UO will not be UL, it will be a new unit type equivalent ot UL*UR or UL/UR, etc

The above runs into a possible issue if both VL and VR are integer types. It's not hard to simply apply the policy above, however doing unit conversions in integer spaces can result in massive loss of precision. For example Converting 1 yard to meters, in integer space, is going to result in 0 meters.

In "coulomb-2 scala", I basically just allowed this, and assumed the user could work these issues out for themselves. That is one reasonable possibility.

Another possibility I'm toying with is to not, by default, support these operations if VO is an integer type, unless it is explicitly enabled.
If I use the scheme above, this could be enforced "after" ValueResolve. I'm not yet sure what the best detailed scheme is yet.

1. Just don't import any algebras over integer types. This is easy but it also disables some cases where this would be OK to do, like adding Quantity[Int, U] + Quantity[Int, U] (i.e. identitcal units)
2. Have another specialized policy context-var, similar to here
3. define another type-class called AllowUnitConversion[V]. By default, this would not be defined for Int, Long, BigInt, etc. The operations requiring unit conversions could all depend on this typeclass.

Currently thinking (3) is cleanest, and easiest for a user to extend for custom types.

[armanbilge]

It seems to me like value and unit conversion in general are not independent. Depending on whether your destination value type is fractional or not, certain conversions may or may not be available. For example (Int, m) => (BigInt, cm) or (Int, cm) => (Double, m) could be allowed but maybe not (Int, m) => (Int, cm) and definitely not (Int, cm) => (Int, m)

[erikerlandson]

4. Slightly refactor how I do value conversions. Using the concepts above, value conversion always preserves input and output types, so the typeclass for value conversion is something like ConvertValueUnit[V, U1, U2] and it defines a function cnv(v: V) => V. So, with this concept we can not (by default) define ConvertValueUnit for integer types.

Separate typeclass for ConvertValue[V1, V2]. For spire types, this just depends on From typeclasses from spire.

So, my older concept of Convert[U1,V2,U2,V2] either no longer exists, or is a composition of ConvertValue -> ConvertValueUnit (and so is gated for integer types that way)

[erikerlandson]

@armanbilge yes - one is always free to enable the integer unit conversions if they feel like they understand what they are doing.

Handling such cases both automatically and properly, while excluding other "bad" cases, feels like a lot of work for relatively small use-case payoff.

[cquiroz]

If I recall correctly many of the conversions where automated via spire's implicit conversions via ConvertTo and ConvertFrom. I think in hindsight we know those are problemmatic as they are invisible.

From a user point of view it would be great that all conversions that have no loss of precission are automatic while the ones that do have to be explicitly allowed in code

[erikerlandson]

@cquiroz +1
My plan is that there will be a coulomb-spire package, and for that the numeric conversions will be driven by spire's conversions, just like they are currently. The ConvertValueUnit[V, U1, U2] and ConvertValue[V1, V2] typeclasses should provide us with decent control over what types we allow conversions over, just by virtue of what the user decides to define (or import).

The basic structure will be:
coulomb-core - no dependencies - provides an out-of-box policy for double/float/long/int
coulomb-algebra - defines the set of policies discussed above - depends only on typelevel/algebra - this package only defines the abstract policy structures, and will not define any particular implementation of them
coulomb-spire - in theory this will only need to provide a set of given definitions that fill out the policies in coulomb-algebra specifically for spire (and probably also double/float/int/long)

[erikerlandson]

@cquiroz @armanbilge I have an implementation of the above ideas, which I am pretty happy with
to see what it looks like, check out:
https://github.com/erikerlandson/coulomb/blob/scala3/core/src/main/scala/coulomb/conversion/standard/standard.scala
https://github.com/erikerlandson/coulomb/blob/scala3/core/src/main/scala/coulomb/conversion/standard/integral/integral.scala
https://github.com/erikerlandson/coulomb/blob/scala3/core/src/main/scala/coulomb/ops/standard/add.scala

[erikerlandson]

the full, unspecialized, logic is as follows:

1. resolve left and right argument types to get an output type e.g. ValueResolution[Int, Double].VO yields an output type Double
2. identify conversions from VL => VO and VR => VO
3. identify a unit conversion in the value type space of VO
4. identify that VO supports some definition of addition
5. allocate the new Add object that defines the full addition operation, with any applicable value conversions and unit conversions

Each of these steps, since they are context typeclasses, are definable as policies, either imported from coulomb or the user can declare their own, by defining the context functions or vars to create the corresponding ValueResolution, ValueConversion and UnitConversion objects

transparent inline given ctx_add_2V2U[VL, UL, VR, UR](using
    nev: NotGiven[VL =:= VR],
    neu: NotGiven[UL =:= UR],
    vres: ValueResolution[VL, VR],
    vlvo: ValueConversion[VL, vres.VO],
    vrvo: ValueConversion[VR, vres.VO],
    ucvo: UnitConversion[vres.VO, UR, UL],
    alg: Algebra[vres.VO]
        ): Add[VL, UL, VR, UR] =
    new Add[VL, UL, VR, UR]:
        type VO = vres.VO
        type UO = UL
        def apply(vl: VL, vr: VR): VO = alg.add(vlvo(vl), ucvo(vrvo(vr)))

[armanbilge]

@erikerlandson I hope you don't mind, I spent this evening starting from tabula rasa to explore some of my own ideas. Obviously I've spent a tiny fraction of the time you've spent thinking about this and don't really have a deep understanding of the problem space. In other words, I apologize in advance for the naivete and stupidity that follows :)

The ideas I explored are extremely underdeveloped, but roughly fell into these two categories:

A "tagless final" style API

Tagless final is the API design adopted by several modern Typelevel libs. Basically the idea is to separate libraries into:

1. a "kernel", which provides typeclasses that describe the fundamental capabilities necessary to write programs
2. a "core", which actually provides data types that implement these typeclasses and actually run programs

The canonical example is Cats Effect 3: the kernel provides typeclasses MonadCancel, Concurrent, Temporal, etc. that describe what it means to be an asynchronous effect. Effectively the entire ecosystem is written solely using these abstraction, including fs2, http4s, skunk, etc.

Then, Cats Effect core provides the IO datatype (implementing said typeclasses) that applications use to run their programs. But IO is for the large part an implementation detail hidden behind those abstractions.

The same pattern roughly applies to algebra/spire: algebra is the kernel, with its algebraic typeclasses, and spire provides concrete numerical types.

So, what does this mean for coulomb? IMHO units are an implementation detail: it should be possible to write code that operates solely in terms of the base quantities (e.g., time, length, mass) without knowing the actual units of measure. Ideally something like:

def f[Q[_]](x0: Q[Length], v0: Q[Length / Time], a: Q[Length / Time^2], t: Q[Time]): Q[Length] =
  x0 + v0 * t + 0.5 * a * t^2

Here Q[_] represents a parameterized Quantity type, that itself is an abstraction.

IMO, this approach has some nice features:

• the function f() is completely agnostic to the units, scale, and numerical representation of the inputs, and can be easily reused/applied across many situations
• there is no concern for any sort of conversions, unit or numerical or otherwise. The code can focus on its purpose, without getting caught up in the mechanics of conversion

I explore what these typeclasses could look like here although I haven't quite figured out how to make the ergonomics as nice as the ideal example above.

Of course, we can't ignore units forever :) I think there are really two situations we need to be concerned with units.

1. when we are doing I/O: either loading values into Q or generating a representation of Q for interop with non-Coulomb code or human-readability. This is where some sort of conversion typeclasses come into play.
2. when we actually want to run our program: we need a concrete implementation of Q. This implementation would determine the numerical types, units, etc. used in its internal representation of quantities. But for the large part this should be an implementation detail hidden from the rest of the program.

A maximally type-level API

The other idea I explored was aggressively static typing the API, taking advantage of several Scala 3 features. This moves a large chunk of the dimensional analysis logic to the type-level and thus compile-time.

A type-level Abelian group for units

For a fixed set of dimensions (e.g., the 7 SI dimensions) it is possible to express an effectively infinite combination of base quantities completely at the type-level in a composable way. This is thanks to Scala 3 support for compile-time arithmetic.

coulomb/core/src/main/scala/coulomb/SIDimension.scala

Lines 23 to 31 in 1dad497

	final type SIDimension[
	+T <: Int,
	+L <: Int,
	+M <: Int,
	+I <: Int,
	+Th <: Int,
	+N <: Int,
	+J <: Int
	]

Here, we are expressing our position as integer-valued coordinates in terms of time, length, mass, electric current, temperature, amount of substance, and luminous intensity. This makes it easy to multiply units completely at the type-level, such that the dimensional analysis is happening completely at the type-level/compile time.

coulomb/core/src/main/scala/coulomb/SIDimension.scala

Lines 38 to 41 in 1dad497

	infix type *[D1 <: SIDimension[Int, Int, Int, Int, Int, Int, Int], D2 <: SIDimension[Int, Int, Int, Int, Int, Int, Int]] =
	(D1, D2) match
	case (SIDimension[x1, x2, x3, x4, x5, x6, x7], SIDimension[y1, y2, y3, y4, y5, y6, y7]) =>
	SIDimension[x1 + y1, x2 + y2, x3 + y3, x4 + y4, x5 + y5, x6 + y6, x7 + y7]

Another bit of magic is that we can now define various aliases and the compiler will automatically be able to recognize the relationships between them.

type Velocity = Length / Time

Statically-typed units

Rational is a central part of coulomb, but a lot of its use is static definitions of constants. Which motivates the implementation of a purely compile-time Rational, which might be used like this:

coulomb/units/src/main/scala/coulomb/units/Unit.scala

Lines 19 to 28 in 1dad497

	trait BaseUnit
	final type DimUnit[R, B <: BaseUnit]
	final type Kilo[B <: BaseUnit] = DimUnit[Rational[1_000, 1], B]
	final type Mega[B <: BaseUnit] = DimUnit[Rational[1_000_000, 1], B]
	final type Giga[B <: BaseUnit] = DimUnit[Rational[1_000_000_000, 1], B]
	final type Milli[B <: BaseUnit] = DimUnit[Rational[1, 1_000], B]
	final type Micro[B <: BaseUnit] = DimUnit[Rational[1, 1_000_000], B]
	final type Nano[B <: BaseUnit] = DimUnit[Rational[1, 1_000_000_000], B]

There are some interesting things we can do with this as well. Suppose we define:

type Inch = DimUnit[Rational[254, 1_000], Meter]

One of the concerns with unit conversion for integer values is whether to support conversions that lead to loss of precision. For example, attempting to convert an integer inch to an integer meter, centimeter, or millimeter.

However, it is possible to convert an integer inch to an integer micrometer or smaller without loss of precision. Because we can do arithmetic on Rationals completely at compile-time, this makes it feasible for the compiler to recognize this completely of its own accord and thus automatically enable that conversion. In fact, we can code this very generically in terms of an arbitrary unit, without specializing to inch etc.

---

That's all for now, thanks for reading :)

[erikerlandson]

A side observation about using type bounds, like Kilo[B <: BaseUnit] - in early versions of coulomb, I made use of these, for example all unit expressions were bounded U <: UnitExpr - it turned out that this did not really add any expressive power (in fact it took some expressive power away), and it tended to metastasize into the code: to make anything compile, I had to use <: UnitExpr absolutely everywhere. Again, it's not really necessary: if someone tries to use some type that isn't a "valid unit expression" - it will simply fail.

Dispensing with this kind of type bound also has the enormous benefit of allowing the unit system to be totally open - if I have a type Book, I can use Book as a unit in unit expressions: I developed this idea here. Note that I allow this behavior to be turned on or off, which is described in this section: https://github.com/erikerlandson/coulomb#declaring-new-units (scala-3 coulomb also has this policy choice)

[armanbilge]

Thanks for the detailed response and all the pointers to your writing :) I definitely have a better understanding of why coulomb is the way it is.

Regarding "moves a large chunk of the dimensional analysis logic to the type-level and thus compile-time." - coulomb also does all of its unit analysis at compile time. Unit analysis "operations" that are incorrect are compile failures: ultimately these boil down to Coefficient[U1, U2] not existing and so resulting in a failure to resolve some implicit context variable

Yes it does, sorry, this was at best a very poorly phrased thought :) I think what I meant is that the compiler is able to "reason" about this for itself, without requiring "hand-holder" implicits/contexts defined by humans. I thought that this would take away the "blindfold", in reference to your comment in #237 (reply in thread):

Scala's type system is sort of blind to the "true" math here. I can define clever implicits to always yield a correct UO output unit type, but scala itself will never recognize that as "being of the possible set of unit expressions" it just knows that the new UO is different than either UL or UR.

But perhaps I misunderstood what you were getting at.

HOWEVER, instead you simply write your function in specific units (which I claim is actually "easier" because people are generally used to thinking about a problem with respect to some particular set of units, but I have no hard data to prove this), and if someone else calls this function, they might call it with different units, but as long as they are properly convertable, coulomb will convert them.

Yes, I completely agree with the easier part, at least for writing functions :) I also agree that in theory as long as units are convertible, it should work, but FWIW all these conversion policies (at both the numerical and unit levels) that may-or-may-not require imports to activate etc. add their own layer of mental overhead (at least for me). Relying on conversions may also force the actual numerical computation to be done on a sub-optimal scale for the problem at hand. Some abstraction around the numerical type could probably help with this. Of course, in practice this may not be a big deal.

Defining these relations purely in terms of the existence of Coefficient[U1,U2] turns out to make the entire unit system very very extensible.

Right, I didn't think much about this. Separating out abstract quantities from measurable units means that to define a new unit on a new dimension requires defining at least two things, which is not so ergonomical at all.

That is, the limit of typelevel integers to Long prevents me from representing some fractions I would require.

Yes, agree with this too. (Scala 3 does support type-level Doubles in the current RC, with the usual caveats :) However, just as we can implement a type-level Rational it may be possible to implement a type-level BigInt, although this could get unwieldy. More realistically, we could define a slightly custom numerical type e.g. n/d * 10^e. There'd still be limits on what we can represent, but it would help reach things like yotta etc.

A side observation about using type bounds

Yeah, I wasn't giving much thought to this at all :)

[erikerlandson]

FWIW all these conversion policies (at both the numerical and unit levels) that may-or-may-not require imports to activate etc. add their own layer of mental overhead (at least for me)

This is completely true, and it is something I'm almost always ambivalent about. Ultimately, I design these policies into the my system because:

1. Any time I try to pick a policy that does not need any configuration, I can start imagining legitimate reasons why someone might desire some other policy: numeric computing in general seems prone to this, partly due to all of the issues caused by the variety of forms of 'loss' that happen whenever you are converting types.
2. values with units make this even worse, since you now have to consider the interactions of conversion between "normal" values and also unit types.

My compromise is that I strive to support out-of-box policies that are "reasonable" or "good enough" for most users to just import, without needing to understand the full details of things like ValueResolution, UnitConversion, etc. "Just import coulomb.conversions.standard.given and coulomb.ops.standard.given and you should be good" (or the eventual corresponding spire ones)

But I try to make them explainable enough so that if someone wants to write their own policy, I can document the process so that it's feasible to figure out. I hope that not many people will really need to customize a policy but I believe I can explain ValueResolution and friends without a major dissertation. We will see!

[erikerlandson]

I suppose that one policy that requires none (or almost none) configuration is the "strict" policy - you can only operate on Quantity if both left and right have same value type and same unit type. If you need to convert, then you just have to use q.toUnit or q.toValue.

There are solid arguments for this, but in the end, I've always felt that forcing people to do this throws away all the benefits of coulomb's ability to do such conversions automatically. But if you make it automatic, it causes you to want to allow the user to define what automatic policy they import .... 😄

[armanbilge]

It might be a matter of personal taste :)

Numerical and unit conversions are fundamentally necessary for coulomb. But, because of the complexities involved, personally I like the idea of pushing these conversions to the boundaries of your code, where it's doing interop and I/O and you can even be explicit about how exactly these conversions should happen.

This should reduce the surface area where users have to think about these conversions at all, from the entire program to very specific regions of the codebase. It also makes it possible to write e.g. a physics library with coulomb without getting bogged down with this at all.

IMHO I like this pattern quite a lot, because of my experience working with Cats Effect and friends.

[erikerlandson]

it may be possible to implement a type-level BigInt, although this could get unwieldy. More realistically, we could define a slightly custom numerical type e.g. n/d * 10^e. There'd still be limits on what we can represent, but it would help reach things like yotta etc.

I'm interested in the possibility of making type-level coefficient definitions work: one weird irony of working in macros meta-programming is that you cannot extract the actual numeric values from Expr and fold them up as you work your way, bottom-up, for example see: https://github.com/erikerlandson/coulomb/blob/scala3/core/src/main/scala/coulomb/infra/meta.scala#L232

And yet (another irony) : I can get the actual numeric vals if they are types!
https://github.com/erikerlandson/coulomb/blob/scala3/core/src/main/scala/coulomb/infra/meta.scala#L315

So, a typelevel representation that allows me to break the Long limit might absolutely allow me to re-write the coefficient metaprogramming to fully roll-up all the coefficient values and that would absolutely yield more efficient code.

I'm unsure of the total extent of the benefit: most unit expressions are not "deep" - but it would not be zero.

One idea I had was to allow string literals: IIUC, we could make the coefficient type a string "12345643353262533 / 432352345235234",
and I could grab this in the metaprogramming, and parse it back into Rational. I think I bailed on this idea because it introduced the possibility of user typos in the strings, and then they'd be getting compile-time errors that might be hard to diagnose. but maybe it's worth it.

Or maybe support a more extensible type structure like you suggested: (N / D) * (10 ^ 21), etc. I could write unapply cases for that "easily" (for some value of easy), and return Rationals.

Part of me wants to explore this later - my brain can only handle so much at one time - on the other hand, if I'm going to really do it, sooner is probably better.

[armanbilge]

Possibly. The Scala 3 compiler is much better at implicit/context resolution anyway, and still hasn't undergone the same kind of optimization the Scala 2 compiler has AFAIK.

There's a maintainability/contributability trade-off here of course, and makes it more challenging for users/casual readers to grok the code.

[erikerlandson]

cool, I don't think they had this last time I tried:
https://dotty.epfl.ch/api/scala/quoted/ToExpr$$BigIntToExpr$.html

[erikerlandson]

It works \o/

scala> summon[RationalTE[1]].value
val res2: coulomb.rational.Rational = 1
                                                                                                                                                               
scala> summon[RationalTE[1 / 100]].value
val res3: coulomb.rational.Rational = 1/100
                                                                                                                                                               
scala> summon[RationalTE[1 / "12345678901234567890"]].value
val res4: coulomb.rational.Rational = 1/12345678901234567890
                                                                                                                                                               
scala> summon[RationalTE[3.1415]].value
val res5: coulomb.rational.Rational = 6283/2000
                                                                                                                                                               
scala> summon[RationalTE[(1 / 3) * (10 ^ 100)]].value
val res6: coulomb.rational.Rational = 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000/3

[armanbilge]

I missed this comment yesterday! This is awesome! 🤩

[erikerlandson]

I have re-written the metaprogramming to "store" unit parameters in types, see:
#245

[erikerlandson]

A separate thread for Abstract Quantities (Length, Mass, Duration)

I can think of a couple possibly interesting directions for this.

1. type Length[V] = Quantity[V, Meter] - in this scheme, def f(x: Length[Double]) is literally equivalent to def f(x: Quantity[Double, Meter]) - so this takes advantage of coulomb's implicit conversion of any unit expression that can convert to Meter as being a valid Length. We might wish to define some higher-kinded type operations like a semantic for Length[V] / Duration[V] - here the scala-3 type lambdas might prove useful.
2. Length, Mass, etc, are context parameters. So possibly def f[U](x: Quantity[Double, U])(using Length[U]), and Length[U] is satisfiable iff Coefficient[U, Meter], etc

Direction (1) seems more promising, especially if we can implement clean semantics for type expressions such as Length[V]/Duration[V], which is equivalent to Quantity[V, Meter] / Quantity[V, Second] (at the type level), so a type lambda that can map this to Quantity[V, Meter / Second] would be very elegant

[armanbilge]

I think I was equally confused by what you meant 😆 I don't think that making all methods inline is a good default, but making all inline methods transparent by default might be.

[erikerlandson]

I see - that might be useful too. In a sense what I care more about is making all methods transparent, but transparent currently only applies to inline. Maybe applying it to non-inline is impossible.

[armanbilge]

It is impossible afaict. A non-inline method is an abstraction that is guaranteed by its type signature. transparent is about looking into the implementation of that method and inferring the specific result type. That breaks the abstraction, and thus is only possible if you inline that method.

[armanbilge]

Here's an example from the docs.
https://dotty.epfl.ch/docs/reference/metaprogramming/inline.html#transparent-inline-methods

class A
class B extends A:
  def m = true

transparent inline def choose(b: Boolean): A =
  if b then new A else new B

val obj1 = choose(true)  // static type is A
val obj2 = choose(false) // static type is B

If you just look at the signature of choose you can't make the inference about obj2. To do that, you need to know what's going on inside of choose, and that requires inline.

[erikerlandson]

aha, yes I forgot that example 👍

[erikerlandson]

[erikerlandson]

It is possible to lift some special cases all the way up to Quantity extension
This may be preferable to specialized Add context functions since it removes one or more layers context vars pulled in via using, and yet still seems to preserve the ability to fall back to the more general expressions

extension[UL](ql: Quantity[Double, UL])
    transparent inline def +[UR](qr: Quantity[Double, UR])(using coef: Coefficient[UR, UL]): Quantity[Double, UL] =
        val c = coef.value.toDouble
        print(s"c= $c\n")
        (ql.value + c*qr.value).withUnit[UL]

extension[VL, UL](ql: Quantity[VL, UL])
    transparent inline def +[VR, UR](qr: Quantity[VR, UR])(using add: Add[VL, UL, VR, UR]): Quantity[add.VO, add.UO] =
        add(ql.value, qr.value).withUnit[add.UO]

scala> 1d.withUnit[Meter] + 1d.withUnit[Kilo * Meter]
c= 1000.0
val res0: coulomb.quantity.Quantity[Double, coulomb.units.si.Meter] = 1001.0
                                                                                                                                                               
scala> 1f.withUnit[Meter] + 1f.withUnit[Kilo * Meter]
val res1: coulomb.quantity.Quantity[Float, coulomb.units.si.Meter] = 1001.0

[erikerlandson]

turns out this is only possible if definitions of + overloadings are in same package scope:

  |        Reference to + is ambiguous,
  |        it is both imported by import coulomb._
  |        and imported subsequently by import coulomb.ops.standard.{...}

I want these definitions to be extensible outside of coulomb-core, so extension based policies are not feasible given that requirement

[erikerlandson]

@cquiroz @armanbilge
So I played with the algebra idiom of making given values trait members, for example:
https://github.com/erikerlandson/coulomb/blob/algebra-in-core/core/src/main/scala/coulomb/ops/algebra/all.scala#L19

I could in theory do this in other places, for one example the unit definitions. Another example would be coulomb.conversions.standard, which might improve re-use in coulomb-spire. Doing this provides some ability to re-mix given definitions as traits, but it also causes duplicate definitions everywhere a trait gets mixed-in, so I'm not sure it's a huge win.

My holy grail has always been the ability to import things such as unit definitions from multiple places, and have it act as a true "union" - so I could get the SI units (meter, kilogram, second, etc) from si, but if I also exposed them via mks, it would be safe to import both si and mks, and Meter would be considered the same single definition. I don't know if there is any way to do that.

[armanbilge]

Scala 3 also now has "export clauses". I haven't really tried them but I think it can replace the trait pattern. Would be amazing if it could solve that duplicate problem you mention as well.

https://docs.scala-lang.org/scala3/reference/other-new-features/export.html

[erikerlandson]

agree, was going to experiment with that over the weekend 👍

[erikerlandson]

This appears to work 🎉

package coulomb.units.mks

import coulomb.units.si.*
export coulomb.units.si.{*, given}

scala> import coulomb.*, coulomb.conversion.standard.given, coulomb.ops.standard.given, coulomb.ops.algebra.all.given, coulomb.units.mks.{*,given}, coulomb.rational.Rational, coulomb.policy.allowImplicitConversions.given, coulomb.rational.typeexpr, scala.util.NotGiven
                                                                                                                                                               
scala> 1f.withUnit[Meter] / 2f.withUnit[Second]
val res0: 
  coulomb.quantity.Quantity[Float, coulomb.units.si.Meter / 
    coulomb.units.si.Second
  ] = 0.5
                                                                                                                                                               
scala> import coulomb.units.si.{*,given}
                                                                                                                                                               
scala> 1f.withUnit[Meter] / 2f.withUnit[Second]
val res1: 
  coulomb.quantity.Quantity[Float, coulomb.units.si.Meter / 
    coulomb.units.si.Second
  ] = 0.5

[erikerlandson]

one caveat, the definitions that are exported have to be defined in a class or object, so si and similar would need to be object si, etc, but that's not really a problem

[armanbilge]

That's amazing!

[erikerlandson]

I feel like these definitions of si units in the form of physical constants will make good unit tests, for the coulomb physical constants. I'm thinking of moving these into the coulomb-units package, if @cquiroz is amenable.

https://en.wikipedia.org/wiki/2019_redefinition_of_the_SI_base_units#Impact_on_base_unit_definitions

[cquiroz]

Sure, sounds very good

[erikerlandson]

@cquiroz what numeric data types do you need supported? I plan on supporting everything I did before, but it might help me prioritize testing or optimization, if I know specifically what you care about for your use cases

[cquiroz]

Not many really, We mostly care about not loosing precision and I'd say we use almost exclusively Int, Long, BigDecimal and spire's Rational

[cquiroz]

And even Rational could be replaced by BigDecimal. We don't have many numeric intensive code where BigDecimal could be a problem, and on those cases we may convert to and back Doubles inside a function

[erikerlandson]

Interestingly, BigDecimal and BigInt are scala native types, and so I could technically support these on coulomb-core, although spire definitely has a more consistent system of conversions between all these types.

[erikerlandson]

@armanbilge @cquiroz
I did an experiment where I make use of scala.Conversion directly in operators. In other words, removing AllowImplicitConversions
#255

I created a parallel version of addition: ++ and Add2, just to see what this would look like. In general it seems pretty good. The only slightly awkward part is that properly respecting Conversion on the optimized operations invokes a conversion object that isn't actually used:

coulomb/core/src/main/scala/coulomb/ops/standard/add.scala

Line 36 in cea67f7

scala.Conversion[Quantity[Double, UR], Quantity[Double, UL]] // redundant!

I don't have any intuition about what scala does with manifestations like this that appear in using args, but are not actually used at runtime.

[erikerlandson]

similar question regarding things like equ: UR =:= UL - does the compiler know it can discard these in the final byte code, or do they add bulk to the compiled code?

[armanbilge]

They will end up in the bytecode, but there's a good chance the Scala.js linker and JVM will optimize them away anyway.

There is an experimental feature for "erased definitions" that are compile-time only.
https://dotty.epfl.ch/docs/reference/experimental/erased-defs.html

[erikerlandson]

erased sounds like the relevant feature. only allowed on nightly builds though. hopefully it makes it into the language

[erikerlandson]

still working out how I feel about this. Add2 seems a bit more semantically correct, at least if one takes the point of view that implicitly converting unit and value types during operations is spiritually equivalent to scala.Conversion. I suspect that the current version, e.g. Add, optimizes slightly better, although I'm not sure how significant the difference is in practice, and I wouldn't trust my intuitions without doing some real benchmarking.

In a perfect world, having Quantity be an opaque type would allow the compiler to completely elide every single instance of .value and .withUnit, with only raw values ending up in the byte code, but I know for a fact that is far from happening now, and I was told that it would be impossible without java valhalla, and maybe not even then.

reference: scala/scala3#12011

[erikerlandson]

Writeup of a benchmarking experiment that clarified my thinking in terms of performance issues:
#255 (comment)

[erikerlandson]

@armanbilge @cquiroz
do either of you know what a good micro-benchmarking idiom or package would be? The only one I found in the scala/sbt ecosystem was scalameter:
https://github.com/scalameter/scalameter-examples/blob/master/basic-with-separate-config/build.sbt

it seems like the kind of thing I have in mind but it's not very active, the last update was a year ago, and the examples are all on sbt 0.13, and scala 2.11

my attempts at code diving munit to add benchmark timing outputs weren't very fruitful, it's pretty oriented around only reporting assertion errors, but it would be cool to add a microbenchmarking facility

[armanbilge]

Checkout the Cats Effect benchmarks, they use JMH as Carlos mentioned :)
https://github.com/typelevel/cats-effect/tree/series/3.x/benchmarks/src/main/scala/cats/effect/benchmarks

[erikerlandson]

#256

[erikerlandson]

if I wanted to add the following to the CI workflow, how would I do that:

     - name: Benchmarks
       run: sbt 'benchmarks/Jmh/run .*Benchmark'

I can make the change, but running sbt githubWorkflowGenerate clobbers it.

[armanbilge]

You can manually modify the workflow according to https://github.com/djspiewak/sbt-github-actions/

Although FWIW I'm not sure how much value there is to run benchmarks in CI, since it's not a very controlled environment.

[erikerlandson]

I tested it out just to see it work, but I commented it out. If I add more benchmarks it could get to be a long CI run fast.

[cquiroz]

Maybe https://github.com/sbt/sbt-jmh

[erikerlandson]

thanks, that looks promising!

[erikerlandson]

@armanbilge @cquiroz as of #257 and #258 coulomb-core for scala 3 is effectively complete 🎉

Needs scaladoc, and any additional unit testing I can think of, but it implements all the functionality I planned for it

[armanbilge]

I'll take a closer look at this later today.

or am I just overstressing scala's type system somehow

I honestly would not be surprised if this is the case 😅 😬

[erikerlandson]

I feel like the trajectory of this conversation totally represents working with scala
"yay! I'm done!"
"oh no..."

[armanbilge]

@erikerlandson ok, I think I got to the bottom of it, see #261.

[erikerlandson]

related, I made some progress by being very specific about types:

    final type EpochTime[V, U] = coulomb.deltaquantity.DeltaQuantity[V, U, coulomb.units.si.Second]

    given ctx_EpochTime[V, U]: scala.Conversion[EpochTime[V, U], coulomb.deltaquantity.DeltaQuantity[V, U, coulomb.units.si.Second]] =
        new scala.Conversion[EpochTime[V, U], coulomb.deltaquantity.DeltaQuantity[V, U, coulomb.units.si.Second]]:
            def apply(t: EpochTime[V, U]): coulomb.deltaquantity.DeltaQuantity[V, U, coulomb.units.si.Second] =
                t.asInstanceOf[coulomb.deltaquantity.DeltaQuantity[V, U, coulomb.units.si.Second]]

    object EpochTime:
        def apply[U](using a: Applier[U]) = a

        abstract class Applier[U]:
            def apply[V](v: V): coulomb.deltaquantity.DeltaQuantity[V, U, coulomb.units.si.Second]
        object Applier:
            given [U]: Applier[U] =
                new Applier[U]:
                    def apply[V](v: V): coulomb.deltaquantity.DeltaQuantity[V, U, coulomb.units.si.Second] =
                        v.withDeltaUnit[U, coulomb.units.si.Second]

    extension[V](v: V)
        def withEpochTime[U]: coulomb.deltaquantity.DeltaQuantity[V, U, coulomb.units.si.Second] =
            v.withDeltaUnit[U, coulomb.units.si.Second]

This worked better but ultimately did not allow me to use show with a variable cast as EpochTime, that still only works if it is explicitly DeltaQuantity

[erikerlandson]

I have a theory that if I tweak q.show so it uses a context var def show[UL](using unitstr[UL]): String = ... it will work, mostly based on the fact that the other methods seem to work, and the only thing different that I see is show involves no parameters. Obviously less elegant than the current definition but if it works properly it's a win.