Issues with std.traits

[ntrel]

@LightBender mentioned this, see also @jmdavis's reply:
https://forum.dlang.org/post/[email protected]

I'm posting this so people can reply with concrete improvements.

[rikkimax]

One thing to note: fullyQualifiedName shouldn't be brought over, it has been added to the compiler.

Other traits of similar complexity should be considered for implementation into the compiler.

Slightly unrelated, but involving getUDAs and getting of overloads https://issues.dlang.org/show_bug.cgi?id=23500

A lot of reflection stuff should be in the compiler: https://issues.dlang.org/show_bug.cgi?id=23559

[jmdavis]

IMO it would be a big mistake to fix this in Phobos without also fixing it in the language.

While I agree that it would be great to fix what __traits is doing, that's going to require Editions, because otherwise, it risks breaking existing code in subtle ways - and not in a way that we can use deprecations either. As it is, we've been stuck with problems in std.traits for years precisely because making any changes to traits breaks code in ways that we can't get around with deprecations. So, the fact that we're replacing std.traits gives us a golden opportunity to finally fix those traits where we couldn't before. We may or may not get a similar chance with __traits thanks to Editions.

So, we really can't assume that anything in __traits is going to be fixed when working on phobos.sys.traits. It's not going to be fixed right now, and it may or may not ever be fixed. Until/Unless __traits is fixed with Editions, we have to design Phobos v3 with what __traits currently does. And I see no reason to make Phobos do the wrong thing just because __traits does.

In the cases where a trait in phobos.sys.traits does something similar to a trait in __traits, the documentation explains the difference, so the situation is then clear (and in almost all cases, the trait in Phobos has a different name as well, reducing potential confusion). And if we are able to later fix the traits in __traits, all the better. I would love to see that happen. But we're throwing away this golden opportunity if we try to make phobos.sys.traits match __traits' behavior, just because we haven't been able to fix __traits.

[Herringway]

LOL. One that I have no intention of porting over is isCallable. Even simply trying to clearly document what it currently does makes it clear that it's way too complicated because of templates, property functions, and opCall. On the other hand, most code really doesn't care about whether a symbol is arbitrarily "callable". Rather, it cares about whether a particular expression will work - e.g. foo(arg1, arg2) - which goes beyond whether the symbol in question is callable in some generic sense.

As much as I'd love a template like that, the best we can do with current language features would be painfully prone to false negatives from __traits(compiles). Something like a template matching trait is needed.

[jmdavis]

I'm specifically talking about using something like __traits(compiles, foo(arg1, arg2)) or is(typeof(foo(arg1, arg2))) and not about adding a trait, because checking whether the call compiles is exactly what needs to be done if you need to check whether a particular function call will actually compile - especially when you're dealing with an arbitrary callable. I don't see how a trait for checking whether a template constraint would pass or not would even be relevant to the situation, since the callable doesn't even necessarily involve a template. Rather, this is about cases where a template constraint needs to check whether a particular callable is callable with a particular set of arguments, and that's very much a question of __traits(compiles, ...). I don't see any other way to do that, particularly if you don't want to try to write a very complicated set of checks for lvalues, rvalues, type qualifiers, etc. where you're basically trying to duplicate the compiler's logic for function calls and inevitably are going to get it wrong because of how many factors are involved (and those factors can change as the language evolves, making such an attempt even more error-prone - e.g. they'll change when move constructors get added like they probably will be).

Of course, there are other situations where checking whether a template constraint passes without actually instantiating the template would be useful. For instance, that would be great for verifying that template constraints are well-written, since then you could use it to try to catch cases where the template constraint allows arguments that then fail to compile with the function body, since in most cases, that's not what you want, and it means that the template constraint needs to be fixed. But I really don't see how checking whether a particular template constraint passes or not really applies to this situation.

Regardless, my main point was that std.traits.isCallable really doesn't make any sense, because it's too complicated, and checking whether a particular symbol is callable in some arbitrary set of circumstances generally isn't useful. What almost all template constraints checking for callability care about is whether a particular symbol is callable with a particular set of arguments, which does inevitably take us to __traits(compiles, ...), but that's a very common thing with traits and template constraints. There are cases where __traits(...) can be used to tell us what we need, and there are cases where we can compare types and infer stuff about them using is expressions, but there are quite a few cases where the only solution is __traits(compiles, ...) or is(typeof(...))), much as it's good to encapsulate that in traits where we can to improve their usability and to reduce how often template constraints have to directly check for compilability. For instance, the traits for ranges are heavily based on whether particular pieces of code compile with the given template argument, and we're basically stuck doing that when checking for more complex stuff.

So, I'm not proposing any traits for checking for callability - quite the opposite really. I'm saying that isCallable doesn't make sense, and any traits that we do have related to whether something is callable need to be specific to the circumstances rather than checking for generic callability.

[Herringway]

I'm specifically talking about using something like __traits(compiles, foo(arg1, arg2)) or is(typeof(foo(arg1, arg2))) and not about adding a trait, because checking whether the call compiles is exactly what needs to be done if you need to check whether a particular function call will actually compile - especially when you're dealing with an arbitrary callable. I don't see how a trait for checking whether a template constraint would pass or not would even be relevant to the situation, since the callable doesn't even necessarily involve a template. Rather, this is about cases where a template constraint needs to check whether a particular callable is callable with a particular set of arguments, and that's very much a question of __traits(compiles, ...). I don't see any other way to do that, particularly if you don't want to try to write a very complicated set of checks for lvalues, rvalues, type qualifiers, etc. where you're basically trying to duplicate the compiler's logic for function calls and inevitably are going to get it wrong because of how many factors are involved (and those factors can change as the language evolves, making such an attempt even more error-prone - e.g. they'll change when move constructors get added like they probably will be).

Of course, there are other situations where checking whether a template constraint passes without actually instantiating the template would be useful. For instance, that would be great for verifying that template constraints are well-written, since then you could use it to try to catch cases where the template constraint allows arguments that then fail to compile with the function body, since in most cases, that's not what you want, and it means that the template constraint needs to be fixed. But I really don't see how checking whether a particular template constraint passes or not really applies to this situation.

Regardless, my main point was that std.traits.isCallable really doesn't make any sense, because it's too complicated, and checking whether a particular symbol is callable in some arbitrary set of circumstances generally isn't useful. What almost all template constraints checking for callability care about is whether a particular symbol is callable with a particular set of arguments, which does inevitably take us to __traits(compiles, ...), but that's a very common thing with traits and template constraints. There are cases where __traits(...) can be used to tell us what we need, and there are cases where we can compare types and infer stuff about them using is expressions, but there are quite a few cases where the only solution is __traits(compiles, ...) or is(typeof(...))), much as it's good to encapsulate that in traits where we can to improve their usability and to reduce how often template constraints have to directly check for compilability. For instance, the traits for ranges are heavily based on whether particular pieces of code compile with the given template argument, and we're basically stuck doing that when checking for more complex stuff.

So, I'm not proposing any traits for checking for callability - quite the opposite really. I'm saying that isCallable doesn't make sense, and any traits that we do have related to whether something is callable need to be specific to the circumstances rather than checking for generic callability.

The problem with checking that something compiles is that it checks that something compiles. Error in the function body? false. Error in the parameters because you forgot to update the statement? false. Some other error? still just false! __traits(compiles) NEVER only tests exactly the thing you mean to test. ANY usage of __traits(compiles) or is(typeof(...)) due to the lack of a precise alternative is a complete and utter failure of the language in my eyes.

The lack of this feature is why one of my phobos PRs has been stalled for years.

[jmdavis]

The problem with checking that something compiles is that it checks that something compiles. Error in the function body? false. Error in the parameters because you forgot to update the statement? false. Some other error? still just false! __traits(compiles) NEVER only tests exactly the thing you mean to test. ANY usage of __traits(compiles) or is(typeof(...)) due to the lack of a precise alternative is a complete and utter failure of the language in my eyes.

Well, __traits(compiles, ...) tests exactly what it's meant to - whether a particular piece of code compiles. And of course, there are cases where a more precise alternative would be nice, but we have the language that we have, and even if we add a few more traits to __traits so that __traits(compiles, ...) is needed less, it'll still be needed in other situations, because that's basically a game of whack-a-mole, and realistically, there are cases where you really do need to check whether a particular piece of code compiles. And a lot of the time, that works perfectly fine.

Of course, there are situations where it becomes problematic - particularly when errors get eaten - and that can result in the wrong overload being used or a particular static if not being compiled in, but there's only so much that we can do about that. And arguably that is a failure of the language, but even then, being able to test a template constraint without instantiating a template would only help us in some of the cases. So, while it might be something that we should add to the language, it's just one improvement.

Ultimately, there's no way that we're going to have individual traits in __traits for everything, and __traits(compiles, ...) is going to need to be used pretty often. If we can reduce that with some additional traits in __traits, then arguably we should, but that's never going to cover everything. And there times where you genuinely want to test whether a particular piece of code compiles, because that's really what the template constraint needs to care about.

Given how template constraints and generic code work, ultimately, the solution is often to simply make sure that we test our code thoroughly (which we need to be doing anyway), and that catches a lot of these problems. Obviously, it's better when we can catch problems at compile time, but there are always going to be cases that we can't catch with compile-time checks - e.g. whether popFront actually pops a single element correctly.

So, while I can certainly see why you would consider it a failure of the language to use __traits(compiles, ...), the reality of the matter is that we need to, and it's highly unlikely that that's ever going to change. We might get some __traits that make it so that we need to use __traits(compiles, ...) less, but it's not going away. There are way too many kinds of things that need to be tested for for the compiler to ever account for them all.