MOP _и_ AST? Por que los dos?

[masak]

Haxe's way of injecting fields into classes at class building time reminded me of a thing I was wondering about years ago, but never opened an issue for: if you have full access to the AST of your classes, do you also separately need a MOP? Or is a meta-object protocol the kind of thing a language has when it doesn't give you that kind of full access to the program code at an AST level?

This is not a rhetorical question where I feel I know the answer already. The ur-MOP, CLOS, came about in an language/system environment (Common Lisp) that classically gives you a ridiculous amount of access to the AST; if nothing else, that's a data point saying "no, you do need both".

And, I mean, I can see that they are somehow separate things. Maybe akin to the distinction in HTML/DOM between attributes (a syntactic concept), and properties (a semantic concept). You can toggle the checked property of your checkbox on and off until you're blue in the face... this does not alter the truth of what the checked attribute of same-said checkbox was, and will remain for the lifetime of the document. Similarly, the AST of the original program is what it is, at some point (during compilation) a MOP could essentially take over and be the "source of truth" for what a class represents, and the AST would presumably turn into a fixed, historical view of things at that point.

The difference here between AST/MOP and attributes/properties, is that in Alma's case, macros (and similar "macro-ish" transformations) introduce the notion of changing the document directly, possibly making a separate MOP level moot.

On the other hand, ASTs are always beholden, to some extent, to the syntax they came from. A MOP can be more its own thing, with its own non-hierarchical structures. On the third hand, an AST could be that too, for some wider definition of AST. (See, as usual, IntelliJ's PSI.)

So, I don't know. I suspect this is a hard question because not enough people have been asking it, and not enough people have been looking for answers. Either that, or I haven't run into those people and their answers yet.

[masak]

As time passes, I find myself leaning more and more strongly towards "they should both exist, and be separate".

I have two new pieces of evidence, for some very loose definition of "evidence":

• Swift macros. I'd need a lot more time and headspace to be able to state clearly what I think is great about them, but... one thing they do well is to (it seems to me) interact with class declarations and their members. While this could still be phrased directly in terms of ASTs (and maybe it is?) the amazing difference compared to Lisp- and Scheme-like macros is that it doesn't feel like we're working on just syntax anymore; we're working with full awareness of the declaration structure of, say, a class declaration.

• Herb Sutter's proposed C++ metaclasses (later renamed to metafunctions, if I understood it right in that keynote). The API here seems like it's in dialogue with a set of "mirror"-like objects, and something like t.add_virtual_destructor(); seems to me to be quite high-level and no longer just AST.

[vendethiel]

I think it's impossible to have a discussion about this without talking about phases and binding times.
What can you see, when? Do macros and classes "appear" into existence at the same time?
If macros were only concerned with syntax and parsing, they'd be considered before any kind of analysis, and classes would just be a node amongst others.
If you consider that macros need to be able to interact with the MOP, you need to interleave both, but how isn't always obvious.

If you have a macro call inside of a class, should it be able to retrieve said class? Either by name or by "surrounding context"? In that case, what should the class look like? A stub, hole-y node, waiting to be completed? (that actually sounds like a quote if you get very very very creative, because it has a hole and because it shouldn't appear in the final AST).

On the other hand, if you collect macros inside of the class body and delay them (that is, after the initial declaration but before "sealing" or - to reuse Raku vocab - "composing" the class) to run them all at once, you might be able to provide a more sensible view of the class.

Early-binding languages and mop<=>macros interactions is not something that's well-researched, I think, and it kind of shows.

[masak]

I think it's impossible to have a discussion about this without talking about phases and binding times.
What can you see, when? Do macros and classes "appear" into existence at the same time?

I think this is the right way to approach the whole thing, yes.

I hereby declare my intention to review a few languages from the above perspective — what can you see, when? — in order to shake out some understanding from that about what modern languages actually do.

My candidate languages will be: Java, Perl 5, Raku, Python, and CLOS. The focus will be on the object metamodel, and at which exact point during the compilation process it becomes accessible.