Things to remove to make progress

[saem]

Project to Track this Work: https://github.com/orgs/nim-works/projects/5/views/1

---

I'm making a list of things that need to be removed in order for us to make progress with language development. Notice, I didn't say compiler development, because I want to dispell any notion of this having anything to do with Nim -- this last point isn't up for debate. The rest, however is.

What follows is a list of things to be disabled/removed, at least temporarily, in order to make progress:

1. ObjectiveC backend
2. C++ backend
3. Code Reordering (removed)
4. Hot Code Reloading
5. Parallel and Spawn (removed)
6. Disallow pragmas on nkVarTuple idents (removed)
7. Remove already deprecated bits (in progress)
8. Remove case and for loop statement macros (in progress)

That's it for now, I'll add more the the above list based on the discussion, if any, that ensues.

Will we bring some or all of these back, likely in some form or another, but it's daunting making progress with just piles of mistakes layered on top of each other.

---

Details:

ObjectiveC Support

It's pretty much untested and I'd rather spare CI cycles and simplify our test matrix than take on more scope.

Code Reordering (removed)

It doesn't work and adds a lot of complexity before we have the prerequisite capabilities to support it.

Hot Code Reloading

This hurts, but beyond the slow test, complexities in testament, in the backend, and a number of other places it's yet another thing that slows down development. The compiler having an understanding of packages and modules, and moreover versions/lineage of these is a must via IC before we can correctly implement it. In the mean time, it's another set of gotchas.

Parallel and Spawn (removed)

This is implemented in the cgen backend, the fact that it creates a cyclic dependency between sem and the backend should be reason alone to kill it with fire. Not to mention this is an experimental feature that's completely mismatched with how NimSkull should approach concurrency and parallelism (hint: structured and built upon CPS).

Disallow pragmas on nkVarTuple idents (removed)

This is plain broken for all sorts of pragmas, resulting in nil pointer issues in the compiler, macro/template pragmas can't return correct AST under any circumstance, built-in pragmas like compileTime don't work, etc...

C++ Backend

• semExprs hack was removed in sem (cpp: remove cpp addr/deref handling in sem #290) and much of the necessary fixes were made in cpp: fix var parameter regression #294.
• this is still rather problematic as the interop/compromises wrt to dispatch, type information, codegen, etc... just makes it too difficult to make real progress

[haxscramper]

Done some searching in the code for similar hacks around c++

nimskull/compiler/sem/transf.nim

Lines 448 to 453 in f62f5b9

	proc transformAddrDeref(c: PTransf, n: PNode, a, b: TNodeKind): PNode =
	result = transformSons(c, n)
	if c.graph.config.backend == backendCpp or sfCompileToCpp in c.module.flags: return
	var n = result
	case n[0].kind
	of nkObjUpConv, nkObjDownConv, nkChckRange, nkChckRangeF, nkChckRange64:

change introduced in nim-lang/Nim@e84834d and makes code like

nimskull/compiler/sem/semexprs.nim

Lines 769 to 773 in f62f5b9

	proc newHiddenAddrTaken(c: PContext, n: PNode): PNode =
	if n.kind == nkHiddenDeref and not (c.config.backend == backendCpp or
	sfCompileToCpp in c.module.flags):
	checkSonsLen(n, 1, c.config)
	result = n[0]

introduced in nim-lang/Nim@d933fde "yet another C++ bugfix"

Inheritance is implemented differently between C and C++ backends - C uses composition, C++ uses native inheritance. This might've been theoretically useful if nim allowed properly interfacing with methods of the base objects (overriding them), but that's not implemented.

nimskull/compiler/backend/ccgexprs.nim

Line 838 in f62f5b9

if not p.module.compileToCpp: r.add(".Sup")

C++ code vs C code for the same object definition

// C++
struct tyObject_Main__PYvVJIOD0tjHqPPO9aT7bjw : public RootObj {
  NI field;
};
struct tyObject_Derive__IW5bJ7hrVb5ctgr0HLGk8A : public tyObject_Main__PYvVJIOD0tjHqPPO9aT7bjw {};

// C
struct tyObject_Main__PYvVJIOD0tjHqPPO9aT7bjw {
  RootObj Sup;
  NI      field;
};
struct tyObject_Derive__IW5bJ7hrVb5ctgr0HLGk8A {
  tyObject_Main__PYvVJIOD0tjHqPPO9aT7bjw Sup;
};

Has to generate different expressions when accessing fields, different methods of passing object procedure that accepts parent object.

    var de = Derive(field: 12)
    echo de
    proc takesDe(a: var Main) =
      discard

    takesDe(de)

de.Sup.field      = ((NI)12);
colontmpD_        = dollar__file_195((&de));
if (NIM_UNLIKELY(*nimErr_)) goto LA5_;
T6_[0] = colontmpD_;
echoBinSafe(T6_, 1);
takesDe_file_220((&de.Sup));

  de.field   = ((NI)12);
  colontmpD_ = dollar__file_195((&de));
  T7_[0]     = colontmpD_;
  echoBinSafe(T7_, 1);
  takesDe_file_220(de);

[saem]

It's a fair line of inquiry, it's something I ask myself often.

Long story short, we need just enough C++ for LLVM, once we have an LLVM backend then we can ideally rid ourselves of directly generating C++ in any first class fashion. If we end up not going the LLVM route the C++ backend will be dropped, because I'm just not interested in carrying it around, as you said C has folks covered and things in a broader ecosystem can help smooth out the experience.

---

The longer version, I'll start by rephrasing what you wrote, perhaps it'll convey a little about how I see things:

deeply mapping semantics - Most, if not all, of the performance gains would come from using specialized features which convey more information about invariants which a C++ compiler could use for optimizations.

conflating FFI and transpiling - having the language map many of its features to C++ in regards to producing or consuing C++ code, results in allowing one to incrementally convert existing projects and/or leverage projects that lack a C interface (LLVM is a bit borderline, Qt is right out, IIRC).

Now on to the matter of whether to C++ backend or not.

Deeply mapping semantics is very seductive until you realize the terrible costs to the language itself, unless of course that's what you're hoping to build, which I'm not. As for FFI, this is handy as there is a lot of good working code out there which would be nice to leverage, but this unfortunately also gets conflated with transpiling (think TypeScript to JavaScript) where the aim is to have a rather seamless TS <> JS experience, again not a goal.

[ShalokShalom]

Hi, thanks for your quick and detailed response.

In general, I think a LLVM interface can be very helpful.

It is one of the reasons, why Nim runs on GraalVM and WebAssembly, obviously.

Why this needs C++, is beyond me. I am a bit baffled, since C and Nim both have a LLVM interface.

Here is the native implementation, that generates LLVM IR directly from Nim code.
That has also a couple of other benefits to it, and the C interface could also work, so I dont know why this is an issue without the Cpp interface.

BUT, if it means giving up on NimQml, it would mean giving up C++ is no option to me, since this is literally the reason why I am interested into the language in the first place.

So far as I understood you, is Qt in any way out of the scope of this project?
This would be horrible. To me.

The performance benefits would be completely uninteresting to me, I would use Cyo also with "C speed" and have no issues with that, whatsoever. 😃

Nim is already a very fast language.

[saem]

Having an understanding of the compiler implementation, it's hot irredeemable garbage and the idea of mapping things in such a fashion is a sisyphean task. Look at Haxe, at least they've demonstrably mapped over far more languages with far more production software shipped with it. Anyways, a C++ codegen target is a bad idea post LLVM and without LLVM it'll not be what it is today (pretending like we're dealing with first class objects and procs).

[ShalokShalom]

So the codegen for C and C++ is combined and it would need to be untangled?

Bad question, why not start of with a presumably clean code base, in the form of the linked nlvm?

Like, dropping the vanilla compiler for C and C++ and changing it for the pure LLVM compiler?

It would get rid of one of the standout features of Nim, namely C and C++ compatibility and therefore offer a cleaner codebase and one can argue, vanilla Nim is still compatible with Cyo's codebase, I guess?

How is it with the Javascript compiler?

Can it be separated from the rest of the chain?

[saem]

Effectively headed in a similar direction with respect to a cleaned up c backed in line with the new VM backend that we have. This also untangled more things.

I'm familiar with nlvm I looked through the code a while back, jumping straight in prior to more cleanup didn't seem like fun.

There isn't a clean story of linearly laid out logical steps, see my comment about the state of affairs of the internals. Not only internals, but just layers of not quite implemented or fully considered features. Anyhow, at some point LLVM can be considered, but we're not quite there yet.

[saem]

More removals in this PR: #393

It's still in progress, but at the very least:

• stdlib: remove deprecated modules
• lang: remove deprecated c/C octal prefix
• macros: remove deprecated TNimSym/TypeKinds
• cpp: remove this pragma, breaks codegen
• dialect: remove old case object define
• pragmas: type pragmas must follow generic params
• stdlib: httpcore ==(string, HttpCode):bool removed
• stdlib: removed deprecated module oswalkdir
• stdlib: times, remove deprecated procs
• stdlib: remove deprecated tables.add and sharedtables
• stdlib: remove deprecated os.existsFile/Dir
• pragma: remove the deprecated unroll pragma

[saem]

@SolitudeSF commented that removal of the for loop macro was unfortunate as it's useful for them. Starting a thread here to capture their thinking so it can be referenced again in the future.

@SolitudeSF could you share your thoughts with a reply to this comment? thank you

[SolitudeSF]

nah, i didnt say that its useful, meaning, functionally irreplacable. i can of course just use regular macro. i said that i like that, because it is a macro, but hidden inside usual for loop syntax. and i think that was the point of for loop macros, to make some cases of macros look less custom.

so i dont have strong reasoning against their removal. just my personal feeling for neat syntax.

[disruptek]

So, what's the "better way" of implementing the following functionality:

import foreach

let a = [1, 2, 3]

foreach n in a.items of int:
  echo n, " is an int"

or

import json
import foreach

let j = %* {
  "one": 1,
  "two": "2",
}

foreach k, v in j.pairs of string and JsonNode:
	echo k, " is a string"
	echo v, " is a JsonNode"

(more examples at https://github.com/disruptek/foreach)

[saem]

To me this is a failure with iterators and associated bits.

The two gaps to cover are:

1. Composition, so calling a map like routine to coerce the types
2. Site specification of generic out parameters to get a compile time failure

[disruptek]

I don't know what any of that means to the user.

We users were able to use foreach, but it was removed becase "better ways exist". What are those better ways which we should use to implement this sort of functionality, now that we cannot implement it as shown?

Are you saying the right solution is to write when key isnot string or val isnot JsonNode in my for loops?
Does Nimskull actually require the formerly-implicit items call?
Are you suggesting that somehow the generic inputs should reflect to the programmer the type of the items call's output?

AFAICT, this "gap" only exists because you removed the functionality without offering a replacement. What was the implementation blocking, anyway?

[saem]

There is no immediate solution that I'm offering, it's simply been removed. On a longer term trajectory better things will exist.

I'm shrinking the language and unwinding things so there is a workable core where the features I believe are necessary go in. I'm not interested in trying to follow the prior vision.