Nim Tooling Roadmap

[jmgomez]

• VSCode Extension:

  • Reliable: Go to definition, Find references, Code completion, Error highlighting as you type, Type Hints
  • Fix perf issues (fix NimSuggest perf issues and make sure nimlangserver and NimSuggest is closed when exiting the extension)
  • Cosmetics/UX
    [x] Unify overloading. Consider, proc add(x: Foo, a:int) and proc add(x: Foo, a:string) when hitting x. it should appear as one navigable entry.
    [ ] Be able to run tests directly
    [ ] Preview template/macro expansion
    [ ] Suggested quick fixes (add missing module, etc)
  • Debug this file
  • Basic refactoring operations
  • Integrated REPL (we could start with a NimScript based one until the next VM is ready)
  • Suggested quick fixes (add missing module, etc)
  • Custom nim panel that shows the nimlangserver status and notifications
  • nimble task runner
  • Improve nimlangserver error reporting
  • nimlangserver
    • Nimble driving the project setup (Nim version selection, entry point files for nimsuggest, etc.)
    • Improve process management (see Rework project discovery langserver#203)
    • chronos migration

• Debugger support

  • Beautify C/C++ backends (i.e. remove mangling and optionally show temps) on LLDB
  • Beautify js
  • Debugger adapter

• Package manager

  • Deterministic builds
  • Dependency resolver
  • Task-specific dependencies (“depend on serialization for tests”)
  • Monorepo / multiple projects per git repo
  • Multi-version support - two versions of the same dependency to turn diamond deps into a tree of deps

• Code formatter

[Clonkk]

Nice

[nikhilsimha]

🔥

[miguelmartin75]

I wanted to emphasize the LSP should be more stable. It crashes pretty frequently. Please prioritize LSP features & stability over VSCode extension features (I am biased, I don't use VSCode).

Here are some questions/feedback:

Beautify C/C++ backends (i.e. remove mangling and optionally show temps) on LLDB

How is this going to be achieved? Is this my RFC proposal (#540)? If not & you remove mangling then how will overloaded symbols get resolved?

Package manager

The list of features here looks more like a build system, which would be a great addition. Here are some suggestions:

1. UX issues when mixing C/C++/ObjC code with Nim (fixing these issues would enable slow migration for users to Nim from C/C++/ObjC): e.g. if one places {.compile: "a.cpp"} in .nim files, it would be good if the compiler outputs a compile_commands.json file. This will enable you to effectively work in a mixed C/C++/ObjC/Nim codebase. Then you would have LSP support right out of the box for your C-based files. Right now, you have to do either (both of the below options are not user-friendly):

   a. Produce a compile_commands.json yourself, which isn't too difficult with, but this is unnecessary friction as you need to set the nimcache dir via --nimcache:dir & parse/transform the <project>.json file to compile_commands.json, or

   b. Use CMake or some alternative, where the Nim compiler is called from this build system. This build system will produce a compile_commands.json for you, but you will not be able to use {.compile.} pragmas: you need to instead build via linking library files or producing object files and link the binary yourself (both are non-optimal).

2. If we want to enable mixed C/C++/ObjC/Nim codebases, have direct CMake support, e.g. https://github.com/stoneface86/example-nim-cmake, alternatively, we could make a CMake alternative written in Nim that would produce:

   • Xcode project files or Visual Studio project files. Xcode is the main feature that would be nice. For MacOS/iOS development & the Metal debugger: Xcode is necessary; not sure about Visual Studio (for debugging perhaps?). Meson has pretty readable code in Python that should be easy to port to produce Xcode project files (I assume the same for Visual Studio)
   • Instead of producing a Makefile/Ninja file (akin to CMake), the build system's internal dependency graph could be used to produce the build script, i.e. just use the nimscript as-is as a Makefile.

3. Request the Nim compiler to output to a build directory local to the project folder by default, e.g. project is ~/repos/my_nim_project and it would build artifacts to ~/repos/my_nim_project/build and the cache for nim can exist in ~/repos/my_nim_project/build/cache or ~/repos/my_nim_project/build/nimcache

   • See (1.a) for why
   • Additionally, this aligns nicely with atlas's approach of placing/maintaining your dependencies relative to your project folder root

[elcritch]

Fantastic!

Debugger support
[ ] Beautify C/C++ backends (i.e. remove mangling and optionally show temps) on LLDB

I started toying with the LLDB / GDB debugger support issue. Folks already provide some scripts for both GDB and LLDB. However, AFAICT, neither debugger appears provide the ability to affect name mangling directly. You can do some custom frame stack printing, etc this SO still seems to be the current state of things.

Perhaps the best route for full demangling while debugging would be to provide a debug-adapter-protocol proxy. Essentially connect to the LLDB/GDB DAP api's then demangle the names and pass it on.

[miguelmartin75]

Perhaps the best route for full demangling while debugging would be to provide a debug-adapter-protocol proxy. Essentially connect to the LLDB/GDB DAP api's then demangle the names and pass it on.

That's one way, but that approach is "in the middle" and wouldn't allow for using lldb or gdb directly through the CLI / their TUI interfaces. Not every user uses DAP. For example, I don't think Xcode or Visual Studio or other LLDB/GDB frontends necessarily use DAP.

A better solution IMO is to follow my suggested changes within the LLDB source itself in #540 (I am biased of course) - GDB appears to be similar (based on @ire4ever1190's comment). LLDB would also be better if it had direct Language support via its language plugin (see links in my RFC), which would tie into Nim's 2024 roadmap for its planned feature: the REPL (i.e. we could have direct REPL support in LLDB). Adding demangling support in debuggers's source directly is the approach other languages take such as D & Rust. Unfortunately, you cannot add demangling support via LLDB's plugin system.

[jmgomez]

@miguelmartin75 we have a few ideas to approach the debugging situations but we dont know yet how we are going to tackle as the initial focus is in the nimlangserver/vscode extension. But your RFC is a good starting point

[Clonkk]

@jmgomez I have several questions if you do not mind :)

• Regarding the VSCode extension, will the improvement be applicable to other IDE (for example, Sublime Text or Intellij) or is the effort focused on LSP + VSCode extension ?

• Regarding nimsuggest, when working with macro-heavy repo (such as https://github.com/SciNim/flambeau) I get regular crashes due to CPU / RAM hogging. What is the plan to have nimsuggest work with macros without using too much resources ?

• About LSP server, what place does nimlsp has, if any (pinging @PMunch since it's his server), in this roadmap ?

[PMunch]

* About LSP server, what place does `nimlsp` has, if any (pinging @PMunch since it's his server), in this roadmap ?

I'm guessing my LSP server isn't considered in this roadmap, since Status seems to be solely focused on the nimlangserver implementation. I would however really like if they properly documented and implemented changes in the nimsuggest module though. Previously I had an issue where changes they made for nimlangserver ended up sending NimLSP into an infinite loop because it entered a codepath they hadn't tested. And the new V3 mode is only available to the external tool and not in the library version of nimsuggest. I tried to have a look at implementing the V3 mode for the library version, but lacking any documentation I couldn't get it to work. So yeah, if anything NimLSP should in theory benefit from all improvements in nimsuggest, but in practice the QC on what is added to nimsuggest has been a bit lackluster and changes are typically only added to the tool version and not the library version which is a real shame.

I really hope that this can change though with this added focus on tooling!

[jmgomez]

@Clonkk the idea is to be as extension agnostics as possible. Obviously some features need to be implemented at the extension level.

We are going to do everything in our hands to make NimSuggest better but it's hard to tell how far we can get it. IC is supposed to help a ton here, we will see. As far a NimLSP vs nimlangserver goes, I cant say much about it. I would love to see all efforts converging into one tool though.

@PMunch Im sorry to hear that changes on the compiler broke NimLSP I know how it feels because that happened to me personally in NimForUE. Anyways, all I can say is that I will document the PR that I do regarding NimSuggest features, cant speak for others.

[evelant]

Total newcomer to nim so please ignore my input if it's not useful. That being said, I'm interested in using nim to write shared code that I can run on any platform.

Since I primarily use react-native at the moment that includes web (js), Android (native) and iOS (native). Looking into nim on iOS/Android I found little other than the brief entries in the compiler guide and a few old experiments. This (at least from the perspective of a newcomer) seems to boil down to "nim compiles to c, and you can use c anywhere, but it's up to you to figure out how to bind to it", or rather you technically could use nim on those platforms but there's no tooling to help with it beyond generating usable c.

Maybe it would be useful to add some tooling that makes it easier for nim to be more seamlessly cross platform such as generating the necessary boilerplate to consume a nim program on mobile platforms?

Another example would be typescript. If tooling could generate .d.ts files for nim compiled to js (or generate typescript code instead of js) it might make using nim in the js ecosystem a whole lot easier. In general tooling to automatically generate types/bindings to/from the standards of other platforms such as the (seemingly unfinished/abandoned) dts2nim might also make it a lot easier to use nim in more contexts.

[ITwrx]

I would like to see the Nim lang file from Aporia updated and merged into GtkSourceView, so that maybe Owlkettle and/or Gintro can support GtkSourceView and people can make gtk-based editors easier.

and here is a GtkSourceView issue that explains the process.

[Araq]

@ITwrx You need to update this file as user defined literals and maybe some keywords have been added since then.

[ITwrx]

@Araq Thanks. I do realize it needs to be updated, but i'm not sure how yet (thanks for the clues on what might have changed). That's why i just added it here.

[ire4ever1190]

One improvement I'd like to see in nimsuggest is having end line/column info for diagnostic messages so that NimLSP and nimlangserver can better display where a hint/warning/error corresponds to

[omentic]

VSCode Extension

Four Nim extensions pop up when searching for Nim in the VSCode marketplace:

The first has not been updated in years. The second is saem's Nim rewrite, and has been the recommended extension for quite some time. I don't know what the third is. The fourth is your fork of saem's extension?

I don't know what the relation between nim-lang/vscode-nim and saem/vscode-nim is, but what strikes me as bad is the old TypeScript extension being first in results and overwhelmingly so - that thing's ancient. I worry that those new to Nim are liable to install it and have a bad time. The authors appear inactive, but judging from rust-lang/vscode-rust#927 there's a chance we can contact Microsoft staff directly and have it be marked deprecated?

[Zectbumo]

As a new comer, I like to install the official one and in your lineup I would choose the fourth, nim-lang.org

[omentic]

I would also add a Language Server section, because there's some serious issues that should be part of the roadmap:

• crashes spawn stuck processes (this issue's been around a long, long time): nimlangserver and nimsuggest cleanup issues on shutdown (leaving processes still running, after exit of VS Code) langserver#70
• breaks with null rootUri: langserver crashes if rootUri is not set langserver#35
• frequent crashes (not much to be done about this one)

I'll also copy the state of non-VSCode editors from my comment in the old RFC #300 (comment) here:

• Neovim
  • language server feat: Add support for the other nim language server neovim/nvim-lspconfig#2581
  • syntax highlighting feat(nim): added nim parser and queries nvim-treesitter/nvim-treesitter#5556
  • regex syntax highlighting (desired?)
  • status of alaviss/nim.nvim vs. the merged tree-sitter and LSP support?
• Helix
  • language server and syntax highlighting merged Initial Nim language support helix-editor/helix#6123
  • needs to be updated to use alaviss/tree-sitter-nim (uses aMOPel/tree-sitter-nim right now)
  • nimlangserver has issues Helix Edior - nimlangserver exits on startup langserver#59
• Sublime
  • rename Update the plugin's metadata on Package Control NimLime#181
  • install LSP by default Install LSP by default NimLime#185
• Emacs
  • nim-mode, README suggests problems?
• Jetbrains
  • plugin, latest reviews suggest problems?
• Others?

[heysokam]

As a person who loves to have my code formatted in a clean and consistent way, and a big fan of how clangd does its formatting with clang-format, I find it really sad to see this section so empty of even any form of planning or ideas.

I understand that NEP1 is standard... but nimpretty doesn't even allow any form of configurability beyond line size.

[auxym]

As a person who loves to have my code formatted in a clean and consistent way, and a big fan of how clangd does its formatting with clang-format, I find it really sad to see this section so empty of even any form of planning or ideas.

@heysokam Have you tried nph? https://github.com/arnetheduck/nph Works great IMO, though not configurable (similar ideology as black for python, but I like it that way).

[heysokam]

@heysokam Have you tried nph? https://github.com/arnetheduck/nph Works great IMO, though not configurable (similar ideology as black for python, but I like it that way).

@auxym nph is a great improvement over nimpretty. Still, just as hardcoded or even more.
No two codebases are the same. Expecting everyone to conform to one single unified codebase is just delusional.
I would rather format everything manually, no matter how painful that might be, than completely lose control over how I organize and read my own code. 🤷

[mratsim]

No two codebases are the same. Expecting everyone to conform to one single unified codebase is just delusional.
I would rather format everything manually, no matter how painful that might be, than completely lose control over how I organize and read my own code.

go fmt and cargo fmt exactly do that.

[Spirarel]

@heysokam Have you tried nph? https://github.com/arnetheduck/nph Works great IMO, though not configurable (similar ideology as black for python, but I like it that way).

@auxym nph is a great improvement over nimpretty. Still, just as hardcoded or even more. No two codebases are the same. Expecting everyone to conform to one single unified codebase is just delusional. I would rather format everything manually, no matter how painful that might be, than completely lose control over how I organize and read my own code. 🤷

Anti-bike-shedding, one-size-fits-none formatters is the current local-maximum in many communities.

[heysokam]

No two codebases are the same. Expecting everyone to conform to one single unified codebase is just delusional.
I would rather format everything manually, no matter how painful that might be, than completely lose control over how I organize and read my own code.

go fmt and cargo fmt exactly do that.

zig fmt even forces the addon to be active by default in your codebase on save, and you have to search for how to disable it or else you are force-formatted into the one-and-only-allowed style.

But community prevalence does not make a decision good, it just makes it prevalent. And lets be real, force-formatting is valued as a standard just because its the simplest/laziest solution to implement.

But prevalence is not quality. Prevalence is just prevalence.
Don't delude yourself into the lazier (and of course simpler) solution to the problem, just because the cost of that solution is lower.

[Araq]

A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines.

[bpr]

No two codebases are the same. Expecting everyone to conform to one single unified codebase is just delusional.
I would rather format everything manually, no matter how painful that might be, than completely lose control over how I organize and read my own code.

go fmt and cargo fmt exactly do that.

cargo fmt is quite configurable, go fmt is not.

[nickysn]

* [ ]  crashes spawn stuck processes (this issue's been around a long, long time): [nimlangserver and nimsuggest cleanup issues on shutdown (leaving processes still running, after exit of VS Code) langserver#70](https://github.com/nim-lang/langserver/issues/70)

This is now fixed.

[nickysn]

I would also add a Language Server section, because there's some serious issues that should be part of the roadmap:
* [ ] breaks with null rootUri: langserver crashes if rootUri is not set langserver#35

Fixed in v1.2.0.

I'll also copy the state of non-VSCode editors from my comment in the old RFC #300 (comment) here:

* Helix
  
  * [ ]  nimlangserver has issues [Helix Edior - nimlangserver exits on startup langserver#59](https://github.com/nim-lang/langserver/issues/59)

Crash is fixed in v1.2.0 and an improved fix is committed in the master branch that takes into account rootPath, instead of rootUri (helix sends rootPath only, without rootUri).

[kraptor]

Did anyone ping @isidorn about the issue of ancient misleading versions of vscode extensions that are no longer maintained? Maybe they can deprecate at least the first one which has no update for 4 years as well... 😅

I don't know about @saem's extension (at least that one is still useful as I see from other users) but the first one should definitely be deprecated, IMHO.