symbols not loading and nimsuggest being opened for each file when project mapping is defined

[emizzle]

Affected versions

1. vscode-nim version: 1.4.1
2. nimlangserver versions:
   • 1.6.0
   • 1.8.1
3. nim version: 2.0.14
4. nimble versions tried (from shell env in VSCode PATH at least):
   • 0.13.1
   • 0.16.4

❯ nimble --version
      Info: Using the environment variable: NIMBLE_DIR='/Users/egonat/repos/codex-storage/nim-codex/vendor/.nimble'
nimble v0.16.4 compiled at 2025-01-31 02:30:26
git hash: 4e7e646b9819b72044e5262c2800e9f7ccfd1b16

My assumption is that the configuration is incorrect, however I do see some "RegEx matched` messages:

This plugin almost never works for me when working inside of an NBS project, could that be the cause of the issue?

In settings, we have:

"nim.projectMapping": [
    {
        "projectFile": "/Users/egonat/repos/codex-storage/nim-codex/tests/testCodex.nim",
        "fileRegex": "tests/codex/.*\\.nim"
    },
    {
        "projectFile": "/Users/egonat/repos/codex-storage/nim-codex/tests/testIntegration.nim",
        "fileRegex": "tests/integration/.*\\.nim"
    },
    {
        "projectFile": "/Users/egonat/repos/codex-storage/nim-codex/codex.nim",
        "fileRegex": "codex/.*\\.nim"
    }
],

Output:

DBG Starting nimlangserver                     version=1.8.1 params="(clientProcessId: none(int), transport: some(stdio), port: 0)"
DBG Starting stdio server                     
DBG [Processsing Message]                      request="\"initialize\""
DBG Initialize received...                    
DBG Registering monitor for process            pid=49553
DBG Initialize completed. Trying to start nimsuggest instances
DBG [Processsing Message]                      request="\"initialized\""
DBG Client initialized.                       
DBG Requesting configuration from the client  
DBG [Processsing Message]                      request="\"textDocument/didOpen\""
DBG New document opened for URI:               uri=file:///Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
DBG Auto-guessing project file for             file=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
DBG getProjectFile                             project=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim fileUri=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
DBG Document associated with the following projectFile uri=file:///Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim projectFile=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
DBG Will create nimsuggest for this file       uri=file:///Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
DBG Starting createOrRestartNimsuggest         projectFile=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim uri=file:///Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
DBG Using {nimVersion}                         nimVersion="Nim Compiler Version 2.0.14 [MacOSX: arm64]"
DBG Creating new nimsuggest project            projectFile=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
INF Starting nimsuggest                        root=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim timeout=120000 path=/Users/egonat/repos/codex-storage/nim-codex/vendor/nimbus-build-system/vendor/Nim/bin/nimsuggest workingDir=/Users/egonat/repos/codex-storage/nim-codex
DBG Parsing nimsuggest capability              capability=con
DBG Nimsuggest Capabilities                    capabilities={con}
ERR NimSuggest Error (stderr)                  err="SIGSEGV: Illegal storage access. (Attempt to read from nil?)\n"
DBG NimSuggest needed to be restarted due to an error 
WRN Server stopped.                            projectFile=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
DBG ShowMessage                                message="Server failed with SIGSEGV: Illegal storage access. (Attempt to read from nil?)\n."
DBG Nimsuggest port                            portLine=
ERR Failed to parse nimsuggest port            portLine=
ERR Nimsuggest nextLine                        nextLine=
DBG NimSuggest needed to be restarted due to an error 
WRN Server stopped.                            projectFile=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
DBG ShowMessage                                message="Server failed with Failed to parse nimsuggest port."
DBG ShowMessage                                message="Opening file:///Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim"
DBG Nimsuggest initialized successfully        projectFile=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim
DBG ShowMessage                                message="Nimsuggest initialized for /Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim"
DBG processQueue                               size=1
DBG Nimsuggest is not working, returning empty result... port=0 command="known \"/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim\":0:0"
DBG ShowMessage                                message="/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim is not compiled as part of project /Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim.\n  In orde to get the IDE features working you must either configure nim.projectMapping or import the module."
DBG [Processsing Message]                      request="\"extension/tasks\""
DBG Received tasks                             rootPath=/Users/egonat/repos/codex-storage/nim-codex
DBG Got response for the didChangeConfiguration registration: res={}
DBG Received the following configuration       configuration="[{\"project\":[\"ethers.nim\"],\"projectMapping\":[{\"projectFile\":\"/Users/egonat/repos/codex-storage/nim-codex/tests/testCodex.nim\",\"fileRegex\":\"tests/codex/.*\\\\.nim\"},{\"projectFile\":\"/Users/egonat/repos/codex-storage/nim-codex/tests/testIntegration.nim\",\"fileRegex\":\"tests/integration/.*\\\\.nim\"},{\"projectFile\":\"/Users/egonat/repos/codex-storage/nim-codex/codex.nim\",\"fileRegex\":\"codex/.*\\\\.nim\"}],\"test-project\":\"\",\"buildOnSave\":false,\"buildCommand\":\"c\",\"runOutputDirectory\":\"\",\"lintOnSave\":false,\"enableNimsuggest\":false,\"provider\":\"lsp\",\"useNimsuggestCheck\":false,\"logNimsuggest\":false,\"licenseString\":\"\",\"nimsuggestRestartTimeout\":60,\"nimprettyIndent\":0,\"nimprettyMaxLineLen\":80,\"lsp\":{\"path\":\"/Users/egonat/.nimble/bin/nimlangserver\"},\"inlayHints\":{\"typeHints\":{\"enable\":false},\"parameterHints\":{\"enable\":true},\"exceptionHints\":{\"enable\":true,\"hintStringLeft\":\"🔔\",\"hintStringRight\":\"\"}},\"debug\":{\"type\":\"lldb\"},\"notificationVerbosity\":\"info\",\"notificationTimeout\":15,\"transportMode\":\"stdio\",\"lspPort\":0,\"formatOnSave\":true,\"useNimCheck\":true,\"nimExpandArc\":false,\"nimExpandMacro\":false,\"nimbleAutoSetup\":true}]"
DBG nimCheck                                   filePath=/Users/egonat/repos/codex-storage/nim-codex/tests/integration/multinodes.nim nimPath=/Users/egonat/repos/codex-storage/nim-codex/vendor/nimbus-build-system/vendor/Nim/bin/nim_commit_bf4de6a394e040d9810cba8c69fb2829ff04dcc6

The SIGSEGV then continues to occur and no symbols are ever loaded.

Have tried multiple versions of fileRegex, all with the same result, eg:

• "fileRegex": "tests/integration/(.*).nim"
• "fileRegex": "tests/integration/.*\\.nim"
• "fileRegex": "tests\/integration\/.*\\.nim"

Have tried enabling nimcheck vs nimsuggest

Do you have any other info that may help us? All of us on my team are struggling to get this to work and it would be a huge boon for us if it did.

[jmgomez]

Notice as long as there is a nimble file, it is preferred as an entry point.
Unfortunately, nimsuggest seems to be crashing. Are you launching code sourcing env? ./env.sh code?

[emizzle]

Notice as long as there is a nimble file, it is preferred as an entry point. Unfortunately, nimsuggest seems to be crashing. Are you launching code sourcing env? ./env.sh code?

Yes, that is exactly how I launch VS Code.

[emizzle]

Which nimsuggest is the plugin using? Is the one from NBS, from my local installation, or the one specified in the VS Code integrated terminal, ie which nimsuggest?

[jmgomez]

It should be using the one your project needs. Its driven by the nimble requires. You can confirm this is the case in the extension panel:

[emizzle]

What I managed to do was use a integrated terminal startup script to use a version of nimble that is compatible with nimlangserver. I also installed nimlangserver on master. The version of nimsuggest should have been fine. Now I'm not getting nimsuggest crashes, but I'm getting far too many instances of nimsuggest. Enough for all 32GB of memory to be consumed.

[jmgomez]

Glad to see you get it working now. The instances should be removed after a period of inactivity. You can control it with the setting: nim.nimsuggestIdleTimeout - the timeout in ms after which an idle nimsuggest will be stopped. If not specified the default is 120 seconds.
By default it tries to reuse the instances via nimsuggest known but if it fails, it will open a new one. It seems the project is too much for nimsuggest unfortunately

[emizzle]

Is nim.nimsuggestIdleTimeout an option for the plugin or is that only on nimlangserver? I tried adding to VS Code settings but it appears to not be honoured:

It seems the project is too much for nimsuggest unfortunately
Is this because of the runaway memory consumption of the nimsuggest instances?

~650MB is not bad if the number of nimsuggest instances was constrained to 3 as I was attempting to do with my project mapping (shown in the issue description). Is there something wrong with my config? Otherwise, why are there so many nimsuggest instances being created?

[jmgomez]

Is nim.nimsuggestIdleTimeout an option for the plugin or is that only on nimlangserver? I tried adding to VS Code settings but it appears to not be honoured:

This should be the case. If they dont close its likely cause they are still active. For every file open that you work on, it opens a new instance. Could it also be the case that nimsuggest chk is waking them up somehow in case you dont have many files open.
To be sure, you could skip nimsuggest chk by setting nim.useNimCheck = true

[emizzle]

For every file open that you work on, it opens a new instance

I'm confused, I thought it was only supposed to open a single nimsuggest instance per project mapping? Also, I don't have as many files open as there are nimsuggest instances.

you could skip nimsuggest chk by setting nim.useNimCheck = true

I had this option enabled and it didn't help my case.

[jmgomez]

Project mapping only applies when not in a nimble project. As I said before, "By default it tries to reuse the instances via nimsuggest known but if it fails, it will open a new one. It seems the project is too much for nimsuggest unfortunately"
In other words, in an ideal scenario where nimsuggest works fine all the time, it will only have one instance. Which, given the size of this project, it isnt the case so it potentially open a new instance per file you have open (instances will be close after a period of inactivity)

[arnetheduck]

when not in a nimble project.

I suspect there's a conflict there with nimbus-build-system projects that have a nimble file but don't use it for package management per se - ie nimbus-build-system sets up a "nimble emulation layer" in order for tests from the nimble file to be runnable, but the nimble file itself is not correct / complete - the expectation is that the nim from n-b-s is used in this case.

[emizzle]

I suspect there's a conflict there with nimbus-build-system projects that have a nimble file but don't use it for package management per se - ie nimbus-build-system sets up a "nimble emulation layer" in order for tests from the nimble file to be runnable, but the nimble file itself is not correct / complete - the expectation is that the nim from n-b-s is used in this case.

I was able to workaround issues with NBS nimble by symlinking nimble from my user home directory into the NBS path in the integrated terminal (codex-storage/nim-codex#1096). It's hard to know what environment the plugin uses to run its commands, however, so I'm not entirely sure if this workaround affects the plugin as much as I think it might. It did start working after that, but that's when I started hitting issues with too many nimsuggest instances opening.

To force project mapping mode, I tried removing ./codex.nimble (and restarting vs code), but alas, no luck in reducing the number of nimsuggest instances and still luck resolving symbols.

Due to #135, OOM errors with too many nimsuggest instances, and no symbol resolution working, I disabled this plugin and re-enabled the older nimsaem extension and low and behold symbol resolution (ie "go to definition") starting working!

[jmgomez]

Since you are in MacOS, maybe the process arent killed due to this: status-im/nim-chronos@36d8ee5