coc-metals spawns wayyy too many CompileServer processes

[rockpunk]

Hi! Not sure where the right place to post this is, but i've been trying to use vim-metals and coc-metals on a production scala monorepo. We have about 40 submodules in the monorepo. After bloop imports the project, things work admirably well, but after a while of jumping between files things turn a bit to mush. The laptop starts heating up and the fans are occasionally pegged. One time, the entire laptop started swapping and crashed as well.

Upon investigating what might be causing this, I noticed that after opening a scala project with coc-metals and working with it, i noticed a ridiculous number of spawned jvm processes, all related to the CompileServer. (Very little documentation about this exists, so i'm not really sure what it's supposed to be doing other than, well... compiling things...)

$ ps -ef | grep CompileServer | wc -l
1903
$ jps -lm | head
61440 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
65537 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
61444 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
71684 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
73734 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
63496 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
69643 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
65550 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
71697 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
67600 scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer

Not sure why so many CompileServers need to be running.

The number of processes tend to fluctuate during use. After a while things start going down and eventually disappear completely.

$ date && ps -ef | grep CompileServer | wc -l
Fri Oct 22 17:38:19 PDT 2021
1557
$ date && ps -ef | grep CompileServer | wc -l
Fri Oct 22 17:39:16 PDT 2021
1277
$ ps -ef | grep vim | grep -v grep
  502 53746 53745   0  4:46PM ??         0:12.84 /usr/local/bin/node --no-warnings /Users/slayland/.vim/plugs/coc.nvim/build/index.js
  502 53748 53745   0  4:46PM ??         0:00.09 /Users/slayland/.pyenv/versions/3.7.3/envs/nvim/bin/python3.7 -c import sys; sys.path = list(filter(lambda x: x != "", sys.path)); import neovim; neovim.start_host() script_host.py
  502 53745  4528   0  4:46PM ttys000    1:13.60 nvim

Strangely, none of the CompileServer processes show up as child processes from vim directly.

─┬= 53745 slayland nvim
 ├─┬= 53746 slayland /usr/local/bin/node --no-warnings /Users/slayland/.vim/plugs/coc.nvim/build/index.js
 │ ├─── 53757 slayland /usr/local/Cellar/node/16.9.1/bin/node /Users/slayland/.config/coc/extensions/node_modules/coc-json/node_modules/vscod
 │ └─┬─ 53995 slayland /Users/slayland/.sdkman/candidates/java/21.1.0.r11-grl/bin/java -Xss4m -Xms100m -classpath /Users/slayland/Library/Cac
 │   └─── 53997 slayland java -Xss4m -XX:MaxInlineLevel=20 -XX:+UseParallelGC -classpath /Users/slayland/Library/Caches/Coursier/v1/https/rep
 └──= 53748 slayland /Users/slayland/.pyenv/versions/3.7.3/envs/nvim/bin/python3.7 -c import sys; sys.path = list(filter(lambda x: x != "", sys.p

And they all seem to be spawned by a bash subshell which is immediately returned to the init process

 502 74544     1   0  5:21PM ??         0:00.02 bash /usr/local/Cellar/scala/2.13.0/libexec/bin/scala scala.tools.nsc.fsc.CompileServer
  502 74557     1   0  5:21PM ??         0:00.02 bash /usr/local/Cellar/scala/2.13.0/libexec/bin/scala scala.tools.nsc.fsc.CompileServer
  502 74570     1   0  5:21PM ??         0:00.02 bash /usr/local/Cellar/scala/2.13.0/libexec/bin/scala scala.tools.nsc.fsc.CompileServer
  502 74584     1   0  5:21PM ??         0:00.01 bash /usr/local/Cellar/scala/2.13.0/libexec/bin/scala scala.tools.nsc.fsc.CompileServer

After waiting for a while and not doing anything in nvim, the CompileServer processes finally die on their own.

$ ps -ef | grep CompileServer | grep -v grep

However, immediately after I execute a simple gd to jump to a definition in an already existing buffer, 3 CompileServer processes are spawned, which soon grow to over 200!

$ ps -ef | grep CompileServer | grep -v grep
  502 77746 77744   0  5:55PM ??         0:00.02 bash /usr/local/Cellar/scala/2.13.0/libexec/bin/scala scala.tools.nsc.fsc.CompileServer
  502 77758 77746   0  5:55PM ??         0:00.95 /Users/slayland/.sdkman/candidates/java/8.0.262.hs-adpt/bin/java -Xmx256M -Xms32M -Xbootclasspath/a:/usr/local/Cellar/scala/2.13.0/libexec/lib/jline-2.14.6.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-compiler.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-library.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-reflect.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scalap-2.13.0.jar -classpath "" -Dscala.boot.class.path=/usr/local/Cellar/scala/2.13.0/libexec/lib/jline-2.14.6.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-compiler.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-library.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-reflect.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scalap-2.13.0.jar -Dscala.home=/usr/local/Cellar/scala/2.13.0/libexec -Dscala.usejavacp=true scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
  502 77760 77744   0  5:55PM ??         0:00.01 bash /usr/local/Cellar/scala/2.13.0/libexec/bin/scala scala.tools.nsc.fsc.CompileServer
  502 77772 77760   0  5:55PM ??         0:00.55 /Users/slayland/.sdkman/candidates/java/8.0.262.hs-adpt/bin/java -Xmx256M -Xms32M -Xbootclasspath/a:/usr/local/Cellar/scala/2.13.0/libexec/lib/jline-2.14.6.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-compiler.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-library.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-reflect.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scalap-2.13.0.jar -classpath "" -Dscala.boot.class.path=/usr/local/Cellar/scala/2.13.0/libexec/lib/jline-2.14.6.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-compiler.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-library.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-reflect.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scalap-2.13.0.jar -Dscala.home=/usr/local/Cellar/scala/2.13.0/libexec -Dscala.usejavacp=true scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer
  502 77773 77744   0  5:55PM ??         0:00.01 bash /usr/local/Cellar/scala/2.13.0/libexec/bin/scala scala.tools.nsc.fsc.CompileServer
  502 77785 77773   0  5:55PM ??         0:00.19 /Users/slayland/.sdkman/candidates/java/8.0.262.hs-adpt/bin/java -Xmx256M -Xms32M -Xbootclasspath/a:/usr/local/Cellar/scala/2.13.0/libexec/lib/jline-2.14.6.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-compiler.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-library.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-reflect.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scalap-2.13.0.jar -classpath "" -Dscala.boot.class.path=/usr/local/Cellar/scala/2.13.0/libexec/lib/jline-2.14.6.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-compiler.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-library.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scala-reflect.jar:/usr/local/Cellar/scala/2.13.0/libexec/lib/scalap-2.13.0.jar -Dscala.home=/usr/local/Cellar/scala/2.13.0/libexec -Dscala.usejavacp=true scala.tools.nsc.MainGenericRunner scala.tools.nsc.fsc.CompileServer

$ ps -ef | grep CompileServer | grep -v grep | wc -l
202

While I'm not sure that CompileServer is for sure the reason my laptop starts consuming a ridiculous amount of memory, it seems like a likely culprit. JVM processes aren't cheap.

Any idea of what craziness might be causing this? Is this a misconfiguration on my end? Or does metals or bloop or something really spawn one server per file that needs to be compiled??

If I can get this memory management under control, VIM + metals might actually be a workable IDE for me in production.

Thanks for any help y'all might have and appreciate all the work you've put in to make scala development featureful on vim.

[ckipp01]

Hey @rockpunk, thanks for sending this in. At first glance, I haven't seen this before. I've transfered this over to the main Metals repo since this isn't really specific to coc-metals, but seems more related to metals/presentation compiler. Possibly one of the other maintainers may have some more insight on this.

[dos65]

@rockpunk which bsp-server do you use? It seems that CompileServer process is spawned by your build tool/bsp-server. I've never seen this process before and as I know Metals/Bloop/Sbt don't spawn such process.

[rockpunk]

Thanks y'all! I'm using bloop server 1.4.10 spawned by vim (coc-metals plugin in particular). Actually, upon further investigation, I also had the sbt-bloop plugin installed for 1.4.8. I updated the sbt bloop plugin to 1.4.10, and reran sbt -Dbloop.export-jar-classifiers=sources bloopInstall. Now, only 2 CompileServer instances are running after multiple metals operations and they do not appear to be exploding.

I'm going to keep an eye on things to see if it's fixed, but it appears this might have all been due to a version mismatch! I'll post after using this for a couple days to see if this indeed fixed the issue, but so far things look promising.