Binary size, speed, and other ideas

[mauricioszabo]

One discussion we always have is how to make binaries smaller, speed up loading process of the editor, etc.

As someone who investigated a little bit on these, I want to share my ideas and somehow explain what could work, what probably won't, and how do can we make spikes/proof of concepts on these ideas.

First things first, how the editor is currently organized: there are some "pain points" for speed and these ideas: first, the fact that "core packages" (packages that come with Pulsar, like settings-view, language-c, welcome, about, command-palette, etc) behave differently than non-core packages - they are bundled in the editor and it uses some "metadata" to avoid parsing things. This speed up things a little bit, with the problem of having some duplication of code and other issues.

The second, is that the editor uses code that's shared via the "main process" and the "renderer process". Trying to bundle both generates a code that's essentially invalid, so we may want to discover where this "shared code" stops and somehow try to bundle all "shared code" together and use then on both processes.

The third is, basically, binary packages. Some bundlers do not like these, and we still bundle node_modules with the final product. Ideally, what we want is to bundle everything that can be bundle and minify this final product, and only include native modules on our node_modules if possible. This can get a much smaller binary for Pulsar - as an example, one of my plug-ins have a final bundle sized 3mb; if I include all node_modules in the final product, it takes about 120mb.

Finally, the most complicated one is that originally, Atom used to have an ATOM_RESOURCE_PATH config to ease development of Atom itself. With our current efforts to run the editor via yarn, this code is not that useful anymore (and it's quite clumsy too, and honestly I feel it won't even work anymore). But the code to support this still exists, with lots of "conditional requires" and other stuff. Most bundlers do not like these conditional requires, so we may want to get rid of them - if we can make all requires "final", we may get a better change at bundling.

[mauricioszabo]

Bundle packages

One idea is to piggieback on this PR to remove the idea of "core" packages behaving differently than others. If we could do that, there are some different paths we can try:

Keep Pulsar's package "the same" but bundle core packages and re-use deps

The idea could be to include a resources/core-packages that essentially behaves the same as user-installed packages. Each of these "core" packages could be bundled and minified with some tool, and we could basically re-use all node_modules' dependencies between all packages. So for example, suppose that tree-view and tabs packages have dependencies pathwatcher (a binary dependency) and react (a non-binary dependency). We could bundle react inside both tree-view and tabs, and install a single version of pathwatcher for both, generating the following directory list:

+- opt
+-- pulsar
+---- core-packages
+------ node_modules
+-------- pathwatcher
+---------- ...
+------ tree-view
+-------- package.json
+-------- index.js
+------ tabs
+-------- package.json
+-------- index.js

Some benchmark I tried on this showed that it's slight slower than bundling the packages inside the editor (as we're currently doing) but it can make the final binary be smaller

Create "meta-packages" that are basically bundles of packages

This could create a meta-package called "core" and inside this meta-package, we could have all "core" packages bundled in a single, final product. This makes it easier to re-use dependencies AND will make the final bundle way smaller, but this needs changes on ppm, on settings-view and basically the package management system.

It also means that disabling a package will not make the editor avoid loading the package's main module, but to be honest, the current implementation of Pulsar already does not allow that.

What I tried

I tried the first option for now. I tried to use webpack because it was the only tool I know that allowed a generic config that works with most (but not all) packages. The end results were not good, honestly - it did bundle some packages, but most packages need more work (some use pegjs to make grammars which needs transpilation, and others use export default which is handled in a special way inside Pulsar but only if we don't transpile the package). It was not faster to load the editor, but the packages that did work indeed got 40% to 80% smaller

[DeeDeeG]

Atom had a script to not include un-needed files from the final bundle, as part of script/build. And there were a lot of not needed files.

If we can somehow curate (delete some of) the files, after building/compiling, but just before the final bundle is packaged, that could be a powerful and flexible (if potentially unwieldy or prone to becoming out of sync) way to decrease the size of the final bundle.

(Yes, this file inclusion/exclusion did need tweaking from time to time over at the Atom main repo to avoid breakages.)

[confused-Techie]

On this note, I added a lot more exclusions on my Fix dugite PR, and I was able to decrease the size I think about 0.8GB, but that was after having to include the build scripts on Windows, which itself increased it a bit over 1GB. So one step forward two steps back.

But that does prove that it's possible to reduce our size a non-insignificant margin by file exclusions alone

[Daeraxa]

Binary size is definitely something we need to slim down. We get enough stick with it on Joplin at 1/3 of the size of the Pulsar binary, nearly 300mb is a bit too much. VSCodium is between 80 and 110mb and the really slim ones like Lite-XL and Lapce come in under 10mb.