Question: Can you bundle dependencies into the main module output?

[millimoose]

I'm attempting to migrate a cobbled-together Electron+Webpack build to electron-webpack and electron-builder to have an integrated solution with HMR. Our current build bundles all dependencies into the main entry point file like you would in a webapp (before bundle splitting) to take advantage of tree-shaking and reduce the final size.

I tried adding my production dependencies to whiteListedModules, but electron-builder still added my entire node_modules to the output and broke native dependency loading in a cryptic way which worked in the ad-hoc build.

Is it possible to make this sort of setup work, or is there a a specific reason why it's not supported?

[loopmode]

@millimoose first of all: sorry that there hasn't been any reaction, at all, to your issue in months. As you've probably seen, issues are piling up and remain unaddressed for months.. we lack manpower..

Based on previous conversations with @develar, I assume that one of his core ideas about an electron app vs a web app is that we just don't have to care about file size or loading times, code splitting and related optimizations. Because hey! It's all bundled up and available in no time.
And I basically agree with that philosophy.
I agree also that reducing bundle size is not a real concern for an electron app because a couple of megabytes are nothing compared to the many dozens of megabytes of electron, chromium etc itself.
We won't ever be able to ship an electron app that is "small" in file size..
(BTW check out Google's Carlo project. It's like electron in many ways, but it relies on chrome or chromium being installed on the user's machine - allowing tiny output almost as small as your Js bundle)

So.. while I cannot provide any solution, I hope I could explain the reason why electron-webpack just bundles up everything :)

[millimoose]

Hey, thanks for getting to me anyway! Fair enough if that’s your project’s call; I have a hunch the difference was significant for my project but it’s been a while I can’t really reproduce it on short notice. My main motivation IIRC was hoping to migrate a cobbled-together build to electron-webpack to hopefully have main process HMR and I was trying to have the new setup work as closely to the old one as possible to avoid some bugs I’d been seeing, so I might try and see if I can figure out how to use that feature on its own with my current build. —dV

…

________________________________ From: Jovica Aleksic <[email protected]> Sent: Friday, May 17, 2019 7:09 pm To: electron-userland/electron-webpack Cc: David Vallner; Mention Subject: Re: [electron-userland/electron-webpack] Question: Can you bundle dependencies into the main module output? (#268) @millimoose<https://github.com/millimoose> first of all: sorry that there hasn't been any reaction, at all, to your issue in months. As you've probably seen, issues are piling up and remain unaddressed for months.. we lack manpower.. Based on previous conversations with @develar<https://github.com/develar>, I assume that one of his core ideas about an electron app vs a web app is that we just don't have to care about file size or loading times, code splitting and related optimizations. Because hey! It's all bundled up and available in no time. And I basically agree with that philosophy. I agree also that reducing bundle size is not a real concern for an electron app because a couple of megabytes are nothing compared to the many dozens of megabytes of electron, chromium etc itself. We won't ever be able to ship an electron app that is "small" in file size.. (BTW check out Google's Carlo project. It's like electron in many ways, but it relies on chrome or chromium being installed on the user's machine - allowing tiny output almost as small as your Js bundle) So.. while I cannot provide any solution, I hope I could explain the reason why electron-webpack just bundles up everything :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#268?email_source=notifications&email_token=AAFFATCFPSP5WASYTOPFPYDPV3RDHA5CNFSM4GWI4U42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVVKKJY#issuecomment-493528359>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAFFATFUNRFEPHYHWFDSMZLPV3RDHANCNFSM4GWI4U4Q>.

[millimoose]

I inadvertently had to look into the difference again because an unrelated issue forced me to investigate using unbundled dependencies.

The difference for me is around 500MB of node_modules versus a bundled index.js with a total of 12MB; a final compressed application archive of 300MB versus 180MB; and a corresponding increase in packaging time.

(That said it's possible said issue might force my hand to use unbundled dependencies anyway, which would ironically mean I might look into electron-webpack as-is again now that that's no longer an issue.)

[loopmode]

Hey @millimoose

I'm still not quite sure I understand the problem :)

The difference for me is around 500MB of node_modules versus a bundled index.js with a total of 12MB

Those 500MB of dependencies didn't appear out of nowhere, right? It must be some depdendencies depdendency etc. My point being: Doesn't your app actually need those dependencies? Wouldn't it crash in some case when some sub-sub-dependency is required but not available?

Also.. is it the point that.. That some of your dependencies claim to need some modules/packages, making the bundle huge, but you would actually like to drop those modules despite of the dependency tree?

I'm not sure how unbundled dependencies work in electron, hadn't heard the term before.
I was under the impression that for anything to be able to run in your production app, it must come from inside the app's own asar file. But then again, I don't really know much in that area :)

[millimoose]

I think this will become clearer when I define my terminology. I'll be using "module" to mean a CJS / ES6 module - a .js file that requires/imports other modules, and exports some functions etc; and "package" to mean say a library you get from NPM, which contains a bunch of modules.

Crucially, by "bundling" I'm referring to what Webpack does in the simplest case: which is take a module / file as a starting point, look at what modules it require and which modules those require transitively etc., and emit a single file - the bundle - containing all the code and only the code you need. (Webpack actually goes further, it looks at individual functions the modules export - the ideal is for the bundle to only include those lines of code your app can possibly need and not include those it never will.)

Webpack also supports external modules, which is telling webpack "hey, when somebody imports these, don't actually include them in the bundle or process them at all, they'll be available to require/import in the usual way. electron-webpack does the thing where it declares all dependencies of your project as external, i.e. the bundle will only contain your project's code, and then the Electron app will contain the bundle, and all the production packages from node_modules.

(The whole reason tools like Webpack were created was to take a NodeJS-like project codebase with NPM dependencies, and allow it to be loaded in a browser with a script tag.)

Doing so is the safer way of bundling Node apps and certainly a reasonable default, because even though Webpack tries very hard, not all Node packages will work correctly when bundled, but it's more than you'd think. But quite a lot of them work - out of said 500mb of crud only three libraries gave me any grief and that was mainly due to loading native bindings. Given this, not bundling them up is a nontrivial amount of bloat even relative to Chromium+Electron.

Long story short, what I mean is that out of all of my node_modules, I literally only actually need the 10MB of Javascript code I mentioned makes up the bundle. Thanks to Webpack, this already includes my project, my dependencies, dependencies of my dependencies etc. Webpack's whole schtick is "surgically" carving out the necessary parts of your dependency tree, down to single functions; those modules aren't "dropped", only the parts that aren't ever needed. It acts like this by default, because that's what it's meant to do; the configuration electron-webpack generates explicitly tells Webpack not to, and electron-builder puts them in the app archive by default
. (It does a whole bunch of other useful stuffs, but this is one of the core features.)

This doesn't have a lot to do with Electron or ASARs, they're handle the archiving and loading from said archive here.

[prcdpr]

Until it's fixed on electron-webpack side, you can patch it

Inside package.json add

"scripts": {
  "postinstall": "node patch.js"
}

Create patch.js file in root directory.

Edit: this script can be shortened further, see this post.

// https://github.com/electron-userland/electron-webpack/issues/268

const ewPath = require("path").join(
  __dirname,
  "node_modules",
  "electron-webpack",
  "out",
  "main.js"
);

script = require("fs").readFileSync(ewPath).toString("utf8");
script = script.replace(/const externals =/, "const externals = [];");
script = script.replace(
  'externals.push("source-map-support/source-map-support.js");',
  [
    'externals.push("source-map-support/source-map-support.js");',
    'externals.push("source-map");',
    'externals.push("buffer-from");',
  ].join("")
);

fs.writeFileSync(ewPath, script);

In electron-builder config filter out almost everything from node_modules

files: [
  "!**/node_modules/**",
  "**/node_modules/source-map-support/**",
  "**/node_modules/source-map/**",
  "**/node_modules/buffer-from/**",
],

Results

Loose node_modules

app_setup.exe 66.9 MB (70,243,704 bytes)
app.asar 82.6 MB (86,655,709 bytes)
585ms startup time (avg of 3 attempts)

Bundled app

app_setup.exe 57.5 MB (60,320,864 bytes) -14% reduce
app.asar 5.28 MB (5,540,487 bytes) -93% reduce
147ms startup time (avg of 3 attempts) -75% reduce

I think that additional comment is not needed.
Even official Electron docs recommend bundling the app https://www.electronjs.org/docs/tutorial/performance#7-bundle-your-code

[loopmode]

@prcdpr so source-map and buffer-from do belong into webpack externals?
Interesting - i have already recommend a couple times to simply set externals to an empty array in your custom config and be done with it. Seems that wasn't the best advice or caused troubles with source maps..

[prcdpr]

@loopmode I'm still learning, so I checked my code twice and it turns out that source-map and buffer-from don't have to be in externals however they must be included in node_modules (appropriate files filter as in my previous post).

source-map-support has two dependencies:

"buffer-from": "^1.0.0",
"source-map": "^0.6.0"

This is why we need those two in node_modules.

Thanks for noticing.

So my patch.js is enough to have only:

// https://github.com/electron-userland/electron-webpack/issues/268

const ewPath = require("path").join(
  __dirname,
  "node_modules",
  "electron-webpack",
  "out",
  "main.js"
);

script = require("fs").readFileSync(ewPath).toString("utf8");
script = script.replace(/const externals =.+/, "const externals = [];");

fs.writeFileSync(ewPath, script);

Not sure what you meant by setting externals to empty array in custom config?

electron-webpack/packages/electron-webpack/src/main.ts

Lines 298 to 300 in 0c0bf2a

	if (this.electronWebpackConfiguration.externals != null) {
	return externals.concat(this.electronWebpackConfiguration.externals)
	}

This would have no effect at all or am I missing something?

[loopmode]

Yeah the trick is to use a custom config module that exposes a function, not an object.
In that case, your config function receives the already processed options object, as argument. At that time, electron-webpack has finished doing whatever it does with the config, e.g. populated the externals array. But your function returns the final config object. So if it sets externals to an empty array, it will really be an empty array.

[loopmode]

See #420 (comment)

[prcdpr]

Thanks, I read that discussion and config.externals = []; in Webpack main and renderer configs seems to do the trick.
At least I learned something 😄 Thanks!

After adding config.externals = [];, those are my requires for main.js bundle

And requires for renderer.js bundle

Everything seems good but one thing keeps me wondering... I think that most of these externals were discovered by Webpack as coming from Node or Electron and marked as such.

But how require("source-map-support/source-map-support.js") made into main.js even though external list was cleared?

Is it this code that hardcodes it into the beginning of main bundle?

electron-webpack/packages/electron-webpack/src/targets/MainTarget.ts

Lines 30 to 43 in 8a9d289

	if (configurator.isProduction) {
	configurator.plugins.push(new DefinePlugin({
	__static: `process.resourcesPath + "/${configurator.staticSourceDirectory}"`
	}))
	// do not add for main dev (to avoid adding to hot update chunks), our main-hmr install it
	configurator.plugins.push(new BannerPlugin({
	banner: 'require("source-map-support/source-map-support.js").install();',
	test: /\.js$/,
	raw: true,
	entryOnly: true,
	}))
	return
	}

I'm happy I finally found out how I can bundle my code but seriously it should be a default or at least documented 🤔

[veeramarni]

I have three renderers, one for tray, one for about and main windows. I have noticed all three bundles have all the packages instead of bundling only required packages. How can we make tree shaking and minimize the bundle size to include only the packages that are used by the component?

Here is the code https://github.com/cdmbase/fullstack-pro/blob/master/portable-devices/desktop/webpack.renderer.additions.js#L41

[loopmode]

Hmm. Interesting problem. I'm not sure but i think it should already work the way you want it to - unless you have references/imports in your code, maybe in unexpected places, that pull in "everything"..
Have you used the official webpack analyse tool? (https://github.com/webpack/analyse) It shows, contrary to bundle size analyzers, why dependencies exist, or at least what the dependency chain is. our should be able to see why or at which point the dependencies are pulled in..

But otherwise: I cannot answer your question about how to enable tree shaking in electron-webpack.
(I remember that when using Babel, one had to enable loose modules for treeshaking to work, but it seems you're using typescript anyway)

[loopmode]

Btw If you use lazy loading and import('...') instead of import '...'; in a few critical places (found via analysis or simply in wrapper components around such components that have heavy external dependencies), you can tackle the problem quite efficiency with manual code splitting.