compress web app using brotli and serve as such #2037
Conversation
this saves ~38k of flash memory, since brotli is much more effective at compressing the web application. as we serve the web application binary as is, the browser needs to take care of decompression. so we have to look at browser support: as it turns out, brotli compression is available on all browsers since more than four years. as per caniuse.com/brotli, the support for brotli is available for 97% of users today. we can safely assume that all OpenDTU users use a browser with support for brotli encoded content. the gzip-compressed transpiled files in webapp_dist are removed from the repo and the new brotli-compressed transpiled files are added to the gitignore list, as build artifacts shall not be tracked.
|
I like the idea. But I would leave the pre-compiled web app files in the repository to simplify the build using just vscode. I mean, its maybe no problem for a lot of users to install nodejs, use yarn to build the web interface etc. but I like to keep the build as simple and compatible as possible. |
|
I totally understand that your approach is different. I think that people should not build firmware themselves (at least since Should I update this changeset considering this, i.e., remove the added lines to |
|
I am just getting the following message when I apply your patch: The request header also just contains gzip: Saw this: https://stackoverflow.com/questions/43862412/why-is-brotli-not-supported-on-http Did you test that it's really supported via http? |
|
I am looking at the same message and I am pondering what could have happened... I change the compression algorithm in vite, rebuilt the firmware, and garbage was decoded, which made sense. After updating the the content encoding header it worked fine. It then did other things, prepared this PR, and wanted to pick up working on the software now. I don't know what's up yet... Sorry about this... I will check your link first. |
|
Then http://localhost:8888/ serves the page just fine in Brave and Firefox. I did use a similar SSH tunnel before... So the decision to accept brotli as content is not based on HTTP vs HTTPS, but on the port, as it seems. Damn, I was so happy that I found this trick to reduce the flash usage 🙈 |
|
Grmpf yes... would have been too nice |
|
More specifically, as per this discussion, brotli is accepted for connections to localhost. The port is irrelevant, as confirmed by a test just now. When using a different host name, the content decoding fails for ports other than 80 as well. Yeah... So... Time to revert the change downstream, as I don't see how this is salvageable 🤷♂️ Thanks for your interest, and sorry for wasting your time 😉 |
|
For the record: Even though it would require some effort, I looked into the possibility of compressing the web application using brotli, but serve it raw through HTTP, such that a stub of the web app would have to decompress the application in Javascript. This would work if the decompressor would only need a couple of kB of code. Unfortunately, this is not the case: https://www.npmjs.com/package/brotli-dec-wasm (~200kB). It seems the dictionary alone is ~100kB. Even if we compressed that with gzip, served it as such, have it decompressed by the browser just to be able to load and decompress the brotli compressed part of the webapp afterwards, we will only complicate things and the firmware blob will still grow. It seems there is no way to utilize the browser's decompressor from using Javascript. |
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion or issue for related concerns. |
I was concerned that OpenDTU-OnBattery would soon hit the flash size limit once more. So much so, that helgeerbe#1025 would be effective much sooner than anticipated, or that I would be unable to merge my refactoring efforts regarding the power meter implementation. Looking for ideas, I targeted the web application, which is embedded in the firmware image. Changing the compression level of gzip did nothing, but switching to brotli did. In case of OpenDTU-OnBattery the savings are even higher at ~48k.
This change saves ~38k of flash memory, since brotli is much more effective at compressing the web application. As we serve the web application binary as is, the browser needs to take care of decompression. So we have to look at browser support: As it turns out, brotli compression is available on all browsers since more than four years. As per https://caniuse.com/brotli, the support for brotli is available for 97% of users today. We can safely assume that all OpenDTU users use a browser with support for brotli encoded content.
This you might not like: The gzip-compressed transpiled files in
webapp_distare removed from the repo and the new brotli-compressed transpiled files are added to the gitignore list, as build artifacts shall not be tracked. This has the added advantage that changes to those files don't prevent switching branches or pulling changes. However, the philosophy in this project has been different and the transpiled web application has always been explicitly updated. Removing the transpiled files will make the build fail if the user forgot to build the web application.All in all, this change might not be interesting for OpenDTU, as the flash size limitation is not concerning as of today. The expected faster decompression by the client is probably irrelevant.