Moving to Electron 21 and beyond

[mauricioszabo]

Just want to draft some of the things before they get lost in Discord messages:

Electron 21 breaks some stuff related to native libraries: electron/electron#35801. Basically, all native libraries, even the ones that are context-aware, will break if they use malloc and memcpy. These techniques are used by some libraries we have. Superstring is one, but that is on the process to migrate to WASM and already works with Electron 19. I suspect Oniguruma is another.

Oniguruma

To move away from Oniguruma means that we need to find a replacement. VSCode have a "Oniguruma" that works with WASM (https://www.npmjs.com/package/vscode-oniguruma-wasm) but on their own words:

This library is used in VS Code and is not intended to grow to have general Oniguruma WASM bindings.

Which can mean they don't intend to have a "stable" API. I checked the API for this Oniguruma WASM, but it's slightly different than what we use in Pulsar.

Another thing we have to consider is that Oniguruma is used on TextMate grammars, which means whatever we choose to have, it must be fast. Looking for different libraries, I found this one: intel/hyperscan#367

It's a very, very fast Regexp library. I want to see if we can adapt it to use inside Pulsar with the WASM bindings

Tree Sitter

Both for Electron 13+ and later, we need to move away from node tree sitter. The reason is simple: it's not context-aware, and none of the grammars are either. So we would either need to keep a fork of all grammars or convince everyone that uses tree-sitter to be context-aware and also to not use any of the newer Electron limitations - bothare completely out of question.

So, we may need to rewrite tree-sitter tokenizer using the WASM bindings. This will probably break all language- packages that use tree-sitter, but these will probably break anyway because of our upgrades, and Pulsar's tree-sitter implementation is different than what it is supposed to be, lacking lots of interesting features like the queries for example.

Terminals

Most terminal packages use native bindings in some way. I want to see if we can also move terminal libraries to something that will keep Pulsar from breaking on newer versions... not sure how

[confused-Techie]

On the note of Tree-Sitter and TextMate, I know VSCode uses TextMate grammars and Oniguruma to accomplish it's syntax highlighting, would it be beneficial for us to do the same?

I mostly say that because if we were to decide TextMate Grammars as our now supported method for tokenization, and especially if we make it compatible with modern grammars, couldn't we then just use all the pre-existing TextMate Grammars rather than have to individually work on each one?

I know this would mean all existing TreeSitter Grammars break, but if they break anyway we could just move this direction to ease the support needed for them. Just an idea