feat: implement v8 sandbox on ppc64 for electron 26

[darkbasic]

With +temp-fix it fails at compilation time:

[12:41:06] Starting compilation...
/usr/lib64/electron-26/node: line 3:    29 Segmentation fault	   (core dumped) ELECTRON_RUN_AS_NODE=1 $(dirname "$0")/electron "${@}"

With -temp-fix it fails at installation time:

>>> Source compiled.
>>> Test phase [not enabled]: app-editors/vscode-9999

>>> Install app-editors/vscode-9999 into /var/tmp/portage/app-editors/vscode-9999/image
[14:29:56] Using gulpfile /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/gulpfile.js
[14:29:56] Starting 'vscode-linux-ppc64-prepare-deb'...
[14:29:56] Starting clean-ppc64el ...
[14:29:56] Finished clean-ppc64el after 7 ms
[14:29:56] Starting vscode-linux-ppc64-prepare-deb ...
Assertion failed: bad export type for `tree_sitter_tsx_external_scanner_create`: undefined
Unhandled Rejection at: Promise Promise {
  <rejected> RuntimeError: abort(Assertion failed: bad export type for `tree_sitter_tsx_external_scanner_create`: undefined). Build with -s ASSERTIONS=1 for more info.
      at se (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:6761)
      at k (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:4462)
      at Pe (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:12582)
      at c (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:10613)
      at /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:10879
      at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:254:10
      at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:243:22
} reason: RuntimeError: abort(Assertion failed: bad export type for `tree_sitter_tsx_external_scanner_create`: undefined). Build with -s ASSERTIONS=1 for more info.
    at se (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:6761)
    at k (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:4462)
    at Pe (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:12582)
    at c (/var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:10613)
    at /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/web-tree-sitter/tree-sitter.js:1:10879
    at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:254:10
    at async /var/tmp/portage/app-editors/vscode-9999/work/vscode-9999/node_modules/@vscode/l10n-dev/dist/main.js:243:22
[14:29:56] The following tasks did not complete: vscode-linux-ppc64-prepare-deb
[14:29:56] Did you forget to signal async completion?

Using system node at compilation time and electron's node during the installation results in a working build.
The shipped ebuild is meant to be used with -temp-fix and force-enables system electron at the installation phase.

Please check if vscode-9999 with electron 26 behaves the same way on amd64.

[PF4Public]

/usr/lib64/electron-26/node: line 3: 29 Segmentation fault

Could Raptor fix this issue as well?

[darkbasic]

Could Raptor fix this issue as well?

Is this ppc64 specific? Doesn't sound like so.
If it ends up being ppc64 specific where do you think this comes from? It looks like a node bug rather than electron's.

[PF4Public]

Is this ppc64 specific? Doesn't sound like so.

It never sagfaulted on amd64 on me, but I'll check it again with the latest electron.

If it ends up being ppc64 specific where do you think this comes from?

Backtrace would be helpful.

[PF4Public]

No issues with temp-fix on x86 and latest electron:

[ebuild     U *] app-editors/vscode-9999::pf4public [1.83.0::pf4public] USE="badge-providers build-online* electron-26 substitute-urls temp-fix -api-proposals -electron-19 -electron-20 -electron-21 -electron-22 -electron-23 -electron-24 -openvsx -reh -reh-web" 0 KiB

Total: 1 package (1 upgrade), Size of downloads: 0 KiB

>>> Verifying ebuild manifests
>>> Emerging (1 of 1) app-editors/vscode-9999::pf4public
>>> Installing (1 of 1) app-editors/vscode-9999::pf4public
>>> Completed (1 of 1) app-editors/vscode-9999::pf4public
>>> Jobs: 1 of 1 complete                        Load avg: 17.7, 18.5, 19.2

And your changes fail vscode check (it is marked as skipped, but it is in fact failed). EDIT: Worker ran out of free space, but chances are it would fail anyway.

[darkbasic]

Thanks for confirming this! I will check if removing the v8 sandbox patch fixes this and eventually if applying it to electron's node's v8 (in addition to electron's v8) helps. If nothing helps I will get a backtrace.

[PF4Public]

Hmm, it gets even more interesting. GitHub workflow for 1.83.1 failed with

[12:39:21] 'vscode-linux-x64-min' errored after 23 min
[12:39:21] Error: Command failed: yarn list --prod --json
/usr/lib64/electron-25/node: line 3:  4070 Trace/breakpoint trap   (core dumped) ELECTRON_RUN_AS_NODE=1 $(dirname "$0")/electron "${@}"

I restarted it and it passed successfully from the second time. It doesn't fail on real hardware though…

Try restarting the build with +temp-fix, maybe it was a one-time incident?

It could be indeed an issue of node or yarn. Sometimes it errs with:

[03:35:10] Error: Command failed: yarn list --prod --json
terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 8197) >= this->size() (which is 2889)
/usr/lib64/electron-26/node: line 3:  3745 Aborted                 (core dumped) ELECTRON_RUN_AS_NODE=1 $(dirname "$0")/electron "${@}"

With -temp-fix it fails 100% times, but with temp-fix it randomly fails in GitHub actions, but does not fail on real hardware.

[darkbasic]

Try restarting the build with +temp-fix, maybe it was a one-time incident?

Unfortunately not, I've tried many times and without the patch it compiles flawlessly but as soon as I add the v8 sandbox patch if segfaults.

How am I supposed to attach a debugger?
I've tried gdb -ex=r --args node "--max_old_space_size=8192" "node_modules/gulp/bin/gulp.js" "vscode-linux-${VSCODE_ARCH}-min" but I get "/usr/lib64/electron-26/node": not in executable format: file format not recognized.

At this point I'm curious to know if adding the patch to node-v18.16.1/deps/v8 as well might solve it.
Unfortunately the patch need to be rebased against it in order to apply.

[madscientist159]

@darkbasic Frankly, the v8 version in Electron 26 is quite buggy even outside of the sandbox patches. Patching it to even compile in the first place (which requires commit 182d9c05 from the current v8 tree) and leaving the backported sandbox patches off yields debug asserts indicating register allocation problems. Patching out the check results in WASM failures similar to the following:

=== cctest/test-gc/RunWasmLiftoff_CallRef ===
#
# Fatal error in ../../src/codegen/register-base.h, line 48
# Debug check failed: reg_code_ >= 0 && reg_code_ < kNumRegisters.

I'm not sure how useful it will be to try to hack this exact v8 version into operation. The fact that you are getting segmentation faults (and I see SIGSEGV/SIGILL on release checks) even without my patches with this v8 version indicates something is seriously wrong in the base code itself. I note that none of these problems show up on the current master version of v8, so something between v8 commits 1626e229 and 72ab23a8 fixed these underlying issues.

Is there any scope to simply updating v8 to a newer version for the ppc64 variant? If not, waiting for Electron 27 might be a better idea at this point. Raptor won't be able to spend significant resources trying to get an obsolete v8 version to work when the current version is operational, unfortunately.

[darkbasic]

I'm all in to use the least amount of work to get this working so I am fine to give up on electron 26 and try electron 27. Yet I am a bit dubious that this would solve the issue, because node version barely changes between electron versions. 27 uses 18.17.1 instead of 18.16.1, so unless they updated deps/v8 that won't do us any good. They won't move out of LTS versions so it will take years to get rid of 18.x.x.

Is there any scope to simply updating v8 to a newer version for the ppc64 variant?

Are you suggesting to simply drop node-v18.16.1/deps/v8 and replace it with git master? If node 18 is capable of working with latest v8 that might be fine for me, but I am a bit worried of unexpected consequences: vscode is stuffed with tons of bugs and it's already a nightmare to be able to build it. Tampering with the node version is known to be especially problematic in that regard.

[madscientist159]

I'm all in to use the least amount of work to get this working so I am fine to give up on electron 26 and try electron 27. Yet I am a bit dubious that this would solve the issue, because node version barely changes between electron versions. 27 uses 18.17.1 instead of 18.16.1, so unless they updated deps/v8 that won't do us any good. They won't move out of LTS versions so it will take years to get rid of 18.x.x.

Is there any scope to simply updating v8 to a newer version for the ppc64 variant?

Are you suggesting to simply drop node-v18.16.1/deps/v8 and replace it with git master? If node 18 is capable of working with latest v8 that might be fine for me, but I am a bit worried of unexpected consequences: vscode is stuffed with tons of bugs and it's already a nightmare to be able to build it. Tampering with the node version is known to be especially problematic in that regard.

Which v8 version is actually being executed here, the one in the embedded Chromium tree or the one in nodejs v18.16.1? If we have multiple v8 trees, they all need patching to enable the same features, otherwise segfaults will be expected.

Given the aging issues mentioned above we may have no choice but to try to fix this on the older versions.

[madscientist159]

@darkbasic The nodejs version here is so ancient that it doesn't even have arch-specific sandbox code in its v8 tree. I'm not confident in my understanding of which components are being used to actually drive the Electron build process, if I could get clarification on that it would be most helpful.

Is there any chance I could get a set of commands required to reproduce the build (and build crash) outside of Gentoo-specific build tooling? Given the complexity of setting up a proper build and debug environment for these components I really need to be able to reproduce from within my Debian v8 development system.

Thanks!

[darkbasic]

Which v8 version is actually being executed here, the one in the embedded Chromium tree or the one in nodejs v18.16.1? If we have multiple v8 trees, they all need patching to enable the same features, otherwise segfaults will be expected.

Electron uses its own bundled v8 (electron/v8) and patching electron/v8 alone is enough to get vscode working with nodejs 20 (the installation phase doesn't work with node 20, but that's another issue). Yet electron also bundles its own node.js (v18.16.1 for electron 26) which in turn bundles its own v8 (nodejs/deps/v8), which is a completely different version. I didn't manage to patch that one because of conflicts and I'm not even sure if it will help doing so (node.js 20 has no v8 sandbox patches yet it doesn't segfault).

I'm not confident in my understanding of which components are being used to actually drive the Electron build process, if I could get clarification on that it would be most helpful.

Is there any chance I could get a set of commands required to reproduce the build (and build crash) outside of Gentoo-specific build tooling? Given the complexity of setting up a proper build and debug environment for these components I really need to be able to reproduce from within my Debian v8 development system.

This is the list of commands required to compile electron 26: https://github.com/PF4Public/gentoo-overlay/blob/master/dev-util/electron/electron-26.4.0.ebuild

And this is the list of commands required to compile vscode git master (you will need to follow the electron-26 and temp-fix code paths for that): https://github.com/PF4Public/gentoo-overlay/blob/master/app-editors/vscode/vscode-9999.ebuild

Gentoo ebuilds are akin to bash scripts so it won't be too hard to reproduce the steps on any distro, yet compiling electron and especially vscode is full of quirks and generally not an easy task so I'm not sure if that would be the easiest path.
I could set up a Gentoo system in a vm where you can run the ebuilds directly if it won't be too hard for you to debug the issue in a distribution you are not familiar with. If Raptor has ready to use ppc64 VMs you could provide me access to one of them so I can set it up for you, otherwise I could create a libvirt vm on my own system and give you the spice credentials.
Or maybe reproducing the same commands on Debian would be easier for you, not sure about that.

[madscientist159]

@darkbasic The more I look at the trees, the more I suspect there is a feature flags conflict between the integrated node build and the Chromium v8 build. From what I can see, the Chromium v8 engine is what is actually running (good), but node itself is built with pointer compression, external code space, and the sandbox all disabled (bad).

Before trying to get me some kind of command sequience to replicate this entire mess on my system, can you try aligning the feature flags in node's v8 BUILD.gn with the flags in Chromium's v8 BUILD.gn? I'm not seeing anything obvious in the ancient node 18 tree that requires any arch-specific patching to turn those features on.

Bear in mind that on x86 the BUILD.gn flags will align across the two trees by default, since they're not being patched...

[darkbasic]

can you try aligning the feature flags in node's v8 BUILD.gn with the flags in Chromium's v8 BUILD.gn?

Wouldn't that require patching node's v8? Otherwise I see no way to enable features which are not even implemented on ppc.

[madscientist159]

can you try aligning the feature flags in node's v8 BUILD.gn with the flags in Chromium's v8 BUILD.gn?

Wouldn't that require patching node's v8? Otherwise I see no way to enable features which are not even implemented on ppc.

The v8 version in nodejs is so old that it doesn't have any arch-specific code for any of those features except pointer sandboxing (which should be off in both the Chromium v8 and nodejs v8) and pointer compression:

git grep -e V8_SANDBOX | grep x64
deps/v8/src/codegen/x64/macro-assembler-x64.cc:#ifdef V8_SANDBOXED_POINTERS
deps/v8/src/codegen/x64/macro-assembler-x64.cc:#ifdef V8_SANDBOXED_POINTERS
deps/v8/src/codegen/x64/macro-assembler-x64.cc:#ifdef V8_SANDBOXED_EXTERNAL_POINTERS
deps/v8/src/codegen/x64/macro-assembler-x64.cc:#endif  // V8_SANDBOXED_EXTERNAL_POINTERS

git grep -e STATIC_ROOTS | grep x64

Pointer compression already has the ppc64 arch-specific code in tree in that nodejs version. Please try turning the features on, and if you hit a build failure let me know.

[PF4Public]

After updating electron to 26.4.0 I couldn't build vscode-1.83.1 and got the same SEGFAULT.

I've just added electron-27. Looks good so far, you can give it a try.

[darkbasic]

After updating electron to 26.4.0 I couldn't build vscode-1.83.1 and got the same SEGFAULT.

Weird, I've built vscode git master 4 consecutive times with electron 26.4.0 without the v8 sandbox patches.

I've just added electron-27. Looks good so far, you can give it a try.

Sure, I'll give it a try.

[darkbasic]

@madscientist159 can you please confirm if the full v8 sandbox patchset made its way into electron 29?

Somehow I'm still getting segfaults but maybe it's unrelated: #317 (comment)