Extend standalone support

[jerbob92]

• Implement _emscripten_throw_longjmp that aborts the program
• Add nice messages to __cxa_throw and __cxa_allocate_exception
• Implement getentropy using WASI
• Implement WASI FS for syscalls lstat64, stat64, newfstatat, getdents64, fstat64 and openat

This is far from complete if you look at the WASI FS spec and the syscalls, but at least it gets basic operations like directory listing, file/directory opening and file/directory stat working. It was enough for me to get pdfium working in standalone mode.

I didn't really know what I should do with AT_FDCWD or what the runtime should do with it when passing AT_FDCWD. AT_FDCWD is also negative while I think fd in WASI is unsigned?

Most of the syscall code was taken from or based on wasi-libc.

[sbc100]

Thanks for working on this!

I left a few nits. Perhaps we can maybe split this up.

[sbc100]

Can we add some some new tests based on these?

[jerbob92]

Can we add some some new tests based on these?

I have never developed in C before, I don't really know where to start, can you give me some hints where you want these tests added and what they should test?

[sbc100]

Just call longjmp would be more accurate I think.

[jerbob92]

I think I could only trigger this by doing an invalid jump, but I think you would know better.

[sbc100]

No.. all longjmp's require callout out of JS IIUC. WebAssembly, as it stands, has no support for longjmp so we call out the JS VM to throw for us.

[jerbob92]

Thanks! Not that it matters but do you have any C to demonstrate this? Just curious.
In my tests I was only able to trigger this by doing an invalid jmp. I think the jumps were being handled by the invoke_* methods?

[sbc100]

Any call to longjmp from C code will called _emscripten_throw_longjmp IIUC.

So here I would just do REPORT_UNSUPPORTED(longjmp)

[sbc100]

I believe the way it works is that there can be any number of pre-opens, starting at 3: https://github.com/WebAssembly/wasi-libc/blob/30094b6ed05f19cee102115215863d185f2db4f0/libc-bottom-half/sources/preopens.c#L212-L215. And non of these pre-opens are necessarily the root, they can be mounted in various places.

Also, can you move this macros (if you keep it), into standalone.c. api.h is copied from wasi-libc, and I think its better to avoid have local modifications to it.

[sbc100]

I think we really needs some tests for this.

The most basic test for the standalone syscalls is in test_standalone_syscalls in test_other.py.

For other tests look for @also_with_standalone_wasm in test_core.py

[codefromthecrypt]

on behalf of wasm people I know I would like to thank @jerbob92 and @sbc100 for progressing this, as it not only unlocks things like pdfium, but also generic wasm utilities that use emscripten (like wabt). Ex being able to run wat2wasm using any wasi runtime instead of whatever was released and packaged.

If any one not currently involved has skills to contribute, please do as this is very likely well into "hobby time" for @jerbob92 and it is at the core a pretty substantial infrastructure change and a lot of work to test it. other wasm people will thank you, but I'll thank in advance!

[sbc100]

on behalf of wasm people I know I would like to thank @jerbob92 and @sbc100 for progressing this, as it not only unlocks things like pdfium, but also generic wasm utilities that use emscripten (like wabt). Ex being able to run wat2wasm using any wasi runtime instead of whatever was released and packaged.

If you just wast a wasi version of the wabt tools I think the simplest path would be to use wasi-sdk to build it. Of course that doesn't mean we shouldn't land this change too.

[codefromthecrypt]

@sbc100 ps your last comment didn't format right. might want to polish it

[codefromthecrypt]

If you just wast a wasi version of the wabt tools I think the simplest path would be to use wasi-sdk to build it. Of course that doesn't mean we shouldn't land this change too.

good idea, on the wabt thing and indeed a separate topic if they are open to decoupling from emscripten.

[sbc100]

If you just wast a wasi version of the wabt tools I think the simplest path would be to use wasi-sdk to build it. Of course that doesn't mean we shouldn't land this change too.

good idea, on the wabt thing and indeed a separate topic if they are open to decoupling from emscripten.

IIUC wabt is not coupled to emscripten, we just happen to build it with emscripten for the web demo.

[sbc100]

(I guess maybe what you are saying is that it would be nice if the web version of wabt, which is run by emscripten, happened to also be WASI compliant.. in that case I agree that would be very cool).

[codefromthecrypt]

yeah sorry it was that we were looking for a wabt compiled to wasm and found the one that was compiled with emscripten for the web. I raised this issue for out of browser wasi binary WebAssembly/wabt#2101

[jerbob92]

Thanks @codefromthecrypt! So in hindsight, I don't think my C skills are good enough to get this PR into a merge-able state, implementing AT_FDCWD, using the preload dirs and not the FD 3, implementing tests.

[codefromthecrypt]

@jerbob92 no worries, I think you got things very far. I'll keep recruiting to whatever end on this.

[jerbob92]

@sbc100 I have done some more progress on this PR now:

• Implemented WASI pre-opens
• Implement mkdirat

I do have some questions though:

• How to get started on adding tests for this?
• The standalone.c file is getting very big right now, how do we want to split this up? Perhaps something like standalone_wasi_preopens.c, standalone_wasi_fs.c, standalone_wasi_random.c go keep things clean?
• Is there an easy way to check which syscalls still need to be implemented in standalone to make it full WASI compliant?
• Code is mostly copied from wasi-libc right now with some changes, is this something that we want?
• wasi-libc has a whole system to maintain a CWD, is this also something that we want to implement?

[sbc100]

@sbc100 I have done some more progress on this PR now:

• Implemented WASI pre-opens
• Implement mkdirat

I do have some questions though:

• How to get started on adding tests for this?
• The standalone.c file is getting very big right now, how do we want to split this up? Perhaps something like standalone_wasi_preopens.c, standalone_wasi_fs.c, standalone_wasi_random.c go keep things clean?

I don't feel strongly about this. Perhaps use the same split that wasi-libc uses? Also I don't think we need standalone in each filename since they will all be in the standalone directory.

• Is there an easy way to check which syscalls still need to be implemented in standalone to make it full WASI compliant?

The ideas is that we will be running the full wasi testsuite. I started adding some of it in #12704 and I have some
more plans.

• Code is mostly copied from wasi-libc right now with some changes, is this something that we want?

I don't think its a problem, but please document the origin and try to document any changes from upstream.

• wasi-libc has a whole system to maintain a CWD, is this also something that we want to implement?

Sure. We could even consider adding wasi-libc as a submodule and including certain files directly?

[jerbob92]

I don't feel strongly about this. Perhaps use the same split that wasi-libc uses? Also I don't think we need standalone in each filename since they will all be in the standalone directory.

Sounds good!

The ideas is that we will be running the full wasi testsuite. I started adding some of it in #12704 and I have some
more plans.

Nice, that will make it a lot easier!

I don't think its a problem, but please document the origin and try to document any changes from upstream.
Sure. We could even consider adding wasi-libc as a submodule and including certain files directly?

If that's a possibility that would be great, it would make it a lot easier because right now I'm just cherry-picking wasi-libc code to get the syscalls/WASI calls working that I require, especially if we want to pass the whole wasi-testsuite then we have to copy a lot from was-libc. Only thing I noticed while copying code is that their WASI function signature is a bit different sometimes, which prevents us from directly using the C code (AFAIK), so we might want to look into making that possible. For example:
wasi-libc __wasi_path_filestat_get = __wasi_errno_t __wasi_path_filestat_get(__wasi_fd_t fd, __wasi_lookupflags_t flags, const char *path, __wasi_filestat_t *retptr0)
Emscripten __wasi_path_filestat_get = __wasi_errno_t __wasi_path_filestat_get(__wasi_fd_t fd, __wasi_lookupflags_t flags, const char *path, size_t path_len, __wasi_filestat_t *buf)

So Emscripten requires the path length and wasi-libc doesn't. But perhaps we can work around this by completely using wasi-libc for the wasi part.

[sbc100]

Can you make this static and make it clear this is an internal helper?

[karelbilek]

A stupid nit - if it was based on wasi libc, shouldn't there be copyright attribution somewhere? (Or is that not necessary?)

[sbc100]

A stupid nit - if it was based on wasi libc, shouldn't there be copyright attribution somewhere? (Or is that not necessary?)

No stupid at all, we should certainly consider how to handle this. I'm not sure why the right answer is. Or how we should deal with keeping the codebases in sync. One easy option wold be to add wasi-libc as a submodule instead of duplicating the code here.

[d3lm]

they couldn't match the emscripten custom FS APIs on performance and code size in all cases

Oh wow, that is very interesting. I would have expected that the WASI FS APIs were more performant than going through Node's JS layer for FS calls.

Do you have an idea why that is? Where does the performance come from with Emscripten's custom FS APIs? Isn't that FS API just going through Node's fs layer?

[sbc100]

they couldn't match the emscripten custom FS APIs on performance and code size in all cases

Oh wow, that is very interesting. I would have expected that the WASI FS APIs were more performant than going through Node's JS layer for FS calls.

I'm not sure what you mean here by WASI FS APIs. Are you talking about a particular implementation? In emscripten the WASI APIs are implemented in JS just like all the other APIs we have: https://github.com/emscripten-core/emscripten/blob/main/src/library_wasi.js.

Perhaps you are referring to some kind of native WASI API that exists on native wasm VMs? Or are you talking some kind of node-provided implementation of WASI? uvwasi maybe?

Do you have an idea why that is? Where does the performance come from with Emscripten's custom FS APIs? Isn't that FS API just going through Node's fs layer?

IIRC it was that the ergonomics of the various APIs such as pathopen had overheads that emscripten syscall later didn't have or could work around. The emscripten syscall layer is modeled after the linux syscall layer used in musl. @kripken can you remember the specific performance issue that stopped us moving foward with switch to WASI internally by default?

[karelbilek]

Just to explain a bit what we are doing (well mostly what @jerbob92 is doing :D ) - we build C code base with emscripten with this patch, then pass that to wazero (go WASI implementation). It mostly works.

https://github.com/tetratelabs/wazero

[sbc100]

Just to explain a bit what we are doing (well mostly what @jerbob92 is doing :D ) - we build C code base with emscripten with this patch, then pass that to wazero (go WASI implementation). It mostly works.

https://github.com/tetratelabs/wazero

That makes sense. Do you know if wazero also implements some of the emscripten syscalls? Or is it only WASI syscalls?

[kripken]

@sbc100

See https://v8.dev/blog/emscripten-standalone-wasm#necessary-api-differences regarding the overhead and other issues that prevented us from moving emscripten to use 100% WASI APIs. But perhaps WASI has improved since then?

[jerbob92]

Just to explain a bit what we are doing (well mostly what @jerbob92 is doing :D ) - we build C code base with emscripten with this patch, then pass that to wazero (go WASI implementation). It mostly works.
https://github.com/tetratelabs/wazero

That makes sense. Do you know if wazero also implements some of the emscripten syscalls? Or is it only WASI syscalls?

Wazero implements WASI and some Emscripten specific things (emscripten_notify_memory_growth, _emscripten_throw_longjmp and invoke_xxx functions). I have also implemented C++ exceptions support, but I'm still waiting for some feedback for that.

Embind support is possible using wazero-emscripten-embind.

Since no other host calls have been implemented, most normal Emscripten builds won't work on Wazero, which is why I added more support for standalone builds. I think it makes sense that Wazero focusses on WASI and not on implementing specific Emscripten host calls, also because they are prone to change between Emscripten versions.

[d3lm]

@sbc100 Sorry for being vague. Yea I was referring to Node's WASI implementation for instance via uvwasi and the implementations for the WASI interface is coming from the internal bindings so they are natively implemented in C++.

but it won't be in the name of more efficiency

That I still don't fully understand. Why would it be less efficient to use native APIs at least when running on Node.js?

[sbc100]

@sbc100 Sorry for being vague. Yea I was referring to Node's WASI implementation for instance via uvwasi and the implementations for the WASI interface is coming from the internal bindings so they are natively implemented in C++.

I see, yes that could be conceivable more efficient. However with emscripten-generated JS we implement WASI in JS since we have our own virtual file system. i.e. the JS normally has run on the web as well as under node so it can't /doesn't assume uvwasi support.

Perhaps you are right that if we push forward with STANDALONE mode and PURE_WASI mode then we could potentially bind uvwasi directly to the wasm module. Especially for users that specify NODERAWFS (i.e. user that already opt out of emscripten's virtual filesystem).

However, given that we have limited time to work on stuff, and given that the direction PURE_WASI seems to be heading in is to essentially copy or import wasi-libc into emscripten I'm inclined to suggest that suggesting folks simply use wasi-sdk where possible.

[sbc100]

@sbc100 Sorry for being vague. Yea I was referring to Node's WASI implementation for instance via uvwasi and the implementations for the WASI interface is coming from the internal bindings so they are natively implemented in C++.

I see, yes that could be conceivable more efficient. However with emscripten-generated JS we implement WASI in JS since we have our own virtual file system. i.e. the JS normally has run on the web as well as under node so it can't /doesn't assume uvwasi support.

Perhaps you are right that if we push forward with STANDALONE mode and PURE_WASI mode then we could potentially bind uvwasi directly to the wasm module. Especially for users that specify NODERAWFS (i.e. user that already opt out of emscripten's virtual filesystem).

However, given that we have limited time to work on stuff, and given that the direction PURE_WASI seems to be heading in is to essentially copy or import wasi-libc into emscripten I'm inclined to suggest that suggesting folks simply use wasi-sdk where possible.

Having said that I will try to take another look at this PR today and see if we can get it landed !

[d3lm]

However with emscripten-generated JS we implement WASI in JS since we have our own virtual file system. i.e. the JS normally has run on the web as well as under node so it can't /doesn't assume uvwasi support.

I see. That's a fair point.

Perhaps you are right that if we push forward with STANDALONE mode and PURE_WASI mode then we could potentially bind uvwasi directly to the wasm module. Especially for users that specify NODERAWFS (i.e. user that already opt out of emscripten's virtual filesystem).

Yea that's what I was thinking if users opt-in to use Node's FS instead of Emscripten's virtual fs.

Having said that I will try to take another look at this PR today and see if we can get it landed!

🎉 🙏

[agnivade]

Just adding another vote of support to this PR. Currently, I am in the process of compiling whisper.cpp to wasi, and without this PR, it's impossible for me to access the model files from the filesystem. Would highly appreciate if this got merged into main.

[sbc100]

Just adding another vote of support to this PR. Currently, I am in the process of compiling whisper.cpp to wasi, and without this PR, it's impossible for me to access the model files from the filesystem. Would highly appreciate if this got merged into main.

Can you explain more about your use case? Is there some reason wasi-sdk doesn't work for you? What platform/engines are you targetting?

[agnivade]

My objective is to use whisper.cpp in a pure Go environment. There's a CGO API of course but that requires libc. Compiling to WASI allows me to bypass all of that.

And as others have alluded, wasi-sdk is nowhere as mature and polished as this project. Documentation is sparse and there's no support for bindings as far as I can see.

IMO, it would be far easier and lead to a larger user base if we were to put effort into emscripten to allow wasi compatibility, than to put effort into wasi-sdk and make it better.

[sbc100]

My objective is to use whisper.cpp in a pure Go environment. There's a CGO API of course but that requires libc. Compiling to WASI allows me to bypass all of that.

Off topic but are you sure you want to be bypassing libc like that? Wouldn't that mean re-implementing a lot of the stuff the libc provides? What do you gain from that?

And as others have alluded, wasi-sdk is nowhere as mature and polished as this project. Documentation is sparse and there's no support for bindings as far as I can see.

I'm not sure I would say its not a polished, more that it has a different set of target users. Users who want to target embedded environment and not the web are often better of using wasi-libc. Users targeting JS and the Web are often better of using emscripten.

IMO, it would be far easier and lead to a larger user base if we were to put effort into emscripten to allow wasi compatibility, than to put effort into wasi-sdk and make it better.

Where are you hoping to run the you code? Are you targeting JS environments such as the browser or node? I'm a little confused since pretty much all of the advanced features of emscripten (such as bindings) rely heavily on JavaScript and non-WASI APIs.

[agnivade]

Off topic but are you sure you want to be bypassing libc like that? Wouldn't that mean re-implementing a lot of the stuff the libc provides? What do you gain from that?

Yeah, this is going slightly off-topic and into the internals of Go, but yes Go has the ability to generate builds that do not depend on libc at all. It has alternate implementations of all libc functionality. The gain is portability. Our software is used in a large variety of customer environments on a plethora of linux flavors (sometimes on very old libc versions). Go gives us the ability to not depend on libc and I'd like to use that leverage.

Where are you hoping to run the you code? Are you targeting JS environments such as the browser or node?

Server side. Again, all of this is purely in Go. Go generates single binaries that you can simply run.

I'm a little confused since pretty much all of the advanced features of emscripten (such as bindings) rely heavily on JavaScript and non-WASI APIs.

Yes, as mentioned earlier in this thread, there are libraries which re-implement all the bindings in pure Go, allowing users to use emscripten generated builds in a pure Go environment. Refer to: #18285 (comment)

Hope that was clear. Happy to answer any further questions :) I fully realize that emscripten is targeted towards JS based environments. But there's a lot of tooling already built that gets us to 90% of what we want to get it to run on non-JS environments. This small PR would almost take it to 99%. Ultimately, it's your decision of course as a maintainer. But this would simply unlock a lot of functionality that would otherwise require a lot of other workarounds.

[sbc100]

Off topic but are you sure you want to be bypassing libc like that? Wouldn't that mean re-implementing a lot of the stuff the libc provides? What do you gain from that?

Yeah, this is going slightly off-topic and into the internals of Go, but yes Go has the ability to generate builds that do not depend on libc at all. It has alternate implementations of all libc functionality. The gain is portability. Our software is used in a large variety of customer environments on a plethora of linux flavors (sometimes on very old libc versions). Go gives us the ability to not depend on libc and I'd like to use that leverage.

Sounds like your libc re-implementations would then be built on top of the syscalls layer which is then, by definition, not portable between OSes.. meaning you would need to re-implement for each OS. But I guess you decided to trade off those two forms of non-portableness?

Where are you hoping to run the you code? Are you targeting JS environments such as the browser or node?

Server side. Again, all of this is purely in Go. Go generates single binaries that you can simply run.

I'm a little confused since pretty much all of the advanced features of emscripten (such as bindings) rely heavily on JavaScript and non-WASI APIs.

Yes, as mentioned earlier in this thread, there are libraries which re-implement all the bindings in pure Go, allowing users to use emscripten generated builds in a pure Go environment. Refer to: #18285 (comment)

Hope that was clear. Happy to answer any further questions :) I fully realize that emscripten is targeted towards JS based environments. But there's a lot of tooling already built that gets us to 90% of what we want to get it to run on non-JS environments. This small PR would almost take it to 99%. Ultimately, it's your decision of course as a maintainer. But this would simply unlock a lot of functionality that would otherwise require a lot of other workarounds.

I'm not necessarily against moving forward with this PR, but I would to ask a few more clarifying questions. For example, if you are already implementing the emscripten ABI/API, why not just continue to do that.. Why does it matter to you if our API is the same as the WASI API or not? Why not implement all the emscripten __syscall_xxx APIs that you need? Is it out of fear that those APIs will not be stable maybe?

[agnivade]

Sounds like your libc re-implementations would then be built on top of the syscalls layer which is then, by definition, not portable between OSes.. meaning you would need to re-implement for each OS. But I guess you decided to trade off those two forms of non-portableness?

Correct. Go has proper cross-compilation support and automatically takes care of this.

For example, if you are already implementing the emscripten ABI/API, why not just continue to do that..

Oh yeah we could. But from what I understand, the openat and other similar functions need to be exposed for us to re-implement it. @jerbob92 can probably provide more details as to why it's not possible to do this from Go. Currently, only these can be overridden:

  (import "wasi_snapshot_preview1" "clock_time_get" (func $__wasi_clock_time_get (type 31)))
  (import "wasi_snapshot_preview1" "proc_exit" (func $__wasi_proc_exit (type 3)))
  (import "wasi_snapshot_preview1" "fd_write" (func $__wasi_fd_write (type 10)))
  (import "wasi_snapshot_preview1" "fd_read" (func $__wasi_fd_read (type 10)))
  (import "wasi_snapshot_preview1" "fd_close" (func $__wasi_fd_close (type 0)))
  (import "wasi_snapshot_preview1" "fd_seek" (func $__wasi_fd_seek (type 66)))
  (import "wasi_snapshot_preview1" "environ_sizes_get" (func $__wasi_environ_sizes_get (type 1)))
  (import "wasi_snapshot_preview1" "environ_get" (func $__wasi_environ_get (type 1)))

[jerbob92]

IMHO it's way better to add support for WASI in Emcripten than to add Emscripten ABI/API support for every runtime out there.

[sbc100]

IMHO it's way better to add support for WASI in Emcripten than to add Emscripten ABI/API support for every runtime out there.

Perhaps, but I think that argument only really makes sense if the resulting binary is WASI-compaible. In this case it sounds like the resulting binary will contain a bunch of emscripten-specific stuff anyway, so won't be WASI-compliant moulde, right? (or am I misunderstanding?)

[jerbob92]

IMHO it's way better to add support for WASI in Emcripten than to add Emscripten ABI/API support for every runtime out there.

Perhaps, but I think that argument only really makes sense if the resulting binary is WASI-compaible. In this case it sounds like the resulting binary will contain a bunch of emscripten-specific stuff anyway, so won't be WASI-compliant moulde, right? (or am I misunderstanding?)

Correct, there are still a few things that are needed to run Emscripten binaries:

1. Embind adds a lot of Emscripten imports, but you don't have to use Embind so I don't think that's a problem
2. emscripten_notify_memory_growth
3. _emscripten_throw_longjmp
4. invoke_xxx generated methods for indirect function calls and catching errors

emscripten_notify_memory_growth could probably be removed in standalone mode (don't know if it adds any value there, in Wazero it doesn't do anything). I suspect that 3 and 4 can be get rid of when more features of WASI get to an implementation phase.