full_posix_sockets #7672

juj · 2018-12-14T10:12:41Z

This adds full POSIX sockets API support via a -s PROXY_POSIX_SOCKETS=1 option. This is implemented by proxying out the sockets API calls to a separate proxy bridge process running on localhost. (that one needs to separately spawn)

This can be useful for special occassions, e.g. when building unit test harnesses or similar, where one just wants to compile existing POSIX sockets code and "have it work" locally. A web browser page can then be simulated to act as a TCP or UDP server, for example.

The feature builds on top of #7670, and can only land after #7670 lands.

Will need splitting out libc to two .bc library parts, one for proxy sockets variant, and another one with the musl stubs.

juj · 2019-01-31T18:17:29Z

This PR would now be good to land, but this is producing some errors on the upstream bots that I do not understand, and am unable to reproduce. @kripken Does any of that make sense to you?

kripken · 2019-01-31T23:17:12Z

I can reproduce those errors locally, using the LLVM wasm backend. E.g. other.test_export_all_ fails on

error: undefined symbol: accept
warning: To disable errors for undefined symbols use `-s ERROR_ON_UNDEFINED_SYMBOLS=0`
error: undefined symbol: bind
error: undefined symbol: connect
error: undefined symbol: getsockname
error: undefined symbol: recv
error: undefined symbol: recvfrom
error: undefined symbol: send
error: undefined symbol: sendto
error: undefined symbol: setsockopt
error: undefined symbol: socket
Error: Aborting compilation due to previous errors

juj · 2019-02-01T00:28:18Z

Hmm, I wonder why that would occur only on the wasm backend.. Those symbols are moved from libc.bc over to libc-sockets.bc, and a libc-sockets.symbols file is provided, which declares those symbols for linking.

juj · 2019-02-01T00:28:41Z

Oh, perhaps I need to also update the libc.symbols and wasm-libc.symbols files, let's see if that does it.

juj · 2019-02-05T23:38:18Z

Updated this PR and it now passes. Ready for a second review?

kripken · 2019-02-06T18:56:15Z

I don't understand the motivation here yet. What does "proxy" mean - what is proxied where? Is this related to threading proxying?

juj · 2019-02-06T19:23:32Z

Proxying here means that instead of implementing the POSIX sockets calls in the browser, each sockets API call is serialized out via WebSockets pipe over to a native "websockify" style intermediate process, which natively performs those exact sockets calls, and returns the results back - making the Emscripten page look like it has full sockets API access available to it. In order to pull it off, it does require threading proxying, because this only works in multithreaded PROXY_TO_PTHREAD builds, since the command proxying is synchronous.

juj · 2019-02-10T16:47:39Z

Updated this PR, and it now looks green to merge. Good for a review pass?

kripken

Thanks, so this lets you use normal C sockets code, and it synchronously proxies commands to a websockets server that does the actual sockets operations? And you say it's useful for things like test harnesses etc. - this seems like quite a lot of work for just that? I feel like I'm missing the use case.

kripken · 2019-02-11T21:43:13Z

site/source/docs/porting/networking.rst

+Emscripten provides a native POSIX Sockets proxy server program, located in directory tools/websocket_to_posix_proxy/, that allows full POSIX Sockets API access from a web browser. This support works by proxying all POSIX Sockets API calls from the browser to the Emscripten POSIX Sockets proxy server (via transparent use of WebSockets API), and the proxy server then performs the native TCP/UDP calls on behalf of the page. This allows a web browser page to run full TCP & UDP connections, act as a server to accept incoming connections, and perform host name lookups and reverse lookups. Because all API calls are individually proxied, this support can be slow. This support is mostly useful for developing testing infrastructure and debugging.
+
+For an example of how the POSIX Sockets proxy server works in an Emscripten client program, see the file tests/websocket/tcp_echo_client.cpp.
+


The docs don't seem to mention the new proxy option from this PR?

Updated docs with linker settings for each case.

kripken · 2019-02-11T21:43:46Z

system/lib/libc.symbols

@@ -1,310 +1,541 @@
-------- C h_errno
-------- C __hwcap


why is this diff so large? I think the symbols should be sorted - maybe you didn't sort them before applying?

This is an as-is result from calling llvm-nm on new generated libc.bc. Is there some special process that one must do instead of dumping them? Perhaps there should be an automatic script that updates the symbol files from already built cache?

We should probably add a script, but basically just pipe it through | sort

Ah gotcha, updated the symbols files (using embuilder to build them, then llvm-nm ~/.emscripten_cache/asmjs/libc.bc | sort > system/lib/libc.symbols

kripken · 2019-02-11T21:46:05Z

tools/system_libs.py

+LIBC_SOCKETS = ['socket.c', 'socketpair.c', 'shutdown.c', 'bind.c', 'connect.c',
+                'listen.c', 'accept.c', 'getsockname.c', 'getpeername.c', 'send.c',
+                'recv.c', 'sendto.c', 'recvfrom.c', 'sendmsg.c', 'recvmsg.c',
+                'getsockopt.c', 'setsockopt.c', 'freeaddrinfo.c']


these are included in those builds - so what is proxied here, the syscalls called by those?

These different API calls are proxied through a WebSocket server. I.e. instead of socket() calling musl which calls socketcall(), the musl functions are replaced with C implemented functions that proxy each call through the WebSockets bridge.

juj · 2019-02-11T22:50:40Z

Thanks, so this lets you use normal C sockets code, and it synchronously proxies commands to a websockets server that does the actual sockets operations? And you say it's useful for things like test harnesses etc. - this seems like quite a lot of work for just that? I feel like I'm missing the use case.

We use this to target an existing sockets utilizing debug suite over to HTML5, retaining the same codebase for portability. This lets one reuse existing POSIX sockets code as is, and one only needs to add one API call at the start of the program to the bridge server, and then you get full sockets. This way we are able to locally connect a Unity page to all sorts of existing harness tools that are running on local developer's machine (e.g. debugger, profiler, inspector)

juj · 2019-11-27T15:27:36Z

For the tests, I think another job like this one, and right after it:

Thanks, good pointer, followed that pattern and added a sockets tests suite, which seems to pass nicely.

jakogut · 2019-11-27T18:30:59Z

Brought this PR up to latest now, looks like all is passing nicely. This should be ready to land now.

@juj There appears to be a problem with a program I'm attempting to compile that usesgetpeername. Upon being called, the proxy server reports: Received too small sockets call message! size: 0 bytes, expected at least 8 bytes, and upon investigating, the size passed to emscripten_websocket_send_binary is negative. I'm looking into it.

Thanks for taking a look at this. I had a detour for a while working on other things, and now coming back to this one. I think if there's a bug in this one, let's look at that as a followup, preferably with a test case? (if you're still interested/active on this?)

Yeah, I think I did end up fixing this one, and I can follow up with a PR and test case after this lands.

jakogut · 2019-11-27T18:51:40Z

I'm unsure what the code under tools/websocket_* is. How does that relate to the other code (in system/)?

This is the server application that proxies socket calls made by clients over websockets to the native system. I've successfully used this, for example, to play a video stream over HTTP using FFplay compiled to WebAssembly.

juj · 2019-12-03T09:55:48Z

Ping @kripken for approval?

kripken

lgtm with switching to upstream as mentioned in the comment.

hopefully these tests are not flakey, but if they are, we can revisit this later...

kripken · 2019-12-09T18:17:38Z

.circleci/config.yml

+            CHROME_FLAGS_NOCACHE: "--disk-cache-dir=/dev/null --disk-cache-size=1 --media-cache-size=1 --disable-application-cache --incognito"
+          command: |
+            export EMTEST_BROWSER="/usr/bin/google-chrome $CHROME_FLAGS_BASE $CHROME_FLAGS_HEADLESS $CHROME_FLAGS_WASM $CHROME_FLAGS_NOCACHE"
+            python tests/runner.py sockets



I wish there was a way to share this code instead of copying it, but I'm not sure if there is in circleci...

.circleci/config.yml

juj · 2019-12-10T13:53:20Z

Test failures are unrelated, looks like LLVM has added indentation checking to freetype code.

hortonelectric · 2020-04-26T11:45:01Z

Hey you guys, I'm working on something that uses POSIX, and I am still getting missing function: sigaltstack . I am doing -s PROXY_POSIX_SOCKETS=1

I'm not sure if I can actually get around this or not.

hortonelectric · 2020-04-26T11:53:12Z

error: undefined symbol: emscripten_websocket_send_binary ...hmmm..?

* Add posix sockets proxying option. * Add test for emscripten_websocket_get_url_length(). * Update docs on which sockets functions are proxied * Add sockets tests for chrome * Install websockets before sockets test run * Install node before ws * Try to fix ws installation * Fix websocket send test * Fix bad _strlen dep in _formatString function. * Add formatString dep

Update `test_websocket_send.c` such that it checks that both text and binary messages are received. Also, delete `websocket.c` which I think was supposed to be removed as part of emscripten-core#7672 (where it was renamed to test_websocket_send.c). Fixes: emscripten-core#21515

Update `test_websocket_send.c` such that it checks that both text and binary messages are received. Also, delete `websocket.c` which I think was supposed to be removed as part of #7672 (where it was renamed to test_websocket_send.c). Fixes: #21515

It appears that ever since 31309a8 "full_posix_sockets (emscripten-core#7672)", the _proxy side sent back a result that included an explicit option_len while the _socket side computes that value from the overall result size.

…socket.c ...that appears to have been bogus ever since 31309a8 "full_posix_sockets (emscripten-core#7672)"

juj added multithreading networking labels Dec 14, 2018

juj mentioned this pull request Dec 19, 2018

Add Emscripten WebSockets API #7670

Merged

juj force-pushed the full_posix_sockets branch from 473d614 to bbf53ca Compare January 16, 2019 20:25

juj force-pushed the full_posix_sockets branch 7 times, most recently from 96f6dff to d36f897 Compare January 31, 2019 14:57

juj force-pushed the full_posix_sockets branch 4 times, most recently from 80189a4 to 52ce4d5 Compare February 5, 2019 10:35

juj force-pushed the full_posix_sockets branch from 024f446 to 38eb8f7 Compare February 6, 2019 09:31

kripken reviewed Feb 11, 2019

View reviewed changes

juj force-pushed the full_posix_sockets branch 4 times, most recently from 0e6b9d9 to 9542cc6 Compare February 14, 2019 15:16

juj added 7 commits November 27, 2019 16:48

Add test for emscripten_websocket_get_url_length().

7410150

Update docs on which sockets functions are proxied

58eeb23

Add sockets tests for chrome

0c5b684

Install websockets before sockets test run

abe95f8

Install node before ws

e1c3bb7

Try to fix ws installation

633c257

Fix websocket send test

499dcff

juj force-pushed the full_posix_sockets branch from 53706a7 to 499dcff Compare November 27, 2019 14:48

kripken approved these changes Dec 9, 2019

View reviewed changes

Fix bad _strlen dep in _formatString function.

4b8208b

juj force-pushed the full_posix_sockets branch from be08762 to 4b8208b Compare December 10, 2019 11:19

Add formatString dep

09bf3c9

juj merged commit 31309a8 into emscripten-core:incoming Dec 10, 2019

kripken mentioned this pull request Dec 23, 2019

C++ sources installed into lib/emscripten/tools/websocket_to_posix_proxy #10093

Closed

meetwudi mentioned this pull request Nov 30, 2020

RTMP live transcoding is available? ffmpegwasm/ffmpeg.wasm#100

Open

sbc100 mentioned this pull request Mar 14, 2024

[wasm64] Fix library_websocket.js #21527

Merged

stbergmann mentioned this pull request Aug 22, 2024

getsockopt mismatch in websocket_to_posix_proxy vs. _socket #22432

Open

stbergmann added a commit to stbergmann/emscripten that referenced this pull request Aug 22, 2024

Fix computation of size to send in getsockname in websocket_to_posix_…

53ad745

…socket.c ...that appears to have been bogus ever since 31309a8 "full_posix_sockets (emscripten-core#7672)"

stbergmann mentioned this pull request Aug 22, 2024

Fix computation of size to send in getsockname in websocket_to_posix_socket.c #22433

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

full_posix_sockets #7672

full_posix_sockets #7672

juj commented Dec 14, 2018

juj commented Jan 31, 2019

kripken commented Jan 31, 2019

juj commented Feb 1, 2019

juj commented Feb 1, 2019

juj commented Feb 5, 2019

kripken commented Feb 6, 2019

juj commented Feb 6, 2019

juj commented Feb 10, 2019

kripken left a comment

kripken Feb 11, 2019

juj Feb 11, 2019

kripken Feb 11, 2019

juj Feb 11, 2019

kripken Feb 25, 2019

juj Feb 26, 2019

kripken Feb 11, 2019

juj Feb 11, 2019

juj commented Feb 11, 2019

juj commented Nov 27, 2019

jakogut commented Nov 27, 2019

jakogut commented Nov 27, 2019

juj commented Dec 3, 2019

kripken left a comment

kripken Dec 9, 2019

juj commented Dec 10, 2019

hortonelectric commented Apr 26, 2020

hortonelectric commented Apr 26, 2020

		Emscripten provides a native POSIX Sockets proxy server program, located in directory tools/websocket_to_posix_proxy/, that allows full POSIX Sockets API access from a web browser. This support works by proxying all POSIX Sockets API calls from the browser to the Emscripten POSIX Sockets proxy server (via transparent use of WebSockets API), and the proxy server then performs the native TCP/UDP calls on behalf of the page. This allows a web browser page to run full TCP & UDP connections, act as a server to accept incoming connections, and perform host name lookups and reverse lookups. Because all API calls are individually proxied, this support can be slow. This support is mostly useful for developing testing infrastructure and debugging.

		For an example of how the POSIX Sockets proxy server works in an Emscripten client program, see the file tests/websocket/tcp_echo_client.cpp.

full_posix_sockets #7672

full_posix_sockets #7672

Conversation

juj commented Dec 14, 2018

juj commented Jan 31, 2019

kripken commented Jan 31, 2019

juj commented Feb 1, 2019

juj commented Feb 1, 2019

juj commented Feb 5, 2019

kripken commented Feb 6, 2019

juj commented Feb 6, 2019

juj commented Feb 10, 2019

kripken left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juj commented Feb 11, 2019

juj commented Nov 27, 2019

jakogut commented Nov 27, 2019

jakogut commented Nov 27, 2019

juj commented Dec 3, 2019

kripken left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juj commented Dec 10, 2019

hortonelectric commented Apr 26, 2020

hortonelectric commented Apr 26, 2020