Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tap_user usability tracking #87

Open
fish4terrisa-MSDSM opened this issue Apr 10, 2023 · 33 comments
Open

tap_user usability tracking #87

fish4terrisa-MSDSM opened this issue Apr 10, 2023 · 33 comments
Labels
discussion Debate for improvement enhancement New feature or request

Comments

@fish4terrisa-MSDSM
Copy link
Contributor

( Sorry for my poor english)
The QEMU has a feature to bind the guest`s port to the host\s port.Will RVVM support it later?It\s impossible to support x11 or SDL on android,so the vnc is the only choice.It`s important to make it possible to connect to the guest using the network port.

@LekKit
Copy link
Owner

LekKit commented Apr 10, 2023

Yeah it will be there in future. The networking is WIP as you can see from README, my attention is pretty scrambled over multiple things.
Can we convert this issue into tap_user usability tracking issue?

P.S. Is there a way to draw a raw framebuffer on Android using native GUI APIs? I am going to add JNI support and then we can have native Android RVVM app without VNC shenanigans. A while ago I expected SDL to be usable on Android and only later realized it isn't...

@LekKit LekKit added the enhancement New feature or request label Apr 10, 2023
@LekKit
Copy link
Owner

LekKit commented Apr 10, 2023

Current tap_user issues/TODOs:

  • No way to do 2-tuple lookup in TCP, because hashmap API is limited. This prevents us from doing 2-tuple reuse and therefore TCP port forwarding is broken.
  • No TCP buffering, it just expects the guest is able to receive all packets in time. Sometimes this results in dead connections, pacman failing to download packages, etc etc.
  • Inefficiend locking, it could be much more scalable and with less overhead
  • Basic firewall/traffic limiting

@LekKit
Copy link
Owner

LekKit commented Apr 10, 2023

Maybe it is worth to add VNC support into RVVM itself. There is such feature in QEMU (It acts like a VNC server), but I am unaware how complicated that might be (At least we have a nice networking lib).

Doing that via guest means is arguably not an ideal solution (We don't see the boot process, requires user intervention, what about non-Linux guests?), and we need a proper Android port anyways. So I guess JNI + android GUI is the best long term solution for this usecase.

@fish4terrisa-MSDSM
Copy link
Contributor Author

Yeah it will be there in future. The networking is WIP as you can see from README, my attention is pretty scrambled over multiple things. Can we convert this issue into tap_user usability tracking issue?

P.S. Is there a way to draw a raw framebuffer on Android using native GUI APIs? I am going to add JNI support and then we can have native Android RVVM app without VNC shenanigans. A while ago I expected SDL to be usable on Android and only later realized it isn't...

Ok i will change the issue name.

P.S. I dont know much about the android api .(in fact,my project heavily depends on alpine-term.I dont found any android app can draw a raw fb,even the termux-x11 or xserver xsdl dont use it).Besides, adding gui support is not so important i think .Many people dont need the gui support on android,they may prefer terminal .And if the port forwarding and the gpu support is done ,the user can start a vnc by themselves.A unused vnc causes profermance decrease. If you want to build the own RVVM app ,editing on my project may be a good idea ,just delete the descriptions of archlinux and you will get one .(My project is still undone ,it will take me about 2 weeks to fully move from QEMU to RVVM .And if its done ,i will notice you in the discussions area)

@fish4terrisa-MSDSM fish4terrisa-MSDSM changed the title Asking for guest port binding tap_user usability tracking Apr 10, 2023
@fish4terrisa-MSDSM
Copy link
Contributor Author

Maybe it is worth to add VNC support into RVVM itself. There is such feature in QEMU (It acts like a VNC server), but I am unaware how complicated that might be (At least we have a nice networking lib).

Doing that via guest means is arguably not an ideal solution (We don't see the boot process, requires user intervention, what about non-Linux guests?), and we need a proper Android port anyways. So I guess JNI + android GUI is the best long term solution for this usecase.

Thats true ,I prefer the vnc method ,its also useful on other platforms.The Android GUI method is a bit complicated (termux-x11 might be a exmaple , it has developed for more than 3y but still have large amounts of bugs) ,but adding a vnc viewer in the app might be a good idea as the user can connect to the machine without using other apps.

@fish4terrisa-MSDSM
Copy link
Contributor Author

A good vnc viewer project is avnc, its license is GPL3.You can insert some parts of it into the app.

@LekKit
Copy link
Owner

LekKit commented May 6, 2023

Commit 833a16e solves all previous TCP traffic issues and running out of UDP ports.
It also includes groundwork for TCP NAT (host->vm connections) but the way to lookup socket via 2-tuple is still in the works.

Please report if you still have any TCP stability issues.

@LekKit
Copy link
Owner

LekKit commented May 7, 2023

Figured there is an issue with keepalive ACKs not being properly handled. Fixed locally, will be upstreamed soon.
(This only matters if you have completely idle connections that live for hours)

@LekKit
Copy link
Owner

LekKit commented May 8, 2023

TCP port forwarding works. However, there is an issue when connecting directly to 127.0.0.1 from the host, the guest gets a connection from 127.0.0.1 as well and gets confused, so the connection is dropped. I don't know how to solve that correctly yet.
Will be upstreamed soon.

image

@fish4terrisa-MSDSM
Copy link
Contributor Author

A bug occured on NetBSD:

[root@archlinux ~]# echo $LANG
C.UTF-8
[root@archlinux ~]# curl www.google.com
WARN: net_poll_add() failed!
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl https://www.google.com
curl: (35) OpenSSL/3.0.8: error:0A0000C6:SSL routines::packet length too long
[root@archlinux ~]#

and when run curl www.google.com --http0.9,it will output the index.html of google,but the connection won`t close before timeout,so i think both the openssl error and the curl error is because that tcp connections cannot close accurately on NetBSD.The ping runs correctly ,so icmp have no problems.
When runs curl ip.sb --http0.9 to get the ip , it outputed

[root@archlinux ~]# curl ip.sb
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl ip.sb --http0.9
TTP/1.1 200 OK
Date: Tue, 09 May 2023 01:38:16 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive
Cache-Control: no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=cgybpWVU5jI3iZjC8NPmm7lMwYhsUfoaNkW7zU5VFKGN%2BxbuCpdO9E1Z%2Fl%2BKK3Rz3gAVbsJx9Pm9q9B9LGOwd0Lrt4dE5hLwJK1y4pfyf3X3ut8XYFoV"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c4638d53bc13087-SEA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

205.166.94.4

The 'H' of "HTTP/1.1" disappeared ,so both the length of the packet and the head of the packets might be wrong(?)
Output of ifconfig

[root@archlinux ~]# ifconfig
enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.100  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::8c2e:4fff:fed9:45f  prefixlen 64  scopeid 0x20<link>
        ether 8e:2e:4f:d9:04:5f  txqueuelen 1000  (Ethernet)
        RX packets 45  bytes 10639 (10.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 79  bytes 6345 (6.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The dhcp just worked well.I haven`t tested udp connections yet,and I also don`t know much about how to test it.
p.s. : now the VMA worked well on sdf.Great!

@fish4terrisa-MSDSM
Copy link
Contributor Author

A bug occured on NetBSD:

[root@archlinux ~]# echo $LANG
C.UTF-8
[root@archlinux ~]# curl www.google.com
WARN: net_poll_add() failed!
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl https://www.google.com
curl: (35) OpenSSL/3.0.8: error:0A0000C6:SSL routines::packet length too long
[root@archlinux ~]#

and when run curl www.google.com --http0.9,it will output the index.html of google,but the connection wont close before timeout,so i think both the openssl error and the curl error is because that tcp connections cannot close accurately on NetBSD.The ping runs correctly ,so icmp have no problems. When runs curl ip.sb --http0.9` to get the ip , it outputed

[root@archlinux ~]# curl ip.sb
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl ip.sb --http0.9
TTP/1.1 200 OK
Date: Tue, 09 May 2023 01:38:16 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive
Cache-Control: no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=cgybpWVU5jI3iZjC8NPmm7lMwYhsUfoaNkW7zU5VFKGN%2BxbuCpdO9E1Z%2Fl%2BKK3Rz3gAVbsJx9Pm9q9B9LGOwd0Lrt4dE5hLwJK1y4pfyf3X3ut8XYFoV"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c4638d53bc13087-SEA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

205.166.94.4

The 'H' of "HTTP/1.1" disappeared ,so both the length of the packet and the head of the packets might be wrong(?) Output of ifconfig

[root@archlinux ~]# ifconfig
enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.100  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::8c2e:4fff:fed9:45f  prefixlen 64  scopeid 0x20<link>
        ether 8e:2e:4f:d9:04:5f  txqueuelen 1000  (Ethernet)
        RX packets 45  bytes 10639 (10.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 79  bytes 6345 (6.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The dhcp just worked well.I havent tested udp connections yet,and I also dont know much about how to test it. p.s. : now the VMA worked well on sdf.Great!

I just tried curl --output ip.sb.out ip.sb --http0.9 -m 20,but the output is:

 /root__curl --output ip.sb.out ip.sb --http0.9 -m 20      05/09/2023 09:47:39 AM
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   602    0   602    0     0     30      0 --:--:--  0:00:20 --:--:--     0
curl: (28) Operation timed out after 20018 milliseconds with 602 bytes received
/root__ls                                                 05/09/2023 09:48:07 AM
_───_───────────_──────_───────_──────────_
│ # │   name    │ type │ size  │ modified │
├───┼───────────┼──────┼───────┼──────────┤
│ 0 │ ip.sb.out │ file │ 602 B │ now      │
_───┴───────────┴──────┴───────┴──────────_
/root__cat ./ip.sb.out                                    05/09/2023 09:48:09 AM
TTP/1.1 200 OK
Date: Tue, 09 May 2023 01:47:50 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive
Cache-Control: no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=%2B9y26TUxB6H%2FDgXjYX%2Fqi1JhdVY9i3aeAMhRmG6SPz8mgvaDPSuw8pk0cJ3tpiAhzive8aTSb3E7kP1BNUmkLCEFn9ZwlvFTcqNG%2BKEi511bq5ceN2Y8"}],"group":"cf-nel","max_age":604800}NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c4646d5be7608a1-SEA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

205.166.94.4

It seems that curl can`t detect the end of the connection.

@LekKit
Copy link
Owner

LekKit commented May 9, 2023

The 'H' of "HTTP/1.1" disappeared

Ideally I'd need a full tcpdump during this curl invocation, thanks. Will look what it might be in the meantime.

WARN: net_poll_add() failed!

Oh, this is unexpected. Usually this indicates host resource exhaustion and only seen to happen on Windows with >1024 open connections. It is unexpected from kqueue to act like this, and this may also be the reason of connections not being closed.
Can you try commenting out this line and re-testing TCP?

#define KQUEUE_NET_IMPL

@LekKit
Copy link
Owner

LekKit commented May 9, 2023

Weird... kqueue udata is not a void* in NetBSD???
image

@fish4terrisa-MSDSM
Copy link
Contributor Author

The 'H' of "HTTP/1.1" disappeared

Ideally I'd need a full tcpdump during this curl invocation, thanks. Will look what it might be in the meantime.

WARN: net_poll_add() failed!

Oh, this is unexpected. Usually this indicates host resource exhaustion and only seen to happen on Windows with >1024 open connections. It is unexpected from kqueue to act like this, and this may also be the reason of connections not being closed. Can you try commenting out this line and re-testing TCP?

#define KQUEUE_NET_IMPL

Commented . It worked!!! Both bug is fixed.Great.Maybe a port of NetBSD can be added.

@fish4terrisa-MSDSM
Copy link
Contributor Author

Yes...I got that warning when compling RVVM on sdf.

@LekKit
Copy link
Owner

LekKit commented May 9, 2023

Yes...I got that warning when compling RVVM on sdf.

This basically means NetBSD violates kqueue API, and either needs workarounds or will have to use slower select implementation which is already used on Windows and Haiku as a safe generic implementation. Commenting out KQUEUE_NET_IMPL disables kqueue.

@LekKit
Copy link
Owner

LekKit commented May 9, 2023

Fixed kqueue issues on NetBSD in 2aba896. Report if something isn't right still.

@LekKit
Copy link
Owner

LekKit commented May 10, 2023

Fixed another (rare) kqueue bug, perhaps it hasn't bothered anyone yet but there it is

@LekKit
Copy link
Owner

LekKit commented May 12, 2023

Weird warning in lynx browser. May be related to FIN/RST flags handling.

image

@fish4terrisa-MSDSM
Copy link
Contributor Author

Now RVVM seems not support multi task downloading (I dont know the right way to describe that),will RVVM support it?
Screenshot_20230617_112723
(As it showed , the other download task is rather slow except one)

@LekKit
Copy link
Owner

LekKit commented Jun 17, 2023

Now RVVM seems not support multi task downloading

I am not sure at all that it's on RVVM side. I occasionally see it's balancing multiple downloads properly, sometimes not, and I have no idea what could be going wrong (I am just pumping data from a socket into ethernet frames).
It might be just that the download speed is peaking on one download, and other ones actually are received at that speed on the host.

However, running iperf in clean conditions over local network seems to distribute bandwidth evenly:
image

I will examine if, for example it accidentally prioritizes one socket over others.
You could also try to comment out this and re-test:

#define EPOLL_NET_IMPL

@LekKit
Copy link
Owner

LekKit commented Jun 18, 2023

Running with epoll on Illumos. Apparently it balances the downloads fine, and it still peaks at around 4.5 MB/s downlink - which makes me believe this is actually peak of the Arch RISC-V mirrors capabilities (per IP?).
So far I am almost convinced this is related environment (mirrors/host) rather than RVVM.

image

@fish4terrisa-MSDSM
Copy link
Contributor Author

fish4terrisa-MSDSM commented Jan 7, 2024

TCP port forwarding works. However, there is an issue when connecting directly to 127.0.0.1 from the host, the guest gets a connection from 127.0.0.1 as well and gets confused, so the connection is dropped. I don't know how to solve that correctly yet. Will be upstreamed soon.

image

Is this problem fixed now? And, there isn't an option matched with port forward, maybe an option like rvvm -hostfwd ${GUEST_PORT}:${HOST_PORT} can be added? That may enable me to mount the guest fs to the host(or a 9pfs like qemu might be a better choice, but currently RVVM didn't support it) And, although nearly nobody need udp port forwarding, -tcpfwd and -udpfwd might be a better choice?

@LekKit
Copy link
Owner

LekKit commented Jan 19, 2024

I've almost finished a fix for localhost connections via port forwarding. Also working on distinguishing graceful FIN versus RST from outside. Will merge soon

@LekKit
Copy link
Owner

LekKit commented Feb 17, 2024

Please test 89a01de, it should implement all the desired (so far) functionality (But no special arguments yet).
You can ssh into the guest with ssh root@localhost -p 2022.

@LekKit
Copy link
Owner

LekKit commented Feb 27, 2024

Fixed a bunch of minor networking issues in staging branch.

@LekKit
Copy link
Owner

LekKit commented Feb 29, 2024

@X547 In upcoming patch it will be possible to have a separate IP address for the guest, so basically a non-NAT scenario without tap_linux being used.

It relies on

  1. Being able to create multiple IP addresses on one adapter
  2. Being able to bind tap_user sockets (both UDP/TCP) on that IP (See networking.h API for details)

The guest will be accessing the network from that address now on. It doesn't need to be a physical adapter either, so you can seat the guest onto a veth/tap/whatever interface you want. Firewall rules also can work specifically on this guest.

It is then possible to disable emulated NAT in tap_user and directly map it's IP to the assigned one, and map any ports directly without port translation (Which should be configured separately)

IPv6 is also worked on, the remaining bits are ICMPv6 and proper guest addr assignment. I am not sure yet if above technique will work for IPv6 ranges, but at least I'm sure /128 addresses will work.



3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:a7:a0:e7:25:2a brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.203/24 brd 192.168.0.255 scope global dynamic noprefixroute wlan0
       valid_lft 40563sec preferred_lft 40563sec
    inet 192.168.0.37/24 scope global secondary wlan0
       valid_lft forever preferred_lft forever

image

image

@LekKit
Copy link
Owner

LekKit commented Feb 29, 2024

When running in unprivileged environment without additional configuration, IPv6 will be still usable for internet access, but will be NATed - there is pretty much no other option in unprivileged case

@fish4terrisa-MSDSM
Copy link
Contributor Author

A separate IP address would be great, but is it possible to choose the ip address to bind with? And, it may cause ip address conflict if the user is not sure about whether the ip address is available in the network.
I think port forwarding is still useful, and options like -udpfwd and -tcpfwd would be great to see. Or, maybe a new option like -ip will help, and we may test whether the ip address is available before start the machine.

@LekKit
Copy link
Owner

LekKit commented Mar 1, 2024

A separate IP address would be great, but is it possible to choose the ip address to bind with? And, it may cause ip address conflict if the user is not sure about whether the ip address is available in the network. I think port forwarding is still useful, and options like -udpfwd and -tcpfwd would be great to see. Or, maybe a new option like -ip will help, and we may test whether the ip address is available before start the machine.

Basically there are going to be 2 options:

  • Default: Guest simply has access to internet (can download stuff, etc) and is behind a virtual NAT (So it's not recognized as a separate network host from the outside point of view). You can use port forwarding to map some host ports to guest ports. Works on all OSes without special privileges.
  • Interface passthrough: Guest sits on a separate address/interface so it is actually treated like a different machine in your LAN or even outside world in case of IPv6. Needs additional privileges, and unsure if it can be done on Windows. This is still WIP.

The latter option will be useful for more advanced usecases like firewalling the guest using host tools (iptables/netfilter/ipfw), accessing it as separate host without NAT, creating a virtual lan or putting it behind a VPN/proxy. It doesn't break simpler usecases (Which is to just get working internet everywhere)

@fish4terrisa-MSDSM
Copy link
Contributor Author

Basically there are going to be 2 options:

  • Default: Guest simply has access to internet (can download stuff, etc) and is behind a virtual NAT (So it's not recognized as a separate network host from the outside point of view). You can use port forwarding to map some host ports to guest ports. Works on all OSes without special privileges.

Great. That will make my current use unaffected. Port forwarding will really help.

  • Interface passthrough: Guest sits on a separate address/interface so it is actually treated like a different machine in your LAN or even outside world in case of IPv6. Needs additional privileges, and unsure if it can be done on Windows. This is still WIP.

That seems attractive for me. Can you tell me which privileges (or, capabilities) in linux are needed?

The latter option will be useful for more advanced usecases like firewalling the guest using host tools (iptables/netfilter/ipfw), accessing it as separate host without NAT, creating a virtual lan or putting it behind a VPN/proxy. It doesn't break simpler usecases (Which is to just get working internet everywhere)

Hmm... It will also make it possible to use tcpdump to monitor the guest.

@LekKit
Copy link
Owner

LekKit commented Mar 1, 2024

That seems attractive for me. Can you tell me which privileges (or, capabilities) in linux are needed?

CAP_NET_ADMIN for adding an address to interface; CAP_NET_BIND_SERVICE if you want to make guest ports below 1024 accessible without mapping them to a different port externally.

Systems without epoll or kqueue support won't be able to bind port ranges efficiently.

@LekKit
Copy link
Owner

LekKit commented Mar 3, 2024

Implemented -portfwd CLI argument in c92727e.

Supported syntax:
-portfwd tcp/2022=22 - Forward TCP 0.0.0.0:2022 to guest port 22 (DHCP guest address is assumed)
-portfwd tcp/127.0.0.1:2022=22 - Forward TCP 127.0.0.1:2022 to guest port 22 (DHCP guest address is assumed)
-portfwd 25565=25565 - Forward both UDP & TCP 0.0.0.0:25565 to the same guest port
-portfwd udp/[::1]:6121=[fd9c:350d:af07::385]:6121 - Forward UDP [::1]:6121 to specific IPv6 address in guest network

@LekKit LekKit added the discussion Debate for improvement label Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion Debate for improvement enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants