add autonat v2 spec #538
add autonat v2 spec #538
Conversation
There was a problem hiding this comment.
Very nice, this is a solid starting point for the spec!
What's the plan for resolving #536? Would you open a new PR that targets this PR here?
|
thanks for your review @thomaseizinger. Proposal: use a list of addresses in priority order for autonat v2 dial requests #539 |
I don't have anything to add to those at the moment :) |
There was a problem hiding this comment.
On a brief skim, this looks good! I'm curious if we'll want to relax the "implementations MUST NOT dial any multiaddress unless it is based on the IP address the requesting node is observed as". Would it be useful to do this, and we can mitigate the amplification attack some other way?
It seems like there's a healthy discussion already going on, so I'll step back here and let other folks stay involved. If there's anything I can help with, please don't hesitate to ping.
|
Thanks for your review @MarcoPolo
The suggested strategy is discussed here: #536 Here's the PR for those changes: #542 |
|
While doing the rust-libp2p implementation, we discovered a race condition, which we are now circumventing by a 100ms delay. You can read the finally comment by @thomaseizinger here: umgefahren/rust-libp2p#1 (comment) It happens when the server successfully performs a dial back, thus sends the confirmation of the address back to the client. However the client hasn't progressed enough to be notified of that successful dial back when receiving the confirmation. In that case the client wrongly assumed an address was confirmed where no dial back occurred. |
Minor correction here: The behaviour is usually that the client discards the "successful" confirmation because it has not yet processed the dial-back so it thinks the server is sending it a confirmation without having actually done the dial. I think the correct way to solve this would be to add an ACK message from the client back to the server for the dial-back where the client can say: "Yes I've processed your dial-back". The server can then proceed to respond on the other stream and thus guarantee that we don't have a race condition between the two streams. |
|
You can read the closing of the stream as the ACK. See: https://github.com/libp2p/go-libp2p/blob/sukun/autonat-v2-2/p2p/protocol/autonatv2/server.go#L251-L257 The spec also dictates closing the stream: https://github.com/libp2p/specs/blame/autonat-v2/autonat/autonat-v2.md#L87 Do you think an explicit ACK is better? |
Yeah I think so. I associate closing a stream with "I have no more data to write". The client never writes data so why wouldn't it immediately close the stream? Also, reading a stream and waiting for that to fail because it has been closed it also somewhat odd 🤷♂️ |
That's a fair point. I'll add an ACK. |
|
Updated the specs with a |
Closes: #4524 This is the implementation of the evolved AutoNAT protocol, named AutonatV2 as defined in the [spec](https://github.com/libp2p/specs/blob/03718ef0f2dea4a756a85ba716ee33f97e4a6d6c/autonat/autonat-v2.md). The stabilization PR for the spec can be found under libp2p/specs#538. The work on the Rust implementation can be found in the PR to my fork: umgefahren#1. The implementation has been smoke-tested with the Go implementation (PR: libp2p/go-libp2p#2469). The new protocol addresses shortcomings of the original AutoNAT protocol: - Since the server now always dials back over a newly allocated port, this made #4568 necessary; the client can be sure of the reachability state for other peers, even if the connection to the server was made through a hole punch. - The server can now test addresses different from the observed address (i.e., the connection to the server was made through a `p2p-circuit`). To mitigate against DDoS attacks, the client has to send more data to the server than the dial-back costs. Pull-Request: #5526.
| same key repeatedly. The only benefit of going via the server to do this attack | ||
| is not spending bandwidth required for a handshake. So the prevention mechanism | ||
| only focuses on bandwidth costs. There is a minor benefit of bypassing IP | ||
| blocklists, but that's made unattractive by the fact that servers may ask 5x |
There was a problem hiding this comment.
I don't think we can simply shrug this off. This is called a reflection attack and has been a huge issue for open DNS resolvers.
Fixing the amplification side does go a long way, but paying a 5x bandwidth cost for a bunch of free IP addresses seems like a pretty reasonable tradeoff from an attacker's standpoint (especially because said attacker isn't paying for the bandwidth, but likely needs to compromise one machine per IP address).
There was a problem hiding this comment.
Also note: home NAT users likely don't need this feature. That is:
- They likely only need 1 dialable address.
- They likely don't care which one.
- Their outbound and inbound IPs are likely identical.
Being willing to dial other addresses does matter for, e.g., AWS and other special settings where there are separate ingress IP addresses. But, in that case, maybe the user should just configure their node correctly rather than relying on AutoNAT? AutoNAT specifically exists to enable home users.
There was a problem hiding this comment.
It simplifies client implementations as they don't need to worry about IPv4 peer vs IPv6 peer. Though the benefit isn't huge since most IPv4 servers won't have IPv6 connectivity so they any way cannot check the IPv6 address.
Fixing the amplification side does go a long way, but paying a 5x bandwidth cost for a bunch of free IP addresses seems like a pretty reasonable tradeoff from an attacker's standpoint (especially because said attacker isn't paying for the bandwidth, but likely needs to compromise one machine per IP address).
Can you elaborate here? why isn't the attacker paying for the bandwidth.
There was a problem hiding this comment.
- This features enables the following attack:
- Contact a large number of autonat v2 servers.
- Give them the target's address to connect to
- When requested for data: provide the data slowly:
* The stream timeout is 1 minute
* In the period, with a 1Gbps connection, you can send 60Gb ~= 6GB
* Maximum dial data requirement is 100kB
* So, in theory, you can run this with 60_000 peers in parallel.
* The servers have a random wait of up to 3 seconds precisely for this scenario. So in theory we can have 20k connections a second for 3 seconds to the target. - We can make a bunch of implementation improvements to reduce the harm here. The simplest ones being: Only wait 10 seconds for the dial data, and wait for 5 seconds before dialing. That would reduce the max new connection rate to 2k / second, which is very manageable.
- The ideal solution is to introduce rate limits for new connections.
- There's another problem with this feature related to implementation difficulty:
- The primary use for this feature is to allow both IPv4 and IPv6 addresses to be tested without worrying about whether we have a v4 or a v6 connection. So you can ask a v4 peer to test your v6 address. This requires correctly reporting an error in case the server has no v6 connectivity, which is majority of servers.
- I'm not sure if the rust implementation correctly handles this case. @umgefahren please correct me if I'm wrong here.
* See discussion around this comment: feat(autonatv2): Implement autonat v2 umgefahren/rust-libp2p#1 (comment) - I'm also not sure if we can rely on other implementations to correctly handle this. They might just make a dial, fail, and report unreachable.
- If we have to ensure that we check v6 addresses with a v6 peer, it might be better to just disable this feature.
@MarcoPolo @Stebalien @umgefahren thoughts?
There was a problem hiding this comment.
The potential attack vector seems to be described correctly, however I'm not sure that rust-libp2p is affected. The whole process is allowed to take a maximum of 10 seconds:
https://github.com/libp2p/rust-libp2p/blob/8ceadaac5aec4b462463ef4082d6af577a3158b1/protocols/autonat/src/v2/server/handler/dial_request.rs#L66
However, we don't wait any time before dealing back. This mitigation is a quick fix, I can prepare.
Regarding IPv4 and IPv6 I stand with @thomaseizinger's comment on that matter: umgefahren/rust-libp2p#1 (comment)
So it correctly handles that case, in that we don't generate false positives.
|
@sukunrt given that AutoNatv2 is merged in two reference implementations libp2p/rust-libp2p#5526 (released in 0.13.0) and libp2p/go-libp2p#2469 (released in 0.36.1) Are there any outstanding comments that need to be addressed before this pull request can be merged? - if there are any that are non-blocking, can they be addressed in follow up PRs? |
@sukunrt and I did interop testing and successfully verified that they are working together. |
|
The implementation is not used in go-libp2p yet. We should merge this after we start inferring reachability in go-libp2p. |
Closes: libp2p#4524 This is the implementation of the evolved AutoNAT protocol, named AutonatV2 as defined in the [spec](https://github.com/libp2p/specs/blob/03718ef0f2dea4a756a85ba716ee33f97e4a6d6c/autonat/autonat-v2.md). The stabilization PR for the spec can be found under libp2p/specs#538. The work on the Rust implementation can be found in the PR to my fork: umgefahren#1. The implementation has been smoke-tested with the Go implementation (PR: libp2p/go-libp2p#2469). The new protocol addresses shortcomings of the original AutoNAT protocol: - Since the server now always dials back over a newly allocated port, this made libp2p#4568 necessary; the client can be sure of the reachability state for other peers, even if the connection to the server was made through a hole punch. - The server can now test addresses different from the observed address (i.e., the connection to the server was made through a `p2p-circuit`). To mitigate against DDoS attacks, the client has to send more data to the server than the dial-back costs. Pull-Request: libp2p#5526.
autonat/autonat-v2.md
Outdated
|
|
||
| This `DialRequest` message has a list of addresses and a fixed64 `nonce`. The | ||
| list is ordered in descending order of priority for verification. AutoNAT V2 is | ||
| only for testing reachability on Public Internet. Client SHOULD NOT send any |
There was a problem hiding this comment.
I think this should be "MUST NOT". and "The server MUST NOT dial any private address".
There was a problem hiding this comment.
It is possible to implement these safely though. Both the client and the server need to check that the peer is connected over a private IP.
client 192.168.0.100 -> server 192.168.0.10
In this case it's reasonable for the client to ask the server to test its private IP reachability.
This is an edge case I'm willing to ignore though. Happy to change this to MUST, just that keeping it SHOULD allows some implementation to provide this feature if they're willing to.
There was a problem hiding this comment.
Can you know that you are indeed on the same private network?
There was a problem hiding this comment.
I'm not completely sure.
If you see local connection address in private IP range and remote connection address in private IP range, is that enough to conclude that you're in some private network?
Note you cannot rely on https://datatracker.ietf.org/doc/html/rfc1918 subnet masks as you can make a private network from a collection of smaller private networks.
There was a problem hiding this comment.
I think if the local address and remote address are in the same private subnet, then it would be okay.
How about adding "The server SHOULD NOT dial any private address"? This leaves the door open in the spec.
I'm not sure the usefulness of doing this though, but maybe others might have a use for it.
First draft for autonat v2. #503
This protocol allows for testing reachability on exactly one address. This helps determine reachability at an address level. This also simplifies the protocol a lot.
I'll change the spec to reflect the discussion on dialing a different ip address from the nodes observed ip address: #536
Discussion for nonce in message is here: libp2p/go-libp2p#1480
and this comment in particular libp2p/go-libp2p#1480 (comment)