Usage question

[lonnietc]

Hello,

I am looking at various Golang project for a massively scaling P2P project. For this work, I have been looking at:

Pinecone
Yggdrasil
Go-Libp2p

and others if I can find them, but so far these "seem" to be the best for what I need.

I am wondering if Pinecone can be compiled in as a library for the core application that I am making of which part of it will have a DHT for some KV data passing for consistency.

Can Pinecone be compiled in as a library?

[lonnietc]

forgot to mention that I also came across the "Noise" library

https://github.com/perlin-network/noise

which is supposed to be extremely good as well.

[lonnietc]

Hello,

What is the simplest way to test out 3 pinecone P2P nodes in a small test cluster?

[neilalexander]

Can Pinecone be compiled in as a library?

It can be used as a regular net.PacketConn interface in Go applications, yes. I don't suppose it will be too easy to use it in a non-Go application though.

What is the simplest way to test out 3 pinecone P2P nodes in a small test cluster?

What exactly do you want to test?

[lonnietc]

Hello and sorry for the late reply.

I am just interested in setting up 3 or more instances (on Windows, for example) to see it in action in a small private cluster. I am assuming that each instance would need to be run on different physical computers.

Also, the application that I am developing is written in Go and is namely targeted toward a desktop P2P database which could have a huge number of nodes unlike other distributed database approaches which have a smaller number of dedicated database servers in a cluster. My project would be a P2P database system where nodes might join an leave often.

Could you please show me some examples of starting up some nodes into the same mesh cluster?

[lonnietc]

Is there any update on this?

I just want to start up 3 - 5 nodes and may ping through them to see the latency.

I actually have a few Linux servers online and some Windows boxes that I will use for the testing.

Can you please list some basic startup commands that I can use on the CLI for this?

I have actually narrowed my project (a game clustering network) down to either trying to use Pinecone or Yggdrasil.

Based upon your site, I would like to be an early adopter of Pinecone since it seems to be a bit more advanced than Yggdrasil but Yggdrasil has a bit more documentation that I can follow to start up and test things out.

Any help that you could provide would be greatly appreciated.
Thanks

Thanks.

[lonnietc]

Hello Neil,

Hope that all is well.

I have been able to easily start up pinecone on a couple of machines for testing with:

-- Sys 1 --
pinecone -listen 192.168.0.23

-- Sys 2 --
pinecone -listen 192.168.0.24:9999 -connect 129.168.0.23

What I am wondering now is how to send data across the mesh network from another application since I do not think that pinecone itself uses TUN network devices so that there is no virtual ethernet connection but could be wrong on this.

Please let me know.
Thanks

[neilalexander]

Sorry I haven't responded to this sooner, buried under notifications.

The ./cmd/pinecone binary is a headless router that is just used for providing routing to other nodes.

There's a separate ./cmd/pineconeip binary that uses TUN to send things over the network, but it's not very reliable, probably only works on Linux and it also doesn't have any encryption (your packets will be sent in the clear). It's really intended for use in the meshnet-lab, not for general purpose use.

For experimentation it might do what you want, but Pinecone is not really meant to be used in this way — it's designed to be a plug-in module for other applications.

[neilalexander]

FYI — for using Pinecone as a net.PacketConn in a Go application: https://pkg.go.dev/github.com/matrix-org/[email protected]/router

[lonnietc]

Thanks for getting back to me on these questions.

So, then it seems that in order to be of any use, I would need to run Pinecone and then PineconeIP. By the way, I did send over a link to what seems like a very simple and useful TUN library that supports multiple platforms to include Windows, Linux, and MacOS.

For my project, I am building a type of online 3D game system which could have thousands of P2P nodes and wanted to use Pinecone to be able to establish Mesh connections for all of the nodes (clients) and was not greatly concerned about needing them to be on a VPN or dedicated TUN devices but mostly to ensure P2P mesh connections to each other.

What would you suggest as I really liked the approach in the documentation of Pinecone over Yggdrasil which is not bad as well, but Pinecone seemed better if I can determine that it could do the job?

As a mesh router, I am wondering if just maybe that would be useful in itself, just a thought.
Thanks again

[neilalexander]

./cmd/pineconeip contains the same router code that ./cmd/pinecone does, so you don't need to run both.

If you are doing this as a part of an application though then I wouldn't think that TUN is really what you want:

1. TUN will expose the system to another network, which might not be what users expect;
2. TUN usually requires admin permissions to set up and configure;
3. You can't really stop anything else from using the TUN interface either.

I would think you want to embed it into your application and use it like a net.PacketConn-style "socket" instead.

[lonnietc]

I see what you mean now and perhaps embedding pinecone is what I need to do so that the nodes can send traffic to each other via the pinecone mesh network.

Also, the pinecone application in the github repo is just to test connections, I guess and would not be useful unless embedded in the server code.

I was just trying to get a feel for what the actual pinecone.exe application that I compiled is used for in its current state since it did not seem to be able to send other messages to the other nodes except for what was hardcoded for heartbeat messages within the mesh. I had figured that the pineconeip.exe used a TUN device but was going to try to stay away from using that if at all possible since to stat up most things that use TUN devices also required the user to elevate their privileges to start up and I did not want them to have to do that when they start their local pinecone node which will be a local server that they connect to when they want to be online in the game. Beyond that, the local node will just run in the background as a P2P node for the mesh network when the user is not connected. That is the idea, at least.

On a side question what is the "pinecone -manhole" switch do?

[neilalexander]

Also, the pinecone application in the github repo is just to test connections, I guess and would not be useful unless embedded in the server code.

Well, we use ./cmd/pinecone to run the public peers at wss://pinecone.matrix.org/public, since ultimately they just need to carry traffic for other nodes.

On a side question what is the "pinecone -manhole" switch do?

The manhole is just a HTTP endpoint that prints out the internal router state for debugging, so you can see information about directly connected peers, paths, DHT entries, queue statistics etc.

[lonnietc]

Hello,

I was working on trying to get PineconeIP integrated with TUN so that it might be used for a massive P2P project in which there could be tens of thousands of users but then I remembered your previous comment:

There's a separate ./cmd/pineconeip binary that uses TUN to send things over the network, but it's not very reliable, probably only works on Linux and it also doesn't have any encryption (your packets will be sent in the clear). It's really intended for use in the meshnet-lab, not for general purpose use.

With this in mind, I was wondering if you had any suggestions for either some other P2P project that you are working on, or perhaps another P2P development that I might investigate?

If "pineconeip" is effectively out of the running, then it seems that I may have to fall back on Yggdrasil or Go-Libp2p since I really do not know of any other possibilities that might be stable and mature enough to handle any really huge scaling needs.

Of course, there is also the SoftEther VPN or other client-server approach that will probably not work for what I need either.

Any thoughts?

[lonnietc]

Just an update.

I think that I have found a reasonable solution for the P2P question.

Although, I do not think that it employs a mesh design, I have come across NEBULA which is put out by Slack development and is said to scale to over 50,000+ nodes easily. If that is true, or not, that will remain to be seen.

https://github.com/slackhq/nebula

In any case, it seems that this will be a good starting point although I had high hopes and a great interest in Pinecone and PineconeIP, in particular, in that I liked the ideas that have gone into it.

I will keep an eye out to see if development continues but I need a solution that I can try to integrate into an evolving project so maybe in the future PineconeIP will be more stable and ready for production use and I will try it again.

Thanks for all of your help and guidance and have a great day.

[neilalexander]

I was working on trying to get PineconeIP integrated with TUN so that it might be used for a massive P2P project in which there could be tens of thousands of users

So I don't know what your application is but unless your application is intending to tunnel traffic, I am fairly certain that your users probably don't want to have to install a TUN driver and effectively open a wormhole into their systems just to use some application. Which is why I'm so confused about why you want TUN and not to just use a library from within your application.

For example, the P2P Matrix demos don't install TUN — they use QUIC over Pinecone. You can open QUIC connections to another Pinecone+QUIC node but you never have to expose the rest of the system to do it.

[lonnietc]

Hello and good morning.

So then, the basic problem set that I am researching is basically that we will have a type of online "game" network with a 3D engine. The difference is that this will be a P2P design such that each user is also effectively running a small server node as a background service and will connect to their local node when they want to join the game (world).

In addition to that, the users will also be running a very lightweight torrent application that will allow them to send/receive game files from other servers in the network that are cached locally in a type of ramdisk cache for faster access by their local game client application. The local torrent server will be signaled from the game client ahead of time to retrieve needed files from other nearby peers if those files are not already on the system and ready for use. It can also serve file to other nodes as needed.

The idea was to protect all of this, as well as to allow for huge scaling, by the use of a P2P VPN or hybrid type system since the local game server and torrent server work together but run independently.

In this approach, along with some other things, the game system is self-sustaining as resources come and go across the network mesh.

With regards to your QUICK connections over Pinecone, I do not know much about how that might work but it is also an intriguing idea that could be considered. How exactly would that work?

This is the idea, at least.

[neilalexander]

You should know that installing any type of VPN tunnelling on the user's system is quite a large security risk, especially to do so "under the radar". It is effectively opening an access route directly into their machine, bypassing any other firewalls or NATs that they have. That might be surprising to users at best and opening a huge attack surface at worst.

So to understand how the QUIC solution works, it's important to understand that Pinecone is effectively just providing you with the equivalent of a datagram/UDP-style socket, so anything that works over UDP (like QUIC, uTP etc) can listen on the Pinecone virtual socket instead of a normal system socket. When doing that, you don't send/receive packets to/from IP addresses — you send/receive packets to/from Pinecone public keys instead and Pinecone figures out how to get the traffic to its destination for you.

So you would effectively link Pinecone into your program as a library, create a QUIC listener on that Pinecone virtual socket (like the sessions package in the Pinecone repository does) and dial/accept QUIC connections using that instead, so that the traffic goes over the Pinecone network.

At the moment though you are probably confined to doing it in Go, or some language that can be easily bridged (i.e. using Cgo, or gobind for Swift/Kotlin), since we only have a Go implementation of Pinecone. Same goes for Yggdrasil.

[lonnietc]

I see what you mean, and it is actually a very interesting possibility that may be a better fit than what I have been considering with the VPN solution. On that note, the software installer was going to strictly let the user know that it would be using a VPN and they would have to agree before the install, but if QUIC/Pinecone could be used in a similar way and still maintain the security needed then that may be a better approach all together since it would still be a private network mesh for the P2P traffic, I think.

But also, did you not mention that the traffic is not encrypted as well? This would have to be considered also.

As far as the development languages, things are being done in Go and also C++ so there is a bit of flexibility and options for cross development.

I want to establish the main game server and torrent application which could be in Go and then the user connects to the server locally as the game client is in C++.

Still need to work out the details on this since it is a work in progress.
Thanks again

[neilalexander]

But also, did you not mention that the traffic is not encrypted as well? This would have to be considered also.

Pinecone doesn't encrypt traffic by itself, but since QUIC brings TLS with it, QUIC over Pinecone is effectively bringing you the transport encryption you'd want.

Yggdrasil, on the other hand, has session encryption. It would encrypt everything sent to another node, but that means if you use something like QUIC, you are actually encrypting twice.

The only thing to bear in mind with both Pinecone and Yggdrasil is that they are unreliable transports (i.e. datagrams can get dropped in transit) in the exact same way that UDP is unreliable. Something like QUIC adds the reliability and stream semantics as well as the encryption.

[lonnietc]

Would you like to discuss things more via email or Slack perhaps so that this issue can be closed since I would like to see how this might be implemented with QUIC/Pinecone?

I did also like Yggdrasil but it seems, based upon the information on your github site, that pinecone is a bit more advanced in the routing which is why I have been focusing on it more.

The current design idea is to basically have things to be as private and direct as possible for the users in a P2P approach but mainly will have just the local game server and the local torrent server as the daemon server on each user physical machine that it always running in the background. Maybe we can figure out how to make this happen with the approach that you are proposing with QUIC/Pincecone.

Maybe could even offer you a small contract to help make it happen, but we would have to discuss it more.

Please let me know your thoughts and how you may, or may not, want to proceed. If you do not want to, or do not have the time, to get involved more then maybe you could point me to some examples with QUIC/Pinecone that might help get me started since my first effort would be to try and figure out how to set up a simple go-torrent (have been looking at the C++ Picotorrent so far) to be able to fire up on some systems.

One more thing is that the game engine not only runs on PC (Windows, Linux, MacOSX) bot also iSO and Android) so there will be an eventual need to support those as well which is good for Go since it does tend to cross-compile well.

[lonnietc]

@neilalexander I've been playing with Yggdrasil for a while now and decided to go back and re-review some of our discussions on Pinecone+QUIC since I find that interesting as well.

In keeping with an overlay network with with qualities of Pinecone (i.e. self-healing, routing mesh topology, etc.), I was wondering if you had any simple examples of Pinecone+QUIC that I might be able to take a look at and investigate further.

I like the principles that Yggdrasil uses, but think that you are correct in that having to do use a TUN/TAP device (i.e. WinTUN, etc.) cold be problematic. On that note, I have designed and almost functional a Windows systray application for Yggdrasil that would allow it to be installed on a Windows machine with elevated privileges and then started/stopped as a service which the user can do from standard privileges. Note, the secret here was to allow the user to have elevated privileges for this application so that it could load/unload the wintun library without needed admin level privileges each time. It works, but I am not completely happy with that solution which is why I have not released it yet.

Anyway, I am interested in possibly seeing some simple Pinecone+QUIC example if you have one.
Thanks