From 709534a8d4361017223296c65fe110de91e42654 Mon Sep 17 00:00:00 2001 From: Moses Narrow Date: Fri, 24 May 2024 09:45:58 -0500 Subject: [PATCH] add updated skywire specs documents from skycoinpro/skywire-specs --- skywire-specs/Drafts.md | 28 + skywire-specs/README.md | 3 + skywire-specs/Specifications.md | 2379 +++++++++++++++++ skywire-specs/VPN/Client.md | 5 + skywire-specs/VPN/Handshake.md | 52 + skywire-specs/VPN/Server.md | 41 + skywire-specs/VPN/Specs.md | 113 + skywire-specs/img/routing-table-diagram.png | Bin 0 -> 33897 bytes skywire-specs/specifications/01-Overview.md | 19 + .../02-HTTP_Authorization_Middleware.md | 98 + skywire-specs/specifications/03-Transport.md | 124 + .../specifications/04-Transport_Discovery.md | 349 +++ .../specifications/05-Messaging_System.md | 514 ++++ skywire-specs/specifications/06-Packets.md | 212 ++ .../specifications/07-Transport_Management.md | 113 + .../specifications/08-Route_Finder.md | 89 + .../specifications/09-Route_Setup_Process.md | 9 + .../specifications/10-Routing_Table.md | 63 + skywire-specs/specifications/11-Router.md | 19 + skywire-specs/specifications/12-App_Server.md | 37 + skywire-specs/specifications/13-Setup_Node.md | 3 + .../specifications/14-Skywire_Visor.md | 429 +++ skywire-specs/specifications/15-Hypervisor.md | 18 + skywire-specs/specifications/16-Glossary.md | 104 + .../specifications/17-App2 module.md | 106 + .../specifications/18-Router2 module.md | 296 ++ skywire-specs/transports/stcpr/Specs.md | 22 + skywire-specs/transports/sudph/Specs.md | 77 + 28 files changed, 5322 insertions(+) create mode 100644 skywire-specs/Drafts.md create mode 100644 skywire-specs/README.md create mode 100644 skywire-specs/Specifications.md create mode 100644 skywire-specs/VPN/Client.md create mode 100644 skywire-specs/VPN/Handshake.md create mode 100644 skywire-specs/VPN/Server.md create mode 100644 skywire-specs/VPN/Specs.md create mode 100644 skywire-specs/img/routing-table-diagram.png create mode 100644 skywire-specs/specifications/01-Overview.md create mode 100644 skywire-specs/specifications/02-HTTP_Authorization_Middleware.md create mode 100644 skywire-specs/specifications/03-Transport.md create mode 100644 skywire-specs/specifications/04-Transport_Discovery.md create mode 100644 skywire-specs/specifications/05-Messaging_System.md create mode 100644 skywire-specs/specifications/06-Packets.md create mode 100644 skywire-specs/specifications/07-Transport_Management.md create mode 100644 skywire-specs/specifications/08-Route_Finder.md create mode 100644 skywire-specs/specifications/09-Route_Setup_Process.md create mode 100644 skywire-specs/specifications/10-Routing_Table.md create mode 100644 skywire-specs/specifications/11-Router.md create mode 100644 skywire-specs/specifications/12-App_Server.md create mode 100644 skywire-specs/specifications/13-Setup_Node.md create mode 100644 skywire-specs/specifications/14-Skywire_Visor.md create mode 100644 skywire-specs/specifications/15-Hypervisor.md create mode 100644 skywire-specs/specifications/16-Glossary.md create mode 100644 skywire-specs/specifications/17-App2 module.md create mode 100644 skywire-specs/specifications/18-Router2 module.md create mode 100644 skywire-specs/transports/stcpr/Specs.md create mode 100644 skywire-specs/transports/sudph/Specs.md diff --git a/skywire-specs/Drafts.md b/skywire-specs/Drafts.md new file mode 100644 index 0000000000..daefd81452 --- /dev/null +++ b/skywire-specs/Drafts.md @@ -0,0 +1,28 @@ +# Drafts + +## Transport State + +The modifiable data associated with a *Transport* is stored in a different structure called the *Transport State*. As a *Transport* has two edges (and hence, two perspectives), a *Transport State* is also updated accordingly. + +## Packet types + +Transports, Routes and the *Route Setup Service* are to deliver data via Packets. Packets are responsible for setting up routes and streams, and delivering data within the constructed routes and streams. + +Here is a summary of all the *Packet Types*. + +| Type | Value | Description | +| ---- | ------ | ----------- | +| `Ping` | `0x0` | Sent between a Transport to check if connection is still open, and to determine latency of the Transport. | +| `InitiateRoute` | `0x1` | First packet sent via a route to have it initiated. | +| `RouteInitiated` | `0x2` | Confirms that a route is set up and functional. | +| `DestroyRoute` | `0x3` | Initiate the destruction of a route. | +| `RouteDestroyed` | `0x4` | Confirm the success of a route's destruction. | +| `OpenStream` | `0x5` | Opens a stream within a route. | +| `StreamOpened` | `0x6` | Confirms that a stream is successfully opened. | +| `CloseStream` | `0x7` | Closes a stream. | +| `StreamClosed` | `0x8` | Informs that a stream has successfully closed. | +| `Forward` | `0x9` | Forwards data via a specified stream. | + +## Route Setup Service + +The *Route Setup Service* is a service which communicates with ... \ No newline at end of file diff --git a/skywire-specs/README.md b/skywire-specs/README.md new file mode 100644 index 0000000000..5dc194fff9 --- /dev/null +++ b/skywire-specs/README.md @@ -0,0 +1,3 @@ +# Skywire Specifications + +The Skywire specifications. diff --git a/skywire-specs/Specifications.md b/skywire-specs/Specifications.md new file mode 100644 index 0000000000..287bd8d1c9 --- /dev/null +++ b/skywire-specs/Specifications.md @@ -0,0 +1,2379 @@ +- [Overview](#overview) +- [HTTP Authorization Middleware](#http-authorization-middleware) + - [Authorization Procedures](#authorization-procedures) +- [Transport](#transport) + - [Transport Module](#transport-module) +- [Transport Discovery](#transport-discovery) + - [Transport Discovery Procedures](#transport-discovery-procedures) + - [Security Procedures](#security-procedures) + - [Code Structure](#code-structure) + - [Database](#database) + - [Endpoint Definitions](#endpoint-definitions) + - [GET Incrementing Security Nonce](#get-incrementing-security-nonce) + - [GET Transport Entry via Transport ID](#get-transport-entry-via-transport-id) + - [GET Transport(s) via Edge Public Key](#get-transports-via-edge-public-key) + - [POST Register Transport(s)](#post-register-transports) + - [POST Status(es)](#post-statuses) +- [Dmsg](#dmsg-system) + - [Dmsg Modules](#dmsg-system-modules) + - [Dmsg Procedures](#dmsg-procedures) + - [Dmsg Discovery](#dmsg-discovery) + - [Instance Entry](#instance-entry) + - [Store Interface](#store-interface) + - [Endpoints](#endpoints) + - [GET Entry](#get-entry) + - [POST Entry](#post-entry) + - [GET Available Servers](#get-available-servers) + - [Dmsg Discovery Client Library](#dmsg-discovery-client-library) + - [Dmsg Discovery Integration Tests](#dmsg-discovery-integration-tests) + - [Dmsg Link](#dmsg-link) + - [Link Handshake Frames](#link-handshake-frames) + - [Dmsg Frames](#dmsg-frames) + - [Noise Implementation in Channels](#noise-implementation-in-channels) + - [Implementation in Code](#implementation-in-code) + - [Dmsg Instance](#dmsg-instance) + - [Configuring an Instance](#configuring-an-instance) + - [Instance Interaction with Dmsg Discovery](#instance-interaction-with-dmsg-discovery) + - [Channel Management](#channel-management) + - [Opening a Channel](#opening-a-channel) + - [Closing a Channel](#closing-a-channel) + - [Handling Disconnections](#handling-disconnections) +- [Packets](#packets) + - [Settlement Packets](#settlement-packets) + - [Foundational Packets](#foundational-packets) + - [`0x30 CreateLoop`](#0x30-createloop) + - [`0x31 LoopCreated`](#0x31-loopcreated) + - [`0x32 LoopNotCreated`](#0x32-loopnotcreated) + - [`0x33 ConfirmLoop`](#0x33-confirmloop) + - [`0x34 LoopConfirmed`](#0x34-loopconfirmed) + - [`0x35 SecureRIDs`](#0x35-securerids) + - [`0x36 RIDsSecured`](#0x36-ridssecured) + - [`0x37 RIDsNotSecured`](#0x37-ridsnotsecured) + - [`0x38 AddRules`](#0x38-addrules) + - [`0x39 RulesAdded`](#0x39-rulesadded) + - [`0x3A RemoveRules`](#0x3a-removerules) + - [`0x3B RulesRemoved`](#0x3b-rulesremoved) + - [Data Packets](#data-packets) + - [Loopback Packets](#loopback-packets) +- [Transport Management](#transport-management) + - [Transport Manager Procedures](#transport-manager-procedures) + - [Logging](#logging) +- [Route Finder](#route-finder) + - [Graph Algorithm](#graph-algorithm) + - [Routing algorithm](#routing-algorithm) + - [Code Structure](#code-structure-1) + - [Database](#database-1) + - [Endpoint Definitions](#endpoint-definitions-1) + - [GET Routes available for the defined start and end key](#get-routes-available-for-the-defined-start-and-end-key) +- [Route Setup Process](#route-setup-process) +- [Routing Table](#routing-table) +- [Router](#router) +- [App Server](#app-server) + - [Loop Encryption](#loop-encryption) +- [Setup Node](#setup-node) +- [App Node](#app-node) + - [App Node Configuration](#app-node-configuration) + - [App Node RPC Interface](#app-node-rpc-interface) + - [Commands](#commands) + - [Ports Management](#ports-management) +- [Manager Node](#manager-node) + - [Manager Node REST API (and User Interface)](#manager-node-rest-api-and-user-interface) +- [Glossary](#glossary) + +# Overview + +Skywire is an decentralized network that attempts to replace the current internet. The *Skywire Network* is made up of *Skywire Nodes*. There are currently two types of *Skywire Nodes*; *App Node* and *Setup Node*. + +Each *Skywire Node* is represented by a unique public key. A direct line of communication between two *Skywire Nodes* is called a *Transport*. Each *Transport* is represented by a unique *Transport ID* which is of a *Transport Type*, and the two *Skywire Nodes* that are connected via the *Transport* is named the *Transport Edges*. + +A *Route* is unidirectional and delivers data units called *Packets*. It is made up of multiple hops where each hop is a *Transport*. Two *Routes* of opposite directions make a *Loop* when associated with the given *Ports* at each *Loop Edge*. *Loops* handle the communication between two *Skywire Apps* and is represented via the *Loop's* source and destination node's public keys and the source and destination ports (similar to how TCP/UDP handles ports). + +A *Packet* is prefixed with a *Route ID* which helps *Skywire Nodes* identify how the *Packet* is to be handled (either to be forward to a remote node, or to be consumed internally). Every *Skywire Node* has a *Routing Table* that has the *Routing Rules* for that particular *Skywire Node*. + +In summary, + +- *Transports* are responsible for single-hop communication between two *Skywire Nodes* and are bidirectional (this may change later on). +- *Routes* are responsible for multi-hop communication between two *Skywire Nodes* and are unidirectional. +- *Loops* are responsible for communication between two *Skywire Apps* and are bidirectional. + +There are many ways in which we can implement a *Transport*. Each unique method is called a *Transport Type*. + +Initially, we need to implement a MVP in which we assume that there are no malicious nodes in the network and discovery of routes, transports and nodes are to be done in a centralized manner. However, basic authentication and encryption is still required. + +# HTTP Authorization Middleware + +Skywire is made up of multiple services and nodes. Some of these services/nodes communicate via restful interfaces, and some of the endpoints require authentication and authorization. + +As nodes in the Skywire network are identified via public keys, an appropriate approach to authentication and authorization is via public/private key cryptography. The curve to use is `secp256k1`, and when referenced by the RESTFUL endpoints, it is to be represented as a hexadecimal string format. + +These HTTP security middleware features should be implemented within the `/pkg/utils/httpauth` module of the `skywire` repository. This module not only provides server-side logic, but also client-side logic to make interaction with the server-side more streamlined. + +## Authorization Procedures + +To avoid replay attacks and unauthorized access, each remote entity (represented by it's public key) is assigned an *Security Nonce* by the `httpauth` module. The remote entity is required to sign the *Security Nonce* alongside the request body on every request. + +For each successful request, the next expected *Security Nonce* is to increment. The `httpauth` module is to provide an interface named `NonceStorer` to keep an record of "remote entity public key" to "next expected nonce" associations. The following is a proposed structure for `NonceStorer`; + +```golang +// NonceStorer stores Incrementing Security Nonces. +type NonceStorer interface { + + // IncrementNonce increments the nonce associated with the specified remote entity. + // It returns the next expected nonce after it has been incremented and returns error on failure. + IncrementNonce(ctx context.Context, remotePK cipher.PubKey) (nonce uint64, err error) + + // Nonce obtains the next expected nonce for a given remote entity (represented by public key). + // It returns error on failure. + Nonce(ctx context.Context, remotePK cipher.PubKey) (nonce uint64, err error) + + // Count obtains the number of entries stored in the underlying database. + Count(ctx context.Context) (n int, err error) +} +``` + +Take note that the only times the next-expected *Security Nonce* (for a given remote entity) is to increment, is when a successful request happens. + +Initially (when no successful requests has been processed for a given remote entity), the next expected *Security Nonce* should always be zero. When it is this value, the underlying database for the `NonceStorer` implementation should not need an entry for it. + +For every request that requires authentication and authorization in this manner, the structure `httpauth.Server` is to handle it. Specifically, it is to "wrap" the original `http.HandlerFunc` to add additional logic for checking the request. Consequently, the `httpauth.Client` appends the needed additional headers to the request. + +The following extra header values are required (`SW` stands for Skywire); + +- `SW-Public` - Specifies the public key (hexadecimal string representation) of the Skywire Node performing this operation. +- `SW-Nonce` - Specifies the incrementing nonce provided by this operation. +- `SW-Sig` - Specifies the of the signature (hexadecimal string representation) of the hash result of the concatenation of the Security Nonce + Body of the request. + +The `httpauth.Server` should also provide the `http.HandlerFunc` which obtains the next expected incrementing nonce for a given public key. This is required when a remote entity looses sync. A successful response of this call should look something of the following; + +```json +{ + "edge": "", + "next_nonce": 0 +} +``` + +The following is a proposed implementation of `httpauth.Server`; + +```golang +package httpauth + +// Server provides server-side logic for Skywire-related RESTFUL authorization and authentication. +type Server struct { + // implementation ... +} + +// NewServer creates a new authentication server with the provided NonceStorer. +func NewServer(store NonceStorer) *Server { + // implementation ... +} + +// WrapConfig configures the '(*Server).Wrap' function. +type WrapConfig struct { + // MaxHTTPBodyLen specifies the max body length that is acceptable. + // No limit is set if the value is 0. + MaxHTTPBodyLen int + + // PubKeyWhitelist specifies the whitelisted public keys. + // If value is nil, no whitelist rules are set. + PubKeyWhitelist []cipher.PubKey +} + +// Wrap wraps a http.HandlerFunc and adds authentication logic. +// The original http.HandlerFunc is responsible for setting the status code. +// The middleware logic should only increment the security nonce if the status code +// from the original http.HandlerFunc is of 2xx value (representing success). +func (as *Server) Wrap(config *WrapConfig, original http.HandlerFunc) http.HandlerFunc { + // implementation ... +} + +// HandleNextNonce returns a http handler that +func (as *Server) NextNonceHandler(remotePK cipher.PubKey) http.HandlerFunc { + // implementation ... +} +``` + +Take note that for the `(*Server).Wrap` function, we will need to define a custom `http.ResponseWriter` to obtain the status code (https://www.reddit.com/r/golang/comments/7p35s4/how_do_i_get_the_response_status_for_my_middleware/). + +The `httpauth.Client` implementation is responsible for providing logic for the following actions; + +- Keep a local record of the next expected *Security Nonce*. +- Adding security header values to a given request (`http.Request`). + +# Transport + +A *Transport* represents a bidirectional line of communication between two *Skywire Nodes* (or *Transport Edges*). A Transport is responsible for ensuring accurate delivery of data and providing symmetric encryption between the two nodes communicating. + +Each *Transport* is represented as a unique 16 byte (128 bit) UUID value called the *Transport ID* and has a *Transport Type* that identifies a specific implementation of the *Transport*. + +A *Transport* has the following information associated with it; + +- **Transport ID:** A `uuid.UUID` value that uniquely identifies the Transport. +- **Edges:** The public keys of the Transport's edge nodes (should only have 2 edges and the initiating edge should come first). +- **Type:** A `string` value that specifies the particular implementation of the *Transport*. +- **Public:** A `bool` that specifies whether the *Transport* is to be registered in the *Transport Discovery* or not. Only public transports are registered. +- **Registered:** A `int64` value that is the epoch time of when the *Transport* is registered in *Transport Discovery*. A value of `0` represents the state where the *Transport* is not (or not yet) registered in the *Transport Discovery*. + +This is a JSON representation of a *Transport Entry*; + +```json +{ + "t_id": "e1808c316b23d1d6119cad1795238ff0", + "edges": ["031d796272349d597d6d3130497ccd11cf8af12c7d186b1726358abfb49edad0c1", "03bd9724f335c5eb5a1011e7862d4af28488102c8edffc84585cf0826ac4864b38"], + "type": "dmsg", + "public": true +} +``` + +## Transport Module + +In code, `Transport` is an interface, and can have many implementations. + +The interface used to generate *Transports* of a certain *Transport Type* is named *Transport Factory* (represented by a `transport.Factory` interface in code). + +The representation of a *Transport* in *Transport Discovery* is of the type `transport.Entry`. + +A `transport.Status` type contains the status of a given *Transport*. Each *Transport Edge* provides such status, and the *Transport Discovery* compares the two statuses to derive the final status. + +```golang +package transport + +// Transport represents communication between two nodes via a single hop. +type Transport interface { + + // Read implements io.Reader + Read(p []byte) (n int, err error) + + // Write implements io.Writer + Write(p []byte) (n int, err error) + + // Close implements io.Closer + Close() error + + // Local returns the local transport edge's public key. + Local() cipher.PubKey + + // Remote returns the remote transport edge's public key. + Remote() cipher.PubKey + + // Type returns the string representation of the transport type. + Type() string + + // SetDeadline functions the same as that from net.Conn + // With a Transport, we don't have a distinction between write and read timeouts. + SetDeadline(t time.Time) error +} + +// Factory generates Transports of a certain type. +type Factory interface { + + // Accept accepts a remotely-initiated Transport. + Accept(ctx context.Context) (Transport, error) + + // Dial initiates a Transport with a remote node. + Dial(ctx context.Context, remote cipher.PubKey) (Transport, error) + + // Close implements io.Closer + Close() error + + // Local returns the local public key. + Local() cipher.PubKey + + // Type returns the Transport type. + Type() string +} + +// Entry is the unsigned representation of a Transport. +type Entry struct { + + // ID is the Transport ID that uniquely identifies the Transport. + ID uuid.UUID `json:"tid"` + + // Edges contains the public keys of the Transport's edge nodes (the public key of the node that initiated the transport should be on index 0). + Edges [2]string `json:"edges"` + + // Type represents the transport type. + Type string `json:"type"` + + // Public determines whether the transport is to be exposed to other nodes or not. + // Public transports are to be registered in the Transport Discovery. + Public bool `json:"public"` +} + +// SignedEntry holds an Entry and it's associated signatures. +// The signatures should be ordered as the contained 'Entry.Edges'. +type SignedEntry struct { + Entry *Entry `json:"entry"` + Signatures [2]string `json:"signatures"` + Registered int64 `json:"registered,omitempty"` +} + +// Status represents the current state of a Transport from the perspective +// from a Transport's single edge. Each Transport will have two perspectives; +// one from each of it's edges. +type Status struct { + + // ID is the Transport ID that identifies the Transport that this status is regarding. + ID uuid.UUID `json:"tid"` + + // IsUp represents whether the Transport is up. + // A Transport that is down will fail to forward Packets. + IsUp bool `json:"is_up"` + + // Updated is the epoch timestamp of when the status is last updated. + Updated int64 `json:"updated,omitempty"` +} +``` + +# Transport Discovery + +The Transport Discovery is a service that exposes a RESTful interface and interacts with a database on the back-end. + +The database stores *Transport Entries* that can be queried using their *Transport ID* or via a given *Transport Edge*. + +The process of submitting a *Transport Entry* is called *Registration* and a Transport cannot be deregistered. However, nodes that are an *Edge* of a *Transport*, can update their *Transport Status*, and specify whether the *Transport* is up or down. + +Any state-altering RESTful call to the *Transport Discovery* is authenticated using signatures, and replay attacks are avoided by expecting an incrementing security nonce (all communication should be encrypted with HTTPS anyhow). + +## Transport Discovery Procedures + +This is a summary of the procedures that the *Transport Discovery* is to handle. + +**Registering a Transport:** + +Technically, *Transports* are created by the Skywire Nodes themselves via an internal *Transport Factory* implementation. The *Transport Discovery* is only responsible for registering *Transports* in the form of a *Transport Entry*. + +When two Skywire Nodes establish a Transport connection between them, it is at first, unregistered in the *Transport Discovery*. The node that initiated the creation of the Transport (or the node that called the `(transport.Transport).Dial` method), is the node that is responsible for initiating the *Transport Settlement Handshake*. + +If two nodes; **A** and **B** establish a *Transport* between them (where **A** is the *Transport Initiator*), **A** is then also responsible for sending the first handshake packet for the *Transport Settlement Handshake*. The procedure is as follows: + +1. **A** sends **B** a proposed `transport.Entry` and also **A**'s signature of the Entry (in the form of `transport.SignedEntry`). + +2. **B** checks the `transport.SignedEntry` sent from **A**; + + 1. The `Entry.ID` field should be unique (check via *Transport Discovery*). + 2. The `Entry.Edges` field should be ordered correctly and contain public keys of **A** and **B**. + 3. The `Entry.Type` field should have the expected Transport Type. + 4. The `Signatures` field should contain **A**'s valid signature in the correct location (in the same index as **A**'s public key in `Entry.Edges`). + 5. The `Registered` field should be empty. + +3. **B** then adds it's only signature to the `transport.SignedEntry` and registers it to the *Transport Discovery*. Both public and private Transports are registered in the *Transport Discovery* (however only public *Transports* are publicly available). + +4. **B** then informs **A** on the success/failure of the registration, or just that the `transport.SignedEntry` is accepted by itself (depending on whether the Transport is to be public or not). + +**Submitting Transport Statuses:** + +If a given *Transport* is public, the associated *Transport Edges* is responsible for submitting their individual *Transport Statuses* to the *Transport Discovery* whenever the follow events occur; + +- Directly after a *Transport* is first successfully registered in the *Transport Discovery*. +- Whenever the *Transport* comes online/offline (connected/disconnected). + +**Obtaining Transports:** + +There are two ways to obtain transports; either via the assigned *Transport ID*, or via one of the *Transport Edges*. There is no restriction as who can access this information and results can be sorted by a given meta. + +## Security Procedures + +**Incrementing Security Nonce:** + +An *Incrementing Security Nonce* is represented by a `uint64` value. + +To avoid replay attacks and unauthorized access, each public key of a *Skywire Node* is assigned an *Incrementing Security Nonce*, and is expected to sign it with the rest of the body, and include the signature result in the http header. The *Incrementing Security Nonce* should increment every time an" endpoint is called (except for the endpoint that obtains the next expected incrementing security nonce). An *Incrementing Security Nonce* is not operation-specific, and increments every time any endpoint is called by the given Skywire Node. + +The *Transport Discovery* should store a table of expected next *Incrementing Security Nonce* for each public key of a *Skywire Node*. There is an endpoint `GET /security/nonces/{public-key}` that provides the next expected *Incrementing Security Nonce* for a given Node public key. This endpoint should be publicly accessible, but nevertheless, the *Skywire Nodes* themselves should keep a copy of their next expected *Incrementing Security Nonce*. + +The only times an *Incrementing Security Nonce* should not increment is when: + +- An invalid request is submitted (missing/extra fields, invalid signature). +- An internal server error occurs. + +Initially, the expected *Incrementing Security Nonce* should be 0. When it is this value, the *Transport Discovery* should not have an entry for it. + +Each operation should contain the following extra header entries: + +- `SW-Public` - Specifies the public key of the Skywire Node performing this operation. +- `SW-Nonce` - Specifies the incrementing nonce provided by this operation. +- `SW-Sig` - Specifies the hex-representation of the signature of the hash result of the concatenation of the *Incrementing Security Nonce* + Body of the request. + +If these values are not valid, the *Transport Discovery* should reject the request. + +## Code Structure + +The code should be in the `skywire-services` repository. + +- `/cmd/transport-discovery/transport-discovery.go` is the main executable for the *Transport Discovery*. +- `/pkg/transport-discovery/api/` contains the RESTFUL API definitions. +- `/pkg/transport-discovery/store/` contains the definition of the `Storer` interface and it's implementations. +- `/pkg/transport-discovery/client/` contains the client library that interacts with the *Transport Discovery* server's RESTFUL API. + +## Database + +The *Transport Discovery* should work with a variety of databases and the following interfaces should be defined for such implementations; + +- `TransportStorer` should store *Transport Signed Entries* and it's associated *Transport Statuses*. +- `NonceStorer` should store expected *Incrementing Nonces*. + +## Endpoint Definitions + +The following is a summary of all the *Transport Discovery* endpoints. + +- `GET /security/nonces/edge:` +- `GET /transports/id:` +- `GET /transports/edge:` +- `POST /transports` +- `POST /statuses` + +All endpoints should include an `Accept: application/json` field and the response header should include an `Content-Type: application/json` field. + +All requests (except for obtaining the next expected incrementing nonce) should include the following fields. + +``` +Accept: application/json +Content-Type: application/json +SW-Public: +SW-Nonce: +SW-Sig: +``` + +### GET Incrementing Security Nonce + +Obtains the next expected incrementing nonce for a given edge's public key. + +**Request:** + +``` +GET /security/nonces/ +``` + +**Responses:** + +- 200 OK (Success). + ```json + { + "edge": "", + "next_nonce": 0 + } + ``` +- 400 Bad Request (Malformed request). +- 500 Internal Server Error (Server error). + +### GET Transport Entry via Transport ID + +Obtains a *Transport* via a given *Transport ID*. + +Should only return a single `"transport"` result. + +**Request:** + +``` +GET /transports/id: +``` + +**Responses:** + +- 200 OK (Success). + ```json + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "" + }, + "is_up": true, + "registered": 0 + } + ``` +- 400 Bad Request (Malformed request). +- 500 Internal Server Error (Server error). + +### GET Transport(s) via Edge Public Key + +Obtains *Transport(s)* via a given *Transport Edge* public key. + +**Request:** + +``` +GET /transports/edge: +``` + +**Responses:** + +- 200 OK (Success). + ```json + [ + { + "entry": { + "t_id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "is_up": true, + "registered": 0 + }, + { + "entry": { + "t_id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "is_up": false, + "registered": 0 + } + ] + ``` +- 400 Bad Request (Malformed request). +- 500 Internal Server Error (Server error). + +### POST Register Transport(s) + +Registers one or multiple Transports. + +**Request:** + +``` +POST /transports +``` + +```json +[ + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "signatures": [ + "", + "" + ] + }, + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "signatures": [ + "", + "" + ] + } +] +``` + +**Responses:** + +- 200 OK (Success). + ```json + [ + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "signatures": [ + "", + "" + ], + "registered": 0 + }, + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "signatures": [ + "", + "" + ], + "registered": 0 + } + ] + ``` +- 400 Bad Request (Malformed request). +- 401 Unauthorized (Invalid signature/nonce). +- 408 Request Timeout (Timed out). +- 500 Internal Server Error (Server error). + +### POST Status(es) + +Submits one or multiple *Transport Status(es)* from the perspective of the submitting node. The returned result is the final *Transport Status(es)* determined by the *Transport Discovery* that is generated using the submitted *Transport Status(es)* of the two edges. + +When a Transport is registered, it is considered to be *up*. Then after, every time a node's *Status* is submitted, the *Transport Discovery* alters the final state *Status* with the following rules: + +- If there is only one edge's *Status* submitted, the final status is of that of the submitted *Status*. +- If there are two *Status*es submitted and they both agree, final *Status* will also be the same. +- If the two submitted *Status*es disagree, then the final *Status* is always *Down*. + +**Request:** + +``` +POST /statuses +``` + +```json +[ + { + "id": "", + "is_up": true + }, + { + "id": "", + "is_up": true + } +] +``` + +**Responses:** + +- 200 OK (Success). + ```json + [ + { + "id": "", + "is_up": true, + "updated": 0 + }, + { + "id": "", + "is_up": false, + "updated": 0 + } + ] + ``` +- 400 Bad Request (Malformed request). +- 401 Unauthorized (Invalid signature/nonce). +- 408 Request Timeout (Timed out). +- 500 Internal Server Error (Server error). + +# Dmsg + +[Dmsg](https://github.com/skycoin/dmsg) is an initial implementation of the `Transport` and associated interfaces. To work, dmsg requires an active internet connection and is designed to be horizontally scalable. + +Three services make up dmsg: *Dmsg Client* (or *Client Instance*), *Dmsg Server* (or *Service Instance*) and *Dmsg Discovery*. + +*Dmsg Clients* and *Dmsg Servers* are represented by public/private key pairs. *Dmsg Clients* deliver data to one another via *Dmsg Servers* which act as relays. + +The *Dmsg Discovery* is responsible for allowing *Dmsg Clients* to find other advertised *Dmsg Clients* via their public keys. It is also responsible for finding appropriate *Dmsg Servers* that either "advertise" them, or "advertise" other *Dmsg Clients*. + +``` + [D] + + S(1) S(2) + // \\ // \\ + // \\ // \\ + C(A) C(B) C(C) C(D) +``` + +Legend: +- ` [D]` - Discovery Service +- `S(X)` - Dmsg Server (Server Instance) +- `C(X)` - Dmsg Client (Client Instance) + +## Dmsg Modules + +There are two modules of dmsg. + +- `dmsg-discovery` contains the implementation of the *Dmsg Discovery*. +- `dmsg` contains the implementation of either a *Client Instance* or a *Server Instance* of a *Dmsg Client* or *Dmsg Server*. + +## Dmsg Procedures + +This is a summary of the procedures that the *Dmsg* is to handle. + +**Advertising a client:** + +To be discoverable by other clients, a client needs to advertise itself. + +1. Client queries the Discovery to find available Servers. +2. Client connects to some (or all) of the suggested Servers. +3. Client updates it's own record in Discovery to include it's delegated Servers. + +**Client creates a channel to another client:** + +In order for two clients to communicate, both clients need to be connected to the same dmsg server, and create a channel to each other via the server. + +1. Client queries discovery of the remote client's connected servers. The client will connect to one of these servers if it originally has no shared servers with the remote. +2. The client sends a `OpenChannel` frame to the remote client via the shared server. +3. If the remote client accepts, a `ChannelOpened` frame is sent back to the initiating client (also via the shared server). A channel is represented via two *Channel IDs* (between the initiating client and the server, and between the responding client and the server). The associated between the two channel IDs is defined within the server. +4. Once a channel is created, clients can communicate via one another via the channel. + +## Dmsg Discovery + +The *Dmsg Discovery* acts like a DNS for dmsg instances (*Dmsg Clients* or *Dmsg Servers*). + +### Instance Entry + +An entry within the *Dmsg Discovery* can either represent a *Dmsg Server* or a *Dmsg Client*. The *Dmsg Discovery* is a key-value store, in which entries (of either server or client) use their public keys as their "key". + +The following is the representation of an Entry in Golang. + +```golang +// Entry represents an Instance's entry in the Discovery database. +type Entry struct { + // The data structure's version. + Version string `json:"version"` + + // A Entry of a given public key may need to iterate. This is the iteration sequence. + Sequence uint64 `json:"sequence"` + + // Timestamp of the current iteration. + Timestamp int64 `json:"timestamp"` + + // Public key that represents the Instance. + Static string `json:"static"` + + // Contains the node's required client meta if it's to be advertised as a Dmsg Client. + Client *Client `json:"client,omitempty"` + + // Contains the node's required server meta if it's to be advertised as a Dmsg Server. + Server *Server `json:"server,omitempty"` + + // Signature for proving authenticity of of the Entry. + Signature string `json:"signature,omitempty"` +} + +// Client contains the node's required client meta, if it is to be advertised as a Dmsg Client. +type Client struct { + // DelegatedServers contains a list of delegated servers represented by their public keys. + DelegatedServers []string `json:"delegated_servers"` +} + +// Server contains the node's required server meta, if it is to be advertised as a Dmsg Server. +type Server struct { + // IPv4 or IPv6 public address of the Dmsg Server. + Address string `json:"address"` + + // Port in which the Dmsg Server is listening for connections. + Port string `json:"port"` + + // Number of connections still available. + AvailableConnections int `json:"available_connections"` +} +``` + +**Definition rules:** + +- A record **MUST** have either a "Server" field, a "Client" field, or both "Server" and "Client" fields. In other words, a Dmsg Node can be a Dmsg Server Node, a Dmsg Client Node, or both a Dmsg Server Node and a Dmsg Client Node. + +**Iteration rules:** + +- The first entry submitted of a given static public key, needs to have a "Sequence" value of `0`. Any future entries (of the same static public key) need to have a "Sequence" value of `{previous_sequence} + 1`. +- The "Timestamp" field of an entry, must be of a higher value than the "Timestamp" value of the previous entry. + +**Signature Rules:** + +The "Signature" field authenticates the entry. This is the process of generating a signature of the entry: +1. Obtain a JSON representation of the Entry, in which: + 1. There is no whitespace (no ` ` or `\n` characters). + 2. The `"signature"` field is non-existent. +2. Hash this JSON representation, ensuring the above rules. +3. Create a Signature of the hash using the node's static secret key. + +The process of verifying an entry's signature will be similar. + +### Store Interface + +The underlying database of the *Dmsg Discovery* is a key-value store. The `Store` interface allows many databases to be used with the *Dmsg Discovery*. + +```golang +type Store interface { + // Entry obtains a single dmsg instance entry. + // 'static' is a hex representation of the public key identifying the dmsg instance. + Entry(ctx context.Context, static string) (*Entry, error) + + // SetEntry set's an entry. + // This is unsafe and does not check signature. + SetEntry(ctx context.Context, entry *Entry) error + + // AvailableServers discovers available dmsg servers. + // Obtains at most 'maxCount' amount of available servers obtained randomly. + AvailableServers(ctx context.Context, maxCount int) ([]*Entry, error) +} +``` + +### Endpoints + +Only 3 endpoints need to be defined; Get Entry, Post Entry, and Get Available Servers. + +#### GET Entry +Obtains a dmsg node's entry. +> `GET {domain}/discovery/entries/{public_key}` + +**REQUEST** + +Header: +``` +Accept: application/json +``` + +**RESPONSE** + +Possible Status Codes: +- Success (200) - Successfully updated record. + - Header: + ``` + Content-Type: application/json + ``` + - Body: + > JSON-encoded entry. +- Not Found (404) - Entry of public key is not found. +- Unauthorized (401) - invalid signature. +- Internal Server Error (500) - something unexpected happened. + +#### POST Entry +Posts an entry and replaces the current entry if valid. +> `POST {domain}/discovery/entries` + +**REQUEST** + +Header: +``` +Content-Type: application/json +``` +Body: +> JSON-encoded, signed Entry. + +**RESPONSE** + +Possible Response Codes: +- Success (200) - Successfully registered record. +- Unauthorized (401) - invalid signature. +- Internal Server Error (500) - something unexpected happened. + +#### GET Available Servers +Obtains a subset of available server entries. +> `GET {domain}/discovery/available_servers` + +**REQUEST** + +Header: +``` +Accept: application/json +``` + +**RESPONSE** + +Possible Status Codes: +- Success (200) - Got results. + - Header: + ``` + Content-Type: application/json + ``` + - Body: + > JSON-encoded `[]Entry`. +- Not Found (404) - No results. +- Forbidden (403) - When access is forbidden. +- Internal Server Error (500) - Something unexpected happened. + +### Dmsg Discovery Client Library + +The module is named `client`. It contains a `HTTPClient` structure, that defines how the client will interact with the *Dmsg Discovery* API. + +A new `HTTPClient` object can be instantiated using the public function `New(address string)`. + +It exposes the following public methods: + +```go +// Entry retrieves an entry associated to the given public key from the discovery server. +func (*HTTPClient) Entry(ctx context.Context, static string) (*Entry, error) { + // definition ... +} + +// SetEntry tries to set the given entry on the discovery server. It must be signed. +// If the entry is modifying a previous one, must be signed by the same private key. +func (*HTTPClient) SetEntry(ctx context.Context, entry *Entry) error { + // definition ... +} + +// AvailableServers gets a list of server entries from the skywire discovery server. +// The amount is determined by the discovery server. +func (*HTTPClient) AvailableServers(ctx context.Context) ([]*Entry, error) { + // definition ... +} +``` + +The module also provides public functions to instantiate valid `Entry` objects. + +### Dmsg Discovery Integration Tests + +> **TODO:** Fix wording. + +This package does not uses another `messenger` package, so integration tests are defined for the external services that `discovery` is using. In this case, the external store and `discovery` itself for testing the `client` library. + +The cases for the store integration testing: + + 1. Its able to set an entry on the database without error by calling the `storer.SetEntry` method. + 2. Its able to retrieve the previously set entry by calling the `storer.Entry` method. + 3. Creates multiple service entries and store them by calling `storer.SetEntry`, then it should be able to retrieve them with `store.AvailableServers`. + 4. `store.AvailableServers` receives a `maxCount int` argument. We also test it passing an integer which value is less than the amount of server entries we have set in the database, it should return this exact amount of server entries. + 5. Same as in number 4, but we set `maxCount` to number bigger than the number of server entries we have set, now we should get an slice of the size of the server entries we have set. + +In order to run the test we preferably create a clean new instance of the store database using Docker, and the test code should connect to it. We remove it after we have tested. + +In order to test the client library we do integration test with an instance of the discovery server. +The test cases for the client integration testing: + +1. By using the method SetEntry the client can set a new entry on the discovery server. +2. By using the method SetEntry the client can update a previously set entry on the discovery server. +3. If using SetEntry to update a previously set Entry, but the new sequence is not the previous sequence + 1 it should return an error with status code 500, Something unexpected happened. +4. If using SetEntry to update a previously set Entry, but the signature of the new entry has been made by a different secret key it should return an error with status code 401, Invalid signature. +5. By calling the method Entry with the public key of a previously set Entry it should return that entry. +6. By calling the method Entry with the public key of a previously non-set Entry it should return an error with code 404, Entry of public key is not found. +7. By calling the method AvailableServers when there are previously set server entries it should return them. + +## Dmsg Link + +The `link` provides two *Dmsg Instances* a means to establish a connection with one another, and also handle a pool of connections. + +Using the *Dmsg Discovery*, a *Dmsg Instance* can discover other instances via only their public key. However, a *Link* requires both a public key and an address:port. + +Data sent via a *Link* is encapsulated in *Frames*. A *Link* is implemented using a TCP connection. + +### Link Handshake Frames + +When setting up a *Link* between two instances, the instance that initiates is called the *Initiator* and the instance that responds is called the *Responder*. Each instance is represented by a public key. + +To set up a *Link*, the *Initiator* first dials a TCP connection to the listening *Responder*. Once the TCP connection is established, the *Responder* sends the first *Frame*. It is expected that the *Initiator* knows the public key of the *Responder* + +Given a situation where instances 'A' and 'B' are to establish a link with one another (where 'A' is the initiator), the following *Frames* are delivered to perform a handshake. + +Link Handshake Frames are to be in JSON format. + +**Link Handshake Frame 1 (A -> B):** + +```json +{ + "version": "0.1", + "initiator": "036b630436972743d4b3f5bb39cd451da29d222b5d30893684a40f34a66c692157", + "responder": "02e18998f0174631710e47052927e13bddc712ca0c11289e0150fbf57570e31151", + "nonce": "853ea8454d11bd3b59cb31f8572a3779" +} +``` + +The initiator is responsible for sending the first frame. +- `"version"` specifies the version of dmsg protocol that the initiator is using (`"0.1"` for now). +- `"initiator"` should contain the hex representation of the public key of the initiator (the instance that is sending the first handshake frame). +- `"responder"` should contain the hex representation of the public key of the expected responder (the responder should disconnect TCP if this is not their public key). +- `"nonce"` is the hex-string representation of a 16-byte nonce that the responder should sign (alongside the initiator's public key) to check authenticity of the responder and whether the responder. + +**Link Handshake Frame 2 (B -> A):** + +```json +{ + "version": "0.1", + "initiator": "036b630436972743d4b3f5bb39cd451da29d222b5d30893684a40f34a66c692157", + "responder": "02e18998f0174631710e47052927e13bddc712ca0c11289e0150fbf57570e31151", + "nonce": "853ea8454d11bd3b59cb31f8572a3779", + "sig1": "df8a978f0ea681e218cfd8127692dbe4190441567181b9057ab15da34b08ff610d9060e5195419e1744bb57d50373c1dd444b5c2753a80dba32b292fa306e9df01" +} +``` + +This frame allows the responder agree with the initiator and prove it's ownership of it's claimed public key. + +The `"sig1"` field contains a hex representation of the result of signing the concatenation of the version, initiator, responder and nonce fields. Note that before concatenation, hex representations should be decoded and the concatenation result needs to be hashed before being signed. `"sig1"` should be signed by the responder. + +**Link Handshake Frame 3 (A -> B):** + +```json +{ + "version": "0.1", + "initiator": "036b630436972743d4b3f5bb39cd451da29d222b5d30893684a40f34a66c692157", + "responder": "02e18998f0174631710e47052927e13bddc712ca0c11289e0150fbf57570e31151", + "nonce": "853ea8454d11bd3b59cb31f8572a3779", + "sig1": "df8a978f0ea681e218cfd8127692dbe4190441567181b9057ab15da34b08ff610d9060e5195419e1744bb57d50373c1dd444b5c2753a80dba32b292fa306e9df01", + "sig2": "fc17928d5a3f7691434282fb3108d1603889f996e8e45adc2e35362e08009b8611abb9f45e511b9931f0f04b37ff1057fd69554befe534ad28c77ff0c44121ab00" +} +``` + +This frame allows the initiator to inform the responder that `"sig1"` is accepted, and to prove the initiator's ownership of it's public key. + +`"sig2"` is the signature result of the concatenation of the version, initiator, responder, nonce and sig1 fields. Concatenation rules are the same as that of `"sig1"`. + +**Link Handshake Frame 4 (B -> A):** + +```json +{ + "version": "0.1", + "initiator": "036b630436972743d4b3f5bb39cd451da29d222b5d30893684a40f34a66c692157", + "responder": "02e18998f0174631710e47052927e13bddc712ca0c11289e0150fbf57570e31151", + "nonce": "853ea8454d11bd3b59cb31f8572a3779", + "sig1": "df8a978f0ea681e218cfd8127692dbe4190441567181b9057ab15da34b08ff610d9060e5195419e1744bb57d50373c1dd444b5c2753a80dba32b292fa306e9df01", + "sig2": "fc17928d5a3f7691434282fb3108d1603889f996e8e45adc2e35362e08009b8611abb9f45e511b9931f0f04b37ff1057fd69554befe534ad28c77ff0c44121ab00", + "accepted": true +} +``` + +Sent by the responder, this frame concludes the handshake if the value of `"accepted"` is `true`. + +### Dmsg Frames + +After the handshake phase, frames have a reoccurring format. These are the *Dmsg Frames* of dmsg. + +``` +| FrameType | PayloadSize | Payload | +| 1 byte | 2 bytes | ~ bytes | +``` + +- The `Type` specifies the frame type. Different frame types are used for opening and closing channels as well as sending packets via the channels. +- `PayloadSize` contains an encoded `uint16` value that represents the Payload's length (the max size is 65535). +- `Payload` has a length determined by `PayloadSize`. + +The following is a summary of the frame types. + +| FrameTypeValue | FrameTypeName | FrameBody | +| -------------- | ------------- | --------- | +| `0x0` | `OpenChannel` | ChannelID + RemoteStatic + NoiseMessage1 | +| `0x1` | `ChannelOpened` | ChannelID + NoiseMessage2 | +| `0x2` | `CloseChannel` | ChannelID | +| `0x3` | `ChannelClosed` | ChannelID | +| `0x4` | `Send` | ChannelID + CipherText | + +The `FrameBody` has the following sub-fields. A `FrameBody` with multiple sub-fields have the sub-fields concatenated. + +- The `ChannelID` sub-field is represented by a single byte. This restricts a *Client Instance* to have at most 256 channels via a single *Server Instance*. +- The `RemoteStatic` sub-field is represented by 33 bytes. It contains a public key of a remote *Client Instance*. +- `NoiseMessage1` and `NoiseMessage2` are both represented by 49 bytes. It contains the noise handshake messages for establishing symmetric encryption between the two client instances of the channel. The noise handshake pattern used is KK. +- The `CipherText` sub-field is the only sub-field with a modular length. It contains size of the encrypted payload followed by payload that is to be delivered. + +### Noise Implementation in Channels + +As stated above, a channel is established using the `OpenChannel` and `ChannelOpened` frames. Then, after a channel is established, the two *Client Instances* of the channel can communicate with each over via `Send` frames (which includes a `CipherText` component). + +The protocol used to establish the symmetric encryption of the `CipherText` is the [Noise Protocol](http://noiseprotocol.org/). + +The curve used will be `secp256k1` for the key pair, and `chacha20poly1305` will be used for the symmetric encryption itself. + +Note that, the noise protocol requires the public key length and the ECDH result length (shared secret) to be equal. Because for `secp256k1`, public keys have a length of 33, and the ECDH result has a length of 32, so an empty byte (`0x0`) should be appended to all generated ECDH results. Hence, the `DHLEN` constant for the noise protocol should be 33. + +After the handshake, the CipherState object will be used by the *Client Instances* to encrypt and decrypt the `CipherText` contained within the `Send` frame. + +**Handshake pattern:** + +Only the `KK` [interactive handshake pattern (fundamental)](http://noiseprotocol.org/noise.html#interactive-handshake-patterns-fundamental) will be supported. + +``` +-> s +<- s +... +-> e, es, ss +<- e, ee, se +``` + +The `-> e, es, ss` message is the `NoiseMessage1` of a `OpenChannel` frame, while the `<- e, ee, se` message is the `NoiseMessage2` of a `ChannelOpened` frame. + +### Implementation in Code + +Within the `dmsg` module: + +- `Link` structure should represent a link between two instances. +- `Pool` structure should handle multiple `Links` (with different instances). +- `Client` which implements a *Client Instance*. +- `Server` which implements a *Server Instance*. + +*Client Instances* communicate with each other via a *Server Instance* (which acts as a relay). + +Both structs will use `link.Pool` to handle links, but *Frames* are handled differently. *Client Instances* are to implement `TransportFactory` while a *Server Instance* is not required to. A *Client Instance* should also represent an established *Channel* as a `Transport` implementation. + +### Configuring an Instance + +When creating an *Instance*, the following options should be available via the following structure. + +```golang +// Config configures an instance. +type Config struct { + // Public determines whether the instance is to advertise itself to the dmsg discovery servers. + Public bool + + // DiscoveryAddresses contains the dmsg discovery services to be used (in order of preference). + DiscoveryAddresses []string +} +``` + +The above structure is to be an input when creating a *Server Instance* or a *Client Instance*. + +### Instance Interaction with Dmsg Discovery + +On startup `Server` that is supposed to be publicly available should register itself in dmsg discovery. To do so it first has to fetch current version of an `Entry`, if entry doesn't exist it should create one. If entry exists it may update it if necessary. + +On startup `Client` may connect to necessary amount of servers by fetching list of available servers from the discovery. Once initial connections are established `Client` should update discovery entry to advertise it's relays. + +### Channel Management + +The following is a proposal of how a Channel can be represented in code. + +```golang +// Channel represents a channel that implements Transport. It can be from the perspective of a Server or Client Instance. +type Channel struct { + // ChannelID represents the ID that is associated of the adjacent link. + ChannelID uint8 + + // Destination is the public key of the instance that is the final destination. + // This should always contain the public key of a client instance (as a server cannot be the final destination). + Destination cipher.PubKey + + // Link contains the adjacent link of the channel. + Link *link.Link +} +``` + +Both the client and server instances needs to manage channels. Channels are associated with a channel ID and also the public key(s) of the remote instances that the channel interacts with. Channels are hence identified by *Link* + *Channel ID*. + +From the perspective of a *Client Instance*, the assignment of *Channel IDs* are unique to a given link with a *Server Instance*. For example, let's say client 'A' is connected with server 'B' and server 'C', hence we have links 'AB' and 'AC'. We can have 'AB' and 'AC' share the same Channel ID, but because the channel itself is associated with a different link, they are considered different channels. + +From the perspective of a *Server Instance*, the assignment of *Channel IDs* are unique to a given link with a *Client Instance*. + +### Opening a Channel + +A channel in it's entirety handles the communication between two client instances via a server instance (which acts as a relay). Within the link between a single client instance and the server instance, a channel is represented using a *Channel ID*. The *Channel ID* of the two *links* of the same "channel" can be different, and the *Server Instance* is responsible for recording this association of the *Channel IDs* (coupled with the client instance's public key). + +When a *Client Instance* wishes to communicate with another *Client Instance*, it is responsible for initiating the creation of a channel. To do so, t sends a `OpenChannel` frame to the *Server Instance* in which: + +- `ChannelID` contains a ChannelID that the client wishes to associate with the channel. +- `RemoteStatic` contains the public key of the remote *Client Instance* that the local client wishes to communicate via this channel. +- `NoiseMessage1` is the first noise handshake message (the handshake pattern used is KK). + +If the *Server Instance* wishes to reject the request to open channel, it can send a `ChannelClosed` frame back to the initiating client with the `ChannelID` sub-field containing the value of the channel ID suggested by the initiating client. + +If the *Server Instance* wishes to go forward with opening of a channel, it sends a `OpenChannel` frame to the second *Client Instance*, in which `ChannelID` is an ID that's unique between the server and the second client and public key of the first client. + +If the second *Client Instance* wishes to reject the request, it can send a `ChannelClosed` frame back to the server, and the server can subsequently send a `ChannelClosed` frame to the initiating client (the `ChannelID` sub-fields of these `ChannelClosed` frames should be the unique channel IDs of the associated links). + +If the second *Client Instance* accepts the request, it sends a `ChannelOpened` back to the *Server Instance* (with the `NoiseMessage2`). Subsequently, the *Server Instance* sends a `ChannelOpened` back to the initiating client (the `ChannelID` sub-fields of these `ChannelOpened` frames should be the unique channel IDs of the associated links). + +### Closing a Channel + +A *Client Instance* can safely close any of it's channels by sending a `CloseChannel` (with the associated `ChannelID`) to the *Server Instance*. + +After a *Client Instance* sends a `CloseChannel`, no more frames are to be sent by that instance. However, the remote instance can still send frames until it receives the `CloseChannel` to it. The "close-responding" client then sends a `ChannelClosed` instance back to the "close-initiating" client. Once the `ChannelClosed` channel is sent by the "close-responding" client, it will no longer send or receive frames. Once the "close-initiator" receives the `ChannelClosed` frame. it will no longer receive frames. + +In summary, + +- When a client instance sends a `CloseChannel` frame, the channel is "partially-closed" and the client instance will only receive and not send via the channel. If a `ChannelClosed` frame is not received after a given timeout, the channel sends a `ChannelClosed` itself and the channel is "fully-closed". +- When a client instance receives a `CloseChannel` frame, it delivers a `ChannelClosed` frame and the channel is "fully-closed" and the client will no longer receive or send via the channel. +- When a client instance receives a `ChannelClosed` frame, the channel is "fully-closed". + +### Handling Disconnections + +In any given situation, there may be a possibility that the *Server Instance* unexpectedly disconnects with a *Client Instance*, or that a *Client Instance* unexpectedly disconnects with a *Server Instance*. This should directly affect the channels associated with the *Link* in question. + +When a *Client Instance* detects that a *Server Instance* has disconnected from it. All associated channels with that *Server Instance* should be closed. When a channel closes, the associated *Transport* should also be closed. + +When a *Server Instance* detects a disconnection from a *Client Instance*, it should send a `ChannelClosed` frame to all the other *Client Instances* that shares a channel with the disconnected client. After so, the *Server Instance* should dissociate all relations with the closed channels. + +# Packets + +The *Node Module* handles data encapsulated within data units called *Packets*. *Packets* can be grouped within the following categories based on their use-case; + +- ***Settlement Packets*** are used by the *Transport Manager* to "settle" Transports. Settlement, allows the two nodes that are the edges of the transport to decide on the *Transport ID* to be used, and whether the Transport is to be public. Only after a *Transport* is settled, can the *Router* have access to the Transport. + + *Settlement Packets* contain `json` encoded payload. + +- ***Foundational Packets*** are used by a *Router* to communicate with a remote *Setup Node* and is used for setting up, establishing and destroying routes. + + *Foundational Packets* are prefixed by 3 bytes: the packet size (2 bytes) and a Type (1 byte) that contains the foundational packet type. + +- ***Data Packets*** are Packets that are actually used to encapsulate data delivered between two Apps. + + *Data Packets* are prefixed by 6 bytes; including the packet size (2 bytes) and the Route ID (4 bytes) which can have any value other than `0x00` or `0x01`. + +- ***Loopback Packets*** are packets that are consumed locally by the node. + + *Loopback Packets* are structurally similar to data packets but their Route ID links to a rule that specifies which app to forward the packet to. + +## Settlement Packets + +After a Transport is established between two nodes, the nodes needs to decide on the *Transport ID* that describes the Transport and whether the Transport is to be public or private (public Transports are to be registered in the *Transport Discovery*). This process is called the *Settlement Handshake*. + +The Packets of this handshake contain `json` encoded messages. + +*Settlement Handshake* packets do not need a field for Packet-type are they are expected in a specific order. + +- Request to settle transport is sent by the *Transport Initiator* to the *Transport Responder* after a *Transport* connection is established. + + JSON Body: Contains a `transport.SignedEntry` structure with the *Transport Initiator*'s signature. + +- *Transport Responder* should validate submitted `transport.SignedEntry`, and if entry is valid it should add sign it and perform transport registration in transport discovery. If registration was successful responder should send updated `transport.SignedEntry` back to initiator. + + JSON Body: Contains a `transport.SignedEntry` structure with signatures from both the *Transport Initiator* and the *Transport Responder*. If the transport is registered in *Transport Discovery*, the `SignedTransport.Registered` should contain the epoch time of registration. + +If transport will fail at any step participants can chose to stop handshake procedures and close corresponding transport. Transport disconnect during the handshake should be handled appropriately by participants. Optional handshake timeout should also be supported. + +## Foundational Packets + +Foundational packets are used for the communication between *App Nodes* and *Setup Nodes*. + +The *Setup Node* is responsible for fulfilling Route initiating and destroying requests by communicating with the initiating, responding and intermediate nodes of the proposed route. + +The following is the expected format of a Foundational Packet; + +``` +| Packet Len | Type | JSON Body | +| 2 bytes | 1 byte | ~ | +``` + +- ***Packet Len*** specifies the total packet length in bytes (exclusive of the *Packet Len* field). +- ***Type*** specifies the *Foundational Packet Type*. +- ***JSON Body*** is the packet body (in JSON format) that is unique depending on the packet type. + +**Foundational Packet Types Summary:** + +| Type | Name | +| ---- | ---- | +| 0x00 | `AddRules` | +| 0x01 | `RemoveRules` | +| 0x02 | `CreateLoop` | +| 0x03 | `ConfirmLoop` | +| 0x04 | `CloseLoop` | +| 0x05 | `LoopClosed` | +| 0xfe | `ResponseFailure` | +| 0xff | `ResponseSuccess` | + +### `0x00 AddRules` + +Sent by the *Setup Node* to all *Nodes* of the route. This packet informs nodes what rules are to be added to their internal routing table. + +**JSON Body:** + +```json +[, ] +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with + ```json + [, ] + ``` + +### `0x01 RemoveRules` + +Sent by the *Setup Node* to *Node* of the route. + +**JSON Body:** + +```json +["", "rid-2"] +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with + ```json + [, ] + ``` + +### `0x02 CreateLoop` + +Sent by the *Route Initiator* to a *Setup Node* to have a *Loop* created. + +**JSON Body:** + +```json +{ + "local-port": , + "remote-port": , + "forward": [ + { + "from": "", + "to": "", + "tid": "" + }, + { + "from": "", + "to": "", + "tid": "" + } + ], + "reverse": [ + { + "from": "", + "to": "", + "tid": "" + }, + { + "from": "", + "to": "", + "tid": "" + } + ], + "expiry": "" +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty payload. + +### `0x3 ConfirmLoop` + +Sent by the *Setup Node* to Responder and Initiator *Node* to confirm notify about route in opposite direction. + +**JSON Body:** + +```json +{ + "remote-pk": "", + "remote-port": , + "local-port": , + "resp-rid": +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty payload. + +### `0x4 CloseLoop` + +Sent by a Responder or Initiator *Node* to a *Setup Node* to notify about closing a loop locally. + +**JSON Body:** + +```json +{ + "port": "", + "remote": { + "port": , + "pk": + } +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty payload. + +### `0x5 LoopClosed` + +Sent by a *Setup Node* to a Responder or Initiator to notify about closed loop on the opposite end. + +**JSON Body:** + +```json +{ + "port": "", + "remote": { + "port": , + "pk": + } +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty payload. + + +## Data Packets + +The follow is the structure of a *Data Packet*. + +``` +| Packet Len | Route ID | Payload | +| 2 bytes | 4 bytes | ~ | +``` + +# Transport Management + +For all Skywire Node types, we need a universal way for managing and logging Transports. The structure that is responsible for this is the `TransportManager` (which should be within the `/pkg/node` package of the `skywire` package). + +As the `TransportManager` needs to interact with the *Transport Discovery* and other Skywire Nodes, it should have access to the local node's public and private key identity. + +The following is a proposed implementation of `TransportManager`; + +```golang +package node + +// Transport wraps a 'transport.Transport' implementation and contains +// associated/useful for the 'transport.Transport' implementation. +type Transport struct { + transport.Transport + ID uuid.UUID + // more fields ... +} + +// TransportManagerConfig configures a TransportManager. +type TransportManagerConfig struct { + PubKey cipher.PubKey // Local PubKey + SecKey cipher.SecKey // Local SecKey + DiscoveryClient client.Client // Transport discovery client + LogStore TransportLogStore // Store for transport's transfer rates +} + +// TransportManager manages Transports. +type TransportManager struct { + // Members... +} + +// NewTransportManager creates a TransportManager with the provided configuration and transport factories. +// 'factories' should be ordered by preference. +func NewTransportManager(config *TransportManagerConfig, factories ...transport.Factory) (*TransportManager, error) { /* ... */ } + +// Start starts the transport manager. +// - 'ctx' can end the transport listening operation. +func (tm *TransportManager) Serve(ctx context.Context) error { /* ... */ } + +// Observe returns channel for notifications about new Transport +// registration. Only single observer can listen for on a channel. +func (tm *TransportManager) Observe() <-chan *Transport { /* ... */ } + +// Factories returns all the factory types contained within the TransportManager. +func (tm *TransportManager) Factories() []string { /* ... */ } + +// Transport obtains a Transport via a given Transport ID. +func (tm *TransportManager) Transport(id uuid.UUID) (*Transport, bool) { /* ... */ } + +// RangeTransports ranges all Transports. +// Should return when 'action' returns a non-nil error. +func (tm *TransportManager) RangeAllTransports(action TransportAction) error { /* ... */ } + +// CreateTransport begins to attempt to establish transports to the given 'remote' node. +// This should be a non-blocking operation and any failures or future Transport disconnections +// should be dealt with with retries (under a given time interval). +// - 'remote' specifies the remote node to attempt to establish the Transports with. +// - 'tpType' is the transport type that is to be created. +// - 'public' determines whether the Transports established should be advertised to Transport Discovery. +// If a transport is not to be public, a random transport ID is assigned. +func (tm *TransportManager) CreateTransport(ctx context.Context, remote cipher.PubKey, tpType string, public bool) (*Transport, error) { /* ... */ } + +// DeleteTransport disconnects and removes the Transport of Transport ID. +func (tm *TransportManager) DeleteTransport(id uuid.UUID) error { /* ... */ } +``` + +## Transport Manager Procedures + +The transport manager is responsible for keeping track of established transports (via the `transport.Entry` and the `transport.Status` structures). The `transport.Entry` structure describes and identifies transports, while `transport.Status` keeps track of whether the transport is up or down (based on the perspective of the local node). + +If the *Transport Manager* wishes to confirm transport information, it can query the *Transport Discovery* via the `GET /transports/edge:` endpoint. Note that it is expected of the *Transport Manager* to call this endpoint on startup. + +When a transport is "closed" it is only considered "down", not "destroyed". + +The following highlights detailed startup and shutdown procedures of a *Transport Manager*; + +**Startup:** + +On startup, the `TransportManager` should call the *Transport Discovery* to ensure that it is up to date. Then it needs to attempt to establish (or re-establish) transports to the relevant remote nodes. + +When re-establishing a Transport, the `transport.Entry` used should be that also previously stored in the *Transport Discovery*. + +Once connected, the `TransportManager` should update it's *Status* of the given Transport and set `is_up` to `true`. + +The startup logic is triggered when `Start` is called. + +**Shutdown:** + +On shutdown, the first step is to update the *Transport Statuses* to "down" via the *Transport Discovery*. Then Transports to remote nodes is to be closed (with a timeout, in which after, the transport in question is forcefully closed). + +## Logging + +A *Transport Manager* is responsible for logging incoming and outgoing communication for each transport. Initially, only the total incoming and outgoing bandwidth (in bytes) is to be logged per transport. + +```golang +// TransportLogEntry represents a logging entry for a given Transport. +// The entry is updated every time a packet is received or sent. +type TransportLogEntry struct { + ReceivedBytes big.Int // Total received bytes. + SentBytes big.Int // Total sent bytes. +} +``` + +Logs for each transport is to be stored using `TransportLogStore`. `TransportLogStore` is to be specified within `TransportManagerConfig`. + +```golang +// TransportLogStore stores transport log entries. +type TransportLogStore interface { + Entry(id uuid.UUID) (*TransportLogEntry, error) + Record(id uuid.UUID, entry *TransportLogEntry) error +} +``` + +# Route Finder + +The *Route Finder* (or *Route Finding Service*) is responsible for finding and suggesting routes between two Skywire Nodes (identified by public keys). It is expected that an *App Node* is to use this service to find possible routes before contacting the *Setup Node*. + +In the initial version of the *Route Finder*, it should use a basic algorithm to choose and order the best routes. This algorithm should find the x amount (limited by the max routes parameter) of "fastest" routes determined by the least amount of hops needed, and order it by hops ascending. + +The implementation of *Route Finder* requires only a single REST API endpoint. + +## Graph Algorithm + +In order to explore routing we need to create a graph that represents the current skywire network, or at least the network formed by all the reachable nodes by `source node`. + +For this purpose, we use the `mark and sweep` algorithm. Such an algorithm consists of two phases. + +In the first phase, every object in the graph is explored in a `Deep First Search` order. This means that we need to explore every transport starting from route node, accessing the `transport-discovery` database each time that we need to retrieve new information. + +In the second phase, we remove nodes from the graph that have not been visited, and then mark every node as unvisited in preparation for the next iteration of the algorithm. + +An explanation and implementation of this algorithm can be found [here](https://www.geeksforgeeks.org/mark-and-sweep-garbage-collection-algorithm/). + +## Routing algorithm + +Given the previous graph we can now explore it to find the best routes from every given starting node to destiny node. + +For this purpose we use a modification of `Dijkstra algorithm`. + +An implementation can be found [here](http://rosettacode.org/wiki/Dijkstra%27s_algorithm#Go). + +Route-finder modifies this algorithm by keeping track of all the nodes that reached to destination node. This allows the ability to backtrack every best route that arrives from a different node to destination node. + +## Code Structure + +The code should be in the `watercompany/skywire` repository; + +- `/cmd/route-finder/route-finder.go` is the main executable for the *Route Finder*. +- `/pkg/route-finder/api/` contains the RESTFUL API definitions. +- `/pkg/route-finder/store/` contains the definition of the `Storer` interface and it's implementations [**TODO**]. +- `/pkg/route-finder/client/` contains the client library that interacts with the *Route Finder* service's RESTFUL API. + +## Database + +The *Route Finder* only accesses the Transport database already defined in the *Transport Discovery* specification. + +## Endpoint Definitions + +All endpoint calls should include an `Accept: application/json` field in the request header, and the response header should include an `Content-Type: application/json` field. + +### GET Routes available for the defined start and end key + +Obtains the routes available for a specific start and end public key. Optionally with custom min and max hop parameters. + +Note that each transport is to be represented by the `transport.Entry` structure. + +**Request:** + +``` +GET /routes +``` + +```json +{ + "src_pk": "", + "dst_pk": "", + "min_hops": 0, + "max_hops": 0, +} +``` + +**Responses:** + +- 200 OK (Success). + ```json + { + "routes": [ + { + "transports": [ + { + "tid": "", + "edges": ["", ""], + "type": "", + "public": true, + } + ] + } + ] + } + ``` +- 400 Bad Request (Malformed request). +- 500 Internal Server Error (Server error). + +# Route Setup Process + +1. Route paths are uni-directional. So, the whole route between 2 visors consists of forward and reverse paths. *Setup node* receives both of these paths in the routes setup request. +2. For each node along both paths *Setup node* calculates how many rules are to be applied. +3. *Setup node* connects to all the node along both paths and sends `ReserveIDs` request to reserve available rule IDs needed to setup the route. +4. *Setup node* creates rules the following way. Let's consider visor A setting up route to visor B. This way we have forward path `A->B` and reverse path `B->A`. For forward path we create `Forward` rule for visor `A`, `IntermediaryForward` rules for each node between `A` and `B`, and `Consume` rule for `B`. For reverse path we create `Forward` rule for visor `B`, `IntermediaryForward` rules for each visor between `B` and `A`, and `Consume` rule for `A`. +5. *Setup node* sends all the created `IntermediaryForward` rules to corresponding visors to be applied. +6. *Setup node* sends `Consume` and `Forward` rules to visor `B` (remote in our case). +7. *Setup node* sends `Forward` and `Consume` rules to visor `A` in response to the route setup request. + +## Loop Setup Process: + +In the code `Loop` is represented by the following structure: + +```golang +// Hop defines a route hop between 2 nodes. +type Hop struct { + From cipher.PubKey // Sender's pk + To cipher.PubKey // Receiver's pk + Transport uuid.UUID // Transport ID between sender and receiver +} + +// Route defines a route as a set of Hops. +type Route []*Hop + +// Loop defines a loop over a pair of routes. +type Loop struct { + LocalPort uint16 // Initiator's port + RemotePort uint16 // Responder's port + Forward Route // Initial route + Reverse Route // Route in opposite direction + ExpireAt time.Time // Expiration time +} + +``` + +Setup procedures: + +1. *Initiating Node* sends `CreateLoop` command to a *Setup Node*. +2. *Setup Node* contacts all nodes along a `Reverse` route using a separate dmsg channel and setups routing rules using `AddRules` command. +3. *Setup Node* performs the same operation for a `Forward` route. +4. *Setup Node* sends `ConfirmLoop` to initiator with reverse id for a `Forward` route. +5. *Setup Node* sends `ConfirmLoop` to responder with reverse id for a `Reverse` route. + +If at any point *Setup Node* will be unable to proceed it will issue requests to remove rules for the loop. + +## Loop Close Process: + +Loop can be closed by edge nodes at any time by sending `CloseLoop` command to a *Setup Node*. + +Close procedures: + +1. *Node* sends `CloseLoop` command to a *Setup Node*. +2. *Setup Node* sends `LoopClosed` to an opposite *Node*. +3. Forward routes are not removed since it's not possible to reconstruct the route. They will be removed by expiration timeout. + +# Routing Table + +A *Routing Table* (located within the `skywire/pkg/node` module) is unique for a given Node's public key. It is basically a key-value store in which the key is the *Route ID* and the value is the *Routing Rule* for the given *Route ID*. + +Initially, there will be two types of Routing Rules: *App* and *Forward*. + +- *App* rules are identified by their unique `` value of `0x00`. A packet which contains a *Route ID* that associates with a *App* rule is to be sent to a local App. +- *Forward* rules are identified by their unique `` value of `0x01`. A packet which contains a *Route ID* that associates with a *Forward* rule is to be forwarded. + +| Action | Key (Route ID) | Value (Routing Rule) | +| ------ | -------------- | -------------------- | +| *App* | ``
*4 bytes* | ``
*48 bytes* | +| *Forward* | ``
*4 bytes* | ``
*29 bytes* | + +- `` is the *Route ID* `uint32` key (represented by 4 bytes) that is used to obtain the routing rules for the Packet. +- `` contains the epoch time (8 bytes) of when the rule is to be discarded (or becomes invalid). +- `` specifies the type of Routing Rule (1 byte). Currently there are two possible routing rule types; *App* (`0x00`) and *Forward* (`0x01`). +- `` is the *Route ID* (4 bytes) that is the Route ID key for the reserve Route of the loop. +- `` identifies and classifies the loop. It contains the following sub-fields; ``. + - `` is the remote edge public key in which this route/loop is associated with. It is represented by 33 bytes. + - `` is the remote port in which this route/loop is associated with. It is represented by 2 bytes. + - `` is the local port in which this route/loop is associated with. It is represented by 2 bytes. +- `` is the *Route ID* that is to replace the `` before the Packet is to be forwarded. +- `` represents the transport which the packet is to be forwarded to. A Transport ID is 16 bytes long. + +Every time a Skywire Node receives a packet, it performs the following steps: + +1. Obtain the `` from the Packet, and uses this value to obtain a routing rule entry from the routing table. If no routing rule is found, or the routing rule has already expired (via checking the `` field), the Packet is then discarded. +2. Obtains the `` value to determine how the packet is to be dealt with. If the `` value is `0x00`, the packet is then to be sent to the local *App Server* with the Routing Rule. If the `` value is `0x01`, the packet is to be forwarded; continue on to step 3. +3. Obtain the `` from the *Routing Rule* and replace the `` from the *Route ID* field of the Packet. +4. Forward the Packet to the referenced transport specified within ``. + +The routing table is to be an interface. + +```golang +package node + +// RangeFunc is used by RangeRules to iterate over rules. +type RangeFunc func(routeID transport.RouteID, rule RoutingRule) (next bool) + +// RoutingTable represents a routing table implementation. +type RoutingTable interface { + // AddRule adds a new RoutingRules to the table and returns assigned RouteID. + AddRule(rule RoutingRule) (routeID transport.RouteID, err error) + + // SetRule sets RoutingRule for a given RouteID. + SetRule(routeID transport.RouteID, rule RoutingRule) error + + // Rule returns RoutingRule with a given RouteID. + Rule(routeID transport.RouteID) (RoutingRule, error) + + // DeleteRules removes RoutingRules with a given a RouteIDs. + DeleteRules(routeIDs ...transport.RouteID) error + + // RangeRules iterates over all rules and yields values to the rangeFunc until `next` is false. + RangeRules(rangeFunc RangeFunc) error + + // Count returns the number of RoutingRule entries stored. + Count() int +} +``` + +Potential improvement we could consider is to move ports from the rules into the data packet header, aligning this with `tcp`. By doing so we will be able to re-use intermediate forward rules across multiple loops which can drastically improve loop establishment time for complex loops. + +# Router + +The `Router` (located within the `skywire/pkg/node` module) uses the *Transport Manager* and the *Routing Table* internally, and is responsible for the handling incoming Packets (either from external nodes via transports, or internally via the `AppServer`), and also the process for setting up routes. + +Regarding the Route setup process, a router should be able to interact with multiple trusted *Setup Nodes*. + +Every transport created or accepted by the *Transport Manager* should be handled by the *Router*. All incoming packets should be cross-referenced against *Routing Table* and either be forwarded to next *Node* or to a local *App*. + +*Transport Manager* is also responsible for managing connections on local ports for node's apps. *App Node* will request new local connection from the *Router* on *App* startup. All incoming packets from the app's connections should be forwarded based on *App* rules defined in a local routing table. *Transport Manager* should also be capable of requesting new loops from a *Setup Node*. + +## Port management + +Router is responsible for port management. Port allocation algorithm should work similarly to how `tcp` manages ports: + +- Certain range of ports should be reserved and be inaccessible for general purpose apps. Default staring port is `10`. +- All allocated local ports should be unique. +- App should be able to allocate static ports that it will be accessible on for remote connections. Static port allocation is performed during `app` init. +- App should be able to dynamically allocate local port for newly created loops. +- Allocated ports should be closed on app shutdown or disconnect from the node. + +# App Server + +The `AppServer` (located within the `skywire/pkg/node` module) handles communication between local and remote Apps. It also manages and administers local Apps. It is to interact with a `Router` and identify loops via the *App* routing rule retrieved from the routing table. The *App* rule is structured as follows; + +``` +| expiry | r-type | resp-rid | remote-pk | remote-port | local-port | +| 8 bytes | 1 byte | 4 bytes | 33 bytes | 2 bytes | 2 bytes | +``` + +The *App Server* not only forwards Packets between Apps and remote entities, but it can also be instructed by the end user to send *Quit* signals to the Apps. *Apps* can also request to open *Loops* with remote/local Apps. + +Each *Loop* is identified uniquely via the local and remote public keys, paired with the local and remote ports. + +Within the `AppServerConfig` file, local ports are reserved for certain Apps. The following rules are to be opposed: +- Ports are either "reserved" or "unreserved". + - No two Apps are allowed to "reserve" the same port. + - Ports are reserved via the `AppServerConfig` file. +- Reserved ports are either "active" or "inactive". + - A port is "active" when the port is "reserved" for an App, and that App is running. + - A port is "inactive" either when the port is "unclaimed", or when the port is "claimed" but the App is not running. + +The following communication processes between a given App and the *App Server* is to exist: + +- **App requests to open loop** +- **App Server asks App whether it wishes to respond to a remotely-initiated loop** +- **App Server informs App that loop is to be closed (with given reason)** + - Reasons include: Route timeout, remotely closed, locally closed, etc. +- **App informs App Server it wishes to close a loop (with given reason)** + - Reasons include: App is shutting down, loop no-longer used, etc. +- **App Server forwards packets received from remote App** + - If local app does not exist or does not accept, the loop is to be closed, and routes destroyed. +- **App Server forwards packets received from local App** + - If the rule does not exist, or the remote app nodes not accept, the loop is to be closed and routes destroyed. + +## Loop Encryption + +Each loop is to be symmetrically encrypted with [Noise](http://noiseprotocol.org/noise.html). Specifically, we are to use the `KK` fundamental [handshake pattern](http://noiseprotocol.org/noise.html#handshake-patterns). The first noise message should be provided by initiator in the request to create a new loop, this messages will be setup to responder in a loop confirmation request. Responder should send second noise message which will be returned to initiator in a loop confirmation request. + +# Setup Node + +The *Setup Node* (located within the `skywire/pkg/node` module) uses *Transport Manager* internally and is responsible for propagation of routing rules to nodes along the *Route*. *Setup Node* should be only addressed by a public key and should work over dmsg transport using multiple channels. Each channel can be used to issue route setup commands by initiator. For *Loop* setup requests *Node* will be an initiator, for *Rule* setup related operations *Setup Node* will be a channel initiator. *Setup Node* is only responsible for handling *Foundational Packets* and doesn't perform any forwarding functions. + +# App Node + +An App Node is a node that is part of the Skywire network and is represented by a key pair (using the `secp256k1` curve). It handles Transports to remote nodes, sets up routes and loops (via Routing Rules and interaction with the *Setup Node*), and manages Apps. + +Each App is it's own executable that communicates with an *App Node* using a pair of *POSIX* pipes. Piped connection is setup on *App* startup and inherited by a forked *App* process using file descriptor `3` and `4`. Setup process for a forked *App* is handled by the `app` package. + +``` + [Skywire Node] + / | \ + / | \ +[App 1] [App 2] [App 3] +``` + +## Communication reliability + +Since dmsg and loop communication is dependent on intermediate servers we have to provide acknowledgment mechanism between edge nodes. This will be done via wrapper `io.ReadWriter` (`AckReadWriter`) that will be able to augment existing communication channels with `ack` packets. `Write` calls on `AckReadWrite` should block until corresponding `ack` packet is received. + +`ack` logic should be working in `tcp`-like way: all pending `ack` packets should be sent with subsequent write on the opposite edge. If no write is happened within a certain interval then all pending `ack` packets should be flushed. Outstanding `ack` packets should also be flushed on `Close` call. + +`AckReadWriter` should be able to send and receive 2 types of packets: `payload`(`0x0`) and `ack` (`0x1`): + +Format of a `payload` packet: + +``` +| Packet Type | Packet ID | Payload | +| 0x0 | 1 byte | ~ | +``` + +Format of an `ack` packet: + +``` +| Packet Type | Packet ID | SHA256 | +| 0x1 | 1 byte | 32 bytes | +``` + +`AckReadWriter` should be able to prepend any amount of `ack` packets to a `payload` packet. Sequences without `payload` packet should also be valid. Example packet sequence: + +``` +| 0x1 | 0x0 | hash | 0x1 | 0x1 | hash | 0x0 | 0x2 | payload | +``` + +This packet sequence will acknowledge received packets with ids `0` and `1` and will send packet with id `2`. + +Upon reading `ack` packets receiver should validate received hash for each packet. + +## App Programming Interface + +*App* programming interface (located within the `skywire/pkg/app` module) should expose methods for *Apps* to connect to a piped connection, perform handshake and exchange data with remote nodes. + +*App* interface should expose following methods: + +```golang +// Addr implements net.Addr for App connections. +type Addr struct { + PubKey transport.PubKey + Port uint16 +} + +// LoopAddr stores addressing parameters of a loop package. +type LoopAddr struct { + Port uint16 + Remote Addr +} + +// Packet represents message exchanged between App and Node. +type Packet struct { + Addr *LoopAddr + Payload []byte +} + +// Config defines configuration parameters for an App +type Config struct { + AppName string + AppVersion string + ProtocolVersion string +} + +// Setup sets up an app using default pair of pipes and performs handshake. +func Setup(config *Config) (*App, error) {} + +// Accept awaits for incoming loop confirmation request from a Node and +// returns net.Conn for a received loop. +func (app *App) Accept() (net.Conn, error) {} + +// Dial sends create loop request to a Node and returns net.Conn for created loop. +func (app *App) Dial(raddr *Addr) (net.Conn, error) {} + +// Addr returns empty Addr, implements net.Listener. +func (app *App) Addr() net.Addr {} + +// Close implements io.Closer for an App. +func (app *App) Close() error {} +``` + +## App to Node Communication protocol + +Communication between *Node* and an *App* happens over the piped connection using binary multiplexed protocol. + +The following is the expected format of a App Packet: + +``` +| Packet Len | Type | Message ID | JSON Body | +| 2 bytes | 1 byte | 1 byte | ~ | +``` + +- ***Packet Len*** specifies the total packet length in bytes (exclusive of the *Packet Len* field). +- ***Type*** specifies the *App Packet Type*. +- ***Message ID*** specifies multiplexing ID of a message, response for this message should contain the same ID. +- ***JSON Body*** is the packet body (in JSON format) that is unique depending on the packet type. + +**App Packet Types Summary:** + +| Type | Name | +| ---- | ---- | +| 0x00 | `Init` | +| 0x01 | `CreateLoop` | +| 0x02 | `ConfirmLoop` | +| 0x03 | `Send` | +| 0x04 | `Close` | +| 0xfe | `ResponseFailure` | +| 0xff | `ResponseSuccess` | + +### `0x00 Init` + +Sent by an *App* to a *Node*. This packet is used to handshake connection between an *App* and a *Node*. *Node* will typically check if app is allowed by the config file and which port should be statically allocated it. + +**JSON Body:** + +```json +{ + "app-name": "foo", + "app-version": "0.0.1", + "protocol-version": "0.0.1" +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` without body. + +### `0x01 CreateLoop` + +Sent by an *App* to a *Node*. This packet is used to open new *Loop* to a remote *Node*. + +**JSON Body:** + +```json +{ + "pk": "", + "port": +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with + ```json + { + "pk": "", + "port": + } + ``` + +### `0x02 ConfirmLoop` + +Sent by a *Node* to an *App* to notify about request to open new *Loop* from a remote *Node* + +**JSON Body:** + +```json +[ + { + "pk": "", + "port": + }, + { + "pk": "", + "port": + } +] +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty body. + +### `0x03 Send` + +Sent by a *Node* and an *App*. This message is used to exchange messages through a previously established *Loop*. + +**JSON Body:** + +```json +{ + "addr": { + "port": , + "remote": { + "pk": "", + "port": + } + }, + "payload": "" +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty body. + +### `0x04 Close` + +Sent by a *Node* and an *App*. *App* uses this message to notify about closed *Loop*. *Node* sends this message after remote node is requested to close established *Loop*. + +**JSON Body:** + +```json +{ + "port": , + "remote": { + "pk": "", + "port": + } +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty body. + +## App Node Configuration + +The following is the JSON representation of a Skywire configuration. + +```json +{ + "version": "1.0", + "node": { + "static_public_key": "024ec47420176680816e0406250e7156465e4531f5b26057c9f6297bb0303558c7", + "static_secret_key": "42bca4df2f3189b28872d40e6c61aacd5e85b8e91f8fea65780af27c142419e5" + }, + "dmsg": { + "discovery_addresses": ["http://localhost:9090"], + "server_count": 1 + }, + "apps": [ + { + "app": "helloworld", + "version": "1.0", + "auto_start": true, + "port": 10, + "args": [] + } + ], + "transport_discovery": "http://localhost:9091", + "setup_nodes": ["02603d53d49b6575a0b8cee05b70dd23c86e42cd6cba99af769d61a6196ea2bcb1"], + "trusted_nodes": ["0348c941c5015a05c455ff238af2e57fb8f914c399aab604e9abb5b32b91a4c1fe"], + "dmsg_path": "./dmsg", + "apps_path": "./apps", + "local_path": "./local", + "log_level": "info", + "interfaces": { + "rpc": ":3436" + } +} +``` + +- `"version"` represents the version of the Skywire Node (and also the configuration format version). + +- `"node"` includes the public/private keys that identify the node. + +- `"dmsg"` configures the dmsg client instance included within the Skywire Node. + - When `"public"` is set, the Dmsg Client Instance will advertise itself to discovery. + - `"discovery_addresses"` specifies the Dmsg Discovery Services that the Skywire Node is to try. + - `"server_count"` specifies the number of servers to ensure connection with on first startup. + +- `"apps"` lists all available Skywire Apps. These configurations include; the App's name, whether the specified App should auto-start, and the ports that is reserved for the App. If these are not defined for an App, the App will not auto-start, nor have ports reserved for the App. + - If `"version"` is not specified, the highest stable version will be selected. + +- `"node_path"` stores logs, routing tables, and any data that the node may use. + +- `"dmsg_path"` holds the path which the Dmsg Client Instance can use to store cache or additional configurations. + +- `"apps_path"` holds all the app executables. App executable files should be named with no spaces or weird characters (TODO: define properly). They should also be appended with the semantic version of the App after a dot; `{app_name}.v{semantic_version}`. + +- `"local_path"` contains the working directories of the apps. An app named `chat` of version `v1.0` should have a working directory within `{root_dir}/{local_path}/chat/v1.0/`. The contents of the App's working directory is specified by the App. + +## App Node RPC Interface + +The App node should attempt to connect to the assigned *Manager Node* on startup. The connection is to be encrypted via Noise (KK handshake pattern) so that the nodes can identify one another. + +For the *App Node* to connect to the *Manager Node*, it needs the public key and tcp address of the *Manager Node* in it's configuration. + +After connection has been established, the *App Node* becomes the RPC Server and the *Manager Node* becomes the RPC client that can execute commands on the *App Node* (the RPC Server). + +Additionally, the App Node should listen on a port so that a local command-line executable (`skywire-cli`) can interact with it. This local port should hence, only accept connections from localhost. + +### Commands + +The following sub-commands should be supported. Note that command-line actions are listed below, but they should be served via RESTFUL interfaces. + +**General:** + +- **`skywire cli visor info`** obtains a summary of the current state of the ~~App Node~~ running skywire visor. + +**App Management:** + +- **`skywire cli visor app ls`** lists applications and applications stats (running/not running) (auto-start/non-auto-start) (local/remote ports). There should be flags for filtering (to be defined later). + +- **`skywire cli visor app start `** starts a Skywire app. + +- **`skywire cli visor app stop `** stops a Skywire app if running. + +- **`add-autostart-app [--start-now]`** adds a Skywire app to auto-start. After calling this command, the actual app does not actually start unless `--start-now` is set - **NOT IMPLEMENTED** - set autostart behavior with `skywire cli config gen` or `skywire cli config update` or directly in the config + +- **`rm-autostart-app [--stop-now]`** removes an app from auto-starting. After calling this command, the actual app does not stop running unless `--stop-now` is set - **NOT IMPLEMENTED** - set autostart behavior with `skywire cli config gen` or `skywire cli config update` or directly in the config + +**Dmsg Management:** + +- **`messaging list-discoveries`** lists saved discoveries and their statuses - **NOT IMPLEMENTED** + +- **`messaging add-discovery `** connects to and saves a discovery - **NOT IMPLEMENTED** + +- **`messaging rm-discovery `** disconnects from and removes a discovery - **NOT IMPLEMENTED** + +- **`skywire cli mdisc servers`** lists dmsg servers (just [available_servers](https://dmsgd.skywire.skycoin.com/dmsg-discovery/available_servers) not [all_servers](https://dmsgd.skywire.skycoin.com/dmsg-discovery/all_servers)) from dmsg discovery ~~and their statuses (connected/disconnected) (auto-connect/non-auto-connect)~~. + +- **`messaging connect-server (|--auto)`** connects to a dmsg server for this session (does not save server for auto-connect). If `--auto` is set, the transport discovery is queried for a random available dmsg server - **NOT IMPLEMENTED** + +- **`messaging disconnect-server `** disconnects from a dmsg server for this session (does not affect auto-connect settings) - **NOT IMPLEMENTED** + +- **`messaging add-autoconnect-server [--connect-now]`** adds a dmsg server to auto-connect. This command does not connect to the specified dmsg server unless `--connect-now` is set - **NOT IMPLEMENTED** + +- **`messaging rm-autoconnect-server [--disconnect-now]`** removes a dmsg server from auto-connecting. This command does not disconnect from the specified dmsg server unless `--disconnect-now` is set - **NOT IMPLEMENTED** + +**Transport Management:** + +- **`skywire cli tp --types `** lists all transports by type used by the visor (represented as strings). +- **`skywire cli tp`** lists all transports associated with the visor. +- **`skywire cli tp add -t `** adds a transport from the local visor to the remote public key of a given type. +- **`skywire cli tp rm [--tid=|--remote-pk=|--all]`** removes a transport; either for a given transport ID, or all transports connected to a remote node (identified via the remote node's public key). + +**Routes Management:** + +- **`skywire cli route`** lists all routing rules. A route ID range filter can be specified. +- **`skywire cli route add`** add routing rules +- **`skywire cli route rm`** removes routing rules; either via a list of route ID keys, or via a range of route ID keys (note that routing rules are identified via their `` key). This action may consequently destroy loops, and may cause the *Setup Node* to request destruction of more routing rules. +- **`skywire cli route find`** Query the route finder to find routes between two keys. + +**Loops Management:** + +- **`list-loops [--local-port=] [--remote-addr=[:]]`** lists all loops. A local port filter can be specified, where the returned loops will only be of the specified local port (there is an equivalent remote address filter) - **NOT IMPLEMENTED** +- **`add-loop --local-port= --remote-addr=: [--setup-node=]`** attempts to create a loop with the assigned setup node. The setup node is automatically chosen if not specified - **NOT IMPLEMENTED** + +## Ports Management + +Within the `AppsConfig` file, ports are reserved for certain Apps. The following rules are to be opposed: +- Ports are either "reserved" or "unreserved". + - No two Apps are allowed to "reserve" the same port. + - Ports are reserved via the `AppsConfig` file. +- Reserved ports are either "active" or "inactive". + - A port is "active" when the port is "reserved" for an App, and that App is running. + - A port is "inactive" either when the port is "unclaimed", or when the port is "claimed" but the App is not running. + +## App Example + +Simple `ping-pong` client and server apps can be implemented in such way: + +Server: + +```golang +package server + +import ( + "log" + + "github.com/watercompany/skywire/pkg/app" +) + +func main() { + // Open connection with Node + helloworldApp, err := app.Setup(&app.Config{AppName: "helloworld-server", AppVersion: "1.0", ProtocolVersion: "0.0.1"}) + if err != nil { + log.Fatal("Setup failure: ", err) + } + defer helloworldApp.Close() + + log.Println("listening for incoming connections") + // Start listening loop + for { + // Wait for new Loop + conn, err := helloworldApp.Accept() + if err != nil { + log.Fatal("Failed to accept conn: ", err) + } + + log.Println("got new connection from:", conn.RemoteAddr()) + // Handle incoming connection + go func() { + buf := make([]byte, 4) + if _, err := conn.Read(buf); err != nil { + log.Println("Failed to read remote data: ", err) + } + + log.Printf("Message from %s: %s", conn.RemoteAddr().String(), string(buf)) + if _, err := conn.Write([]byte("pong")); err != nil { + log.Println("Failed to write to a remote node: ", err) + } + }() + } +} +``` + +Client: + +```golang +package server + +import ( + "log" + "os" + + "github.com/watercompany/skywire/pkg/app" + "github.com/watercompany/skywire/pkg/cipher" +) + +func main() { + // Open connection with Node + helloworldApp, err := app.Setup(&app.Config{AppName: "helloworld-client", AppVersion: "1.0", ProtocolVersion: "0.0.1"}) + if err != nil { + log.Fatal("Setup failure: ", err) + } + defer helloworldApp.Close() + + // Read remote PK from stdin + remotePK := cipher.PubKey{} + if err := remotePK.UnmarshalText([]byte(os.Args[1])); err != nil { + log.Fatal("Failed to construct PubKey: ", err, os.Args[1]) + } + + // Dial to remote Node + conn, err := helloworldApp.Dial(&app.Addr{PubKey: remotePK, Port: 10}) + if err != nil { + log.Fatal("Failed to open remote conn: ", err) + } + + // Send payload + if _, err := conn.Write([]byte("ping")); err != nil { + log.Fatal("Failed to write to a remote node: ", err) + } + + // Receive payload + buf := make([]byte, 4) + if _, err = conn.Read(buf); err != nil { + log.Fatal("Failed to read remote data: ", err) + } + + log.Printf("Message from %s: %s", conn.RemoteAddr().String(), string(buf)) +} +``` + +# Manager Node + +__Note: this section is using old terminology to refer to a visor running with hypervisor UI and remote visors which are configured to connect to that hypervisor__ + +The *Manager Node* is responsible for managing *App Nodes* and is identified via it's public key and TCP address. + +The *App Node* is responsible for including a trusted *Manager Node* in it's configuration file, and attempt to connect to it on startup. The connection is authenticated and encrypted via the Noise protocol where the `XK` handshake pattern is used with the *App Node* being the initiator. + +After the connection is successfully established between an *App Node* and a *Manager Node*, the *App Node* acts as the RPC server while the *Manager Node* is the RPC client. In this way, the *Manager Node* can execute commands on the *App Node*. + +The *Manager Node* serves a REST API which the end user can interact with. To access the API, the user is required to log in via a username and password. + +The *Manager Node* should be implemented in `/pkg/manager-node`. + +## Manager Node REST API (and User Interface) + +- Login/logout. +- Change password. +- List connected *App Nodes*, each App Node should contain the following summary; Public key, local address, number of transports established, number of apps running, Uptime. +- The user can click into a listed *App Node* and perform node-specific actions; specifically, the [RPC commands as specified above.](#app-node-rpc-interface) + +# Glossary + +**Skywire Transport:** + +Identified via a Transport ID, a Skywire Transport is represented as an interface and can be implemented as different transport types. It is a bi-directional line of communication between two Skywire nodes (transport edges) and constructs a single hop of a route. + +It is the responsibility of a Skywire Factory to generate Skywire Transports. + +**Skywire Transport ID:** + +A Skywire Transport is an uint16 integer that refers to a Skywire Transport and identifies it in the transport discovery. Skywire Transport IDs are assigned uniquely by and for a transport edge. Therefore, the same transport can be referenced by two different Transport IDs, assigned by the 2 edges. + +**Transport Edge:** + +A transport edge is one of the two Skywire nodes that make up a transport. It is represented by a unique public key that identifies the Skywire node. + +**Transport Type:** + +A Transport Type (represented by a string) refers to the underlying implementation of a Transport Factory or a Transport. A Transport Factory of a certain type can only construct Transports of that type. + +Initially, we will have two transport types; dmsg and native. The dmsg transport is implemented by dmsg. The native transport is non-dependent on the current internet and in the future will be the main transport implementation for Skywire. + +**Transport Factory:** + +A transport factory is used by a Skywire node which constructs transports of a certain transport type. It is an interface that can either dial or listen for remotely initiated transports. + +**Transport Perspective:** + +A transport perspective is the assumed state of a transport (number of packets and bandwidth sent and received over that transport; whether transport is up or down). It only represents the perspective (on a transport) from a single transport edge and therefore the perspectives of two edges on the same transport might conflict. + +**Transport Discovery:** + +The transport discovery is a service that registers transports and the associated transport perspectives. It thereby provides the basis for the route finding service because it represents the public network topology. It is queried by the route finding service to discover routes. + +**Hop:** + +A hop is equivalent to a transport and is a single unit of a route. + +**Skywire Route:** + +A Skywire Route is a unidirectional network path that allows apps and services to communicate. It can be made up of one or more hops between Skywire Nodes. It is identified by route IDs, interpreted by individual Skywire Nodes via the routing table. + +**Skywire Route ID:** + +A route ID is represented by 32 bits and is the basis for a Skywire Node’s routing rules. Route IDs are changed along every hop of a route and there is a unique set of route IDs for every node. + +**Route Finder:** + +The route finder is a service that evaluates the network topology via the information of the transport discovery to provide possible routes to inquiring Skywire nodes. Currently it evaluates possible routes only on the basis of the hop metric. + + +**Routing Table:** + +The Routing Table is a key-value store that determines the action to be performed on an incoming Packet. + +Using the Routing ID (from the Packet) as the key, a Skywire Node can obtain either the next transport that the packet is to be sent over (and the new Routing ID) or whether the packet is to be consumed by the node itself. + +**Stream:** + +A stream represents a bi-directional line of communication between two Skywire Apps. A stream can use multiple Routes to establish itself. + +**Skywire App:** + +A Skywire App is an executable that interacts with the Skywire App Node via Unix pipes. Skywire apps provide services for the end user such as a proxy, ssh, chat, etc. + +**Dmsg:** + +dmsg is primitive fallback implementation of Transport and Transport Factory over the Internet. It consists of a dmsg discovery, client and server instances, where the server instance relays messages between clients. The communication between two clients is called a dmsg channel. + +**Dmsg Instance:** + +Dmsg instances (alongside the dmsg discovery) are the main components of dmsg. A dmsg instance can be either a dmsg server or dmsg client. + +**Dmsg Link:** + +A dmsg link is the direct line of communication between a server instance and a client instance. A dmsg channel between two client instances can be constructed from two dmsg links to a shared server instance. + +**Dmsg Discovery:** + +The dmsg discovery is a key value store that registers the messenger servers that a given messenger client is connected to. It allows clients to get the information necessary to establish a dmsg transport with another node by querying the public keys associated dmsg servers. + +**Dmsg Channel:** + +A Dmsg Channel represents the bi-directional connection-oriented line of communication between two client instances of dmsg. + +**Skywire Node:** + +A Skywire Node (now skywire __visor__) is the general term for nodes that make up the Skywire Network. Any entity that manages Transports and Packets is considered to be a Skywire Node. + +Examples of Skywire Nodes include; App Node (which manage Skywire Apps), Control Node (which administers App Nodes) and the Route Setup Node (which coordinates the construction of Routes with App Nodes). + +**Skywire App Node:** + +An App Node is a Skywire Node that runs, stops, monitors and sets permissions for Skywire Apps. Internally, it handles and coordinates packets incoming and outgoing between the Skywire App and the external Skywire network. + +An App Node can also forward packets to external Skywire Nodes (based on the set Routing Rules). + +**Skywire Control Nodes:** + +A Skywire Control node is similar to an app node, with the difference that it has administrative permissions for other nodes in a cluster and is being sent logs from other nodes. It should not run user-based Skywire Apps, nor should it forward Packets. + +**Skywire Route Setup Node:** + +The Route Setup Node is a Skywire Node which runs a service that allows it to set up routes for other Skywire nodes. It does this by relaying the routing rules to individual Skywire App Nodes. + +**Skywire Transsport Setup Node:** + +The Transport Setup Node is a service which is permissioned via inclusion of it's public key in the visor's config by other visors on the network that allows it to set up transports between remote visors or Skywire nodes. It does this via a special api which is exposed over dmsg to any keys which are whitelisted in the transport setup nodes array in the visor's config. diff --git a/skywire-specs/VPN/Client.md b/skywire-specs/VPN/Client.md new file mode 100644 index 0000000000..b481066822 --- /dev/null +++ b/skywire-specs/VPN/Client.md @@ -0,0 +1,5 @@ +# VPN Client + +Client is mostly the same as a [server](./Server.md). What really differs server from the client is overall routing. Only outbound traffic should be going through the interface. Not sure if the inbound traffic matters. So, client reads all the outbound from the adapter and passes it to the remote VPN server through the open connection. + +Since internal Skywire traffic actually provides the tunnel functionality, we pass all the outbound traffic through the interface except for packets targeting Skywire services. To exclude, we pass IPs of our services to the client on startup via ENVs. During its work we add or remove such direct IPs via network hooks between visor and app. \ No newline at end of file diff --git a/skywire-specs/VPN/Handshake.md b/skywire-specs/VPN/Handshake.md new file mode 100644 index 0000000000..23a5ea1a16 --- /dev/null +++ b/skywire-specs/VPN/Handshake.md @@ -0,0 +1,52 @@ +# Client/Server Handshake + +Before client and server start exchange the actual traffic the handshake process must take place. Client and server exchange their specific hello messages to agree on exchange details. Both messages are just JSON objects being sent over the app connection. Client sends its message first. + +As stated in the [general server description](./Server.md), we need to choose 4 different IP addresses in the same subnetwork to give these to client- and server-side TUN interfaces. These IPs must be in the same network. Interfaces of different clients must not share the same subnet. To make this process deterministic, server will be responsible for choosing these addresses for each connecting client. Obviously for the system to work these generated IPs must not clash neither with any of the IP of the local client network interfaces nor with its default network gateway. + +### Client Hello Message + +``` +type ClientHello struct { + UnavailablePrivateIPs []net.IP `json:"unavailable_private_ips"` + Passcode string `json:"passcode"` +} +``` + +Here we have only one field - IPs that server must exclude from the generation. Usually client includes in this field IPs of all its local network interfaces plus its default network gateway. Client may also want to include some of the IPs that it's going to connect to directly without VPN. + +### Server Hello Message + +``` +type ServerHello struct { + Status HandshakeStatus `json:"status"` + TUNIP net.IP `json:"tun_ip"` + TUNGateway net.IP `json:"tun_gateway"` +} +``` + +Status represents the handshake process result. May be one of the following: + +- 0 - OK +- 1 - Client message was malformed +- 2 - No free IPs left to give +- 3 - Internal server error +- 4 - Forbidden (invalid passcode) + +`TUNIP` and `TUNGateway` fields are used by the client to set up its local TUN interface + +### Server-Side IP Generation + +We need to generate 4 different IPs lying in the same network. For this we'll use the `/29` (`255.255.255.248`) mask. Server iterates over all private IP ranges: +- `192.168.0.0` - `192.168.255.255` +- `172.16.0.0` - `172.31.255.255` +- `10.0.0.0` - `10.255.255.255` + +Generation step is 8, so the IPs will be generate like: +`192.168.0.0, 192.168.0.8, 192.168.0.16, ...` + +This way the generated IP address will be the address of the subnet. Let's say we have the generated IP - `192.168.0.0`. Then server will assign the following addresses: +- `192.168.0.1` - Server-side TUN gateway +- `192.168.0.2` - Server-side TUN IP +- `192.168.0.3` - Client-side TUN gateway +- `192.168.0.4` - Client-side TUN IP \ No newline at end of file diff --git a/skywire-specs/VPN/Server.md b/skywire-specs/VPN/Server.md new file mode 100644 index 0000000000..0fa5ced2d3 --- /dev/null +++ b/skywire-specs/VPN/Server.md @@ -0,0 +1,41 @@ +# VPN Server + +Server is responsible for listening to incoming VPN client connections over Skywire network, creating TUN/TAP adapter, setting routing up, reading packets from adapter and passing them to the remote VPN server. + +## Implementation Note + +Due to the firewall used on MacOS server cannot be implemented for the system. Windows needs to be investigated. + +## Routing + +We allocate TUN interface for each VPN client. This way we may easily distribute traffic between clients. For the system to work both client and server TUN interfaces must be in the same subnetwork. And they must have different IPs. Gateway probably may be the same, but just to be sure, we're giving different ones. For the generation process details please consult [handshake](./Handshake.md) section. + +Let's say we have server-side TUN `tun0` with IP `192.168.255.2` and gateway `192.168.255.1`. + +- Linux +``` +/sbin/ifconfig tun0 192.168.255.2 192.168.255.1 mtu 1500 netmask 255.255.255.248 up +``` + +Then we set up routing. First, we need to allow kernel pass packets from one interface to another. This is done like this: +- Linux +``` +sudo sysctl -w net.ipv4.ip_forward=1 // for IPv4 +sudo sysctl -w net.ipv6.conf.all.forwarding=1 // for IPv6 +``` + +Then we need to let the system work as NAT, so that packets flow as expected with their source IPs changed to the IP of the default interface. +``` +sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE +``` +Here `wlan0` is a default network interface in the system. May be fetched from the output of `netstat -rn` on Unix-like systems. + +For cleanup we may fetch the old value of forwarding like: +``` +sudo sysctl net.ipv4.ip_forward +sudo sysctl net.ipv6.conf.all.forwarding +``` +and then we may assign old values on cleanup. Routing rule may be removed like this: +``` +sudo iptables -t nat -D POSTROUTING -o wlan0 -j MASQUERADE +``` diff --git a/skywire-specs/VPN/Specs.md b/skywire-specs/VPN/Specs.md new file mode 100644 index 0000000000..42624a7012 --- /dev/null +++ b/skywire-specs/VPN/Specs.md @@ -0,0 +1,113 @@ +# VPN + +Basically VPN consists of 2 applications - [client](./Client.md) and [server](./Server.md). Both applications are made in form of a Skywire app (like skychat currently). They are run under control of a Skywire visor. + +## TUN/TAP adapter + +We use TUN/TAP to create a virtual interface for both client and server. Client and server are connected through a tunnel (SkyTunnel?). This is not a tunnel in common understanding like direct connection, apps are connected over Skywire network through visors like all other apps. + +All the system traffic is routed through created virtual network interface to the application which had created it. For the initial implementation I suggest that we go with TUN adapter which allows us to inspect and handle IP packets, so we may concentrate on the overall VPN functionality. + +### TUN/TAP creation + +To create the adapter we may use the following library. Linux is fully supported, MacOS is only for TUN adapter (initial implementation). + +https://github.com/songgao/water + +Also, these links may help: +- http://tuntaposx.sourceforge.net/ + +For Windows this can only be achieved with the installation of `WinTUN` driver and if running client with the `SYSTEM` account, apparently having administrator privileges is not enough. Library wrapper to create the actual interface we use: + +- https://golang.zx2c4.com/wireguard/tun + +The easiest way to get `WinTUN` installed is to install the `Wireguard` itself. Also acquiring `SYSTEM` rights is a plain pain, so, unfortunately I can not guide through these processes. This is not user-ready in any case. But the only thing I'm sure of is that it works, so this approach can be used as a proof of concept. + +The route setup trick is done by OpenVPN on all platforms, it's the same, and we do the same. + +- Linux/MacOS + +This one sets up the interface itself with `192.168.255.6` as interface address and `192.168.255.5` as a destination address for P2P connection: +``` +/sbin/ifconfig utun2 192.168.255.6 192.168.255.5 mtu 1500 netmask 255.255.255.255 up +``` + +Example of setting routing up: +``` +/sbin/route add -net 134.209.17.43 192.168.1.1 +/sbin/route add -net 0.0.0.0 192.168.255.5 128.0.0.1 +/sbin/route add -net 128.0.0.0 192.168.255.5 128.0.0.1 +``` + +And the corresponding cleanup: +``` +/sbin/route delete -net 134.209.17.43 192.168.1.1 +/sbin/route delete -net 0.0.0.0 192.168.255.5 +/sbin/route delete -net 128.0.0.0 192.168.255.5 +``` + +Setup: +- `134.209.17.43` is IP of one of Skywire services (we add all the IPs we need to connect directly. Like setup node, Dmsg servers, discoveries, etc). +- `192.168.1.1` is our router IP which serves as a default gateway when VPN is down. This IP can be fetched on Unix machines by: `netstat -rn | grep default | grep {DEFAULT_INTERFACE_NAME} | awk '{print $2}` +- `192.168.255.6` is a TUN interface's IP (question is how this one is being chosen. on this current run my laptop IP in the local network is 192.168.1.5. Probably OpenVPN just gets this addr from `ifconfig` and changes its 3rd octet to 255) +- `192.168.255.5` is a gateway for TUN interface (destination IP for P2P connection) (can't say for now how this one is determined, probably TUN interface's IP and plus 1 to the 4th octet) + +Basically in this example we do the following. Route all the traffic to the Skywire services through our router, like a usual connection does by default. Then we route all traffic from subnets `0.0.0.0` and `128.0.0.0` to the VPN gateway `192.168.255.5`. So, we cover all the IPv4 range of addresses with this. Netmask `128.0.0.0` should be applied to both half ranges. So first half range covers `0.0.0.0` through `127.255.255.255` and the second one covers `128.0.0.0` through `255.255.255.255`. We could use a single route `0.0.0.0/0` but this would override the default route in the system and will make cleanup more complicated. This way we will be routing all the IPv4 traffic from the system to gateway `192.168.255.5`, it will go to `192.168.255.6` by the P2P connection and we'll be reading this traffic out of TUN interface in the app. + +This command set should work for all Unix systems, the only difference is the binaries location. + +Localhost traffic shouldn't be affected by all this routing. So app/visor communication will be going on as usual. The part that bothers us is visor-to-other-services communication. All of the used services are put into visor's config. So, when visor starts apps, it's fully initialized itself. So, it may take all of external services and pass their domains/IPs to the VPN app. This way VPN app can resolve IPs and add needed routes. The problem for now is Dmsg servers and other visors that are being add to the local STCP table. These entities are being added at runtime, so we need to pass these to the VPN app and to update the routing table. Based on this link https://unix.stackexchange.com/questions/188584/which-order-is-the-route-table-analyzed-in , routing table is being consulted from the most specific rules to the least specific. We're adding highly specific routes, so it should work like a charm. App should have a mechanism to get new values from the visor on the fly. + +- Windows + +We provide just the examples of commands we use, without specific IPs, cause it is already demonstrated above. + +Setting up interface and its MTU requires 2 separate commands: + +``` +netsh interface ip set address name="${INTERFACE_NAME}" source=static addr=${IP} mask=${MASK} gateway=${GATEWAY} +netsh interface ipv4 set subinterface "${INTERFACE_NAME}" mtu=${MTU} +``` + +After we use these commands there's a lag before we can set up routes, cause interface doesn't get ready instantly (Windows, what can I say). Just be aware, that it may take several seconds (we wait for 10 in our code just to be sure). + +Setting and removing routes: +``` +route add ${IP} mask ${MASK} ${GATEWAY} +route delete ${IP} mask ${MASK} ${GATEWAY} +``` + +#### Cleanup + +Regardless of other cleanup routines that need to be run on app shutdown, I guess all the possible interruption signals should be caught so we could at least remove the routes and let the system network stack work as usual not to ruin UX. + +### MTU + +MTU setup is not yet clear. I see that my OpenVPN instance uses 1500 which is an Ethernet MTU. Is it fixed for all hardware configurations possible? We'll have it fixed for now. These links may be useful: +- https://community.spiceworks.com/topic/217130-mtu-issues-in-vpn-connections +- https://www.sonassi.com/help/troubleshooting/setting-correct-mtu-for-openvpn + +## Configuration + +Both client and server can be configured like any other VPN app. + +Server flags: +- `--pk` - server pub key; +- `--sk` - server secret key; +- `--passcode` - password for the client to authenticate; +- `--secure` - by default client can access machines in the server's local network (SSH in, for example). Some people may use this as a feature, while others consider this a security breach. So, setting this flag forbids access to the local network. + +Client flags: +- `--srv` - server's pub key; +- `--pk` - client's pub key; +- `--sk` - client's secret key; +- `--passcode` - password to authenticate; +- `--killswitch` - If VPN tunnel goes down and client tries to reconnect. By default during this process direct Internet access gets restored. If we set this flag, there won't be any direct Internet access, user will wait till VPN tunnel is up again. + +## Encryption + +We rely on underlying Skywire transports for encryption. + +## Authentication + +Authentication is implied by the Skywire protocol itself. No further actions needed. \ No newline at end of file diff --git a/skywire-specs/img/routing-table-diagram.png b/skywire-specs/img/routing-table-diagram.png new file mode 100644 index 0000000000000000000000000000000000000000..633e41268b68858f517f8d4bfb6beba793428747 GIT binary patch literal 33897 zcmeFYRajh2(=H4N1WAD4!3pl}P6+P7J-9Q#K=8qWySsaEcSvvga2?J?77y=>a|*{x~san?wT+aC22HNLR2_7I5b(A_aETko)5v&9mp?WzlA6v zGH`IHq}CD=DzXw1_x)C69vx#>d)wT!4#c{v4oy+6p}TV zSON&h8b60NQ7h;uC~}e8p5_10h0{@gkMlCZ@7XW3I%K+u{s6jC59~$HUmei#yYZXx zhcRB@6Eve0ntAhFEPSi5nC9CDII0Uy1aVT9NR185S1X)ia4zI1sb6Dz6_s7lkxvn2 z?nNGj*J9vggx0umY#%NjebKB9T?Fvq=-@VR3I&+GqdODA0m)cWs;tA zh;jWef4^%Minu^uMuc!y>qPL%(hR?Vz|gyDk|Q`RjQ%CuS(>&kg!$o6?)Af5P9w>C zxSux^aitd%=@ZZ3c4y*FDDLYAh@Xj3;QA4HwWbAB6s5a+!M}JNS1Q=T>Vi%9nN-a5JJ;n4T|ZSeg5qxsHRyJ)>OWd8AfX4(_tZKP zX1?6*ZQp=(CxX)4qxEC?l#S`LQ4f&Lk0Ni6&NcCEc#uiKy((tun?$%qM2%1e>Djz= zRgzM}8H2>v+<~{D1NmiEFP2{0hx~dUlH0n+b&2P~-iEd6=O78qJ;@*ZUcbls233l{ z)UUSR-B{DugncpfnF2qA49Hc0c4*Ce7gq=tEc42Bm-8Tnl;gKO#J0fsjSR zjHqNV8&H}Me4oYC_^Z6*liDL+d8_amFKp^P&1Ci)1qE^kTFuuPVe_FeVE|dG_!q;$ zI5Oe1_LLg2N(m0J12Sa89A-Wo86ClyvTKS}v~F=^vFjh`a7eDc*smnIVy^`&Sr;@Re7m%+?04Fh}2r+s{J4=O=7g6>nZ_3!meTi=) z=VsuhRZQYZ`jw)T%#tkf)vGd$ns!=gHiE7H}QD?-lPq)LyEZGSZW-dxn=(o+uBmd_H%Pmr(37;WIbO zSLry&I2anruE~n%cBynpg0w;6{&_!&V@qqa@)S41D0Eeg*j3o|&T2Nf`w6Ji-R8{3 z+$Lq#kj;!>Jd#QZu^@!KysF~3EW6CGe66~x^xF*blm3$8+UeUhEx9at zH#F9b)d-v3m1pG{V@}C1`%#EZ1CeOn6*-taw~c znoLS0+$O}HEt2geEZwQz(P9{7Sid&6meJ|uJtV^D!{C#Bb8v_1EqMP?6wvJhiqW#F z{!wjpf%Itjcz2(J!1jFV8R%8tix&u*h$0vqXiGRV?eT5S4qzYgDF~vdlW4=Q z_R*7G8DKBH5g>yQY7)}un|Ih;d|BQI-#FVi!fXls9=e4IEML_;F%qsFa8~@59@hwN zsp+RQ9B+CKUu9ldQnJSx$I()VI^FZTX13B*mmZYruw;1o-NoWbJFH5IL zDGw|RCJrd{FAW|?WJLob8DlcSEC;m)x|L29N@Y`(FO@5laj1i+8>zMuC=*yu&9B4m zvvFnibZJvQk0$NX>?NbKV26b=l&_rBwBf&^q1cO=mpS}EVDsp65AM*mfqtoIp$gRA|*amw?bFm#*8y< zd9F>&Rf-FwYa;I7?%uhD7!2N!MS&_#)K4nf=)7vTvCcj`Qcv!zi~OV|3B z?yKmltY7dM{?Xrd0#kyCL#kF>JG~yV=R22s8H16e-$@a7Tg zh-nfY5MuGBxGKCaJD+aLxdJ(0>Y}hlw?)rgEV z2EFt-zw5v0!9XP=6sZ&W#qyb1fyKhN$LGs+1+;Z5Vj^WQ-z<(UiSmbc<1;#{jnp@A zYctQqP>}ot{H{Z0<+G1?W1c_sAKfDvp1T_LqWW2wA!D^?u)rx%mer$8N`0^+2+V@B zb9j_~#M@_wSjg*|!J8yn^$qYM`#BLucB|jY$o) zmO1(<>b=wrFP_+QF+hhG4>K519A$KXaBz6kzrXOZAE-{@;NY{Y)wNx;6%_U~#f_5%;B3jt&d<-! z%ErOU!NCkG!3^|pa4~jgb^ucT*~s7Ryf+7$I$JxsSOXl$f46IF0&sN^rl9!U(LbL* z$7$|v{Xac90RJKjMj-3&99DJ~Hr9XIhE)~%{Z>H5+TGkv`@OZjxdRZ^hX_9ppU@xW z|CRGUJ^n{co&TxH&iB7+{zuMVHHBDz6Yw7d{mHFA-on@=f-1!N59>uxJ5mEuVY5MO z{a#rec7OJJYG5Bj*q83FJ1ot&=H)?T2fKffeJ`%=4*xR?se?rFYLM3D9WBDM&se&p zQ}6aVX}Kh%-`BpRz+=L}Revw3j>8lh)J-nwDT~|*jfcw%#)|)r@>xZq_36T3-Ogh7 zclqM;{U27h7jq8W7hfNzstj_e?%xGKjeN5kNK~Qk{nT|+8;U8@4v4%@bp8TX#YJh z*igSB%ZSg}{@0lPc$5-DtuuW7?>;cuo_&bDBK&tBp?;V!Qs)0n|Nl=?_S6tWZucmm z)xHm-2A=yx4$Z5dfJWCn_$xve;OUa{+`jO?>AF;cjiG&`H4`|_f6Uu`qeysR^mvZn zd^SK-a<`N!jM4#1IcyKb9d|<4p3 zsX@)kPhaj!AFs!Sq55v$#gU5OCxk8(p#7{M_SJbQAOcJGy)!f+=bkFcEXU`U(;%sr zUE6wi)CA~P{c)T$scb$WmsM>0&$`i>;Yrk*aEU)AGuQeVvsSD!{|*lCr=JU4#zpS3rYkK@$9L;bAorn%J!w&AJ%?6vh_=yEnvh4Go@JKn%H{rag4s#O#rW|- zdG)v%JSl7EM!{%VT(4?F+lL~dTW|Z$ZO5NiwTL?(JzC?UN;@_ zji%Z?pk?C)bkyjPqcb?_B}>a!=C-~3G}EYKp}U{;!@Qf-M)r-fE7;0H8*X<$t4Cs% zNLP17pFG^=bS#Vfp{s7|+YfRWmw8b(JFj(gtnz~5Z~X*}=MCIkN15ssnjxn-PxqD_ z3@3o|q-yW}hpTGet9Is_&i&$ym8NBfi~-lnN$GZ@`%V1)E}p+uYVII%qAt^%BfXBr zl&G3J@Z|J-lr<&U2z0|mJimY}=CIJ^@?8pTTIlL*NqS|LxIz;&!@Y2*S+fp;|YLWkGF@M)*i>LyEY(np$w1RtY$N`mZ=aj zP-+5q8V-_D@|$D#W(&EPmd4s+x79@H={=5`t=*ZM4G6E z&{oTA4;t)l`XSfBNg`I28&$m zwfiW9+dq@=gizLTuS;j!eY#^~S??&J-Y;*bY)UTHdbckbs# z*_plRxZaEqPVPtr)}z+>(h%9zhz~0~e|g~wW*;LV|-pl!udaMR0QSK^{%x%rXavw?bT6}*_4^Roc+Sk}P)QEj>Gq;#n zO^*3QKQyloR4mS{p=Vf4rj=y+OWieMS+#~;<%N)@^C{QL31KP}S2i!7&~$@w>rwfH zg9*y7h})@?wJC(>@OghN14NdedPx}-AohLAK9M3$^Bq>f5x5ktK6wc$}*UBVsk4-lqDiQ?#rx}Q701GLv~SU{5VF;r?!i5=h_g2p#k;0`JK_7N*7j@3R(>u| z<(kxa%Z74DXzp$sgfH8QPIvBg*> zLC3fSG;VQU76^-XzJm2XmKlB+lh45gRGn9<+=;!Zq)TJ{3~uJOTc}Y;tn@q~A`q`C zaX`nm2gz`F+>X0-VQ0@Z{k3T}WQpl1R>Z_oEyC)z@9`|!7L09lY-(jN4nY}1GiNTj$Msm zvWc1Xly&&$FA8_hAt}b1fC6f2;96p9v@O2A<2Nke0ugs-^+4-fFug}%Z(ru1`P7j9 z*ZkIvR8TN+*X?XImSPUBfK$(XO;mgwtwIWNFJ79~h5lHtm&5M!`HQu5(_DSY*7Sh; z>#Z2esTvsjcc~lANDDYOz;cwl`fgn%D=ItEK08Q2$6mL{2bDcDlnLL5LnCFS3xjk? zc?hxj%*}2NnW16Rm`6Q=Y;{5XR0UN$H0A61 z%XC&;;qQ#qjcfbyM%BAK#`WLWhvITcl^n(`&`)CcUibElmeV_I1Q{1Zc1*=a?=*Km z*l&uB)YILr0w;ulBvRbv@i^PRZ^fgr94Zk8I7_>IBS-G6D`^VKi+16DdUO9 z;GiQGp1dIL_Ay9#7$*NZb0}vQ&^zOs$YhKi3}z$5&+W z)TRY~EC^)7qlv(^sVFQSbK!c0x9bdz%;$Dl4b@p_ss*+=q^NA(n5xq%+#R={WEVHj zMcHJ+^m^+_C?+Mxe*Re|t-CeGbc~6CS!b-94?xJ`BAU}<$s~e#BtFdAH}!<#;s9nE zuwzHt+N~L*j7Ho*GOB;BNa7nLwrc&woz#(-V~yA{M(XtPD&uPT?3=kv*haj4!EcrC7GbhG;vV3ZS&1yNl7(;oJT5o=}~a=%

lYioD$li`HdszCP!H3H^gp z4u4`TJr|HJqA-0EQZ0BjqsF+U=F+@zlZZCB{}Hrfitk6L&W2m? z!0DC}5!OJyLu8`o>-P}e(E#H%l3;Q1mqDu97hf|_db#uZqH;SLBYIEIcqjvn2^Xrk zbl)&KL>7DOs^6P5Md*0geHs|Dp^N+w0yS{o2uhIBdlP)*4r=t68%MC#6i25lpOwXz zrw+27;PiP~UVltRsegLB3u*q+Mgok^%Hz%_=d|QeaHF;jbXV`|_^zo41fa)4nb||; zU!q;RgJ6~w_lA?pj-bZ*J<^H~bd_M-CHM0b3t}tDNf}Jql$++x3osFH!WpYlT`8A6 zceC6^ii)rJunSW@cB-_-}$j&T8U9@PIlr11Z`Jv{>|=u z*j6x3d=|?c-A?t_f$O^(LYsj&I%B6}MqG;xR*cY=V(N|DnbAsUmWYoh%>iYM5Qo~$ z<5`Ta!h~`L4>RZOsQ0`lwL^s|D5Oq>s&*64nSM#?Ogiu7PGN%5$o8SF(Veclz-sh- zdSp`6C&rXk;{)%$Xs?W|8r)^myhmF`rRiE2u+69e*C zAP8rU>_k72f<4Om!?Aj7EtA0Yk3e~sm~uem*R0!Y&{(Rz?7F_c zUQY!w;%q<15Mt}Ci>H2n?>!DXgXT1d+(mr&eF}N?T-28@`-{rHUVblaW{CcN8nB8M zXZurU32c^q&{vDdPudQv>K(A`e4y>?V;!S;0bnfPhF99|kMGjB%BC<;{B;H2T?@Tl zcs5kxzy4b}RAk6PVV2j$D_a4$|Gn-9*u!&TMmZGDI+pTXrQrjf9TCy-A>2;qJgQHx z*BcyO1gpdd|61T;*}Mb}pXwi+h?^E-dh8daO!0Z^!*rvev3e2?uN0!A6l7}y>a~7n z=%CN!iyds*S|Sv-yHr=GL020PNxLZ&Hk~!tn*4%gK4>38Qg*d?8y(eg**~~>$x}|g zm6kr9#C8OhWx-PD>8aflHMw_brsTkUTW}%s)#90J`enUm97ob&VwSaVfIe74oXxL&zk!d z_->^lfVj%uyBCeuY!lKRrp0+W9EQ6swdS??#lmEDq%G~P-&`)c8?)zc4^J-1Zv|0| zFOt}cA=*(<7n9<6wy}lTLa$bQt>*f~VS>vuWIZGLYr!k*RWw#KCxneG;%zD4Xt99vRjOjtq>a@%j%AYq8Fxd>`0SEQ#4FKH%;DPj6i8p&-o zT<}zGW|*~0K*$)PvT3PS9S~U{oP!4_IowO#7uU&M-xPyOHuO# zns$y*lohwI_cSe-L=;(5kJTpNYz{D%PhfPvWNuIq&?Q^T@TfE9gRq43AiKJYOmK1* zOdG#xvQ>?ZNPKs^vDN$KBg~^>96x2xce@*&(BxT2xT}pqDeQsDal&+Hd3Skhgae@v z-$xOIZ-JDRWHX?y?M8=C%50+{NoBw6P6C&k+C~eFhQLfkUTmXIR1VwTfjI_zLwDm~ z&e1;h5CH|ZQgZATk`sEUD(nREfnGB*Y$#SU~Ge za-G+AdR+PPwL2SB=kh~{J34Ox6zeZf+8+Hgd#%`EIysC2fN{aIixF)${Fl1|#)QNf z!zN8*LIz~`z^Dt)xiCR}?9>5HpznighFzVk_g&jB=z?hsOya<{mv_i8Yxi@+uGz*S zk6||Iu1u{-58*VZAuT#Uo8WD>G^s})E%lALL&fe7-~DgW{(bieMc#XTr@Qxn>;U2n z+^yG9lU`E?S^Q4ZE${=O$ZOu6r>{(3)S8_1;yq|?WtNaX-Awqh81sy+qtIrmnD{6W zOWz3H840k3LxK~sVGasfAlkhJZJ3~j5IzVdguQ;zRcu1QI7@J9+@^6@iM#NTC$1Sq z#IBsFgD1>oLAy3Z=DC+w*O)A;R`J`sp{`^Q%~fHT)de+3h|>tC-iMqpEb|@qPzsNp zsRH_U7Xgg{3c8!Bdd2kLG4bWyI-pKS_Kb%X0@V8}X#5cVqR5LfpTAkk@=Q3NJb^$V zvllh)sN*KnNrJc^&)f#)|1($W4h?Lrki=pw!CLDcdhVjCe|EO8LeT$3<_o=peh4;m zhjEDY3ZVO8B)>0CMd-r-H=V2TZa_>(t6^=&Y0pb@G#s+ECk|GDPj$&9OyRlh9{uhe)z)1rl}MC^F%sb{j6=jk`3=|kJa(gA*Dar6_NhA8~gd~0|SgOY4d z8=41q>+G!UHLsJgZB{dl;SRzVOh~xh^+HC;m3h6iBGM~SbVIezi`k+;g2Znj+JEpV zC~O3SE)nYl{J7ZH4Leh?AQE&xujN+GWa+5pw5JwYBSGaL8HIsm|A$@{c3tx)rh4LFi^_V1v~YU14Oe= zGLd5AHm`LFw?Q-9C+WJ;#*ah;@~ogZb+wgySEqtBmwf%co>EI8Bo>8;b&Ihj$3m7! zS{`2L{OPEV4z|l6i5qyEU0K{h$w5`O=LHMYqDx=M1ZQm?VER^NJCOz~(P0n;q(tR- zrTtiv4%-hO2{fa|0o9*!vW3VzgP{}Q>vE7;k{CzTyxMy;_XgCjvkU&Va9&OSwLqb%)-Y>ictn{mRMFzv)_)7oJ3MheS2{klI*HTySXxWTj$yEJ%}yRwwg z1K3%wj0xGyrCUpjz6D)|WrpeWnav#kW7jS2Unh3hK_Vu!GX99u1kKX&A}5occwEXli(Y&@ zF)n=?Fz%ZfRcO(Z<1|1pE+l3d&0S}T269_44k=1WzrY);{e&7bjU#G#k(5PGHlD2F zTx>O$iNHzk%ER%~4(EH?O&MniKetf^e(dEb80@{YDC1%nd$~Xa=-8l9NR9o>yR(#r zzt=XEEj80i#ACUvgCw4gy2cwVdyaC-yzK2@DOXd^e}3biE@V?+GoE?FzXs5gp3^Cj z>}CHn@Ac)p;683f4U}lK+C{nS&|b+IVzQ4`$@yK!QnhVeji}Q5qE`&J36F(q27+UEwOs#p5b_yN^?YIr*Ocu%6&U&+vU%+Qyr`Zo%Q@^~k)M5RD}lg(3G-!#DcwyNqhRU!E{q_J3+c5R_XhJb zp-9D$9@*WnSwe2DH^HNgoUHROxBWTF*NNg&K0Rpxl$viZh^Vr{R?Q{jb5KeCUM|kh zNb<-tc&#jf{UgqR?4Yc%G1p3-U5*lyafmrJwEiYKOqLJwBYrW%Z!F7=>)b9Zo~oc} zt-1axD{HJWtCO>8r7_WOBqn*&j)CLi=m%R44{bDezp3Fu)|l-)uhDbnV#*tO8Aw5o zaq_rK$KQI-GhIxYVF;{ap}shGdz2guh5;;U)L}@GvWE3Q53gIv$%k_6ukZ1K7G7SI zrm4|l`g*=905{~X%x}bBx|6{}QUz78jG}AjC-_Q=n zG*8^b9h0R3(MmQ)%l1I=fl6?!CW;I8S+G_8Jzd^Bo z5IHj#rZ@iMd-Q*UfT3a~Sg`>sWy*#(e>vR#140~v;gFf%@<{##`1vbQlOwjO<#e1g z|8L|HhEK%4B}e2Blo0&uboe)70ej4n6GL?_5%Jj3``6IPC16=ELO%UN=06F6BtM*O zC7<(==6?@87?$-4{pcSe|4Fzoz_87%+wCO4zlZ(`mPJ6kg!vzI{y9r~>M#IN@M@TG z{og|;fMs#=HBkPG+V5pB;MMIQL90vhUmzz3!tGJ(ZgvqE^Z#Rrp<+nC8Sjr_p!09W z)6tS+A3D-IYJu@C!r*!OW-(f5EJ`dD7whEc2tB+7D$*8@RUy`-Ndp<PG1dPdX3eEt0h2vfS>hEx->@$KUe5o?;yJ&73DgNy}tgSsh#G5OpEv)kgBsAxX zWuyA@+dz!AwCR#|4?KezG;KhqMZxqHY!Ue+ehmS1AFZYHvCBU6lviv*3?09Tq+?U- zA+)9V9c<`$m+1R6w(rMESdi>3E!`{oI!pXoEcw}j?t%_~fPH{_fOU{)VLnhl(bi0> z=v-wa+I%^u$4g{ze6{8KU<*A3fD-t52UmdYnDYj^cm}3XV5g%{F#yeX9aQJC?Quh! zvTcuwi`A1hSKrgX;+(Ms82Dek28_|OiC|ud#y47V@eYSgTrL31{ibPorLjPYI&6I- zKVl`9x%p=Xf$A5@TtSpl((&(|-t}Yg(cA)yA zYNul_+Hsd@jB8Ph=%>-+hJ@u`KkF}8CfPCfl3-HGd!r5eK~YJG_-gk~4E5aZoZL%? z=UDjG3i@bUO&ztJXLN7EDsS+7I4&Ub-QC`*+o}i5VfnNADhkkY<$J$%naOiNdAhA* z$v^VLW8aGJX8V@K!FW#-RPCcG9<91|-#XDcY&y_#WAA8l5<(WOY-@zz$?PAk7CT`8 zy4g0a@Y#-Y$SR1ox&0<1?m(DF@!Eee;OnT{%7(*&?-Ld4a+X`Rp3TLWWrR09wy^w< zDdXOiHRe1FQ3gS(inzHMbk`TM>j7W@dGT#Yu|*a>V4Y8|$m)MA9t z)K`xFsVSGlU^>us2-a+Jfd0ec9N|0I;|Jk9a%Gq&VkHn3v=GlnKQx;Y_wJJPnQ#?( zxAiE|M~lRH;O!b?b<6y5w`g8tB0qhqWo`e8^hxJ#Hm6#Gb&SxaNG5wPvlY#Bp#Axi z9~#e4(d~$V8-M1DzPcZ%j8yXjgBQOnyUv2Wh9OO{c#u~cZ15%e7p$#?j9*fR@kyT} zozw&#^3LmfOdg~ngPRZ-eCEw=dz;sstJmCeyi9u#2C--#QgjBGyT%2=bj~{h+zob&yHiHKd^;S+KAna48}tm~lm3~M2Xo(T%CV?k8Be6jFltk>h0@)bdg zcpm7xm#zJC`kd^wS+@R&@fUnz-}uRUI`Q5J@h!64N(;~{@3NJM^2v*5^FQ5F<;+$w zhjX4>x2u)VOw8nD03UDkPAb3(E=CQz+|9MxCcS76mGPFLgY%C(^HH~cyHDwzrm0;Q zk;W}Q#|7Ji+92M=_;_})Df6W6^}b zDGQRE$60NoWXI_%BAtQacHHd_=j$kGGDRT*(}T8(MrFAi@6ELa(?p4eay&wgzO!fy zE=%J^8A|h3){FRat3eX%5>*fA?OvAKN<{NYe5T1x9QQSW|^35p0RR}F6=z*9mRX95R?h{BlPB6;}) zU3WOncFa+F$5CBk7w(CxHC zME7H3BqHKO6cSk`@%kcR1ed@-^qV;M!HDO8(}bKBZVXW&k)JGY1ZMJn416fZKeXma z*%_Ds&2>VjyMwD-3yHF zd(a-HQaHp}ID*cFFMhPfkKr4B>-fBOe={LSh?~dMc6;EUXUnGkfpnbv3D<<6^72fL^Br?$H>)d^GI8~M~sGom0bfbG}(%K~IWvnb)jOWaD;`u?0JeY`cY z+%t)6dL*QbfR?DF_MtawiK>0Zo#9DNE?hiCfM=YNgIQ3g{)*3z)Ppa|i6()D?zdeC%dQsY~lL8a=9$cjQ{)LWHWm zlrZH0u_Cv&&==}G9sydzc}Vrhnt?9io35X7$H=0CEP0l2u#+A03O*Im&(6>qj@O%d z%Qwoou_6_Ew%p6=+sPr3&>8Y@bAC9u@eH~tns=#mU17jJwAncoZt_I3Wl_uiX>0yw zXFajjv|hH^*){9LyW~KQEGl`&<#E--c8fyNo_Bl|1dBS+A$pxBVT0cu444>=LL%8N zUYZhbM76I`1l=v`2F?>#!4{jsM{M~@!-{>re){ASA;qoQ-PU3H5-Jj)hhYVMf+HSp zimO#s&Ag$kPpsCbqO;lBIMvuniOj~hKlE~ghIqCG=}(pLoJMMuHxF0uBwRT&`pD$9 z*|&0kz}xao-3=p8=kpMt3=aVnYJDs8o80_m`+7hBipFP5(u}9C%Q2Z zA2&vFkGHa-nKHaU#g=k;V-T9+;duthSiQPCJ_Ou#yY4nEG0?kPW$r|Yfl(`%G4})! z4sNP+$ee^or4&)p<~?r-R?YhY0HFX>>QHV%Uf4lugo*6(#I$vNBly~I1&`?DqY z3u30s>}`s@3g;uVVs#7;1Ffgy-NNk!_}>up_A>W`-j_OUIZMlhKE1GY?;aBT*u!}` z_l|D8LPtL2xzkS@Imr$ldQ@&r$%>0nAL%vs0@F)coTru2=byUH<#<1KUiC~51JQ6V zh6|)=C$ceX{2>*z$+HUI%3K>*yPA@HW_x!Mb&JSD)*;b6RyB#tH#4O(y67dSb)6}r zM{VU^0|GnON_#logNdLena79avv_&bEHoayWX_harL0-bf_enFdhqd8f^9E^&jkeb z`_47ruoW;BNJO5qO;BJq(G6GPby-eCv}D{Erd3DGMxEI_27Xr^DcEaLFv z5J<&C8vsno`|?DZlcmqXM`wB7jri2ey|Io8=s4^Qcm-5w*Xj0ttM|&`w*^Hb#~!@(3Ruw~SDF zI`Plj5bBqB%zYsydnMi<1B4c(7>7>eEe2#j*tG-LG|5~vDV)^ln1{MW8Zhh>h^+{V z>LM;H$m8h~U@qhBnWUQ9#WibUr|nDNlPgMbq%!TzdE}bczX|JpTCG)h*;8+{PIzW) zNt0E;yU@Arvk_#r?Q{2%<#e*7n`^4N4oW<&MtBhU%F|=~AauD+z%@8+D}A>6-rV=g zX>-Xgc{wSIFIVM<*b~8HKIk=pB^B?&sDeo^wDG=bN1Bj6q%%Ol&gB$fKT4@e>LosD zrk-*BvBnhwFwL^5ZOCVmF6&qt!Cwjrqek3mbQOB5pxmpAjD46NJ*y=Bp_S`d#(^ly%`2cqlTrFB+gt=#r zkbYBD%{|Qw3B-%4h?{pa@}4ws5Psl_#KP_-d#rgSzEERQ2t=UyVj)1pFg_gmp3R6k zl1e=`0<6~gLAE2ONp;5^hb?jf0D~v7F)ZvAXpM0?@mi;(K&h5P9s34$E}VFfdIK|y zu*g>x<`zeXPkYi%v1VjJCj@H0RFW-WfG}aODoCo)nwYt6$!7m7Kgt5fY-)&({ptlJ zsiASng$n@@LEhTe_d@F0y)K_&5grEONF)6gHjnuPDHJ1Xi@e$Y_R9Ov_koe$=*S50Ay(FwblJvZ zC^HA`U_+B`C~{`ahJ=a?7HVkntu9NEtW+bNK12eswN&Aw4bc&+z-;OqrhNIxc@5{V zZ-ViSk~Hm(7xNB)mT}1Clc9+#+>WsQPfY&eUE_zoi%8tiL5_oPTE#A=5HD<5w~Ik45nSYS5)Zx)b_J(<)duxh`^`{ z51n4Bymx|+;1JtIYU{05SG*SqQfENqd3(%NqG>$-ODHXgj8!sm$V+JN3z@i0oM7lO z2qEpkvx-uvKsoZ=+xF!luZlhq3yV;wTa@34({2{GP6eB9)W9T_DVb7`F|*Q03GS~2 z$`k@?_x|H=#Kx=_3odUIHW*PR;%!?jt*j*sOF7-UxGKJcX~jkmUg=K^Y1dVi()WNd z(Sx{F%r96l$5nT6hJ!T8*f#6NRi#^9VbKRGA0yv$QjFb6mJABR5c$o^@@y$lOehVa z&eiD?Dfc?DYb(A$EL}ANN>(5?_%ojQ3tK)WO2?m)of9~sM~tWN8HT)4O<5|K zO$Y$Vo*2vCs~wHDdIrfM7LW==;Z+>^kxDbvXT|e~Bw0@Bf;!^GU<(_h4_0X=N9%=5 zwVxP2$cigHTM!jsEY_ArF$YdrTaU2x)Ej>xzY;g`Q7Mv0L0kz5{`hZyFs}r!$wx%_6aO%pB$ODcx z`5QTYqLJqo)R(1cO_VMzpV3D*-Eb0IaJIki>%75V7Q$BeK@fFzV%Zr5+APx&2YnTI ze|n)>%{C9xvVAp}@5x6qwIC8L3phGxDG+|I6NJJU9?Z*Jg#(1m=q3j4_H#v@k$U}? zC4@iiPgvI;I$fl7K-D^^!p3TtGB4NBcElT_m2ya*^r{2E`o`K^6p5^o=){K)hK#G- zn%K5do@dR0>gl}EJ1_CrWMB{*R?Q*V+D~k3yK;;g-SdDgPJz%bDD%qPXu#3xr`cMf73x`POrilf8WtH}FkT2ojJ4kr%3`89a4^9}_PAU_r zake`=`W^|_YfN%(yePq3%}X+oTCv#53y?DbWs9zx;`ZFpLO$g)aC@;1Vp<%rl*>WV zvfVz}y-*)8ve%j-BkNPTo^|)bL6iVHaA5E`9(SbQ+cliJ;~{63(=|2zpsAYTvAcdw z*G?dN>l)*bDr<0`MX9|+MNP!N^KM6Ld0A$R;0ENxqBW)(zj!}(cW>YV;y;E+KqRxsYg#5VawKSg4TH~fiuPvJiF&R z`0i_Uakm{GowBLpVon_D-95h8DtW+y9Gt#O6$LtVxDsw@W0!3|QJ{Q$YT13zd;$t5 zi@Wem4;1yrnxmF8m+fs$RwFkgh{@xcq0UNI<_?;Tzl`civhV77%}L2{{^GEApxd~= zBOz_+yEIkS3(u4BF4CAl=1BU6bLekI@&Q7_DNT?Un~VK1l|Qzu z5C^A7auDXZLvGk?2Ym-;m-S&nF7UZwVMPUf!IU0n9*GWwcF0Oq@i#*T(T1BfpmoNf zYw-mNOMdE}GsW!bozC=0g=D8PnlXkUKKQ=AHbZg$e)yL?Q(B-b)9&rp=EG}E0e(=B zf&4v_`1_mBf#Tq;jvxM)YIBYj==O}CZ>*t<2MCc0-|6E*(+-);#FMT&G7-=jZu5GpXf3j$+aTDrgmq}22={Y1omq1Y%!9AP9M|&ff><}-wBje zmo~Q?2Ud9hn9aGLrrZO95u0FSW;v}jtEz6j-{N+T$}uW*gwJPx4&8vYyh38mZyOrZ z2w2r4=ZG?uzi>2&nSq5q*?z08)t~S|*0b*X0h>iRXqARI=zDRk!k#XQMIyg2qm8Yp z!Mnj%AL;|_jw~{#MC`A=UGUr`))@_WK7!{^3L)h19J{Q?5l31PB|sw?ZITjbg-klYrZD%?OL_8ebC(-XLt5l=`;# zg6B{r_QBI7QtHUEvr96-^s`x72{G&388_X^M5@OKkSAb+NZ~W=yx&SJQSr?%qD+Ih z`j=$xG78s?0jI$?)~m1a_IdL&QqxK|J?SyXYUC<5wfc)A5FbsfYVw2^Q}MK1hO9qz zqfPAf)I2&_CI&o=A8KLow%zD;G1U+wFCOrTMZ=VrchNUl5o>Qw3VBvv?mPt(7u*Ou z97$F>B^R0s8w;1laiq)nx!VYjd#Xc(9pOB7NKi0NcCHgnu;md2(5nY#bD8E3m105p zhNPq!T*23u8n;Eq@`MjO42SE=LIG+i{iH37{nH~a+=}jSvebn2-xuF(fiKLvSnD#< zj_MnfB(3X+X^t&mk=}HpoxFgjtXL&TG5V3S%OHpwCVN$QMg_ZiH=VGJrd}UUi6Kr- zo_^JM8fX9d{TKQnK3oL>*)b1OzP_3ZUlC@7>!vUdw+nF9z;vAbS1k@3Je#jQ8XWzW& z0&+t`Ln%a_(;k>4Z<#gATz56+5JuwW0<5f++VZ>^ zG{raEV_7y&arBtJ7zY9?yFB@2y`Juhv)LwuHqvmRYs)^Tdg~)1zFTfxTgL*odyc*( z(qyjF<7^TfN5e0fx7R$j(0C_8M1S(Y&TlV@Jg&|LR|`4E9id*tBX5|Rj+z{Xm3^1S zYNzA^v!@Z?vVFiT<*d$WgAlq+@+^O*@wqN7STR|)pI5yw1YH}uJq8oC@xqL7=9;qs z2h1EOH!Q?<2zVud;X3O5c(Hgtq3#uuJpLx za40iyQ!I{P6>+_qJ*E&X$%`5nnjX80i|K2w@1B5O`5w>dt$JhAw4f=A3TIRu40MAY zfo`L3Lt3ow1+K!D9e@ed?c)Zu%MKfHF*mL*<4IadB8D9o(O#S9qpJ@k*$1Q#%j-S5 z7AU*==D5v?M!xu0_UR7EIzOE!x=U8Ps@q}lw!LfLv^mqwzag$}3jUw=&blwEuYdP4 z3?)cNgS2!G-5@9k(j(mrASEdv-5?FpFo2+dlyr9uUDDkk-CbvUes#aj{e1-I9R6dk z*|B1;wf1LS*L&?P6FsgoE)<2cnggfws1-u5cDhlJ1FJS4j{U!%_3U1pMBE-fvb&n?wEj45vMpIx0iV!AGD zEH?De+Md4Sa2RU=v@m9FlU9X72oa~?I-c<#UbU`YBwUWW4$9MAu~CW+yG5u(co_cuJlwLF?yGy^DL zxU=_l|LmaG<)FX#IkcJl}}c#}Aw%M8Ha&#%&wUe4K%n z&p3}_1!oO>ZhkE;+lR{SN5iQ=3q~KPA$p{Gno%5&093)*(x`8HP5x%t+2GoOw;zOA zTMWe3A3pA;c?Y|hFMSh36vbkMsn3#PKBPbG*;CY1QZFjH4>ts(*y<8SJ(t)cy=9fS zov}CDu=ANK2I7bZD4dOtgP4+sjSuy@)6q-EEw}n-uOGIq$v>`VoXX<_G6w$4V%YBp zz?0}_-BtA_FLv(4d*uLothu>AdnQne$-gA}@`ETB$Bs(jE|;4t3$?2gyX>)3##ZrS zDmOj4^O26-?RA@|+bHJYK2w~n@R6kJR*4VARRINHqQ#ETcgsyXtx`uK?V78DUjE!@ zd2p`jd+*0QlC1Vb=aZcuH<@_+j&0&tW3f{m+xpsZ6hGLARA4Y4?%^h1{kaRzu5m*A@Ob zthc9sfYq^s^!jnNuJhY{%`0d=lI2-6E(t4oC3neV-Q}XO(r4DF)$4|X0-GBVyJqdi zhk2)7-U0ea>=QSj*CMxEbidwME#jNPx_l_#Dd-~SY#ga2eiVcS5t2EbR(Ie;+XXEh zHlHdv90X#}L=p>Q4vS*R$JH+<7)N@px##>wObyLhd_S7+n)^s>XnT9P(QrU^f)s$i zl__!8e3}d^zrKpwmGOFPzTONXw#6KNZ8lcp*pYe1d-5Z?wS=4(FjB5=2HfflH6O7|4gv%eXMl3vHv$=ny-Aby`JdSLatE_sVI^Sx}T!odl?9!Su%J)`q- zT9RXYrZnD8ctw`6bvh!nReAK<`K!j{%I?JsV?3JS*Y6GY^X2YMcRh3gU!7s>mx-=H z#a?-wHG0Rh>%EbV84YVut3ObPv7CF34I?|DK3thaLr}`k-#k5f9u7A>n~VyJrP4U2 z9s6_7C49ydrDu6G^dNN{ZtsRr#g_@jbW0k4*LE*DbZv=iUd3&;Qyp~nTHRkvDn>RM zdTgfXxc#A_;d+7oIp^+ZbPVt?{#{h=s`A)RDCz_{mG?a)4n%Rf*kYA!OdW6AwB}3( zT;(Or9XyuL#?rWTdJ((XxFk8~>zjO_!~A!g6q|#V54R71F{KYnqP>efghGS5)3HTi z@vabZ$Uy1%_S)rmuegSYk)u@dZ5)2Q3DR}o_2#Xz5R?=tFk9=`d8(`%n}yERxE5ty zi@7@$j4ecu{IS+Fgd&U_TkzXa9!?!Yl~oy6er-4_T648>6yl^Rh<#xsR1*iOdFGtD#7j*Jzt#HzAFw zqZG1~Ge7bDF$N^sQT$!mJO%pm4!nT3K6W?JcR`}z1A`KqN-F=R9C2WiV@m#G zgOK4IUJRA+<-E)-C!oDu9b|@*O?aGThlj-x>HKt{f}2Q5%9Ocxmw2l;UYv~`7f1Ol zd3WMPIu>4hkHS<#mHO6?j+GMXypbj3jlG6=CjUF%a*FvXKS8f((9<+^JI~eAq#Y|N zdURJ=VqX-0JELVS${r)`&rG7hY||ev)1!&jfoQvfB*h}>x=?=Y#Tz0JjMA^O@dEUg zE697lmr?rkct6wmxT`}C?)Z3*JkT(C%+MMRz=V()I7QJ~U<#d^C*CPnrYlR@xzk&g zRRK6I{lMse{53iB+;g$}$}E%ddWh?G511;W*nxsAIRllC+!q^NXH(3{jzk}q4g4fD z6|PR7M?Xn*L6D+dTN;ah;nYs4UkOmih(A=8AZS9VLT(af2&Sn`F^ z!{&p7{M@nK+i7x4oAS^FE6+6npNvbqv-YZvK!=KVoX$K0{JLx2*J!Yv7gTE?r*Q)3 zjy#Ek68$l8o40q5cJrhLUVCMTpeiq{Cj%;T^fv^V>K4xqkz3BjYmxRhp()M0%EEkQ z)#bw4{l`vCJD7&(+U%DTCBmN3(Dh;y|3dVKp0t|}n|C82YQ1Drzb>rr146a9UTU2{ zGaaq(_dY~cHr=SYHXH182G z+bx@YNVVn=xCB55Ax*b<1Zi3!m|{0hW3DaadTxI@(y82D_U>N(v@l}@3riW9qW7dQ ze5!|p`0CkUvp;Vh@idWoVm{iIS+eqO4`s=1N?!o%Z>;JudpkY2_PBI4>=Q*bVMjv^ zevaOF-Hu6P(JlQUlj082FL&fcKk~f!=88lbot4;g4|Y&DXNS`k$$#l5gUjLgDdj^j zWy7Cr>d$sNUhiOtKCr20mgb2Ah~v`oc!gWaA$VzG$Kb6z#`2GV*fxnGirHzjkW>H- zfhzW!9VsP((LJb@0(#%z1e*Q!^wV*B%+sD6ch`5447ztzCASEH7vkAWlqMz$0~Hsu zZHAvPBwW~hVFPEJy8(VHbm??H^;;bCR)xzv&C9Y9JCnC-R8Ddo&hb)L5%@-EZDbNE z(1)l`NN#Kg9ShH6RI5KjS4Wc45#~Q%NRAHLV)U45e_&@6$Mmj`urxu(^{w*MGM?}{ zz1~`=Ge`++aaCcsUUmT!_v1Mw2oivIO)?sH2+KOne1qjBbBlye6P`qOmRTgAQuHZ! z#TW%#-KUl(_E&J!|4dt+B8f+Pl#*L^o7Qh!7A5%Px74qFsOEC@fBYrpto2lAlf#FFA)uD)slV^gwM#t>ApgsD8%SslYN*%X4$N{ABkd$ zNfew&Vf>{$lw}y>$l@h|OSA{W^DYol&ul2aFxlMPuJHHdm3?GrW2tGM@ItQi$Gsok~!cU%DMZ(Nem4Vux)GeP!lBu-g; z8(a9#H+##+`%4@zyrrqbhu6!13DH?#;Zxl0GMdC67XU{v^ol9i@2@9r6+lNhmI34~ z0r6<@8;j!9&xpn&rodr!O?`D%9I=mt&% zQJwcZ9o#q5lD`j;xT9GY8dKmQ%s6IWm}vQq;>VcrIUD#=V<+Fe1GqX`4LfA`RaxYd zXv$*eey)^gVS-O)HyneQkVZ4P)6%o{si-1ED`$A?3NV4A3# zqghpxujU<)X{JOy$6kaWxqz)++tpRCvN!kU1=Gz10GF^?AzX|U&+xw44p-OE&JhIg zF05tcX(de;IxB?6NZ&u(x+B$1Tjr&Q0f?TZuhsYWK$SMAmW1LnR^K;LlBA*Um3Cko z>)AA&PwHEK+m+||Xue8+i&!={_B!-f(xv8|${GVAk?VZ>m19IDjVJQyY{K4~Ln4Q4 z50^fV0-lZ4#cOx*ez+vP=&hgJpDp+biFRP|U|msduEQtTn zIeRs@JA@mE0no(M(IUd5z0bZ^@lasE+B5xwogaW&Fy41L8w*I%bCg4Z4T`t)(1sH% zT_FU*05B3SM%!PYnhAhivW+125!R%Bvh@c5@2~@AvUC}DXngn55z$trk2gnn-!8gh z;uL^$y6mfLst`Hbnq{kzb_Y6G(*-~>50{+zOCxm!lqZw+V(V}Epa_LT!GCYrZ0 zJWz8&mMY8I1fG)l^ra5Mco!s}TxgIW&k|$SRuaG20x()>A1472gBlwEDnmYzbhBLx z6kR4&ai7eJrKP6wzFIN%&H!rMITf-uTpB!k18Nz2C%ju!A*BrnxgMt~Ok|pTvBqrT zU|B&#Pg@ujl2ypL`1MY7EprmAwnn!D{_+-ncGy%5YSdYL(uv9-26aR}!AgVVzz--t zFZ=n`8C95$Nu^;|w8Bk!?ttd6R@hfqT&>fC%>HN}yXow;uKL+Tu2oR-4}>Q4SS?AW zJ$c&dZ`k3Ah);Hzj_UetD<%SP{zZQTFb6agYhB`t`Ye}k{ZPWikWZU07d?oR487fs zf4}1-1iWP|0Blo;}yxb!m$;hRYyVJoL50=Gz0-zl0$Ez~({zIh*lQpMmttrb92| zfvnep0C?^x0nK7jV=aI;3p<}OY=KbX01N|D&A7?5`H)YG`z-KAC)#H;F_034SAOR+ z0KzGGAa7^&Endw2HzNR%mtS!3rRfr2MG4$g{Gu(JK>YQ_WU(P+N-W-E_WX6jYbhAP zl|_#`jefPSo(CC$f7?mrr&+lOA9%7u?2Djg7dZ!+n2}E!Ky#=ySh68_$UDap1<5d zM{TnbAE5tub`E&i>B}wMz1EUpQ5epWBR_=|51qwQHxBPBq{V80e%M?tWJCNH<)W4@00n8h0!;lAR}zfWXwoo~=P=nmdmb}ZJ3T?eQ-+Y3_9TQTOVjmB zb>;oi#*UldAN$K~narhmGytH9(jv|G`*t{PGb}3TSoEOWtU}?1EO7v_W1WnD#Z~o= zi?*77-g!!YUwpCZEOqMqTjycKLxA7v`$GT(;q+Z%HGu}$$uNTjb)dDX=u%b3Njmn;~H*ey7T zNQ8uej*YBVAK;(_d`&S4jr+sGW3!5@s$uIz67TTo703}RgC;>wtr;Nj5>y_or;W=S z(0lZk(>RWvN%^o9rwcDAG$5#1bhoC%8HnaulYHBS1$Np3@OZx@D^f7W{p@;CfdIca zmxTK3<{$TGc@h*1_nbc0{fLTO`p1&Tn-WA7{JQrqz^Zim`5qEgA)$sA4OrS2@{^W` z?9L0_ruuI$$aZFPV`1~xBd;M=;34(R8XY%2TnMv6^-bngC!>_`z}RdKKs(%8(RR{q z{Sa}|6OR9#Z%VJ?3LuTejT=CT$OWCMQp{6ZS-B*C@F8-C8fC(Vm*!${f3gA4LjCxW;5N|8GkFhf6Gp<%}d<5t7aQ5mr zI8PA*=q;}@0F#Nj2&zHvla_0f^C28XGfxZ$I>MIJPPNIk|=)dny< z=n*Gh`sO*?$LOk~ESmmD-BR$^MKm$LCuip9Y7T?7^kD0k7 zeca<=+Qh&PZLzkxL!4I3+_iqh^1Rn|l$wb7V8g*Z+}gg0Ii-kTu0i%j77q}Tb2p{^ z4awl5&isrvw(^5CF$QnO)%D7f(8XCHrhis_@z z{fw_UMq$=bd!C$ zGF3gtYCi$MVX~}_tpcM@29sFTiKc{)_sLa~AMikg=*g2ceQ0rRUD=4*BJQ{W!Va0v z3Ub?8V_X+i)EtBuraQ}x*ejX0%G8AXIWHixa@}%_j=MOVN!f9Lx=Tg$CQYP zcuC5nr5(HQB|q*$V$(j}cqL{NjyC(QPWM-7+7fDkZ$gdLLLdqoDfKcET0+V8YvN{) z-AlrrSJ&0G=Lbp`S#a0xNw_xaI?As%s zq8dmQEp_*{Q_GA9R$;v#vx8VDU6Zc&MI>$tRD9V-S@^;fUwB`r`QEzEQN6@@Vc?-x z#92`Kz1x9tq5=IC-Z}i!9?rg+wxkTHqCq~+(-222B(PGr%45o~Cxfo2Q~$hGM3S+D zasq5h?KtL0Gn%j{ak5#p$k1&gq1U|>ZHCfnyMJjAo2`#DZC6r=>Q;jAHs^qZZzE+x z|3QX73}Y&};KTCnPoFTE1mA&dD=bDok7?cr8^`yM*+l-!k7QQK+)v(|1`Q))^+^D% zDG(HguDL8J*`D<>7D9&z>WB78%uQ{ZM;F zF@oCO3ZcY9^pQhZojdCNw~W6sW3r3G$Q1>1i#N#?$asI6g^?2R|0aMi=2N+GQAN(C zOrZx>$2k2Ckf6858KpL+WqZ!#l&SbjL0YK9gPrjuVac#l~3FZ=;)?x)?6a%<6-wt=yCI!SW6lmnW#Rx53#TW=^vzG z$TZ=`%u46V^im#OesGpG6M8! zA}0y6DBBXHHSs*VBK#wc_@D{+MtJ7OsDbwx7f}p5iM}>r8l~+8W%~Y_d~nq$2ls9a zj`l482CU8XYu{V`3pdh?QpAdLXlyGp4bc0}dw9|Vj{w0o5?V;$ut{*6Kw z<@^(>S6!g;-sf79T*`vRKlaTjm4&nt5_=We%1n&+74artXedJiX=O(~c-k0B`B_i( zm8AdW+QN!`8GV(hpM8`xD0B7?crsbR?LezY5rb0-|Ndf=Y4T^T;O2uvH-)A*@2$iw z+O=;}RrC{=$zVqn>0i3_F5OIP$y?i0C%#nWj#t&vP?qqV!1TWFyuKf(pIOtEKTmX9 z_j{4>^nkwrY)Yg@7Qo|8YR_vj{}*6fo<6-73}IXEz_7N8;Zd#4WbNneNDlgePScu^ z_r1?FRwwjl3SEbvr_}7HsQjc;C3qwCZ3?{P03T0inNzb*WNm{=+PJ5b=?YJ$xLn6)j^5(RYO;7{%+j!48G(vXbFqVtK-9r(x2=uVZv(;hA9QKE}q1o-I%bP->*sf~;^7;ajXrX13x zRBaMjf_3jAOOw^Yl1>%xhcUu7y2xU=Kgu$jo^QlgDWsne6_Ji-_3d;7LVs`2ZnqQL zQ@x@#m&rfu7QZO7$3bwvsUAA9`9-B@Im*WIhQ;Ss+mwC%hS~LwDOMors!y4 z@p>vVahfIB;#1S>abNAtz$Lz8+|+e}isP#I)SD+p2MYRf&1}FIqOmQ|&Rp56yZBs= z1cRwSNqw-hvgcNQE-VJT}7~irtH0Zm9*GBm_#<#?s$kC+2I9#FLI>0ec_-V|uIhRO{2=JR9y?j}n?KN(Dme_Y`Cp zFj}HBsCiIj;TXhNie5fpTn=Q$_1Fr@!P@i|8D~Wj@Aop%RXKfuxTaa6E6j3g&qTDClXzZa1(Q$ z^1V+Np(Zb?k`+&6CItT_gIWr@y`|r92B2MUKvwzZLYb+ z^bISxyXB~)*bJ=&t1>9~=1ZjMTERY=8IU7K%Go58nv{sFf~j+XTCVO@PnYRBw?3~a zwu2MovZ12uAWufb*7H6`1YS>+MRhZ-JERFy;5U94zj&gAOzC;lCV2bn@u1o_LJclt zrh(6tls33htIAYpAu-&|A)gqG59`EoJ0U_brBablhi9QZxJDkiBy#h{VL*Tqk*Q7wM~w#Yf0mMbZ5!U7%;l)L;K}zl<9xLJs;G>w26?q#@4RFe*E2KWa*US4<*9%D zoNr&%Hh?>e$V`&uIk{QNAp}3h2&pbwk#`3aKneBaZo9|8X8>1Bo&>*;_`W3dEe-wK zxJeY(13HN+Q7|sqNK9%kO&*nVGCOQqP|Q|_p(t;jjt!mq7cpdo)dI`&AU$Dllkars zq*aqwAg|6=0kmYcGtnqxFKJEbvzH!Mq<-vtn|`9qHdS-ivH~Wm8^gg=_^M_Sf7*X9 z0n|LR@E7UWY}?4tX*+CV_%kZ_Zbmay6D+g4bu|~S!&LyBJ!r6qJwwv5585o(ml@KF zbsV*W`JtMZ^~uu{mD3VH&5qKYoI8<&C$ZS}{Ic6JTdKq(SnIw4TAG_C>ou8xLI?VPE*@h(G;4S*`L$uyS$qH-g&v@oE%RICM3& z#nqR}d?T`dh7E9sILOUFARh)(i$sS9KtK9E&0Jl;L#wWJV6VD72fb{3vZA!~yG%8- z=GQuIzLfT6g!|56025$WScF=)3)nP1zDRL7#6O^Uc_J%$pIkYC?2t}J(!7JaMKt*j zeA&uR(86Bd1mC>$Ta}#*(HjxI5g1gFH$?_v)WJs0me*r|8=WgE%YSWSUvrW^dj)CZ zdX~L`gp3lb9t(0h|LsgO!UEzF$?fV?$d^Cb2y90%9q<%dP6Bfr2Jg{zvnen6fd$Ig_{K&pO(dxM zZ1%|FT!auK#aG2em}D!!IR=G+rH1_>6I%W^XpF~7?`%SuN5$N-`?P0M99;wc%UQ}u zL0lbT!{)O%#(>3>}~8 zlWU^w!5+Z{N2(lX+w&siqEaz>@_(*sBmv@as2MGFQUalbY$i2tVgl~QmeXU+f7GIP zA&5Kib+Fg`5z*L@@G4u&Aq(4nQAd;=3Dkr zav7W#*%((efwfM@Q!61WG}s=VU)6b~MPe&IXz?%wD2Hu_d&wwT z+aO&ELj1CZRV&;S1NB(Rr@}NgKM~q4WK3~^!Hf|SE~{U^dGHl^6gS&G^S= zG2b}@y(kzzZ{tjx2enWy1emS~HdVFq3bhr@!9Y2T{$ye1B9Tb_!gzVcj>Va4c9_`# zUxk`&TRtwv-@ZJTh=AlF;8@z3_>u+cVX&}u1qPVm#P}TLYWdITl}TG_mH44bB)=xo z6mFKTqGKEV#dg}aJj0HrSUd{Gtk$@e1y^kIlb`h?w1@nV8g+)YGTb4VA8l=bRS_B~ zyM{!xx{LE0cM7DGG0_Uj_RD%GcQ9pPtGQ|4LOIt0D-tQq=whWLRgzaG^c*m_Ah128 zc(@DWTwo}jAnvv5Fa>R;bTy461Tn5`Yth@X7c^qO4Adt@_|cpgd#ceKRS=Y;eT?Ei zS!=>QIz?Z#*&V*k%V0uwqtI_4=x54*(!M?w36U2sM(;5{QA)%88hbd6mQtwo6u0NM zQYF;-*F+V|BEGX_f?gqpu3Wbz>oe<1>8;JXdbJJ2rmW3Nbpq??=~igx>g#ko6B-_U ze}|rwP)O|FClbT&1P;$_w7693<*;mckg~AaTfze+IdMx-FffBPdi>Xqp8C7=D7cUg zHxA{8ksSVzR*e_VH}Rguc+L=qjzMG}{j*TW%FLhwZ>CWO_mJV-%D{CDO(^QYQn2zy z3}6pdF9VaUcF$1@m=f3SMzZ8`>6f1IEPZ-u>?7M-YAE(?PE$d_i&#EQoL}9LH;yGi zQi5d;iB6W~-1a%8l@a$Y=u?~-Z&Wqjpr0d<*R%gjuCOzW1!fhvhNRts7{eD{PRd^q zio9fM$nP}Mz7;lpyV4+dWU<9#`TIvLS(SGUD>WT;>hGMHHibe$?i?Hr%(FQgEO$$zkb^x`KHlv_c!du#iEB}## zhFT$N;4$;hAy6fJEF0}cZh)v1=A@=H=l%+K@fHYT-1LKs7u1(>zb$tz{@xfUb0PsE zyv7U(=+bq^OS=%il;ZU9WAR1X7)YzLQchuC&D7+&iR+IH`dQIa%H4_=Faos3SKI1u z;BRB8JY_^i9m_h)&oL1^-8?GQ%-DF;j;aZ}M??`2%z^6;st2w)g-P*Qt&PgDx|6HhKqQO&O2&Z|A(Dy3gWcPcRu2>>Kt7lfR?G&M?qk=dLGjD6DfI%5aVzVX$hn9=HnSBJSQx^2Y} zA@a4kq8-JF0t3sr*cG%Bj1Ei>17FkW5$d5JG8^rSxhh_HdKdmoa0B+SnX{<~Y79+G zJE@QoLL(a3Jz;$x3^QfaDb6zzFjnQyJLM z+I5N`y7gy%F$>Y>SR;L0!bB1xI+%rAS0vM_=V8L9vn{eXxpMmP821=zM-eXQhHp=5 zlsTS22Y5Bo8GzTz$g$WnV>p)C_H7!>R!G6<*=j&Z8D1C*g8XR*ydkt{DLJm6z*y?t zOi>5CeS&_{9-DOb`bSalEPD$e#crEW8iTY97bEQ_D!X4`;3K}o#;Nt#N4AC=MM`Q{ zXK<}>&~uIF$EDP9hd8cDByrl3ZvK92-^Eg7Z&>M&QPDn-fDe4Ts(N>qiFN0C%Fb}Wun^woaV}f;0?!&v-t)nIc_!ea=L#zA@&Ohb} z*P&0T56_BuT)uF*g1`K`=aI4hG;O412!sy{ZqD$wjFf*{`)KWz)dpe-LKkVDdh(H4 z3?HXh@cKC$mWVrvm4Z;{Z9B^oB8{7zm>QvpK8exY203^@MM)48oO{rRz$ICYu2ui5Q*IvCig zh{he8*o(btgiH1bIo88b=RiTFJr&xzMwkalYb03Rjg(eMDd!{hj0KhyWXv$y#dlwH z;2vEHj4;sV^pO-(`P2)DL7CMwu!Py}b=kB&r>`KqV1|Q{WhY<<2~( zWZkkq?9gDNpQX62%3l?HX%<-_)Hz+0D($d%Awm?Bf{Y$A09ATTGpXEHdOR$u$Xw1Y z(VTKPKC^0FX|iv6{InJ8)kI(PsTo$vIZoTrY8^snD8uAZNF^LYd?`={LFE}d2%V%Q zFya@7zWTD(=-Bdc|4mk;THFIOG!GVkN7BGQ6#K|PKmT&?2-%D`BsCYy`qW~sPe+xZ(h{4ULW)0?{WkMZ&)3)d4c>==l?t9V zYd-A}?b`iM;T{8Z=KdCSdi>AG$CBexwFRbd`_}3Ze4r3My2|oI#-)u*%-jsS4+eAj zNeON36vIxx@;vcFU zAEVUE#tl|UB)=$fLlikaGc>y?4%;~*TK|xn7T*!#($1C?Dx50BzpdpH;aB+qk6LZO z!iMy1Sg@@`fbJqUJeMb{EB^!1ccK;0Rs+kI=0<{v^HR{&i zM}rxXv&mah7s-I~qi#C!fm3#tVdN#Cjy?InzkVd1m}w& zWk#^`_4FTb$_j`2l>cmT3x;rvwJM&yr2UA&n7urQMEjff0;T;@m`|ui`(xy*MrSa5cj^{A;Pgw3D$qjrIGKtpc#(-9g(Wo#FNf|M> zW%$%rVTlzuabh~Lj;?K-fU(nJi0_xhp|oh>N3xv6OnMr#TRF16v$!88+AS-B5~?q@ z8!c;3XP5fUjurd9MSA18c&}hA9edpqfu#L>m)^oZn09EL^}XB*(%B+Z8%o;|89%trqEDYv(RvD720pEoQslg4iBATds%xq zO)n9CnzKTOi1j%ycrEbn&Y>5Cf)oFN6T3(?%&vIN<)dtrDg~r#Zr0o}og!GXUpTeOhPa56==KkP zjkylFTS>SXV&yL%{#~-X!frK}k+&ySBt1?Dgj3GSH^%GQQg;?KK-lWj&aeo7Lcy}? z9CXjtUe=k0#i93Z`PMKQNhQ8t3D#H|b{8v+`D#}e=E=pPxx9dmFTx1iSf@Hot!>)0 zIqC-8!qd}L|HyEiy!}U9{7Y&*U%>(W%(6Mj};g5?K2r&J}Ws#0FT)nqQ`BFam!1am9M_tW{>V01j6ezK)GX>_x$PF_*^%$z4+(I zQ(JUts2`qMpVm5i<8|Zd=8u)vY-jnifFKzgNy*x?m1_%G6x>GQhh8C} zB+`hZxwMjn>LJS#8EQ<5x>z!Ctw|LmMnpsEBuOxGF9}PikzC#~eFn8rfZcmoQ}_p7 z>h6Wus*`s!fgYZr451x8wI$=ldqZi9S1TVHC~5e#hizxkLE^00XFvWF51bw?Uoc7y z4s2z5^CZ*a4u|IMb`;`?7M-geZ@Kdd z*kKbJiQjX4#(ox3ulr-xHOv?BIuCv4+652SZj{4Bo)a95gTyv~&MQBn>+kL?((~u2 zN`c-L*{_)(rU40l@_$Ny|Ekx^{))WICNixe|4kVzi9Y@s(4#9ZpC$i8S_C^FVva6; zcyIkrf^-Hzb8ZdF>;8u{BO5>-?q+kIr}Hl=`u~=X^Wnw#>fsBDF@XR5RM=mMGptZz zg6toJ(?3IjjHI@116tRAJC#KDudrYDK7aDRgzx`S6q1TSMy9|@(Lh1}pQpA?O9A5k z=^f=NnSam+ThRhKcLsZRg?~sx5CJ97t>II^_iq|_Kv|CuWQ5Oe4E7IcT-AUcTeSKH jT8sz!w`9Lm^gD^V!%#fT#iaBnz@OY3rPsftjQsx>gghFl literal 0 HcmV?d00001 diff --git a/skywire-specs/specifications/01-Overview.md b/skywire-specs/specifications/01-Overview.md new file mode 100644 index 0000000000..55d67b5e92 --- /dev/null +++ b/skywire-specs/specifications/01-Overview.md @@ -0,0 +1,19 @@ +# Overview + +Skywire is an decentralized network that attempts to replace the current internet. The *Skywire Network* is made up of physical *Skywire Nodes* which run the *Skywire-Visor*. There are currently two types of *Skywire Nodes*; *Skywire Visor* and *Setup Node*. + +Each *Skywire Visor* is represented by a unique public key. A direct line of communication between two *Skywire Visors* is called a *Transport*. Each *Transport* is represented by a unique *Transport ID* which is of a *Transport Type*, and the two *Skywire Visors* that are connected via the *Transport* are named the *Transport Edges*. + +A *Route* is unidirectional and delivers data units called *Packets*. It is made up of multiple hops where each hop is a *Transport*. Two *Routes* of opposite directions make a *Loop* when associated with the given *Ports* at each *Loop Edge*. *Loops* handle the communication between two *Skywire Apps* and are represented via the *Loop's* source and destination visor's public keys and the source and destination ports (similar to how TCP/UDP handles ports). + +A *Packet* is prefixed with a *Route ID* which helps *Skywire Visors* identify how the *Packet* is to be handled (either to be forward to a remote node, or to be consumed internally). Every *Skywire Visor* has a *Routing Table* that has the *Routing Rules* for that particular *Skywire Visor*. + +In summary, + +- *Transports* are responsible for single-hop communication between two *Skywire Visors* and are bidirectional. +- *Routes* are responsible for multi-hop communication between two *Skywire Visors* and are unidirectional. +- *Loops* are responsible for communication between two *Skywire Apps* and are bidirectional. + +There are many ways in which we can implement a *Transport*. Each unique method is called a *Transport Type*. + +Initially, we need to implement a MVP in which we assume that there are no malicious nodes in the network and discovery of routes, transports and nodes are to be done in a centralized manner. diff --git a/skywire-specs/specifications/02-HTTP_Authorization_Middleware.md b/skywire-specs/specifications/02-HTTP_Authorization_Middleware.md new file mode 100644 index 0000000000..1cb11aacb1 --- /dev/null +++ b/skywire-specs/specifications/02-HTTP_Authorization_Middleware.md @@ -0,0 +1,98 @@ +# HTTP Authorization Middleware + +Skywire is made up of multiple services and nodes. Some of these services/nodes communicate via restful interfaces, and some of the endpoints require authentication and authorization. + +As nodes in the Skywire network are identified via public keys, an appropriate approach to authentication and authorization is via public/private key cryptography. The curve to use is `secp256k1`, and when referenced by the RESTFUL endpoints, it is to be represented as a hexadecimal string format. + +These HTTP security middleware features should be implemented within the `/pkg/utils/httpauth` module of the `skywire` repository. This module not only provides server-side logic, but also client-side logic to make interaction with the server-side more streamlined. + +## Authorization Procedures + +To avoid replay attacks and unauthorized access, each remote entity (represented by it's public key) is assigned an *Security Nonce* by the `httpauth` module. The remote entity is required to sign the *Security Nonce* alongside the request body on every request. + +For each successful request, the next expected *Security Nonce* is to increment. The `httpauth` module is to provide an interface named `NonceStorer` to keep an record of "remote entity public key" to "next expected nonce" associations. The following is a proposed structure for `NonceStorer`; + +```golang +// NonceStorer stores Incrementing Security Nonces. +type NonceStorer interface { + + // IncrementNonce increments the nonce associated with the specified remote entity. + // It returns the next expected nonce after it has been incremented and returns error on failure. + IncrementNonce(ctx context.Context, remotePK cipher.PubKey) (nonce uint64, err error) + + // Nonce obtains the next expected nonce for a given remote entity (represented by public key). + // It returns error on failure. + Nonce(ctx context.Context, remotePK cipher.PubKey) (nonce uint64, err error) + + // Count obtains the number of entries stored in the underlying database. + Count(ctx context.Context) (n int, err error) +} +``` + +Take note that the only times the next-expected *Security Nonce* (for a given remote entity) is to increment, is when a successful request happens. + +Initially (when no successful requests has been processed for a given remote entity), the next expected *Security Nonce* should always be zero. When it is this value, the underlying database for the `NonceStorer` implementation should not need an entry for it. + +For every request that requires authentication and authorization in this manner, the structure `httpauth.Server` is to handle it. Specifically, it is to "wrap" the original `http.HandlerFunc` to add additional logic for checking the request. Consequently, the `httpauth.Client` appends the needed additional headers to the request. + +The following extra header values are required (`SW` stands for Skywire); + +- `SW-Public` - Specifies the public key (hexadecimal string representation) of the Skywire Node performing this operation. +- `SW-Nonce` - Specifies the incrementing nonce provided by this operation. +- `SW-Sig` - Specifies the of the signature (hexadecimal string representation) of the hash result of the concatenation of the Security Nonce + Body of the request. + +The `httpauth.Server` should also provide the `http.HandlerFunc` which obtains the next expected incrementing nonce for a given public key. This is required when a remote entity looses sync. A successful response of this call should look something of the following; + +```json +{ + "edge": "", + "next_nonce": 0 +} +``` + +The following is a proposed implementation of `httpauth.Server`; + +```golang +package httpauth + +// Server provides server-side logic for Skywire-related RESTFUL authorization and authentication. +type Server struct { + // implementation ... +} + +// NewServer creates a new authentication server with the provided NonceStorer. +func NewServer(store NonceStorer) *Server { + // implementation ... +} + +// WrapConfig configures the '(*Server).Wrap' function. +type WrapConfig struct { + // MaxHTTPBodyLen specifies the max body length that is acceptable. + // No limit is set if the value is 0. + MaxHTTPBodyLen int + + // PubKeyWhitelist specifies the whitelisted public keys. + // If value is nil, no whitelist rules are set. + PubKeyWhitelist []cipher.PubKey +} + +// Wrap wraps a http.HandlerFunc and adds authentication logic. +// The original http.HandlerFunc is responsible for setting the status code. +// The middleware logic should only increment the security nonce if the status code +// from the original http.HandlerFunc is of 2xx value (representing success). +func (as *Server) Wrap(config *WrapConfig, original http.HandlerFunc) http.HandlerFunc { + // implementation ... +} + +// HandleNextNonce returns a http handler that +func (as *Server) NextNonceHandler(remotePK cipher.PubKey) http.HandlerFunc { + // implementation ... +} +``` + +Take note that for the `(*Server).Wrap` function, we will need to define a custom `http.ResponseWriter` to obtain the status code (https://www.reddit.com/r/golang/comments/7p35s4/how_do_i_get_the_response_status_for_my_middleware/). + +The `httpauth.Client` implementation is responsible for providing logic for the following actions; + +- Keep a local record of the next expected *Security Nonce*. +- Adding security header values to a given request (`http.Request`). diff --git a/skywire-specs/specifications/03-Transport.md b/skywire-specs/specifications/03-Transport.md new file mode 100644 index 0000000000..84c5aafdc7 --- /dev/null +++ b/skywire-specs/specifications/03-Transport.md @@ -0,0 +1,124 @@ +# Transport + +A *Transport* represents a bidirectional line of communication between two *Skywire Nodes* (or *Transport Edges*). + +Each *Transport* is represented as a unique 16 byte (128 bit) UUID value called the *Transport ID* and has a *Transport Type* that identifies a specific implementation of the *Transport*. + +A *Transport* has the following information associated with it; + +- **Transport ID:** A `uuid.UUID` value that uniquely identifies the Transport. +- **Edges:** The public keys of the Transport's edge nodes (should only have 2 edges and the initiating edge should come first). +- **Type:** A `string` value that specifies the particular implementation of the *Transport*. +- **Public:** A `bool` that specifies whether the *Transport* is to be registered in the *Transport Discovery* or not. Only public transports are registered. +- **Registered:** A `int64` value that is the epoch time of when the *Transport* is registered in *Transport Discovery*. A value of `0` represents the state where the *Transport* is not (or not yet) registered in the *Transport Discovery*. + +This is a JSON representation of a *Transport Entry*; + +```json +{ + "t_id": "e1808c316b23d1d6119cad1795238ff0", + "edges": ["031d796272349d597d6d3130497ccd11cf8af12c7d186b1726358abfb49edad0c1", "03bd9724f335c5eb5a1011e7862d4af28488102c8edffc84585cf0826ac4864b38"], + "type": "messaging", + "public": true +} +``` + +## Transport Module + +In code, `Transport` is an interface, and can have many implementations. + +The interface used to generate *Transports* of a certain *Transport Type* is named *Transport Factory* (represented by a `transport.Factory` interface in code). + +The representation of a *Transport* in *Transport Discovery* is of the type `transport.Entry`. + +A `transport.Status` type contains the status of a given *Transport*. Each *Transport Edge* provides such status, and the *Transport Discovery* compares the two statuses to derive the final status. + +```golang +package transport + +// Transport represents communication between two nodes via a single hop. +type Transport interface { + + // Read implements io.Reader + Read(p []byte) (n int, err error) + + // Write implements io.Writer + Write(p []byte) (n int, err error) + + // Close implements io.Closer + Close() error + + // Local returns the local transport edge's public key. + Local() cipher.PubKey + + // Remote returns the remote transport edge's public key. + Remote() cipher.PubKey + + // Type returns the string representation of the transport type. + Type() string + + // SetDeadline functions the same as that from net.Conn + // With a Transport, we don't have a distinction between write and read timeouts. + SetDeadline(t time.Time) error +} + +// Factory generates Transports of a certain type. +type Factory interface { + + // Accept accepts a remotely-initiated Transport. + Accept(ctx context.Context) (Transport, error) + + // Dial initiates a Transport with a remote node. + Dial(ctx context.Context, remote cipher.PubKey) (Transport, error) + + // Close implements io.Closer + Close() error + + // Local returns the local public key. + Local() cipher.PubKey + + // Type returns the Transport type. + Type() string +} + +// Entry is the unsigned representation of a Transport. +type Entry struct { + + // ID is the Transport ID that uniquely identifies the Transport. + ID uuid.UUID `json:"tid"` + + // Edges contains the public keys of the Transport's edge nodes (the public key of the node that initiated the transport should be on index 0). + Edges [2]string `json:"edges"` + + // Type represents the transport type. + Type string `json:"type"` + + // Public determines whether the transport is to be exposed to other nodes or not. + // Public transports are to be registered in the Transport Discovery. + Public bool `json:"public"` +} + +// SignedEntry holds an Entry and it's associated signatures. +// The signatures should be ordered as the contained 'Entry.Edges'. +type SignedEntry struct { + Entry *Entry `json:"entry"` + Signatures [2]string `json:"signatures"` + Registered int64 `json:"registered,omitempty"` +} + +// Status represents the current state of a Transport from the perspective +// from a Transport's single edge. Each Transport will have two perspectives; +// one from each of it's edges. +type Status struct { + + // ID is the Transport ID that identifies the Transport that this status is regarding. + ID uuid.UUID `json:"tid"` + + // IsUp represents whether the Transport is up. + // A Transport that is down will fail to forward Packets. + IsUp bool `json:"is_up"` + + // Updated is the epoch timestamp of when the status is last updated. + Updated int64 `json:"updated,omitempty"` +} +``` diff --git a/skywire-specs/specifications/04-Transport_Discovery.md b/skywire-specs/specifications/04-Transport_Discovery.md new file mode 100644 index 0000000000..3055c02ebd --- /dev/null +++ b/skywire-specs/specifications/04-Transport_Discovery.md @@ -0,0 +1,349 @@ +# Transport Discovery + +The Transport Discovery is a service that exposes a RESTful interface and interacts with a database on the back-end. + +The database stores *Transport Entries* that can be queried using their *Transport ID* or via a given *Transport Edge*. + +The process of submitting a *Transport Entry* is called *Registration* and a Transport cannot be deregistered. However, nodes that are an *Edge* of a *Transport*, can update their *Transport Status*, and specify whether the *Transport* is up or down. + +Any state-altering RESTful call to the *Transport Discovery* is authenticated using signatures, and replay attacks are avoided by expecting an incrementing security nonce (all communication should be encrypted with HTTPS anyhow). + +## Transport Discovery Procedures + +This is a summary of the procedures that the *Transport Discovery* is to handle. + +**Registering a Transport:** + +Technically, *Transports* are created by the Skywire Nodes themselves via an internal *Transport Factory* implementation. The *Transport Discovery* is only responsible for registering *Transports* in the form of a *Transport Entry*. + +When two Skywire Nodes establish a Transport connection between them, it is at first, unregistered in the *Transport Discovery*. The node that initiated the creation of the Transport (or the node that called the `(transport.Transport).Dial` method), is the node that is responsible for initiating the *Transport Settlement Handshake*. + +If two nodes; **A** and **B** establish a *Transport* between them (where **A** is the *Transport Initiator*), **A** is then also responsible for sending the first handshake packet for the *Transport Settlement Handshake*. The procedure is as follows: + +1. **A** sends **B** a proposed `transport.Entry` and also **A**'s signature of the Entry (in the form of `transport.SignedEntry`). + +2. **B** checks the `transport.SignedEntry` sent from **A**; + + 1. The `Entry.ID` field should be unique (check via *Transport Discovery*). + 2. The `Entry.Edges` field should be ordered correctly and contain public keys of **A** and **B**. + 3. The `Entry.Type` field should have the expected Transport Type. + 4. The `Signatures` field should contain **A**'s valid signature in the correct location (in the same index as **A**'s public key in `Entry.Edges`). + 5. The `Registered` field should be empty. + +3. **B** then adds it's only signature to the `transport.SignedEntry` and registers it to the *Transport Discovery*. Both public and private Transports are registered in the *Transport Discovery* (however only public *Transports* are publicly available). + +4. **B** then informs **A** on the success/failure of the registration, or just that the `transport.SignedEntry` is accepted by itself (depending on whether the Transport is to be public or not). + +**Submitting Transport Statuses:** + +If a given *Transport* is public, the associated *Transport Edges* is responsible for submitting their individual *Transport Statuses* to the *Transport Discovery* whenever the follow events occur; + +- Directly after a *Transport* is first successfully registered in the *Transport Discovery*. +- Whenever the *Transport* comes online/offline (connected/disconnected). + +**Obtaining Transports:** + +There are two ways to obtain transports; either via the assigned *Transport ID*, or via one of the *Transport Edges*. There is no restriction as who can access this information and results can be sorted by a given meta. + +## Security Procedures + +**Incrementing Security Nonce:** + +An *Incrementing Security Nonce* is represented by a `uint64` value. + +To avoid replay attacks and unauthorized access, each public key of a *Skywire Node* is assigned an *Incrementing Security Nonce*, and is expected to sign it with the rest of the body, and include the signature result in the http header. The *Incrementing Security Nonce* should increment every time an" endpoint is called (except for the endpoint that obtains the next expected incrementing security nonce). An *Incrementing Security Nonce* is not operation-specific, and increments every time any endpoint is called by the given Skywire Node. + +The *Transport Discovery* should store a table of expected next *Incrementing Security Nonce* for each public key of a *Skywire Node*. There is an endpoint `GET /security/nonces/{public-key}` that provides the next expected *Incrementing Security Nonce* for a given Node public key. This endpoint should be publicly accessible, but nevertheless, the *Skywire Nodes* themselves should keep a copy of their next expected *Incrementing Security Nonce*. + +The only times an *Incrementing Security Nonce* should not increment is when: + +- An invalid request is submitted (missing/extra fields, invalid signature). +- An internal server error occurs. + +Initially, the expected *Incrementing Security Nonce* should be 0. When it is this value, the *Transport Discovery* should not have an entry for it. + +Each operation should contain the following extra header entries: + +- `SW-Public` - Specifies the public key of the Skywire Node performing this operation. +- `SW-Nonce` - Specifies the incrementing nonce provided by this operation. +- `SW-Sig` - Specifies the hex-representation of the signature of the hash result of the concatenation of the *Incrementing Security Nonce* + Body of the request. + +If these values are not valid, the *Transport Discovery* should reject the request. + +## Code Structure + +The code should be in the `skywire-services` repository. + +- `/cmd/transport-discovery/transport-discovery.go` is the main executable for the *Transport Discovery*. +- `/pkg/transport-discovery/api/` contains the RESTFUL API definitions. +- `/pkg/transport-discovery/store/` contains the definition of the `Storer` interface and it's implementations. +- `/pkg/transport-discovery/client/` contains the client library that interacts with the *Transport Discovery* server's RESTFUL API. + +## Database + +The *Transport Discovery* should work with a variety of databases and the following interfaces should be defined for such implementations; + +- `TransportStorer` should store *Transport Signed Entries* and it's associated *Transport Statuses*. +- `NonceStorer` should store expected *Incrementing Nonces*. + +## Endpoint Definitions + +The following is a summary of all the *Transport Discovery* endpoints. + +- `GET /security/nonces/edge:` +- `GET /transports/id:` +- `GET /transports/edge:` +- `POST /transports` +- `POST /statuses` + +All endpoints should include an `Accept: application/json` field and the response header should include an `Content-Type: application/json` field. + +All requests (except for obtaining the next expected incrementing nonce) should include the following fields. + +``` +Accept: application/json +Content-Type: application/json +SW-Public: +SW-Nonce: +SW-Sig: +``` + +### GET Incrementing Security Nonce + +Obtains the next expected incrementing nonce for a given edge's public key. + +**Request:** + +``` +GET /security/nonces/ +``` + +**Responses:** + +- 200 OK (Success). + ```json + { + "edge": "", + "next_nonce": 0 + } + ``` +- 400 Bad Request (Malformed request). +- 500 Internal Server Error (Server error). + +### GET Transport Entry via Transport ID + +Obtains a *Transport* via a given *Transport ID*. + +Should only return a single `"transport"` result. + +**Request:** + +``` +GET /transports/id: +``` + +**Responses:** + +- 200 OK (Success). + ```json + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "" + }, + "is_up": true, + "registered": 0 + } + ``` +- 400 Bad Request (Malformed request). +- 500 Internal Server Error (Server error). + +### GET Transport(s) via Edge Public Key + +Obtains *Transport(s)* via a given *Transport Edge* public key. + +**Request:** + +``` +GET /transports/edge: +``` + +**Responses:** + +- 200 OK (Success). + ```json + [ + { + "entry": { + "t_id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "is_up": true, + "registered": 0 + }, + { + "entry": { + "t_id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "is_up": false, + "registered": 0 + } + ] + ``` +- 400 Bad Request (Malformed request). +- 500 Internal Server Error (Server error). + +### POST Register Transport(s) + +Registers one or multiple Transports. + +**Request:** + +``` +POST /transports +``` + +```json +[ + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "signatures": [ + "", + "" + ] + }, + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "signatures": [ + "", + "" + ] + } +] +``` + +**Responses:** + +- 200 OK (Success). + ```json + [ + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "signatures": [ + "", + "" + ], + "registered": 0 + }, + { + "entry": { + "id": "", + "edges": [ + "", + "" + ], + "type": "", + "public": true + }, + "signatures": [ + "", + "" + ], + "registered": 0 + } + ] + ``` +- 400 Bad Request (Malformed request). +- 401 Unauthorized (Invalid signature/nonce). +- 408 Request Timeout (Timed out). +- 500 Internal Server Error (Server error). + +### POST Status(es) + +Submits one or multiple *Transport Status(es)* from the perspective of the submitting node. The returned result is the final *Transport Status(es)* determined by the *Transport Discovery* that is generated using the submitted *Transport Status(es)* of the two edges. + +When a Transport is registered, it is considered to be *up*. Then after, every time a node's *Status* is submitted, the *Transport Discovery* alters the final state *Status* with the following rules: + +- If there is only one edge's *Status* submitted, the final status is of that of the submitted *Status*. +- If there are two *Status*es submitted and they both agree, final *Status* will also be the same. +- If the two submitted *Status*es disagree, then the final *Status* is always *Down*. + +**Request:** + +``` +POST /statuses +``` + +```json +[ + { + "id": "", + "is_up": true + }, + { + "id": "", + "is_up": true + } +] +``` + +**Responses:** + +- 200 OK (Success). + ```json + [ + { + "id": "", + "is_up": true, + "updated": 0 + }, + { + "id": "", + "is_up": false, + "updated": 0 + } + ] + ``` +- 400 Bad Request (Malformed request). +- 401 Unauthorized (Invalid signature/nonce). +- 408 Request Timeout (Timed out). +- 500 Internal Server Error (Server error). diff --git a/skywire-specs/specifications/05-Messaging_System.md b/skywire-specs/specifications/05-Messaging_System.md new file mode 100644 index 0000000000..f8d8f4a62c --- /dev/null +++ b/skywire-specs/specifications/05-Messaging_System.md @@ -0,0 +1,514 @@ +# Messaging System + +The messaging system is an initial implementation of the `Transport` and associated interfaces. To work, the messaging system requires an active internet connection and is designed to be horizontally scalable. + +Three services make up the messaging system: *Messaging Client* (or *Client Instance*), *Messaging Server* (or *Service Instance*) and *Messaging Discovery*. + +*Messaging Clients* and *Messaging Servers* are represented by public/private key pairs. *Messaging Clients* deliver data to one another via *Messaging Servers* which act as relays. + +The *Messaging Discovery* is responsible for allowing *Messaging Clients* to find other advertised *Messaging Clients* via their public keys. It is also responsible for finding appropriate *Messaging Servers* that either "advertise" them, or "advertise" other *Messaging Clients*. + +``` + [D] + + S(1) S(2) + // \\ // \\ + // \\ // \\ + C(A) C(B) C(C) C(D) +``` + +Legend: +- ` [D]` - Discovery Service +- `S(X)` - Messaging Server (Server Instance) +- `C(X)` - Messaging Client (Client Instance) + +## Messaging System Modules + +There are two modules of the messaging system. + +- `messaging-discovery` contains the implementation of the *Messaging Discovery*. +- `messaging` contains the implementation of either a *Client Instance* or a *Server Instance* of a *Messaging Client* or *Messaging Server*. + +## Messaging Procedures + +This is a summary of the procedures that the *Messaging System* is to handle. + +**Advertising a client:** + +To be discoverable by other clients, a client needs to advertise itself. + +1. Client queries the Discovery to find available Servers. +2. Client connects to some (or all) of the suggested Servers. +3. Client updates it's own record in Discovery to include it's delegated Servers. + +**Client creates a channel to another client:** + +In order for two clients to communicate, both clients need to be connected to the same messaging server, and create a channel to each other via the server. + +1. Client queries discovery of the remote client's connected servers. The client will connect to one of these servers if it originally has no shared servers with the remote. +2. The client sends a `OpenChannel` frame to the remote client via the shared server. +3. If the remote client accepts, a `ChannelOpened` frame is sent back to the initiating client (also via the shared server). A channel is represented via two *Channel IDs* (between the initiating client and the server, and between the responding client and the server). The associated between the two channel IDs is defined within the server. +4. Once a channel is created, clients can communicate via one another via the channel. + +## Messaging Discovery + +The *Messaging Discovery* acts like a DNS for messaging instances (*Messaging Clients* or *Messaging Servers*). + +### Instance Entry + +An entry within the *Messaging Discovery* can either represent a *Messaging Server* or a *Messaging Client*. The *Messaging Discovery* is a key-value store, in which entries (of either server or client) use their public keys as their "key". + +The following is the representation of an Entry in Golang. + +```golang +// Entry represents an Instance's entry in the Discovery database. +type Entry struct { + // The data structure's version. + Version string `json:"version"` + + // A Entry of a given public key may need to iterate. This is the iteration sequence. + Sequence uint64 `json:"sequence"` + + // Timestamp of the current iteration. + Timestamp int64 `json:"timestamp"` + + // Public key that represents the Instance. + Static string `json:"static"` + + // Contains the node's required client meta if it's to be advertised as a Messaging Client. + Client *Client `json:"client,omitempty"` + + // Contains the node's required server meta if it's to be advertised as a Messaging Server. + Server *Server `json:"server,omitempty"` + + // Signature for proving authenticity of of the Entry. + Signature string `json:"signature,omitempty"` +} + +// Client contains the node's required client meta, if it is to be advertised as a Messaging Client. +type Client struct { + // DelegatedServers contains a list of delegated servers represented by their public keys. + DelegatedServers []string `json:"delegated_servers"` +} + +// Server contains the node's required server meta, if it is to be advertised as a Messaging Server. +type Server struct { + // IPv4 or IPv6 public address of the Messaging Server. + Address string `json:"address"` + + // Port in which the Messaging Server is listening for connections. + Port string `json:"port"` + + // Number of connections still available. + AvailableConnections int `json:"available_connections"` +} +``` + +**Definition rules:** + +- A record **MUST** have either a "Server" field, a "Client" field, or both "Server" and "Client" fields. In other words, a Messaging Node can be a Messaging Server Node, a Messaging Client Node, or both a Messaging Server Node and a Messaging Client Node. + +**Iteration rules:** + +- The first entry submitted of a given static public key, needs to have a "Sequence" value of `0`. Any future entries (of the same static public key) need to have a "Sequence" value of `{previous_sequence} + 1`. +- The "Timestamp" field of an entry, must be of a higher value than the "Timestamp" value of the previous entry. + +**Signature Rules:** + +The "Signature" field authenticates the entry. This is the process of generating a signature of the entry: +1. Obtain a JSON representation of the Entry, in which: + 1. There is no whitespace (no ` ` or `\n` characters). + 2. The `"signature"` field is non-existent. +2. Hash this JSON representation, ensuring the above rules. +3. Create a Signature of the hash using the node's static secret key. + +The process of verifying an entry's signature will be similar. + +### Store Interface + +The underlying database of the *Messaging Discovery* is a key-value store. The `Store` interface allows many databases to be used with the *Messaging Discovery*. + +```golang +type Store interface { + // Entry obtains a single messaging instance entry. + // 'static' is a hex representation of the public key identifying the messaging instance. + Entry(ctx context.Context, static string) (*Entry, error) + + // SetEntry set's an entry. + // This is unsafe and does not check signature. + SetEntry(ctx context.Context, entry *Entry) error + + // AvailableServers discovers available messaging servers. + // Obtains at most 'maxCount' amount of available servers obtained randomly. + AvailableServers(ctx context.Context, maxCount int) ([]*Entry, error) +} +``` + +### Endpoints + +Only 3 endpoints need to be defined; Get Entry, Post Entry, and Get Available Servers. + +#### GET Entry +Obtains a messaging node's entry. +> `GET {domain}/discovery/entries/{public_key}` + +**REQUEST** + +Header: +``` +Accept: application/json +``` + +**RESPONSE** + +Possible Status Codes: +- Success (200) - Successfully updated record. + - Header: + ``` + Content-Type: application/json + ``` + - Body: + > JSON-encoded entry. +- Not Found (404) - Entry of public key is not found. +- Unauthorized (401) - invalid signature. +- Internal Server Error (500) - something unexpected happened. + +#### POST Entry +Posts an entry and replaces the current entry if valid. +> `POST {domain}/discovery/entries` + +**REQUEST** + +Header: +``` +Content-Type: application/json +``` +Body: +> JSON-encoded, signed Entry. + +**RESPONSE** + +Possible Response Codes: +- Success (200) - Successfully registered record. +- Unauthorized (401) - invalid signature. +- Internal Server Error (500) - something unexpected happened. + +#### GET Available Servers +Obtains a subset of available server entries. +> `GET {domain}/discovery/available_servers` + +**REQUEST** + +Header: +``` +Accept: application/json +``` + +**RESPONSE** + +Possible Status Codes: +- Success (200) - Got results. + - Header: + ``` + Content-Type: application/json + ``` + - Body: + > JSON-encoded `[]Entry`. +- Not Found (404) - No results. +- Forbidden (403) - When access is forbidden. +- Internal Server Error (500) - Something unexpected happened. + +### Messaging Discovery Client Library + +The module is named `client`. It contains a `HTTPClient` structure, that defines how the client will interact with the *Messaging Discovery* API. + +A new `HTTPClient` object can be instantiated using the public function `New(address string)`. + +It exposes the following public methods: + +```go +// Entry retrieves an entry associated to the given public key from the discovery server. +func (*HTTPClient) Entry(ctx context.Context, static string) (*Entry, error) { + // definition ... +} + +// SetEntry tries to set the given entry on the discovery server. It must be signed. +// If the entry is modifying a previous one, must be signed by the same private key. +func (*HTTPClient) SetEntry(ctx context.Context, entry *Entry) error { + // definition ... +} + +// AvailableServers gets a list of server entries from the skywire discovery server. +// The amount is determined by the discovery server. +func (*HTTPClient) AvailableServers(ctx context.Context) ([]*Entry, error) { + // definition ... +} +``` + +The module also provides public functions to instantiate valid `Entry` objects. + +### Messaging Discovery Integration Tests + +> **TODO:** Fix wording. + +This package does not uses another `messenger` package, so integration tests are defined for the external services that `discovery` is using. In this case, the external store and `discovery` itself for testing the `client` library. + +The cases for the store integration testing: + + 1. Its able to set an entry on the database without error by calling the `storer.SetEntry` method. + 2. Its able to retrieve the previously set entry by calling the `storer.Entry` method. + 3. Creates multiple service entries and store them by calling `storer.SetEntry`, then it should be able to retrieve them with `store.AvailableServers`. + 4. `store.AvailableServers` receives a `maxCount int` argument. We also test it passing an integer which value is less than the amount of server entries we have set in the database, it should return this exact amount of server entries. + 5. Same as in number 4, but we set `maxCount` to number bigger than the number of server entries we have set, now we should get an slice of the size of the server entries we have set. + +In order to run the test we preferably create a clean new instance of the store database using Docker, and the test code should connect to it. We remove it after we have tested. + +In order to test the client library we do integration test with an instance of the discovery server. +The test cases for the client integration testing: + +1. By using the method SetEntry the client can set a new entry on the discovery server. +2. By using the method SetEntry the client can update a previously set entry on the discovery server. +3. If using SetEntry to update a previously set Entry, but the new sequence is not the previous sequence + 1 it should return an error with status code 500, Something unexpected happened. +4. If using SetEntry to update a previously set Entry, but the signature of the new entry has been made by a different secret key it should return an error with status code 401, Invalid signature. +5. By calling the method Entry with the public key of a previously set Entry it should return that entry. +6. By calling the method Entry with the public key of a previously non-set Entry it should return an error with code 404, Entry of public key is not found. +7. By calling the method AvailableServers when there are previously set server entries it should return them. + +## Messaging Link + +The `link` provides two *Messaging Instances* a means to establish a connection with one another, and also handle a pool of connections. + +Using the *Messaging Discovery*, a *Messaging Instance* can discover other instances via only their public key. However, a *Link* requires both a public key and an address:port. + +Data sent via a *Link* is encapsulated in *Frames*. A *Link* is implemented using a TCP connection. + +### Link Handshake Frames + +When setting up a *Link* between two instances, the instance that initiates is called the *Initiator* and the instance that responds is called the *Responder*. Each instance is represented by a public key. + +To set up a *Link*, the *Initiator* first dials a TCP connection to the listening *Responder*. Once the TCP connection is established, the *Responder* sends the first *Frame*. It is expected that the *Initiator* knows the public key of the *Responder* + +Given a situation where instances 'A' and 'B' are to establish a link with one another (where 'A' is the initiator), the following *Frames* are delivered to perform a handshake. + +Link Handshake Frames are to be in JSON format. + +**Link Handshake Frame 1 (A -> B):** + +```json +{ + "version": "0.1", + "initiator": "036b630436972743d4b3f5bb39cd451da29d222b5d30893684a40f34a66c692157", + "responder": "02e18998f0174631710e47052927e13bddc712ca0c11289e0150fbf57570e31151", + "nonce": "853ea8454d11bd3b59cb31f8572a3779" +} +``` + +The initiator is responsible for sending the first frame. +- `"version"` specifies the version of messaging protocol that the initiator is using (`"0.1"` for now). +- `"initiator"` should contain the hex representation of the public key of the initiator (the instance that is sending the first handshake frame). +- `"responder"` should contain the hex representation of the public key of the expected responder (the responder should disconnect TCP if this is not their public key). +- `"nonce"` is the hex-string representation of a 16-byte nonce that the responder should sign (alongside the initiator's public key) to check authenticity of the responder and whether the responder. + +**Link Handshake Frame 2 (B -> A):** + +```json +{ + "version": "0.1", + "initiator": "036b630436972743d4b3f5bb39cd451da29d222b5d30893684a40f34a66c692157", + "responder": "02e18998f0174631710e47052927e13bddc712ca0c11289e0150fbf57570e31151", + "nonce": "853ea8454d11bd3b59cb31f8572a3779", + "sig1": "df8a978f0ea681e218cfd8127692dbe4190441567181b9057ab15da34b08ff610d9060e5195419e1744bb57d50373c1dd444b5c2753a80dba32b292fa306e9df01" +} +``` + +This frame allows the responder agree with the initiator and prove it's ownership of it's claimed public key. + +The `"sig1"` field contains a hex representation of the result of signing the concatenation of the version, initiator, responder and nonce fields. Note that before concatenation, hex representations should be decoded and the concatenation result needs to be hashed before being signed. `"sig1"` should be signed by the responder. + +**Link Handshake Frame 3 (A -> B):** + +```json +{ + "version": "0.1", + "initiator": "036b630436972743d4b3f5bb39cd451da29d222b5d30893684a40f34a66c692157", + "responder": "02e18998f0174631710e47052927e13bddc712ca0c11289e0150fbf57570e31151", + "nonce": "853ea8454d11bd3b59cb31f8572a3779", + "sig1": "df8a978f0ea681e218cfd8127692dbe4190441567181b9057ab15da34b08ff610d9060e5195419e1744bb57d50373c1dd444b5c2753a80dba32b292fa306e9df01", + "sig2": "fc17928d5a3f7691434282fb3108d1603889f996e8e45adc2e35362e08009b8611abb9f45e511b9931f0f04b37ff1057fd69554befe534ad28c77ff0c44121ab00" +} +``` + +This frame allows the initiator to inform the responder that `"sig1"` is accepted, and to prove the initiator's ownership of it's public key. + +`"sig2"` is the signature result of the concatenation of the version, initiator, responder, nonce and sig1 fields. Concatenation rules are the same as that of `"sig1"`. + +**Link Handshake Frame 4 (B -> A):** + +```json +{ + "version": "0.1", + "initiator": "036b630436972743d4b3f5bb39cd451da29d222b5d30893684a40f34a66c692157", + "responder": "02e18998f0174631710e47052927e13bddc712ca0c11289e0150fbf57570e31151", + "nonce": "853ea8454d11bd3b59cb31f8572a3779", + "sig1": "df8a978f0ea681e218cfd8127692dbe4190441567181b9057ab15da34b08ff610d9060e5195419e1744bb57d50373c1dd444b5c2753a80dba32b292fa306e9df01", + "sig2": "fc17928d5a3f7691434282fb3108d1603889f996e8e45adc2e35362e08009b8611abb9f45e511b9931f0f04b37ff1057fd69554befe534ad28c77ff0c44121ab00", + "accepted": true +} +``` + +Sent by the responder, this frame concludes the handshake if the value of `"accepted"` is `true`. + +### Messaging Frames + +After the handshake phase, frames have a reoccurring format. These are the *Messaging Frames* of the messaging system. + +``` +| FrameType | PayloadSize | Payload | +| 1 byte | 2 bytes | ~ bytes | +``` + +- The `Type` specifies the frame type. Different frame types are used for opening and closing channels as well as sending packets via the channels. +- `PayloadSize` contains an encoded `uint16` value that represents the Payload's length (the max size is 65535). +- `Payload` has a length determined by `PayloadSize`. + +The following is a summary of the frame types. + +| FrameTypeValue | FrameTypeName | FrameBody | +| -------------- | ------------- | --------- | +| `0x0` | `OpenChannel` | ChannelID + RemoteStatic + NoiseMessage1 | +| `0x1` | `ChannelOpened` | ChannelID + NoiseMessage2 | +| `0x2` | `CloseChannel` | ChannelID | +| `0x3` | `ChannelClosed` | ChannelID | +| `0x4` | `Send` | ChannelID + CipherText | + +The `FrameBody` has the following sub-fields. A `FrameBody` with multiple sub-fields have the sub-fields concatenated. + +- The `ChannelID` sub-field is represented by a single byte. This restricts a *Client Instance* to have at most 256 channels via a single *Server Instance*. +- The `RemoteStatic` sub-field is represented by 33 bytes. It contains a public key of a remote *Client Instance*. +- `NoiseMessage1` and `NoiseMessage2` are both represented by 49 bytes. It contains the noise handshake messages for establishing symmetric encryption between the two client instances of the channel. The noise handshake pattern used is KK. +- The `CipherText` sub-field is the only sub-field with a modular length. It contains size of the encrypted payload followed by payload that is to be delivered. + +### Noise Implementation in Channels + +As stated above, a channel is established using the `OpenChannel` and `ChannelOpened` frames. Then, after a channel is established, the two *Client Instances* of the channel can communicate with each over via `Send` frames (which includes a `CipherText` component). + +The protocol used to establish the symmetric encryption of the `CipherText` is the [Noise Protocol](http://noiseprotocol.org/). + +The curve used will be `secp256k1` for the key pair, and `chacha20poly1305` will be used for the symmetric encryption itself. + +Note that, the noise protocol requires the public key length and the ECDH result length (shared secret) to be equal. Because for `secp256k1`, public keys have a length of 33, and the ECDH result has a length of 32, so an empty byte (`0x0`) should be appended to all generated ECDH results. Hence, the `DHLEN` constant for the noise protocol should be 33. + +After the handshake, the CipherState object will be used by the *Client Instances* to encrypt and decrypt the `CipherText` contained within the `Send` frame. + +**Handshake pattern:** + +Only the `KK` [interactive handshake pattern (fundamental)](http://noiseprotocol.org/noise.html#interactive-handshake-patterns-fundamental) will be supported. + +``` +-> s +<- s +... +-> e, es, ss +<- e, ee, se +``` + +The `-> e, es, ss` message is the `NoiseMessage1` of a `OpenChannel` frame, while the `<- e, ee, se` message is the `NoiseMessage2` of a `ChannelOpened` frame. + +### Implementation in Code + +Within the `messaging` module: + +- `Link` structure should represent a link between two instances. +- `Pool` structure should handle multiple `Links` (with different instances). +- `Client` which implements a *Client Instance*. +- `Server` which implements a *Server Instance*. + +*Client Instances* communicate with each other via a *Server Instance* (which acts as a relay). + +Both structs will use `link.Pool` to handle links, but *Frames* are handled differently. *Client Instances* are to implement `TransportFactory` while a *Server Instance* is not required to. A *Client Instance* should also represent an established *Channel* as a `Transport` implementation. + +### Configuring an Instance + +When creating an *Instance*, the following options should be available via the following structure. + +```golang +// Config configures an instance. +type Config struct { + // Public determines whether the instance is to advertise itself to the messaging discovery servers. + Public bool + + // DiscoveryAddresses contains the messaging discovery services to be used (in order of preference). + DiscoveryAddresses []string +} +``` + +The above structure is to be an input when creating a *Server Instance* or a *Client Instance*. + +### Instance Interaction with Messaging Discovery + +On startup `Server` that is supposed to be publicly available should register itself in messaging discovery. To do so it first has to fetch current version of an `Entry`, if entry doesn't exist it should create one. If entry exists it may update it if necessary. + +On startup `Client` may connect to necessary amount of servers by fetching list of available servers from the discovery. Once initial connections are established `Client` should update discovery entry to advertise it's relays. + +### Channel Management + +The following is a proposal of how a Channel can be represented in code. + +```golang +// Channel represents a channel that implements Transport. It can be from the perspective of a Server or Client Instance. +type Channel struct { + // ChannelID represents the ID that is associated of the adjacent link. + ChannelID uint8 + + // Destination is the public key of the instance that is the final destination. + // This should always contain the public key of a client instance (as a server cannot be the final destination). + Destination cipher.PubKey + + // Link contains the adjacent link of the channel. + Link *link.Link +} +``` + +Both the client and server instances needs to manage channels. Channels are associated with a channel ID and also the public key(s) of the remote instances that the channel interacts with. Channels are hence identified by *Link* + *Channel ID*. + +From the perspective of a *Client Instance*, the assignment of *Channel IDs* are unique to a given link with a *Server Instance*. For example, let's say client 'A' is connected with server 'B' and server 'C', hence we have links 'AB' and 'AC'. We can have 'AB' and 'AC' share the same Channel ID, but because the channel itself is associated with a different link, they are considered different channels. + +From the perspective of a *Server Instance*, the assignment of *Channel IDs* are unique to a given link with a *Client Instance*. + +### Opening a Channel + +A channel in it's entirety handles the communication between two client instances via a server instance (which acts as a relay). Within the link between a single client instance and the server instance, a channel is represented using a *Channel ID*. The *Channel ID* of the two *links* of the same "channel" can be different, and the *Server Instance* is responsible for recording this association of the *Channel IDs* (coupled with the client instance's public key). + +When a *Client Instance* wishes to communicate with another *Client Instance*, it is responsible for initiating the creation of a channel. To do so, t sends a `OpenChannel` frame to the *Server Instance* in which: + +- `ChannelID` contains a ChannelID that the client wishes to associate with the channel. +- `RemoteStatic` contains the public key of the remote *Client Instance* that the local client wishes to communicate via this channel. +- `NoiseMessage1` is the first noise handshake message (the handshake pattern used is KK). + +If the *Server Instance* wishes to reject the request to open channel, it can send a `ChannelClosed` frame back to the initiating client with the `ChannelID` sub-field containing the value of the channel ID suggested by the initiating client. + +If the *Server Instance* wishes to go forward with opening of a channel, it sends a `OpenChannel` frame to the second *Client Instance*, in which `ChannelID` is an ID that's unique between the server and the second client and public key of the first client. + +If the second *Client Instance* wishes to reject the request, it can send a `ChannelClosed` frame back to the server, and the server can subsequently send a `ChannelClosed` frame to the initiating client (the `ChannelID` sub-fields of these `ChannelClosed` frames should be the unique channel IDs of the associated links). + +If the second *Client Instance* accepts the request, it sends a `ChannelOpened` back to the *Server Instance* (with the `NoiseMessage2`). Subsequently, the *Server Instance* sends a `ChannelOpened` back to the initiating client (the `ChannelID` sub-fields of these `ChannelOpened` frames should be the unique channel IDs of the associated links). + +### Closing a Channel + +A *Client Instance* can safely close any of it's channels by sending a `CloseChannel` (with the associated `ChannelID`) to the *Server Instance*. + +After a *Client Instance* sends a `CloseChannel`, no more frames are to be sent by that instance. However, the remote instance can still send frames until it receives the `CloseChannel` to it. The "close-responding" client then sends a `ChannelClosed` instance back to the "close-initiating" client. Once the `ChannelClosed` channel is sent by the "close-responding" client, it will no longer send or receive frames. Once the "close-initiator" receives the `ChannelClosed` frame. it will no longer receive frames. + +In summary, + +- When a client instance sends a `CloseChannel` frame, the channel is "partially-closed" and the client instance will only receive and not send via the channel. If a `ChannelClosed` frame is not received after a given timeout, the channel sends a `ChannelClosed` itself and the channel is "fully-closed". +- When a client instance receives a `CloseChannel` frame, it delivers a `ChannelClosed` frame and the channel is "fully-closed" and the client will no longer receive or send via the channel. +- When a client instance receives a `ChannelClosed` frame, the channel is "fully-closed". + +### Handling Disconnections + +In any given situation, there may be a possibility that the *Server Instance* unexpectedly disconnects with a *Client Instance*, or that a *Client Instance* unexpectedly disconnects with a *Server Instance*. This should directly affect the channels associated with the *Link* in question. + +When a *Client Instance* detects that a *Server Instance* has disconnected from it. All associated channels with that *Server Instance* should be closed. When a channel closes, the associated *Transport* should also be closed. + +When a *Server Instance* detects a disconnection from a *Client Instance*, it should send a `ChannelClosed` frame to all the other *Client Instances* that shares a channel with the disconnected client. After so, the *Server Instance* should dissociate all relations with the closed channels. diff --git a/skywire-specs/specifications/06-Packets.md b/skywire-specs/specifications/06-Packets.md new file mode 100644 index 0000000000..70b92d6438 --- /dev/null +++ b/skywire-specs/specifications/06-Packets.md @@ -0,0 +1,212 @@ +# Packets + +The *Node Module* handles data encapsulated within data units called *Packets*. *Packets* can be grouped within the following categories based on their use-case; + +- ***Settlement Packets*** are used by the *Transport Manager* to "settle" Transports. Settlement, allows the two nodes that are the edges of the transport to decide on the *Transport ID* to be used, and whether the Transport is to be public. Only after a *Transport* is settled, can the *Router* have access to the Transport. + + *Settlement Packets* contain `json` encoded payload. + +- ***Foundational Packets*** are used by a *Router* to communicate with a remote *Setup Node* and is used for setting up, establishing and destroying routes. + + *Foundational Packets* are prefixed by 3 bytes: the packet size (2 bytes) and a Type (1 byte) that contains the foundational packet type. + +- ***Data Packets*** are Packets that are actually used to encapsulate data delivered between two Apps. + + *Data Packets* are prefixed by 6 bytes; including the packet size (2 bytes) and the Route ID (4 bytes) which can have any value other than `0x00` or `0x01`. + +- ***Loopback Packets*** are packets that are consumed locally by the node. + + *Loopback Packets* are structurally similar to data packets but their Route ID links to a rule that specifies which app to forward the packet to. + +## Settlement Packets + +After a Transport is established between two nodes, the nodes needs to decide on the *Transport ID* that describes the Transport and whether the Transport is to be public or private (public Transports are to be registered in the *Transport Discovery*). This process is called the *Settlement Handshake*. + +The Packets of this handshake contain `json` encoded messages. + +*Settlement Handshake* packets do not need a field for Packet-type are they are expected in a specific order. + +- Request to settle transport is sent by the *Transport Initiator* to the *Transport Responder* after a *Transport* connection is established. + + JSON Body: Contains a `transport.SignedEntry` structure with the *Transport Initiator*'s signature. + +- *Transport Responder* should validate submitted `transport.SignedEntry`, and if entry is valid it should add sign it and perform transport registration in transport discovery. If registration was successful responder should send updated `transport.SignedEntry` back to initiator. + + JSON Body: Contains a `transport.SignedEntry` structure with signatures from both the *Transport Initiator* and the *Transport Responder*. If the transport is registered in *Transport Discovery*, the `SignedTransport.Registered` should contain the epoch time of registration. + +If transport will fail at any step participants can chose to stop handshake procedures and close corresponding transport. Transport disconnect during the handshake should be handled appropriately by participants. Optional handshake timeout should also be supported. + +## Foundational Packets + +Foundational packets are used for the communication between *App Nodes* and *Setup Nodes*. + +The *Setup Node* is responsible for fulfilling Route initiating and destroying requests by communicating with the initiating, responding and intermediate nodes of the proposed route. + +The following is the expected format of a Foundational Packet; + +``` +| Packet Len | Type | JSON Body | +| 2 bytes | 1 byte | ~ | +``` + +- ***Packet Len*** specifies the total packet length in bytes (exclusive of the *Packet Len* field). +- ***Type*** specifies the *Foundational Packet Type*. +- ***JSON Body*** is the packet body (in JSON format) that is unique depending on the packet type. + +**Foundational Packet Types Summary:** + +| Type | Name | +| ---- | ---- | +| 0x00 | `AddRules` | +| 0x01 | `RemoveRules` | +| 0x02 | `CreateLoop` | +| 0x03 | `ConfirmLoop` | +| 0x04 | `CloseLoop` | +| 0x05 | `LoopClosed` | +| 0xfe | `ResponseFailure` | +| 0xff | `ResponseSuccess` | + +### `0x00 AddRules` + +Sent by the *Setup Node* to all *Nodes* of the route. This packet informs nodes what rules are to be added to their internal routing table. + +**JSON Body:** + +```json +[, ] +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with + ```json + [, ] + ``` + +### `0x01 RemoveRules` + +Sent by the *Setup Node* to *Node* of the route. + +**JSON Body:** + +```json +["", "rid-2"] +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with + ```json + [, ] + ``` + +### `0x02 CreateLoop` + +Sent by the *Route Initiator* to a *Setup Node* to have a *Loop* created. + +**JSON Body:** + +```json +{ + "local-port": , + "remote-port": , + "forward": [ + { + "from": "", + "to": "", + "tid": "" + }, + { + "from": "", + "to": "", + "tid": "" + } + ], + "reverse": [ + { + "from": "", + "to": "", + "tid": "" + }, + { + "from": "", + "to": "", + "tid": "" + } + ], + "expiry": "" +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty payload. + +### `0x3 ConfirmLoop` + +Sent by the *Setup Node* to Responder and Initiator *Node* to confirm notify about route in opposite direction. + +**JSON Body:** + +```json +{ + "remote-pk": "", + "remote-port": , + "local-port": , + "resp-rid": +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty payload. + +### `0x4 CloseLoop` + +Sent by a Responder or Initiator *Node* to a *Setup Node* to notify about closing a loop locally. + +**JSON Body:** + +```json +{ + "port": "", + "remote": { + "port": , + "pk": + } +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty payload. + +### `0x5 LoopClosed` + +Sent by a *Setup Node* to a Responder or Initiator to notify about closed loop on the opposite end. + +**JSON Body:** + +```json +{ + "port": "", + "remote": { + "port": , + "pk": + } +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty payload. + + +## Data Packets + +The follow is the structure of a *Data Packet*. + +``` +| Packet Len | Route ID | Payload | +| 2 bytes | 4 bytes | ~ | +``` diff --git a/skywire-specs/specifications/07-Transport_Management.md b/skywire-specs/specifications/07-Transport_Management.md new file mode 100644 index 0000000000..e2e9be976b --- /dev/null +++ b/skywire-specs/specifications/07-Transport_Management.md @@ -0,0 +1,113 @@ +# Transport Management + +For all Skywire Node types, we need a universal way for managing and logging Transports. The structure that is responsible for this is the `TransportManager` (which should be within the `/pkg/node` package of the `skywire` package). + +As the `TransportManager` needs to interact with the *Transport Discovery* and other Skywire Nodes, it should have access to the local node's public and private key identity. + +The following is a proposed implementation of `TransportManager`; + +```golang +package node + +// Transport wraps a 'transport.Transport' implementation and contains +// associated/useful for the 'transport.Transport' implementation. +type Transport struct { + transport.Transport + ID uuid.UUID + // more fields ... +} + +// TransportManagerConfig configures a TransportManager. +type TransportManagerConfig struct { + PubKey cipher.PubKey // Local PubKey + SecKey cipher.SecKey // Local SecKey + DiscoveryClient client.Client // Transport discovery client + LogStore TransportLogStore // Store for transport's transfer rates +} + +// TransportManager manages Transports. +type TransportManager struct { + // Members... +} + +// NewTransportManager creates a TransportManager with the provided configuration and transport factories. +// 'factories' should be ordered by preference. +func NewTransportManager(config *TransportManagerConfig, factories ...transport.Factory) (*TransportManager, error) { /* ... */ } + +// Start starts the transport manager. +// - 'ctx' can end the transport listening operation. +func (tm *TransportManager) Serve(ctx context.Context) error { /* ... */ } + +// Observe returns channel for notifications about new Transport +// registration. Only single observer can listen for on a channel. +func (tm *TransportManager) Observe() <-chan *Transport { /* ... */ } + +// Factories returns all the factory types contained within the TransportManager. +func (tm *TransportManager) Factories() []string { /* ... */ } + +// Transport obtains a Transport via a given Transport ID. +func (tm *TransportManager) Transport(id uuid.UUID) (*Transport, bool) { /* ... */ } + +// RangeTransports ranges all Transports. +// Should return when 'action' returns a non-nil error. +func (tm *TransportManager) RangeAllTransports(action TransportAction) error { /* ... */ } + +// CreateTransport begins to attempt to establish transports to the given 'remote' node. +// This should be a non-blocking operation and any failures or future Transport disconnections +// should be dealt with with retries (under a given time interval). +// - 'remote' specifies the remote node to attempt to establish the Transports with. +// - 'tpType' is the transport type that is to be created. +// - 'public' determines whether the Transports established should be advertised to Transport Discovery. +// If a transport is not to be public, a random transport ID is assigned. +func (tm *TransportManager) CreateTransport(ctx context.Context, remote cipher.PubKey, tpType string, public bool) (*Transport, error) { /* ... */ } + +// DeleteTransport disconnects and removes the Transport of Transport ID. +func (tm *TransportManager) DeleteTransport(id uuid.UUID) error { /* ... */ } +``` + +## Transport Manager Procedures + +The transport manager is responsible for keeping track of established transports (via the `transport.Entry` and the `transport.Status` structures). The `transport.Entry` structure describes and identifies transports, while `transport.Status` keeps track of whether the transport is up or down (based on the perspective of the local node). + +If the *Transport Manager* wishes to confirm transport information, it can query the *Transport Discovery* via the `GET /transports/edge:` endpoint. Note that it is expected of the *Transport Manager* to call this endpoint on startup. + +When a transport is "closed" it is only considered "down", not "destroyed". + +The following highlights detailed startup and shutdown procedures of a *Transport Manager*; + +**Startup:** + +On startup, the `TransportManager` should call the *Transport Discovery* to ensure that it is up to date. Then it needs to attempt to establish (or re-establish) transports to the relevant remote nodes. + +When re-establishing a Transport, the `transport.Entry` used should be that also previously stored in the *Transport Discovery*. + +Once connected, the `TransportManager` should update it's *Status* of the given Transport and set `is_up` to `true`. + +The startup logic is triggered when `Start` is called. + +**Shutdown:** + +On shutdown, the first step is to update the *Transport Statuses* to "down" via the *Transport Discovery*. Then Transports to remote nodes is to be closed (with a timeout, in which after, the transport in question is forcefully closed). + +## Logging + +A *Transport Manager* is responsible for logging incoming and outgoing communication for each transport. Initially, only the total incoming and outgoing bandwidth (in bytes) is to be logged per transport. + +```golang +// TransportLogEntry represents a logging entry for a given Transport. +// The entry is updated every time a packet is received or sent. +type TransportLogEntry struct { + ReceivedBytes big.Int // Total received bytes. + SentBytes big.Int // Total sent bytes. +} +``` + +Logs for each transport is to be stored using `TransportLogStore`. `TransportLogStore` is to be specified within `TransportManagerConfig`. + +```golang +// TransportLogStore stores transport log entries. +type TransportLogStore interface { + Entry(id uuid.UUID) (*TransportLogEntry, error) + Record(id uuid.UUID, entry *TransportLogEntry) error +} +``` diff --git a/skywire-specs/specifications/08-Route_Finder.md b/skywire-specs/specifications/08-Route_Finder.md new file mode 100644 index 0000000000..b0c42df13c --- /dev/null +++ b/skywire-specs/specifications/08-Route_Finder.md @@ -0,0 +1,89 @@ +# Route Finder + +The *Route Finder* (or *Route Finding Service*) is responsible for finding and suggesting routes between two Skywire Nodes (identified by public keys). It is expected that an *App Node* is to use this service to find possible routes before contacting the *Setup Node*. + +In the initial version of the *Route Finder*, it should use a basic algorithm to choose and order the best routes. This algorithm should find the x amount (limited by the max routes parameter) of "fastest" routes determined by the least amount of hops needed, and order it by hops ascending. + +The implementation of *Route Finder* requires only a single REST API endpoint. + +## Graph Algorithm + +In order to explore routing we need to create a graph that represents the current skywire network, or at least the network formed by all the reachable nodes by `source node`. + +For this purpose, we use the `mark and sweep` algorithm. Such an algorithm consists of two phases. + +In the first phase, every object in the graph is explored in a `Deep First Search` order. This means that we need to explore every transport starting from route node, accessing the `transport-discovery` database each time that we need to retrieve new information. + +In the second phase, we remove nodes from the graph that have not been visited, and then mark every node as unvisited in preparation for the next iteration of the algorithm. + +An explanation and implementation of this algorithm can be found [here](https://www.geeksforgeeks.org/mark-and-sweep-garbage-collection-algorithm/). + +## Routing algorithm + +Given the previous graph we can now explore it to find the best routes from every given starting node to destiny node. + +For this purpose we use a modification of `Dijkstra algorithm`. + +An implementation can be found [here](http://rosettacode.org/wiki/Dijkstra%27s_algorithm#Go). + +Route-finder modifies this algorithm by keeping track of all the nodes that reached to destination node. This allows the ability to backtrack every best route that arrives from a different node to destination node. + +## Code Structure + +The code should be in the `watercompany/skywire` repository; + +- `/cmd/route-finder/route-finder.go` is the main executable for the *Route Finder*. +- `/pkg/route-finder/api/` contains the RESTFUL API definitions. +- `/pkg/route-finder/store/` contains the definition of the `Storer` interface and it's implementations [**TODO**]. +- `/pkg/route-finder/client/` contains the client library that interacts with the *Route Finder* service's RESTFUL API. + +## Database + +The *Route Finder* only accesses the Transport database already defined in the *Transport Discovery* specification. + +## Endpoint Definitions + +All endpoint calls should include an `Accept: application/json` field in the request header, and the response header should include an `Content-Type: application/json` field. + +### GET Routes available for the defined start and end key + +Obtains the routes available for a specific start and end public key. Optionally with custom min and max hop parameters. + +Note that each transport is to be represented by the `transport.Entry` structure. + +**Request:** + +``` +GET /routes +``` + +```json +{ + "src_pk": "", + "dst_pk": "", + "min_hops": 0, + "max_hops": 0, +} +``` + +**Responses:** + +- 200 OK (Success). + ```json + { + "routes": [ + { + "transports": [ + { + "tid": "", + "edges": ["", ""], + "type": "", + "public": true, + } + ] + } + ] + } + ``` +- 400 Bad Request (Malformed request). +- 500 Internal Server Error (Server error). diff --git a/skywire-specs/specifications/09-Route_Setup_Process.md b/skywire-specs/specifications/09-Route_Setup_Process.md new file mode 100644 index 0000000000..85218ffdb9 --- /dev/null +++ b/skywire-specs/specifications/09-Route_Setup_Process.md @@ -0,0 +1,9 @@ +# Route Setup Process + +1. Route paths are uni-directional. So, the whole route between 2 visors consists of forward and reverse paths. *Setup node* receives both of these paths in the routes setup request. +2. For each node along both paths *Setup node* calculates how many rules are to be applied. +3. *Setup node* connects to all the node along both paths and sends `ReserveIDs` request to reserve available rule IDs needed to setup the route. +4. *Setup node* creates rules the following way. Let's consider visor A setting up route to visor B. This way we have forward path `A->B` and reverse path `B->A`. For forward path we create `Forward` rule for visor `A`, `IntermediaryForward` rules for each node between `A` and `B`, and `Consume` rule for `B`. For reverse path we create `Forward` rule for visor `B`, `IntermediaryForward` rules for each visor between `B` and `A`, and `Consume` rule for `A`. +5. *Setup node* sends all the created `IntermediaryForward` rules to corresponding visors to be applied. +6. *Setup node* sends `Consume` and `Forward` rules to visor `B` (remote in our case). +7. *Setup node* sends `Forward` and `Consume` rules to visor `A` in response to the route setup request. diff --git a/skywire-specs/specifications/10-Routing_Table.md b/skywire-specs/specifications/10-Routing_Table.md new file mode 100644 index 0000000000..08b06b4e7e --- /dev/null +++ b/skywire-specs/specifications/10-Routing_Table.md @@ -0,0 +1,63 @@ +# Routing Table + +A *Routing Table* (located within the `skywire/pkg/node` module) is unique for a given Node's public key. It is basically a key-value store in which the key is the *Route ID* and the value is the *Routing Rule* for the given *Route ID*. + +Initially, there will be two types of Routing Rules: *App* and *Forward*. + +- *App* rules are identified by their unique `` value of `0x00`. A packet which contains a *Route ID* that associates with a *App* rule is to be sent to a local App. +- *Forward* rules are identified by their unique `` value of `0x01`. A packet which contains a *Route ID* that associates with a *Forward* rule is to be forwarded. + +| Action | Key (Route ID) | Value (Routing Rule) | +| ------ | -------------- | -------------------- | +| *App* | ``
*4 bytes* | ``
*48 bytes* | +| *Forward* | ``
*4 bytes* | ``
*29 bytes* | + +- `` is the *Route ID* `uint32` key (represented by 4 bytes) that is used to obtain the routing rules for the Packet. +- `` contains the epoch time (8 bytes) of when the rule is to be discarded (or becomes invalid). +- `` specifies the type of Routing Rule (1 byte). Currently there are two possible routing rule types; *App* (`0x00`) and *Forward* (`0x01`). +- `` is the *Route ID* (4 bytes) that is the Route ID key for the reserve Route of the loop. +- `` identifies and classifies the loop. It contains the following sub-fields; ``. + - `` is the remote edge public key in which this route/loop is associated with. It is represented by 33 bytes. + - `` is the remote port in which this route/loop is associated with. It is represented by 2 bytes. + - `` is the local port in which this route/loop is associated with. It is represented by 2 bytes. +- `` is the *Route ID* that is to replace the `` before the Packet is to be forwarded. +- `` represents the transport which the packet is to be forwarded to. A Transport ID is 16 bytes long. + +Every time a Skywire Node receives a packet, it performs the following steps: + +1. Obtain the `` from the Packet, and uses this value to obtain a routing rule entry from the routing table. If no routing rule is found, or the routing rule has already expired (via checking the `` field), the Packet is then discarded. +2. Obtains the `` value to determine how the packet is to be dealt with. If the `` value is `0x00`, the packet is then to be sent to the local *App Server* with the Routing Rule. If the `` value is `0x01`, the packet is to be forwarded; continue on to step 3. +3. Obtain the `` from the *Routing Rule* and replace the `` from the *Route ID* field of the Packet. +4. Forward the Packet to the referenced transport specified within ``. + +The routing table is to be an interface. + +```golang +package node + +// RangeFunc is used by RangeRules to iterate over rules. +type RangeFunc func(routeID transport.RouteID, rule RoutingRule) (next bool) + +// RoutingTable represents a routing table implementation. +type RoutingTable interface { + // AddRule adds a new RoutingRules to the table and returns assigned RouteID. + AddRule(rule RoutingRule) (routeID transport.RouteID, err error) + + // SetRule sets RoutingRule for a given RouteID. + SetRule(routeID transport.RouteID, rule RoutingRule) error + + // Rule returns RoutingRule with a given RouteID. + Rule(routeID transport.RouteID) (RoutingRule, error) + + // DeleteRules removes RoutingRules with a given a RouteIDs. + DeleteRules(routeIDs ...transport.RouteID) error + + // RangeRules iterates over all rules and yields values to the rangeFunc until `next` is false. + RangeRules(rangeFunc RangeFunc) error + + // Count returns the number of RoutingRule entries stored. + Count() int +} +``` + +Potential improvement we could consider is to move ports from the rules into the data packet header, aligning this with `tcp`. By doing so we will be able to re-use intermediate forward rules across multiple loops which can drastically improve loop establishment time for complex loops. diff --git a/skywire-specs/specifications/11-Router.md b/skywire-specs/specifications/11-Router.md new file mode 100644 index 0000000000..9adb5f915c --- /dev/null +++ b/skywire-specs/specifications/11-Router.md @@ -0,0 +1,19 @@ +# Router + +The `Router` (located within the `skywire/pkg/node` module) uses the *Transport Manager* and the *Routing Table* internally, and is responsible for the handling incoming Packets (either from external nodes via transports, or internally via the `AppServer`), and also the process for setting up routes. + +Regarding the Route setup process, a router should be able to interact with multiple trusted *Setup Nodes*. + +Every transport created or accepted by the *Transport Manager* should be handled by the *Router*. All incoming packets should be cross-referenced against *Routing Table* and either be forwarded to next *Node* or to a local *App*. + +*Transport Manager* is also responsible for managing connections on local ports for node's apps. *App Node* will request new local connection from the *Router* on *App* startup. All incoming packets from the app's connections should be forwarded based on *App* rules defined in a local routing table. *Transport Manager* should also be capable of requesting new loops from a *Setup Node*. + +## Port management + +Router is responsible for port management. Port allocation algorithm should work similarly to how `tcp` manages ports: + +- Certain range of ports should be reserved and be inaccessible for general purpose apps. Default staring port is `10`. +- All allocated local ports should be unique. +- App should be able to allocate static ports that it will be accessible on for remote connections. Static port allocation is performed during `app` init. +- App should be able to dynamically allocate local port for newly created loops. +- Allocated ports should be closed on app shutdown or disconnect from the node. diff --git a/skywire-specs/specifications/12-App_Server.md b/skywire-specs/specifications/12-App_Server.md new file mode 100644 index 0000000000..25fb3ab9d4 --- /dev/null +++ b/skywire-specs/specifications/12-App_Server.md @@ -0,0 +1,37 @@ +# App Server + +The `AppServer` (located within the `skywire/pkg/node` module) handles communication between local and remote Apps. It also manages and administers local Apps. It is to interact with a `Router` and identify loops via the *App* routing rule retrieved from the routing table. The *App* rule is structured as follows; + +``` +| expiry | r-type | resp-rid | remote-pk | remote-port | local-port | +| 8 bytes | 1 byte | 4 bytes | 33 bytes | 2 bytes | 2 bytes | +``` + +The *App Server* not only forwards Packets between Apps and remote entities, but it can also be instructed by the end user to send *Quit* signals to the Apps. *Apps* can also request to open *Loops* with remote/local Apps. + +Each *Loop* is identified uniquely via the local and remote public keys, paired with the local and remote ports. + +Within the `AppServerConfig` file, local ports are reserved for certain Apps. The following rules are to be opposed: +- Ports are either "reserved" or "unreserved". + - No two Apps are allowed to "reserve" the same port. + - Ports are reserved via the `AppServerConfig` file. +- Reserved ports are either "active" or "inactive". + - A port is "active" when the port is "reserved" for an App, and that App is running. + - A port is "inactive" either when the port is "unclaimed", or when the port is "claimed" but the App is not running. + +The following communication processes between a given App and the *App Server* is to exist: + +- **App requests to open loop** +- **App Server asks App whether it wishes to respond to a remotely-initiated loop** +- **App Server informs App that loop is to be closed (with given reason)** + - Reasons include: Route timeout, remotely closed, locally closed, etc. +- **App informs App Server it wishes to close a loop (with given reason)** + - Reasons include: App is shutting down, loop no-longer used, etc. +- **App Server forwards packets received from remote App** + - If local app does not exist or does not accept, the loop is to be closed, and routes destroyed. +- **App Server forwards packets received from local App** + - If the rule does not exist, or the remote app nodes not accept, the loop is to be closed and routes destroyed. + +## Loop Encryption + +Each loop is to be symmetrically encrypted with [Noise](http://noiseprotocol.org/noise.html). Specifically, we are to use the `KK` fundamental [handshake pattern](http://noiseprotocol.org/noise.html#handshake-patterns). The first noise message should be provided by initiator in the request to create a new loop, this messages will be setup to responder in a loop confirmation request. Responder should send second noise message which will be returned to initiator in a loop confirmation request. diff --git a/skywire-specs/specifications/13-Setup_Node.md b/skywire-specs/specifications/13-Setup_Node.md new file mode 100644 index 0000000000..de49eb20c7 --- /dev/null +++ b/skywire-specs/specifications/13-Setup_Node.md @@ -0,0 +1,3 @@ +# Setup Node + +The *Setup Node* (located within the `skywire/pkg/node` module) uses *Transport Manager* internally and is responsible for propagation of routing rules to nodes along the *Route*. *Setup Node* should be only addressed by a public key and should work over messaging transport using multiple channels. Each channel can be used to issue route setup commands by initiator. For *Loop* setup requests *Node* will be an initiator, for *Rule* setup related operations *Setup Node* will be a channel initiator. *Setup Node* is only responsible for handling *Foundational Packets* and doesn't perform any forwarding functions. diff --git a/skywire-specs/specifications/14-Skywire_Visor.md b/skywire-specs/specifications/14-Skywire_Visor.md new file mode 100644 index 0000000000..2aa57b68f5 --- /dev/null +++ b/skywire-specs/specifications/14-Skywire_Visor.md @@ -0,0 +1,429 @@ +# Skywire Visor + +A visor is a node that is part of the Skywire network and is represented by a key pair (using the `secp256k1` curve). It handles Transports to remote visors, sets up routes and loops (via Routing Rules and interaction with the *Setup Node*), and manages Apps. + +Each App is it's own executable that communicates with an *App Node* using a pair of *POSIX* pipes. A piped connection is setup on *App* startup and inherited by a forked *App* process using file descriptor `3` and `4`. Setup process for a forked *App* is handled by the `app` package. + +``` + [Skywire Visor] + / | \ + / | \ +[App 1] [App 2] [App 3] +``` + +## Communication reliability + +Currently, loop ACK's are not implemented. They will need to be implemented at a later stage. ACK's are implemented for the dmsg implementation. + +## App Programming Interface + +*App* programming interface (located within the `skywire/pkg/app` module) should expose methods for *Apps* to connect to a piped connection, perform handshake and exchange data with remote nodes. + +*App* interface should expose following methods: + +```golang +// Addr implements net.Addr for App connections. +type Addr struct { + PubKey transport.PubKey + Port uint16 +} + +// LoopAddr stores addressing parameters of a loop package. +type LoopAddr struct { + Port uint16 + Remote Addr +} + +// Packet represents message exchanged between App and Visor. +type Packet struct { + Addr *LoopAddr + Payload []byte +} + +// Config defines configuration parameters for an App +type Config struct { + AppName string + AppVersion string + ProtocolVersion string +} + +// Setup sets up an app using default pair of pipes and performs handshake. +func Setup(config *Config) (*App, error) {} + +// Accept awaits for incoming loop confirmation request from a Visor and +// returns net.Conn for a received loop. +func (app *App) Accept() (net.Conn, error) {} + +// Dial sends create loop request to a Visor and returns net.Conn for created loop. +func (app *App) Dial(raddr *Addr) (net.Conn, error) {} + +// Addr returns empty Addr, implements net.Listener. +func (app *App) Addr() net.Addr {} + +// Close implements io.Closer for an App. +func (app *App) Close() error {} +``` + +## App to Visor Communication protocol + +Communication between *Visor* and an *App* happens over the piped connection using binary multiplexed protocol. + +The following is the expected format of a App Packet: + +``` +| Packet Len | Type | Message ID | JSON Body | +| 2 bytes | 1 byte | 1 byte | ~ | +``` + +- ***Packet Len*** specifies the total packet length in bytes (exclusive of the *Packet Len* field). +- ***Type*** specifies the *App Packet Type*. +- ***Message ID*** specifies multiplexing ID of a message, response for this message should contain the same ID. +- ***JSON Body*** is the packet body (in JSON format) that is unique depending on the packet type. + +**App Packet Types Summary:** + +| Type | Name | +| ---- | ---- | +| 0x00 | `Init` | +| 0x01 | `CreateLoop` | +| 0x02 | `ConfirmLoop` | +| 0x03 | `Send` | +| 0x04 | `Close` | +| 0xfe | `ResponseFailure` | +| 0xff | `ResponseSuccess` | + +### `0x00 Init` + +Sent by an *App* to a *Visor*. This packet is used to handshake connection between an *App* and a *Visor*. *Visor* will typically check if app is allowed by the config file and which port should be statically allocated it. + +**JSON Body:** + +```json +{ + "app-name": "foo", + "app-version": "0.0.1", + "protocol-version": "0.0.1" +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` without body. + +### `0x01 CreateLoop` + +Sent by an *App* to a *Visor*. This packet is used to open new *Loop* to a remote *Visor*. + +**JSON Body:** + +```json +{ + "pk": "", + "port": +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with + ```json + { + "pk": "", + "port": + } + ``` + +### `0x02 ConfirmLoop` + +Sent by a *Visor* to an *App* to notify about request to open new *Loop* from a remote *Visor* + +**JSON Body:** + +```json +[ + { + "pk": "", + "port": + }, + { + "pk": "", + "port": + } +] +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty body. + +### `0x03 Send` + +Sent by a *Visor* and an *App*. This message is used to exchange messages through a previously established *Loop*. + +**JSON Body:** + +```json +{ + "addr": { + "port": , + "remote": { + "pk": "", + "port": + } + }, + "payload": "" +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty body. + +### `0x04 Close` + +Sent by a *Visor* and an *App*. *App* uses this message to notify about closed *Loop*. *Visor* sends this message after remote node is requested to close established *Loop*. + +**JSON Body:** + +```json +{ + "port": , + "remote": { + "pk": "", + "port": + } +} +``` + +Response: +- `ResponseFailure` with `error`. +- `ResponseSuccess` with empty body. + +## App Node Configuration + +The following is the JSON representation of a Skywire configuration. + +```json +{ + "version": "1.0", + "node": { + "static_public_key": "024ec47420176680816e0406250e7156465e4531f5b26057c9f6297bb0303558c7", + "static_secret_key": "42bca4df2f3189b28872d40e6c61aacd5e85b8e91f8fea65780af27c142419e5" + }, + "messaging": { + "discovery_addresses": ["http://localhost:9090"], + "server_count": 1 + }, + "apps": [ + { + "app": "helloworld", + "version": "1.0", + "auto_start": true, + "port": 10, + "args": [] + } + ], + "transport_discovery": "http://localhost:9091", + "setup_nodes": ["02603d53d49b6575a0b8cee05b70dd23c86e42cd6cba99af769d61a6196ea2bcb1"], + "trusted_nodes": ["0348c941c5015a05c455ff238af2e57fb8f914c399aab604e9abb5b32b91a4c1fe"], + "messaging_path": "./messaging", + "apps_path": "./apps", + "local_path": "./local", + "log_level": "info", + "interfaces": { + "rpc": ":3436" + } +} +``` + +- `"version"` represents the version of the Skywire Visor (and also the configuration format version). + +- `"node"` includes the public/private keys that identify the visor. + +- `"messaging"` configures the dmsg client instance included within the Skywire Visor. + - When `"public"` is set, the dmsg Client Instance will advertise itself to discovery. + - `"discovery_addresses"` specifies the dmsg Discovery Services that the Skywire Visor is to try. + - `"server_count"` specifies the number of servers to ensure connection with on first startup. + +- `"apps"` lists all available Skywire Apps. These configurations include; the App's name, whether the specified App should auto-start, and the ports that is reserved for the App. If these are not defined for an App, the App will not auto-start, nor have ports reserved for the App. + - If `"version"` is not specified, the highest stable version will be selected. + +- `"node_path"` stores logs, routing tables, and any data that the visor may use. + +- `"messaging_path"` holds the path which the dmsg Client Instance can use to store cache or additional configurations. + +- `"apps_path"` holds all the app executables. App executable files should be named with no spaces or weird characters (TODO: define properly). They should also be appended with the semantic version of the App after a dot; `{app_name}.v{semantic_version}`. + +- `"local_path"` contains the working directories of the apps. An app named `chat` of version `v1.0` should have a working directory within `{root_dir}/{local_path}/chat/v1.0/`. The contents of the App's working directory is specified by the App. + +## App Node RPC Interface + +The Visor should attempt to connect to the assigned *Hypervisor* on startup. The connection is to be encrypted via Noise (KK handshake pattern) so that the nodes can identify one another. + +For the *Visor* to connect to the *Hypervisor*, it needs the public key and tcp address of the *Hypervisor* in it's configuration. + +After connection has been established, the *Visor* becomes the RPC Server and the *Hypervisor* becomes the RPC client that can execute commands on the *Visor* (the RPC Server). + +Additionally, the Visor should listen on a port so that a local command-line executable (`skywire-cli`) can interact with it. This local port should hence, only accept connections from localhost. + +### Commands + +The following sub-commands should be supported. Note that command-line actions are listed below, but they should be served via RESTFUL interfaces. + +**General:** + +- **`summary`** obtains a summary of the current state of the Visor. + +**App Management:** + +- **`list-apps`** lists applications and applications stats (running/not running) (auto-start/non-auto-start) (local/remote ports). There should be flags for filtering (to be defined later). + +- **`start-app `** starts a Skywire app. + +- **`stop-app `** stops a Skywire app if running. + +- **`add-autostart-app [--start-now]`** adds a Skywire app to auto-start. After calling this command, the actual app does not actually start unless `--start-now` is set. + +- **`rm-autostart-app [--stop-now]`** removes an app from auto-starting. After calling this command, the actual app does not stop running unless `--stop-now` is set. + +**Messaging System Management:** + +- **`messaging list-discoveries`** lists saved discoveries and their statuses + +- **`messaging add-discovery `** connects to and saves a discovery. + +- **`messaging rm-discovery `** disconnects from and removes a discovery. + +- **`messaging list-servers`** lists connected messaging servers and their statuses (connected/disconnected) (auto-connect/non-auto-connect). + +- **`messaging connect-server (|--auto)`** connects to a messaging server for this session (does not save server for auto-connect). If `--auto` is set, the transport discovery is queried for a random available messaging server. + +- **`messaging disconnect-server `** disconnects from a messaging server for this session (does not affect auto-connect settings). + +- **`messaging add-autoconnect-server [--connect-now]`** adds a messaging server to auto-connect. This command does not connect to the specified messaging server unless `--connect-now` is set. + +- **`messaging rm-autoconnect-server [--disconnect-now]`** removes a messaging server from auto-connecting. This command does not disconnect from the specified messaging server unless `--disconnect-now` is set. + +**Transport Management:** + +- **`transport-types`** lists all transport types used by the visor (represented as strings). +- **`list-transports [--filter-types=,,...] [--filter-pks=,,...] [--no-logs]`** lists all transports associated with the visor. Filters can be used to only show visors associated with specified transport types or public keys. By default, transports are displayed with their logging information. +- **`add-transport [--timeout=]`** adds a transport of a given type and public key. If `--timeout` is not set, the default timeout is used. +- **`rm-transport [--tid=|--remote-pk=]`** removes a transport; either for a given transport ID, or all transports connected to a remote visor (identified via the remote visor's public key). + +**Routes Management:** + +- **`list-rules [--rid-range=:]`** lists all routing rules. A route ID range filter can be specified. +- **`rm-rules [--list=,,...|--range=:]`** removes routing rules; either via a list of route ID keys, or via a range of route ID keys (note that routing rules are identified via their `` key). This action may consequently destroy loops, and may cause the *Setup Node* to request destruction of more routing rules. + +**Loops Management:** + +- **`list-loops [--local-port=] [--remote-addr=[:]]`** lists all loops. A local port filter can be specified, where the returned loops will only be of the specified local port (there is an equivalent remote address filter). +- **`add-loop --local-port= --remote-addr=: [--setup-node=]`** attempts to create a loop with the assigned setup node. The setup node is automatically chosen if not specified. + +## Ports Management + +Within the `AppsConfig` file, ports are reserved for certain Apps. The following rules are to be opposed: +- Ports are either "reserved" or "unreserved". + - No two Apps are allowed to "reserve" the same port. + - Ports are reserved via the `AppsConfig` file. +- Reserved ports are either "active" or "inactive". + - A port is "active" when the port is "reserved" for an App, and that App is running. + - A port is "inactive" either when the port is "unclaimed", or when the port is "claimed" but the App is not running. + +## App Example + +Simple `ping-pong` client and server apps can be implemented in such way: + +Server: + +```golang +package server + +import ( + "log" + + "github.com/watercompany/skywire/pkg/app" +) + +func main() { + // Open connection with visor + helloworldApp, err := app.Setup(&app.Config{AppName: "helloworld-server", AppVersion: "1.0", ProtocolVersion: "0.0.1"}) + if err != nil { + log.Fatal("Setup failure: ", err) + } + defer helloworldApp.Close() + + log.Println("listening for incoming connections") + // Start listening loop + for { + // Wait for new Loop + conn, err := helloworldApp.Accept() + if err != nil { + log.Fatal("Failed to accept conn: ", err) + } + + log.Println("got new connection from:", conn.RemoteAddr()) + // Handle incoming connection + go func() { + buf := make([]byte, 4) + if _, err := conn.Read(buf); err != nil { + log.Println("Failed to read remote data: ", err) + } + + log.Printf("Message from %s: %s", conn.RemoteAddr().String(), string(buf)) + if _, err := conn.Write([]byte("pong")); err != nil { + log.Println("Failed to write to a remote visor: ", err) + } + }() + } +} +``` + +Client: + +```golang +package server + +import ( + "log" + "os" + + "github.com/watercompany/skywire/pkg/app" + "github.com/watercompany/skywire/pkg/cipher" +) + +func main() { + // Open connection with visor + helloworldApp, err := app.Setup(&app.Config{AppName: "helloworld-client", AppVersion: "1.0", ProtocolVersion: "0.0.1"}) + if err != nil { + log.Fatal("Setup failure: ", err) + } + defer helloworldApp.Close() + + // Read remote PK from stdin + remotePK := cipher.PubKey{} + if err := remotePK.UnmarshalText([]byte(os.Args[1])); err != nil { + log.Fatal("Failed to construct PubKey: ", err, os.Args[1]) + } + + // Dial to remote visor + conn, err := helloworldApp.Dial(&app.Addr{PubKey: remotePK, Port: 10}) + if err != nil { + log.Fatal("Failed to open remote conn: ", err) + } + + // Send payload + if _, err := conn.Write([]byte("ping")); err != nil { + log.Fatal("Failed to write to a remote visor: ", err) + } + + // Receive payload + buf := make([]byte, 4) + if _, err = conn.Read(buf); err != nil { + log.Fatal("Failed to read remote data: ", err) + } + + log.Printf("Message from %s: %s", conn.RemoteAddr().String(), string(buf)) +} +``` diff --git a/skywire-specs/specifications/15-Hypervisor.md b/skywire-specs/specifications/15-Hypervisor.md new file mode 100644 index 0000000000..47927c2ed4 --- /dev/null +++ b/skywire-specs/specifications/15-Hypervisor.md @@ -0,0 +1,18 @@ +# Hypervisor + +The *Hypervisor* is responsible for managing *Visors* and is identified via it's public key and TCP address. + +The *Visor* is responsible for including a trusted *Hypervisor* in it's configuration file, and attempt to connect to it on startup. The connection is authenticated and encrypted via the Noise protocol where the `XK` handshake pattern is used with the *Visor* being the initiator. + +After the connection is successfully established between an *Visor* and a *Hypervisor*, the *Visor* acts as the RPC server while the *Hypervisor* is the RPC client. In this way, the *Hypervisor* can execute commands on the *Visor*. + +The *Hypervisor* serves a REST API which the end user can interact with. To access the API, the user is required to log in via a username and password. + +The *Hypervisor* should be implemented in `/pkg/hypervisor`. + +## Hypervisor REST API (and User Interface) + +- Login/logout. +- Change password. +- List connected *Visor*, each Visor should contain the following summary; Public key, local address, number of transports established, number of apps running, Uptime. +- The user can click into a listed *Visor* and perform node-specific actions; specifically, the [RPC commands as specified above.](#visor-rpc-interface) diff --git a/skywire-specs/specifications/16-Glossary.md b/skywire-specs/specifications/16-Glossary.md new file mode 100644 index 0000000000..5ac9ff00bd --- /dev/null +++ b/skywire-specs/specifications/16-Glossary.md @@ -0,0 +1,104 @@ +# Glossary + +**Skywire Transport:** + +Identified via a Transport ID, a Skywire Transport is represented as an interface and can be implemented as different transport types. It is a bi-directional line of communication between two Skywire nodes (transport edges) and constructs a single hop of a route. + +It is the responsibility of a Skywire Factory to generate Skywire Transports. + +**Skywire Transport ID:** + +A Skywire Transport is an uint16 integer that refers to a Skywire Transport and identifies it in the transport discovery. Skywire Transport IDs are assigned uniquely by and for a transport edge. Therefore, the same transport can be referenced by two different Transport IDs, assigned by the 2 edges. + +**Transport Edge:** + +A transport edge is one of the two Skywire nodes that make up a transport. It is represented by a unique public key that identifies the Skywire node. + +**Transport Type:** + +A Transport Type (represented by a string) refers to the underlying implementation of a Transport Factory or a Transport. A Transport Factory of a certain type can only construct Transports of that type. + +Initially, we will have two transport types; messaging and native. The messaging transport is implemented by the messaging system. The native transport is non-dependent on the current internet and in the future will be the main transport implementation for Skywire. + +**Transport Factory:** + +A transport factory is used by a Skywire node which constructs transports of a certain transport type. It is an interface that can either dial or listen for remotely initiated transports. + +**Transport Perspective:** + +A transport perspective is the assumed state of a transport (number of packets and bandwidth sent and received over that transport; whether transport is up or down). It only represents the perspective (on a transport) from a single transport edge and therefore the perspectives of two edges on the same transport might conflict. + +**Transport Discovery:** + +The transport discovery is a service that registers transports and the associated transport perspectives. It thereby provides the basis for the route finding service because it represents the public network topology. It is queried by the route finding service to discover routes. + +**Hop:** + +A hop is equivalent to a transport and is a single unit of a route. + +**Skywire Route:** + +A Skywire Route is a unidirectional network path that allows apps and services to communicate. It can be made up of one or more hops between Skywire Nodes. It is identified by route IDs, interpreted by individual Skywire Nodes via the routing table. + +**Skywire Route ID:** + +A route ID is represented by 32 bits and is the basis for a Skywire Node’s routing rules. Route IDs are changed along every hop of a route and there is a unique set of route IDs for every node. + +**Route Finder:** + +The route finder is a service that evaluates the network topology via the information of the transport discovery to provide possible routes to inquiring Skywire nodes. Currently it evaluates possible routes only on the basis of the hop metric. + + +**Routing Table:** + +The Routing Table is a key-value store that determines the action to be performed on an incoming Packet. + +Using the Routing ID (from the Packet) as the key, a Skywire Node can obtain either the next transport that the packet is to be sent over (and the new Routing ID) or whether the packet is to be consumed by the node itself. + +**Stream:** + +A stream represents a bi-directional line of communication between two Skywire Apps. A stream can use multiple Routes to establish itself. + +**Skywire App:** + +A Skywire App is an executable that interacts with the Skywire App Node via Unix pipes. Skywire apps provide services for the end user such as a proxy, ssh, chat, etc. + +**Messaging System:** + +The messaging system is primitive fallback implementation of Transport and Transport Factory over the Internet. It consists of a messaging discovery, client and server instances, where the server instance relays messages between clients. The communication between two clients is called a messaging channel. + +**Messaging Instance:** + +Messaging instances (alongside the messaging discovery) are the main components of the messaging system. A messaging instance can be either a messaging server or messaging client. + +**Messaging Link:** + +A messaging link is the direct line of communication between a server instance and a client instance. A messaging channel between two client instances can be constructed from two messaging links to a shared server instance. + +**Messaging Discovery:** + +The messaging discovery is a key value store that registers the messenger servers that a given messenger client is connected to. It allows clients to get the information necessary to establish a messaging transport with another node by querying the public keys associated messaging servers. + +**Messaging Channel:** + +A Messaging Channel represents the bi-directional connection-oriented line of communication between two client instances of the messaging system. + +**Skywire Node:** + +A Skywire Node is the general term for nodes that make up the Skywire Network. Any entity that manages Transports and Packets is considered to be a Skywire Node. + +Examples of Skywire Nodes include; App Node (which manage Skywire Apps), Control Node (which administers App Nodes) and the Route Setup Node (which coordinates the construction of Routes with App Nodes). + +**Skywire App Node:** + +An App Node is a Skywire Node that runs, stops, monitors and sets permissions for Skywire Apps. Internally, it handles and coordinates packets incoming and outgoing between the Skywire App and the external Skywire network. + +An App Node can also forward packets to external Skywire Nodes (based on the set Routing Rules). + +**Skywire Control Nodes:** + +A Skywire Control node is similar to an app node, with the difference that it has administrative permissions for other nodes in a cluster and is being sent logs from other nodes. It should not run user-based Skywire Apps, nor should it forward Packets. + +**Skywire Route Setup Node:** + +The Route Setup Node is a Skywire Node which runs a service that allows it to set up routes for other Skywire nodes. It does this by relaying the routing rules to individual Skywire App Nodes. diff --git a/skywire-specs/specifications/17-App2 module.md b/skywire-specs/specifications/17-App2 module.md new file mode 100644 index 0000000000..a30489e4aa --- /dev/null +++ b/skywire-specs/specifications/17-App2 module.md @@ -0,0 +1,106 @@ +# `app2` + +The current `app` module of Skywire is a mess. It is hard to test and it's actual communication logic is split across multiple packages. This proposal details how I would envision an *ideal* `app` package, and also how we can migrate and eventually scrap the old `app` module. + +## Overview + +The visor node is the intermediary of communication between apps and the network(s). The `app2` package will provide structures for use in `visor` and skywire apps. It will facilitate the communication between them. All networks used by skywire should implement the `net.Conn` and `net.Listener` interfaces. Currently, apps only need to dial/accept connections on `dmsg` and `router`. + +The goal of `app2` is to be simple and retain as much of the original form of the communication as possible. This should improve code readability and performance (when compared to the original `app` package). + +`app2` will have two major components: `app2.Client` (which will be used by skywire apps) and `app2.Server` (which will run on the visor node). The communication between `app2.Client` and `app2.Server` should be via unix socket where the `app2.Server` is responsible for listening on a socket file. + +## Types + +The following structures should identify the components that facilitate interation between `app2.Client` and `app2.Server`. + +### Summary + +```golang +package app2 + +// ProcID identifies the current instance of an app (an app process). +// The visor node is responsible for starting apps, and the started process +// should be provided with a ProcID. +type ProcID uint16 + +// HSFrameType identifies the type of handshake frame. +type HSFrameType byte + +// HSFrame is the data unit for socket connection handshakes between Server and Client. +type HSFrame []byte + +// Client is used by skywire apps. +type Client struct { + // TODO: define. +} + +// NewClient creates a new Client. The Client needs to be provided with: +// - localPK: The local public key of the parent skywire visor. +// - pid: The procID assigned for the process that Client is being used by. +// - sockAddr: The socket address to connect to Server. +func NewClient(localPK cipher.PubKey, pid ProcID, sockAddr string) (*Client, error) { + // TODO: define. +} + +// Server is used by skywire visor. +type Server struct { + // TODO: define. +} +``` + +The reason for only having handshake frames (`HSFrame`), is because a connection on a network of choice between a visor node and a remote node, is represented by a socket connection between the visor node (via `app2.Server`) to a skywire app (which uses `app2.Client`). So after the handshake (which is used for determining how we map the socket connection), the data thereafter can just be forwarded as-is to and from the socket connection and the network connection. + +### `HSFrame` + +A `HSFrame` consists of a header and a body. The contents of the body changes based on the `HSFrameType` (specified in the header). + +**`HSFrame` header:** + +The `HSFrame` header consists of a `ProcID`, a `HSFrameType` and a `BodyLen` (5 bytes in total). + +``` +| ProcID (2 bytes) | HSFrameType (1 byte) | BodyLen (2 bytes) | +``` + +**`HSFrame` body:** + +The `HSFrame` body should be a JSON structure. + +| HSFrameType Name | HSFrameType Value | HSFrame Body Contents | Usage(s) | +| --- | --- | --- | --- | +| `DmsgListen` | `10` | localPK, localPort | `Client -> Server` to request listening on a given pk/port pair. | +| `DmsgListening` | `11` | localPK, localPort | `Server -> Client` to inform that the client is now listening the pk/port pair. | +| `DmsgDial` | `12` | localPK, (localPort), remotePK, remotePort | `Client -> Server` to request dialing to a remote.
`Server -> Client` to inform that a remote has dialed.
**Note:** localPort must be `0` when `Client -> Server` as the `app2.Server` will provide an ephemeral port on `DmsgAccept`. | +| `DmsgAccept` | `13` | localPK, localPort, remotePK, remotePort | `Client -> Server` to inform the server that the remote connection is accepted.
`Server -> Client` to inform the client that the connection is accepted by remote. | + +Note that there are no *close* frame types, as one can just close the underlying socket connection. + +## Procedures + +### `app2.Client` listens on given pk/port. + +- `app2.Client` dials a new socket connection to `app2.Server` and sends a `DmsgListen` type `HSFrame` with local pk/port in which the client wishes to listen on. +- `app2.Server` calls `(*dmsg.Client).Listen` and creates a `dmsg.Listener`. + - If there is no error, `app2.Server` sends `DmsgListening` back to the client. + - On failure, the `app2.Server` just closes the socket connection. + +### `app2.Client` accepts a connection from remote. + +Given that an `app2.Client` is listening on a given pk/port, when `app2.Server` accepts a remote connection, the following should happen: + +- `app2.Server` sends `DmsgDial` to `app2.Client` via the socket connection that is initiated with `DmsgListen`. +- `app2.Client` dials a new socket connection to `app2.Server` and sends `DmsgAccept` (with the same body as the `DmsgDial`) via the new socket connection. +- Further data to/from the related remote connection should be directly forwarded by this new socket connection. + +### `app2.Client` dials to a remote pk/port. + +- `app2.Client` dials a new socket connection to the `app2.Server` and sends a `DmsgDial` with an empty localPort. +- `app2.Server` attempts the requested dial. + - On success, the `app2.Server` sends `DmsgAccept` to `app2.Client` via the same socket connection. + - On failure, the socket connection gets closed. + +## Implementation notes + +### `app2.Server` +- We will need to move most of the app-management logic here. Currently, they are mostly located in [`visor`](https://github.com/skycoin/skywire/blob/mainnet-milestone1/pkg/visor/visor.go). On top of this, we need to introduce the idea of `ProcID`. We should probably add a new structure (named `Process`) that holds both the `os.Process` and `net.Conn` socket connection to the app pocess. \ No newline at end of file diff --git a/skywire-specs/specifications/18-Router2 module.md b/skywire-specs/specifications/18-Router2 module.md new file mode 100644 index 0000000000..26a43b69bf --- /dev/null +++ b/skywire-specs/specifications/18-Router2 module.md @@ -0,0 +1,296 @@ +# `router2` + +The goal of this proposal is to split out and define the responsiblities within `router` to allow for easier integration of future features, as well as make the codebase easier to understand. + +The current implementation of `routing.Rule` has fields that have different meanings, based on the rule type. A clearer definition of rule types are addressed in this proposal. + +Currently, setup nodes create *loops* (which is the combination of a forward and a reverse route). We want setup nodes to create *routes* (which is a uni-directional line of communication across x-number of hops). A single `CreateRoute` packet can request to create multiple routes. How routes are to be used is to be handled in an additional structure called `multiplexer`. + +## `routing.Rule` changes + +As the `Router` is to be split up into router/multiplexer, we are to remove the concept of `App` rule types, and work with the concepts of forward/consume. + +### New fields for `routing.Rule` + +A routing rule is structured as follows (note that the `body` field is action-specific). +``` +| keepAlive | rType | keyRtID | body | +| 8[0:8] | 1[8:9] | 4[9:13] | [13:~] | +``` + +#### field `keepAlive` (8 bytes) + +`keepAlive` is the duration of non-use before the rule is to be disregarded/deleted. It is the responsibility of the src edge to send ping packets and the dst edge to respond with pong packets. + +#### field `rType` (1 byte) + +`rType` is the `RuleType`. + +```golang +type RuleType byte ` + +const ( + // ConsumeRule represents a hop to the route's destination node. + // A packet referencing this rule is to be consumed localy. + ConsumeRule = RuleType(0) + + // ForwardRule represents a hop from the route's source node. + // A packet referencing this rule is to be sent to a remote node. + ForwardRule = RuleType(1) + + // IntermediaryForwardRule represents a hop which is not from the route's source, + // nor to the route's destination. + IntermediaryForwardRule = RuleType(2) +) +``` + +#### field `keyRtID` (4 bytes) + +`keyRtID` is the route ID that is used as the key to retreive the rule. + +#### field `body` (~ bytes) + +The contents and size of this field is dependent on the `rType`. + +- `ConsumeRule` `body` field contents: + ``` + | rDesc | + | 70[13:83] | + ``` + +- `ForwardRule` `body` field contents: + ``` + | rDesc | nxtRtID | nxtTpID | + | 70[13:83] | 4[83:87] | 16[87:103] | + ``` +- `IntermediaryForwardRule` `body` field contents: + ``` + | nxtRtID | nxtTpID | + | 4[13:17] | 16[17:33] | + ``` + +#### field `rDesc` (70 bytes) + +```golang +// RouteDescriptor describes a route (from the perspective of the source and destination edges). +type RouteDescriptor [70]byte + +func (d RouteDescriptor) SrcPK() cipher.PubKey { /*TODO*/ } +func (d RouteDescriptor) DstPK() cipher.PubKey { /*TODO*/ } +func (d RouteDescriptor) SrcPort() uint16 { /*TODO*/ } +func (d RouteDescriptor) DstPort() uint16 { /*TODO*/ } +``` + +The decision to go with src/dst instead of local/remote is because it will have the same values independent of the residing edge. + +#### field `nxtRtID` (4 bytes) + +Provides the next `routeID` value for `Forward`-esk type rules. + +#### field `nxtTpID` (16 bytes) + +Provides the next transport ID for `Forward`-esk type rules. + +## `routing.Table` changes + +The nature of rules is going to be temporary and there will no longer be any need to store rules to disk. The only implementation for `routing.Table` should be in memory. + +The new structure of `routing.Table` should have an internal garbage collector, removing the need for `router.managedRoutingTable`. The internal garbage collector should only delete `IntermediaryForwardRule` types. All other rule types should be deleted by external structures. + +```golang +type Table interface { + ReserveKey() (key RouteID, err error) + SaveRule(r Rule) error + Rule(key RouteID) (r Rule, err error) + RulesWithDesc(desc RouteDescriptor) (rs []Rule, err error) + AllRules() []Rule + DelRules([]RouteID) + Count() int +} + +type memTable struct { + // All rules, referenced with 'keyRtID'. + rules map[RouteID]Rule +} +``` + +## Setup node changes + +### Packet types + +We should only have the following packet types: + +| Packet Type | Packet Body | +| --- | --- | +| `CreateRoutes` | `routes []Route` | +| `RoutesCreated` | `routes []Route` | +| `ReserveRtIDs` | `n int` | +| `RtIDsReserved` | `rtIDs []routing.RouteID` | +| `AddRules` | `rules []routing.Rule` | +| `RulesAdded` | `rtIDs []routing.RouteID` | +| `Failure` | `failureCode byte, msg string` | + +Notes: +- We are removing `Close~` packet types as it's no longer needed. Instead, we will use shorter `keepAlive` durations and introduce ping/pong to keep routes alive. + +### `Route` + +```golang +type Hop struct { + TpID uuid.UUID + From cipher.PubKey + To cipher.PubKey +} + +type Route struct { + Desc RouteDescriptor + Hops []Hop + KeepAlive time.Duration +} +``` + +### Use one connection per visor, per request + +Previously, a single request is split between multiple connections per visor. We should only use one connection per request per visor. + +### Failure codes + +We need to define a new type. + +```golang +type FailureCode byte + +func (fc *FailureCode) String() string { + // TODO: define. +} +``` + +And have a list of constants for failure codes. + +## Route Finder Changes + +Instead of finding *paired routes*, we should be able to search for routes independently. + +```golang +type RouteOptions struct { + MinHops int + MaxHops int +} + +// FindRoutes find routes specified in 'rts'. +// As routes are uni-directional, rts[i][0] is the source edge, and rts[i][1] is the destination edge. +// If 'opts' is nil, MinHops is 1 and MaxHops is 3. +func (c *apiClient) FindRoutes(ctx context.Context, rts [][2]cipher.PubKey, opts *RouteOptions) +``` + +Therefore the route finder API also needs modifications. + +## Router Changes + +The `router` is responsible for creating and keeping track of these uni-directional routes. Internally, it uses the *routing table*, *route finder client* and *setup client*. + +```golang +// DialOptions are options when dialing routes. +type DialOptions struct { + MinForwardRts int + MaxForwardRts int + MinConsumeRts int + MaxConsumeRts int +} + +type Router interface { + io.Closer + + // DialRoutes dials to a given visor of 'rPK'. + // 'lPort'/'rPort' specifies the local/remote ports respectively. + // A nil 'opts' input results in a value of '1' for all DialOptions fields. + // A single call to DialRoutes should perform the following: + // - Find routes via RouteFinder (in one call). + // - Setup routes via SetupNode (in one call). + // - Save to routing.Table and internal RouteGroup map. + // - Return RouteGroup if successful. + DialRoutes(ctx context.Context, rPK cipher.PubKey, lPort, rPort uint16, opts *DialOptions) (*RouteGroup, error) + + // AcceptsRoutes should block until we recieve an AddRules packet from SetupNode that contains ConsumeRule(s) or ForwardRule(s). Then the following should happen: + // - Save to routing.Table and internal RouteGroup map. + // - Return the RoutingGroup. + AcceptRoutes() (*RouteGroup, error) +} + +type router { + rt RoutingTable // routing table + tm transport.Manager // transport manager + rfc rfclient.Client // route finder client + rsc rsclient.Client // route setup client + rgs map[RouteDescriptor]*RouteGroup // route groups to push incoming reads from transports. +} +``` + +### `RouteGroup` structure + +A `RouteGroup` is responsible for input/output via rules. In the future, we may read/write via multiple transports, hence we need a way of reordering the packets. + +A `RouteGroup` is created either when we: +- Initiate a set a routes. +- Receive a set of routes. + +```golang +// RouteGroup should implement 'io.ReadWriteCloser'. +type RouteGroup struct { + desc RouteDescriptor // describes the route group + fwd []Rule // forward rules (for writing) + rvs []Rule // reverse rules (for reading) + + // The following fields are used for writing: + // - fwd/tps should have the same number of elements. + // - the corresponding element of tps should have tpID of the corresponding rule in fwd. + // - rg.fwd references 'ForwardRule' rules for writes. + + // 'tps' is transports used for writing/forward rules. + // It should have the same number of elements as 'fwd' + // where each element corresponds with the adjacent element in 'fwd'. + tps []*transport.ManagedTransport + + // 'readCh' reads in incoming packets of this route group. + // - Router should serve call '(*transport.Manager).ReadPacket' in a loop, + // and push to the appropriate '(RouteGroup).readCh'. + readCh <-chan []byte // push reads from Router + readBuf bytes.Buffer // for read overflow +} +``` + +#### Routing packets + +The unit of communication for routing/router is called packets. + +``` +| type (byte) | route ID (uint32) | payload size (uint16) | payload (~) | +| 1[0:1] | 4[1:5] | 2[5:7] | [7:~] | +``` + +packet types: +- `DataPacket` - Payload is just the underlying data. +- `ClosePacket` - Payload is a `type CloseCode byte`. +- `KeepAlivePacket` - Payload is empty. + +#### Reading mechanism of `RouteGroup` + +The `Router`, via `transport.Manager`, is responsible for reading incoming packets and pushing it to the appropriate `RouteGroup` via `(*RouteGroup).readCh`. + +To help with implementing the read logic, within the `dmsg` repo, we have [`ioutil.BufRead`](https://github.com/skycoin/dmsg/blob/master/ioutil/buf_read.go), just in case the read buffer is short. + +#### Writing mechanism of `RouteGroup` + +For the first version, only the first `ForwardRule` (`fwd[0]`) is used for writing. + +#### Closing the `RouteGroup` + +- Send `Close` packet for all `ForwardRule`s. +- Delete all rules (`ForwardRule`s and `ConsumeRule`s) from routing table. +- Close all go channels. + +#### KeepAlive mechanism + +The keepAlive value for routing rules is rather short to ensure that rules are deleted when unused. It is the responsibility for the source edge of each route (or the node with the `ForwardRule`) to send keepAlive packets. + +The `RouteGroup` is responsible for ensuring that at least one packet is sent every `keepAlive/2`. This can be a data packet or a keepAlive packet. If data packets are delivered with a high frequency, we will not need to send keepAlive packets. \ No newline at end of file diff --git a/skywire-specs/transports/stcpr/Specs.md b/skywire-specs/transports/stcpr/Specs.md new file mode 100644 index 0000000000..05dd83e830 --- /dev/null +++ b/skywire-specs/transports/stcpr/Specs.md @@ -0,0 +1,22 @@ +# STCP transport with address resolving + +STCPR transport work the same way as STCP, +but uses the address-resolver service instead of PK table to determine an address by a PK. + +### STCPR description + +Address-resolver has the following HTTP methods for resolving PKs to IPs: + +- `POST` `/bind/stcpr` + +It is used to bind PKs that visors send on start with their addresses. It requires PK authorization. + +The request format is a JSON with a port visor listens on and with a list of visor local addresses. + +- `GET` `/resolve/stcpr/{pk}` + +It is used by dialing visor to resolve public key to address of dialed visor. It requires PK authorization. + +- `/security/nonces/` + +It is used by `httpauth` middleware for public key authorization for both binding and resolving addresses. diff --git a/skywire-specs/transports/sudph/Specs.md b/skywire-specs/transports/sudph/Specs.md new file mode 100644 index 0000000000..1e5fe8fdc8 --- /dev/null +++ b/skywire-specs/transports/sudph/Specs.md @@ -0,0 +1,77 @@ +# Hole-punch transport + +Hole-punch transport uses a point-to-point connection between two parties that are behind NATs or firewalls. + +### UDP hole-punch transport description + +To be able to establish a connection between two visors (`A` and `B`) using UDP hole-punching, +an intermediate server (`S`) between two nodes is necessary. + +The connection to `S` and then connections between `A` and `B` should be created from the same address. + +On start, `A` and `B` listen on ports `AP` and `BP` and send UDP requests to `S` to port `SP` +to register their addresses in `S`. + +This connection should be kept alive within the lifecycle of hole-punching transport to keep NAT mapping. +When `A` tries to hole-punch `B`, +it needs to send packets from the same port `AP` (to reuse existing NAT mapping) to port `BP`. +What about `B`, it listens on the same port `BP` for a connection from `AP`. +So, we need to have 2 connections on `A` (`AP` -> `SP`, `AP` -> `BP`) +and 2 connections on `B` (`BP` -> `SP`, `BP` -> `AP`). + +The local port is same for both connections, therefore UDP listener (`*net.UDPConn`) is shared between both connections. +So in fact, connections to server and other visor reuse the same UDP connection, but remote ports and addresses differ. +When a new [KCP](https://github.com/xtaci/kcp-go) connection is created from a `*net.UDPConn`, +it requires setting remote address and drops packets from different addresses. +Therefore, if `*net.UDPConn` is wrapped by KCP, there's a question how to receive packets from different remote addresses, +which is required for UDP hole punching. + +To solve that, a middleware between `*net.UDPConn` and KCP is used. +It extracts packets sent from different remote addresses. +Packets sent from an expected by KCP address would be passed further to KCP. + +#### Connection from visor to address resolving server + +As written above, visors use the connection for sending a public key they want to connect to +and for receiving an IP they are asked to connect to. Therefore, the connection is bi-directional. +Visor sends an HTTP request which is upgraded to websocket connection. +When visor sends its public key, it should be authenticated. For that, `httpauth` is used for the HTTP request. + +#### Address resolving server + +Address resolving server needs to store visor address and its corresponding address. +The server has two types of storage: + +- non-persistent in-memory storage +- persistent `Redis` storage + +Address-resolver has two types of API that work with UDP hole-punching + +- HTTP +- UDP + +The `--addr` argument specifies an address address-resolver listens both HTTP and UDP on. + +The UDP server is used to bind PKs that visors send on start with their addresses. +PKs are authorized by a handshake procedure same with other handshakes (using noise). + +- `GET` `/resolve/sudph/{pk}` + +It is used by dialing visor to resolve public key to address of dialed visor. It requires PK authorization. + +- `/security/nonces/` + +It is used by `httpauth` middleware for public key authorization for resolving addresses. + +#### Communication format + +Visor starts UDP communication with address-resolver and performs a noise handshake with it. +Then it sends a JSON with a port it listens UDP connections on +and with local IPs to be able to connect with local IPs locally. + +If address resolver sees that external IPs of visors trying to connect to each other are same, +it allows them to try to connect to a local IP first. + +## Useful links + +- https://bford.info/pub/net/p2pnat/index.html \ No newline at end of file