Support new protocol: Earthstar

[cinnamon-bun]

Please review Support New Protocol and address how the considerations can be fulfilled by the suggested protocol. In cases where they cannot be fulfilled, please explain why and offer alternatives.

Notes

Earthstar isn't quite ready for this publishing use-case yet, but we're working in this direction! It might be a couple of months until we're ready.

Background on Earthstar

Earthstar is first and foremost a data format for collaboration using signed documents (specification), along with a server and client library for handling that format, written in Typescript.

You can think of us as "like SSB or Dat, but mutable and deletable instead of append-only, easier collaboration on data, with support for multiple devices per user, and it works in regular browsers". Or "like CouchDb, but with signed documents so anyone can help run servers without requiring full trust."

We've started with HTTP servers and regular browser clients. We'll add direct p2p support later (via hyperswarm). Any way you can get two peers to connect, you can sync their data, we just haven't focused much on the connection part yet.

Earthstar data is stored in workspaces which are like shared folders or Slack groups. Workspaces hold documents which are signed lumps of data, basically files. The data does not form a hash chain or other structure; each document is separate.

The data is hosted on pub server(s). Pub servers have no authority, are interchangeable and redundant. Anyone can run their own pub server to add redundancy to a workspace.

There is no auto-discovery of peers and no global network. Each workspace is a little island. You can only join the workspaces you know about, and you have to know which pub servers are hosting them. Workspace names and pub addresses use "anyone who knows the link" style privacy. So data does not spread very far unless you announce it outside of Earthstar.

What flavor of distributed is this?

Earthstar is best for collaborative tools (wikis, blogs, games, todo lists, etc) but it should also support this publishing use-case.

Without direct p2p support yet, this is more like "Mastodon but with fungible, redundant servers that anyone can host easily". Or like hosting a website mirror, you can easily host a "mirror pub" which syncs with the original pub. "Easily archivable sites".

Users can also sync the entire content down to their device for use offline.

How to operate a node

Pubs vs Apps

The pub server is a small nodeJs program that stores the data in an sqlite file. It serves an HTTP API for syncing, and a basic HTML view for browsing around workspaces. But normally you don't look at pubs directly, you go through apps.

Apps are is a standalone single-page apps hosted in the traditional way, and they use the Earthstar TypeScript library to reach out to pub server(s) to sync data to the browser's localStorage. They can do dynamic things like render chatrooms, let you write comments on blog entries, etc. An example app is Twodays Crossing which is a chat app where messages disappear after 2 days.

To run a pub server, use the earthstar-pub package:

npm install -g earthstar-pub
earthstar-pub --help

Settings: A pub server can be "open" or "closed" (does it accept newly pushed workspaces?). It can also be "discoverable" or not (can visitors learn about existing workspaces?)

URL scheme and domain name system

Mauve and I have been working out a URL scheme here, for use in Agregore browser. This is early work:

https://github.com/earthstar-project/earthstar-fetch/tree/initial-implementation

Usually apps fetch Earthstar data through the pub HTTP API and render it in custom ways. This repo above is a shift to a model where the pub server directly hosts and renders Earthstar documents as webpages.

Publishing process and cost (computation or financial) for content and DNS

Earthstar is not a blockchain and nothing costs money except having some kind of hosting on the internet somewhere.

Each published file must be hashed and signed once, which is very fast. We'd need a little script to read a directory of HTML files and dump them into a local Earthstar pub, and then you'd sync that pub to your production pub servers.

You can update your content as frequently as you like. Clients can even live-stream changes as they're published.

Pub servers need to

• be publically routable
• have HTTPS so browsers won't complain about mixed content
• have a persistent filesystem to store their SQLite files

We've run them on glitch.com, raspberry pis with PageKite, DigitalOcean droplets, etc.

HTTP gateway support

Right now, our apps are hosted anywhere as static sites, and they reach out to pub servers to get content.

A simpler model is to have the pub server also serve the public-facing website content, especially if it's just HTML and not interactive.

It would be pretty easy to add some URLs to the pub server to do this.

• Serve web content out of Earthstar documents right from the pub earthstar-project/earthstar-pub#13
• Supporting a Publishing use-case earthstar-project/earthstar#116

We also need to add a separation between reading and writing permissions, so that you can publish without giving away your write key. This is also fairly simple but it's a few months down our roadmap. We call it "invite-only workspaces" (the invite is for writing; anyone can read if they know the address.)

Thanks!

We're just a couple of people building an non-corporate tool for communities and we want it to be useful. Feedback is very welcome, please reach out with questions to our Discord.

💜 🌱 Cinnamon