feat: add technical overview and walrus site diagram

_typos.toml

@@ -0,0 +1,2 @@
+[files]
+extend-exclude = ["*.svg"]

docs/SUMMARY.md

@@ -39,6 +39,8 @@
   - [Bonus: Set a SuiNS name](./walrus-sites/tutorial-suins.md)
   - [Advanced configuration](./walrus-sites/tutorial-config.md)
 - [Technical overview](./walrus-sites/overview.md)
+  - [Linking from and to Walrus Sites](./walrus-sites/linking.md)
+  - [Redirecting objects to Walrus Sites](./walrus-sites/redirects.md)
   - [The site builder](./walrus-sites/site-builder.md)
   - [The Walrus Sites Portal](./walrus-sites/portal.md)
   - [Known restrictions](./walrus-sites/restrictions.md)

docs/walrus-sites/linking.md

@@ -0,0 +1,42 @@
+# Linking from and to Walrus Sites
+
+Links in Walrus Sites work _almost_ as you would expect in a regular website. We specify here a few
+of the details.
+
+## Linking to resources within the same site
+
+Relative and absolute links (`href="/path/to/resource.html"`) work as usual.
+
+## Linking to resources on the Web
+
+Linking to a resource on the web (`href="https://some.cdn.example.com/stylesheet.css"`) also work as
+usual.
+
+## Linking to resources in other Walrus Sites
+
+Here is the part that is a bit different. Assume there is some image that you can browse at
+`https://gallery.walrus.site/walrus_arctic.webp`, and you want to link it from your own Walrus Site.
+
+Recall that, however, `https://walrus.site` is just one of the possibly many Portals. I.e., the same
+resource is browsable from a local portal (`http://gallery.localhost:8080/walrus_arctic.webp`), or
+from any other portal (e.g., `https://gallery.myotherportal.com/walrus_arctic.webp`). Therefore, how
+can you link the resource in a _Portal independent way_? This is important for interoperability,
+availability, and respecting the user's choice of portal.
+
+### The solution: Walrus Sites links
+
+We solve this problem by having the Portals interpret special links, that are normally invalid on
+the Web, and redirect to the corresponding Walrus Sites resource in the portal itself.
+
+Consider the example above, where the resource `/walrus_arctic.webp` is browsed from the Walrus Site
+with SuiNS name `gallery`, which points to the object ID `abcd123…` (in Base36 encoding). Then,
+the Portal-independent link is: `https://gallery.suiobj/walrus_arctic.webp`.  To fix the object ID
+instead of the SuiNS name, you can use `https://abcd123….suiobj/walrus_arctic.webp`.
+
+Another possibility is to directly point to the Walrus _blob ID_ of the resource, and have the
+browser "sniff" the content type. This works for images, for example, but not for script or
+stylesheets. For example to point to the blob ID (e.g., containing an image) `qwer5678…`, use the
+URL `https://blobid.walrus/qwer5678…`.
+
+With such a link, the Portal will extract the blob ID and redirect the request to the aggregator it
+is using to fetch blobs.

docs/walrus-sites/overview.md

@@ -1 +1,125 @@
-# Overview
+# Technical Overview
+
+In the following sections, we delve deeper in the technical specification of Walrus Sites.
+
+## High-level picture
+
+Walrus Sites are enabled by Sui and Walrus.  The sites' resources (`html`, `css`, `js`, images,
+etc.) are stored on Walrus, while the main entry points to the sites are objects stored on Sui,
+which contain the metadata for the site and point to the Walrus blob IDs.
+
+### The Walrus Sites objects on Sui
+
+A Walrus `Site` is represented on Sui as a very simple object:
+
+``` move
+struct Site has key, store {
+    id: UID,
+    name: String,
+}
+```
+
+The resources associated with this site are then added to this object as [dynamic
+fields](https://docs.sui.io/concepts/dynamic-fields/) of type `Resource`:
+
+``` move
+struct Resource has store, drop {
+    path: String,
+    content_type: String,
+    content_encoding: String,
+    // The walrus blob id containing the bytes for this resource
+    blob_id: u256,
+}
+```
+
+Each resource contains:
+
+- The `path` of the resource, for example `/index.html` (all the paths are always represented as
+  starting from root `/`);
+- the `content_type` of the resource, for example `text/html`;
+- the `content_encoding` of the resource. At the moment the only available value is `plaintext`; and
+- the `blob_id`, which is the Walrus blob ID where the resource can be found.
+
+These `Resource` dynamic fields are keyed with a struct of type `ResourcePath`
+
+``` move
+struct ResourcePath has copy, store, drop {
+    path: String,
+}
+```
+
+This struct just holds the string of the path (`/index.html`); having a separate type ensures that
+we will not have namespace collisions with other dynamic fields, possibly added by other packages.
+
+To see this in action, look at [a Walrus Site in the
+explorer](https://suiscan.xyz/testnet/object/0x049b6d3f34789904efcc20254400b7dca5548ee35cd7b5b145a211f85b2532fa),
+and check its dynamic fields.
+
+### The site rendering path
+
+Given the Sui object ID of a Walrus Site, then, it is easy to look up the resources that compose it
+by looking at the dynamic fields, and then fetch these resources from Walrus using the blob ID
+contained in the `Resource` struct.
+
+The only outstanding question is, therefore, how to perform these lookups on the client. A few
+approaches are possible:
+
+- Having a server that accepts requests for a Sui object ID and possibly a path, and performs this
+  resolution on behalf of the client, serving back the resource as a standard HTML Response.
+- Using a custom application on the client, that has both a web browser and knowledge of how Walrus
+  Sites work, and can locally perform this resolution.
+- A hybrid approach based on service workers, where a service worker that is able to perform the
+  resolution is installed in the browser from a Portal.
+
+All of these approaches are viable (the first has been used for similar applications in IPFS
+gateways, for example), and have trade offs. As an initial step, we have chosen to use the
+service-worker based approach, as it is light weight and ensures that the Portal does not have to
+process all the traffic from clients. In the following, therefore, we present the details of this
+Portal-based approach.
+
+### Browsing and domain isolation
+
+We must ensure that, when browsing multiple sites through a Portal, for example the one hosted at
+<https://walrus.site>, these sites are isolated. Isolation is necessary for security, and to ensure
+that the wallet connection in the browser works as expected.
+
+To do so, we give each Walrus Site a specific _subdomain_ of the Portal's domain. For example, the
+Flatland mint dApp is hosted at <https://flatland.walrus.site>, where the subdomain `flatland` is
+uniquely associated to the object ID of the Walrus Site through SuiNS.
+
+Walrus Sites also work without SuiNS: a site can _always_ be browsed by using as subdomain the
+Base36 encoding of the Sui object ID of the site. For the Flatland dApp, this URL is:
+<https://44terjw9uzwbmtful0387e2bx3k3ro64s0it82hw9x9sz4ttm.blocksite.net> .
+
+Base36 was chosen for two reasons, forced by the subdomain standards:
+
+1. A subdomain can have at most 63, while a Hex-encoded Sui object ID requires 64.
+1. A subdomain is case _insensitive_, ruling out other popular encodings, e.g., Base64 or Base58.
+
+## The end-to-end resolution of a Walrus Site
+
+We now show in greater detail how a Walrus Site is rendered in a client's browser with the service
+worker approach. The steps below all reference the following figure:
+
+![Walrus Site resolution](../assets/walrus-site-diagram.svg)
+
+- **Site publishing** (step 0) The site developer publishes the Walrus Site using the
+  [`site-builder`](./site-builder.md), or making use of a publisher. Assume the developer uses the
+  SuiNS name `dapp.sui` to point to the object ID of the created Walrus Site.
+- **Browsing starts** (step 1) A client browses `dapp.walrus.site/index.html` in their browser.
+- **Service worker installation** (steps 2-6) The browser connects to the Portal hosted at
+  `walrus.site`, which responds with a page that installs the service worker for
+  `dapp.walrus.site`. The page is refreshed to activate the service worker.
+- **Site resolution** (steps 7-10) The service worker, which is now installed, interprets its
+  _origin_ `dapp.walrus.site`, and makes a SuiNS resolution for `dapp.sui`, obtaining the relative
+  object ID. Using the object ID, it then fetches the dynamic fields of the object (also checking
+  [redirects](./portal.md)). From the dynamic fields, it selects the one for `/index.html`, and
+  extracts its Walrus blob ID and content type.
+- **Blob fetch** (steps 11-14) Given the blob ID, the service worker queries a Walrus aggregator for
+  the blob.
+- **Returning the response** (steps 15-16) Now that the service worker has the bytes for
+  `/index.html`, and its `content_type`, it can craft a response that is then rendered by the
+  browser.
+
+These steps are executed for all resources the browser may query thereafter (for example, if
+`/index.html` points to `assets/cat.png`).

docs/walrus-sites/portal.md

@@ -1 +1,48 @@
 # The Walrus Sites Portal
+
+We use the term "Portal" to indicate any technology that is used to access an browse Walrus Sites.
+As mentioned in the [overview](./overview.md#the-site-rendering-path), we foresee three kinds of
+Portals:
+
+1. Server-side Portals;
+1. custom local apps; and
+1. service-worker based Portals in the browser.
+
+Currently, only the service-worker based Portal is available.
+
+## Running the Portal locally
+
+You can run a service worker Portal locally:
+
+- To browse Walrus Sites without accessing external Portals; or
+- for development purposes.
+
+This requires having the [`pnpm`](https://pnpm.io/) tool installed. To start, clone the
+`walrus-sites` repo and enter the `portal` directory. Here, run:
+
+``` sh
+pnpm install
+```
+
+to install the dependencies, and
+
+``` sh
+pnpm serve
+```
+
+to serve the Portal. Typically, you will find it served at `localhost:8080` (but check the output of
+the serve command).
+
+## Configuring the Portal
+
+The most important configuration parameters for the Portal are in `constants.ts`.
+
+- `NETWORK`: The Sui network to be used for fetching the Walrus Sites objects. Currently, we
+  use Sui `testnet`.
+- `AGGREGATOR`: The URL of the [aggregator](../usage/web-api.md) from which the service worker will
+  fetch the Walrus blobs.
+- `SITE_PACKAGE`: "0x514cf7ce2df33b9e2ca69e75bc9645ef38aca67b6f2852992a34e35e9f907f58"
+- `MAX_REDIRECT_DEPTH`: The number of [redirects](./redirects.md) the service worker will follow
+  before stopping.
+- `SITE_NAMES`: Hard coded `name: objectID` mappings, to override the SuiNS names. For development
+  only.

docs/walrus-sites/redirects.md

@@ -0,0 +1,41 @@
+# Redirecting objects to Walrus Sites
+
+We have seen in the [overview](./overview.md) how a Walrus Site object on Sui looks like. We will
+discuss now how you can create ensure that a _set of arbitrary objects_ can all be tied to a
+specific, and possibly unique, Walrus site.
+
+## The goal
+
+Consider a collection of NFTs, such as the one published by <https://flatland.walrus.site>. As we
+show there, each minted NFT has its own Walrus Site, that can be personalized based on the contents
+(e.g., the color) of the NFT itself. How can we achieve this?
+
+## Redirect links
+
+The solution is simple: We add a "redirect" in the NFT's
+[`Display`](https://docs.sui.io/standards/display#sui-utility-objects) property. Each time an NFT's
+object ID is browsed through a Portal, the Portal will check the `Display` of the NFT and, if it
+encounters the `walrus site address` key, it will go fetch the Walrus Site that is at the
+corresponding object ID.
+
+### Redirects in Move
+
+Practically speaking, when creating the `Display` of the NFT, you can include the key-value pair
+that points to the Walrus site that is to beused.
+
+``` move
+...
+const VISUALIZATION_SITE: address = @0x901fb0...;
+display.add(b"walrus site address".to_string(), VISUALIZATION_SITE.to_string());
+...
+```
+
+### How to personalize based on the NFT?
+
+The code above will only open the specified Walrus Site when browsing the object ID of the NFT. How
+do we ensure that the properties of the NFT can be used to personalize the site?
+
+This needs to be done in the `VISUALIZATION_SITE`: Since the subdomain is still pointing to the
+NFT's object ID, the Walrus Site that is loaded can check its `origin` in Javascript, and use the
+subdomain to determine the NFT, fetch it from chain, and use its internal fields to modify the
+displayed site.

docs/walrus-sites/site-builder.md

@@ -1 +1,26 @@
 # The site builder
+
+To facilitate the creation of Walrus Sites, we provide the "site builder" tool. The site builder
+takes care of creating Walrus Sites object on Sui, with the correct structure, and stores the site
+resources to Walrus.
+
+## Site builder commands
+
+The site builder tool exposes the following commands:
+
+- `publish`: Allows to publish a specific directory to Walrus. The directory must contain files that
+  can be served with HTTP, and have an `index.html` file. This command will return a Sui object ID
+  for the newly created site.
+- `update`: After creating a site, you can update it with this command. It takes as input a
+  directory, as above, with the new or updated files, and the object ID of the site to update. Note
+  that the wallet you are using must be the _owner_ of the Walrus Site object to be able to update
+  it. This command will remove and create resources as required, to ensure that the Walrus Sites
+  object on Sui matches the local directory. If run with `--watch`, this command re-updates the site
+  _every time a file in the directory changes_. This is useful during development, but pay attention
+  to costs!
+- `convert`: A utility tool. Given a Sui object ID in Hex format, it converts it to Base36. This is
+  useful if you know the Sui object ID of a site, and want to find the URL.
+- `sitemap`: A utility tool. For a give Walrus Sites Sui object ID, prints out all the resources
+  that compose the site and their object ID.
+
+Check the commands' `--help` for more information.