diff --git a/_typos.toml b/_typos.toml new file mode 100644 index 00000000..01f876a5 --- /dev/null +++ b/_typos.toml @@ -0,0 +1,2 @@ +[files] +extend-exclude = ["*.svg"] diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 30360cc0..c3cc99d1 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -39,6 +39,8 @@ - [Bonus: Set a SuiNS name](./walrus-sites/tutorial-suins.md) - [Advanced configuration](./walrus-sites/tutorial-config.md) - [Technical overview](./walrus-sites/overview.md) + - [Linking from and to Walrus Sites](./walrus-sites/linking.md) + - [Redirecting objects to Walrus Sites](./walrus-sites/redirects.md) - [The site builder](./walrus-sites/site-builder.md) - [The Walrus Sites Portal](./walrus-sites/portal.md) - [Known restrictions](./walrus-sites/restrictions.md) diff --git a/docs/assets/walrus-site-diagram.svg b/docs/assets/walrus-site-diagram.svg new file mode 100644 index 00000000..37ec7746 --- /dev/null +++ b/docs/assets/walrus-site-diagram.svg @@ -0,0 +1 @@ +Web BrowserService WorkerBrowser TabSuiFull NodePortal providerPortalbsite.netLoading PageWorker LogicObject 0x1234...Metadata -> Blob ID(asdf...)Metadata -> Blob ID(ghjk...)User Website DevThe browser requests & processes dataWalrusStorage Node 1Blob ID(asdf...)dapp HTMLBlob ID(ghjk...)dapp ImagesPublisherStorage Node 2CacheAggregatorCache 13a/ The Aggregator reconstructs the blob from the slivers  0b/ The publisher encodes the blob into slivers and reserves space for it on Sui(11b/ Request Walrus blob with blob ID(ghjk...) - same as 11a/)HitMiss8/ Return ObjID "0x1234..."12b/ Return sliver12b/ Return sliver3/ Return loading page Return response6/ GET dapp.walrus.site10/ Return object (BCS bytes)13b/ Cache blob0a/ Push website files to Walrus0c/15/ prepareresponse0e/ Publish site metadata(blob IDs) on Sui16/ Display dapp.walrus.site14/ Return cached blob2/ GET dapp.walrus.site12a/ Request ID(asdf...)5/ Reload0d/ Return blob IDs (asdf...) (ghjk...)4/ Install generic service workerfor *.walrus.site0c/ Slivers to every storage node11a/ Request Walrus blob with blob ID(asdf...)7/ Resolve SuiNs "dapp.sui"12a/ Request ID(asdf...)1/ Request dapp.walrus.site9/ Fetch object 0x1234... diff --git a/docs/walrus-sites/linking.md b/docs/walrus-sites/linking.md new file mode 100644 index 00000000..4a15c83b --- /dev/null +++ b/docs/walrus-sites/linking.md @@ -0,0 +1,42 @@ +# Linking from and to Walrus Sites + +Links in Walrus Sites work _almost_ as you would expect in a regular website. We specify here a few +of the details. + +## Linking to resources within the same site + +Relative and absolute links (`href="/path/to/resource.html"`) work as usual. + +## Linking to resources on the Web + +Linking to a resource on the web (`href="https://some.cdn.example.com/stylesheet.css"`) also work as +usual. + +## Linking to resources in other Walrus Sites + +Here is the part that is a bit different. Assume there is some image that you can browse at +`https://gallery.walrus.site/walrus_arctic.webp`, and you want to link it from your own Walrus Site. + +Recall that, however, `https://walrus.site` is just one of the possibly many Portals. I.e., the same +resource is browsable from a local portal (`http://gallery.localhost:8080/walrus_arctic.webp`), or +from any other portal (e.g., `https://gallery.myotherportal.com/walrus_arctic.webp`). Therefore, how +can you link the resource in a _Portal independent way_? This is important for interoperability, +availability, and respecting the user's choice of portal. + +### The solution: Walrus Sites links + +We solve this problem by having the Portals interpret special links, that are normally invalid on +the Web, and redirect to the corresponding Walrus Sites resource in the portal itself. + +Consider the example above, where the resource `/walrus_arctic.webp` is browsed from the Walrus Site +with SuiNS name `gallery`, which points to the object ID `abcd123…` (in Base36 encoding). Then, +the Portal-independent link is: `https://gallery.suiobj/walrus_arctic.webp`. To fix the object ID +instead of the SuiNS name, you can use `https://abcd123….suiobj/walrus_arctic.webp`. + +Another possibility is to directly point to the Walrus _blob ID_ of the resource, and have the +browser "sniff" the content type. This works for images, for example, but not for script or +stylesheets. For example to point to the blob ID (e.g., containing an image) `qwer5678…`, use the +URL `https://blobid.walrus/qwer5678…`. + +With such a link, the Portal will extract the blob ID and redirect the request to the aggregator it +is using to fetch blobs. diff --git a/docs/walrus-sites/overview.md b/docs/walrus-sites/overview.md index 07dd0c5c..a79be320 100644 --- a/docs/walrus-sites/overview.md +++ b/docs/walrus-sites/overview.md @@ -1 +1,125 @@ -# Overview +# Technical Overview + +In the following sections, we delve deeper in the technical specification of Walrus Sites. + +## High-level picture + +Walrus Sites are enabled by Sui and Walrus. The sites' resources (`html`, `css`, `js`, images, +etc.) are stored on Walrus, while the main entry points to the sites are objects stored on Sui, +which contain the metadata for the site and point to the Walrus blob IDs. + +### The Walrus Sites objects on Sui + +A Walrus `Site` is represented on Sui as a very simple object: + +``` move +struct Site has key, store { + id: UID, + name: String, +} +``` + +The resources associated with this site are then added to this object as [dynamic +fields](https://docs.sui.io/concepts/dynamic-fields/) of type `Resource`: + +``` move +struct Resource has store, drop { + path: String, + content_type: String, + content_encoding: String, + // The walrus blob id containing the bytes for this resource + blob_id: u256, +} +``` + +Each resource contains: + +- The `path` of the resource, for example `/index.html` (all the paths are always represented as + starting from root `/`); +- the `content_type` of the resource, for example `text/html`; +- the `content_encoding` of the resource. At the moment the only available value is `plaintext`; and +- the `blob_id`, which is the Walrus blob ID where the resource can be found. + +These `Resource` dynamic fields are keyed with a struct of type `ResourcePath` + +``` move +struct ResourcePath has copy, store, drop { + path: String, +} +``` + +This struct just holds the string of the path (`/index.html`); having a separate type ensures that +we will not have namespace collisions with other dynamic fields, possibly added by other packages. + +To see this in action, look at [a Walrus Site in the +explorer](https://suiscan.xyz/testnet/object/0x049b6d3f34789904efcc20254400b7dca5548ee35cd7b5b145a211f85b2532fa), +and check its dynamic fields. + +### The site rendering path + +Given the Sui object ID of a Walrus Site, then, it is easy to look up the resources that compose it +by looking at the dynamic fields, and then fetch these resources from Walrus using the blob ID +contained in the `Resource` struct. + +The only outstanding question is, therefore, how to perform these lookups on the client. A few +approaches are possible: + +- Having a server that accepts requests for a Sui object ID and possibly a path, and performs this + resolution on behalf of the client, serving back the resource as a standard HTML Response. +- Using a custom application on the client, that has both a web browser and knowledge of how Walrus + Sites work, and can locally perform this resolution. +- A hybrid approach based on service workers, where a service worker that is able to perform the + resolution is installed in the browser from a Portal. + +All of these approaches are viable (the first has been used for similar applications in IPFS +gateways, for example), and have trade offs. As an initial step, we have chosen to use the +service-worker based approach, as it is light weight and ensures that the Portal does not have to +process all the traffic from clients. In the following, therefore, we present the details of this +Portal-based approach. + +### Browsing and domain isolation + +We must ensure that, when browsing multiple sites through a Portal, for example the one hosted at +, these sites are isolated. Isolation is necessary for security, and to ensure +that the wallet connection in the browser works as expected. + +To do so, we give each Walrus Site a specific _subdomain_ of the Portal's domain. For example, the +Flatland mint dApp is hosted at , where the subdomain `flatland` is +uniquely associated to the object ID of the Walrus Site through SuiNS. + +Walrus Sites also work without SuiNS: a site can _always_ be browsed by using as subdomain the +Base36 encoding of the Sui object ID of the site. For the Flatland dApp, this URL is: + . + +Base36 was chosen for two reasons, forced by the subdomain standards: + +1. A subdomain can have at most 63, while a Hex-encoded Sui object ID requires 64. +1. A subdomain is case _insensitive_, ruling out other popular encodings, e.g., Base64 or Base58. + +## The end-to-end resolution of a Walrus Site + +We now show in greater detail how a Walrus Site is rendered in a client's browser with the service +worker approach. The steps below all reference the following figure: + +![Walrus Site resolution](../assets/walrus-site-diagram.svg) + +- **Site publishing** (step 0) The site developer publishes the Walrus Site using the + [`site-builder`](./site-builder.md), or making use of a publisher. Assume the developer uses the + SuiNS name `dapp.sui` to point to the object ID of the created Walrus Site. +- **Browsing starts** (step 1) A client browses `dapp.walrus.site/index.html` in their browser. +- **Service worker installation** (steps 2-6) The browser connects to the Portal hosted at + `walrus.site`, which responds with a page that installs the service worker for + `dapp.walrus.site`. The page is refreshed to activate the service worker. +- **Site resolution** (steps 7-10) The service worker, which is now installed, interprets its + _origin_ `dapp.walrus.site`, and makes a SuiNS resolution for `dapp.sui`, obtaining the relative + object ID. Using the object ID, it then fetches the dynamic fields of the object (also checking + [redirects](./portal.md)). From the dynamic fields, it selects the one for `/index.html`, and + extracts its Walrus blob ID and content type. +- **Blob fetch** (steps 11-14) Given the blob ID, the service worker queries a Walrus aggregator for + the blob. +- **Returning the response** (steps 15-16) Now that the service worker has the bytes for + `/index.html`, and its `content_type`, it can craft a response that is then rendered by the + browser. + +These steps are executed for all resources the browser may query thereafter (for example, if +`/index.html` points to `assets/cat.png`). diff --git a/docs/walrus-sites/portal.md b/docs/walrus-sites/portal.md index 728c8ca9..c1b92598 100644 --- a/docs/walrus-sites/portal.md +++ b/docs/walrus-sites/portal.md @@ -1 +1,48 @@ # The Walrus Sites Portal + +We use the term "Portal" to indicate any technology that is used to access an browse Walrus Sites. +As mentioned in the [overview](./overview.md#the-site-rendering-path), we foresee three kinds of +Portals: + +1. Server-side Portals; +1. custom local apps; and +1. service-worker based Portals in the browser. + +Currently, only the service-worker based Portal is available. + +## Running the Portal locally + +You can run a service worker Portal locally: + +- To browse Walrus Sites without accessing external Portals; or +- for development purposes. + +This requires having the [`pnpm`](https://pnpm.io/) tool installed. To start, clone the +`walrus-sites` repo and enter the `portal` directory. Here, run: + +``` sh +pnpm install +``` + +to install the dependencies, and + +``` sh +pnpm serve +``` + +to serve the Portal. Typically, you will find it served at `localhost:8080` (but check the output of +the serve command). + +## Configuring the Portal + +The most important configuration parameters for the Portal are in `constants.ts`. + +- `NETWORK`: The Sui network to be used for fetching the Walrus Sites objects. Currently, we + use Sui `testnet`. +- `AGGREGATOR`: The URL of the [aggregator](../usage/web-api.md) from which the service worker will + fetch the Walrus blobs. +- `SITE_PACKAGE`: "0x514cf7ce2df33b9e2ca69e75bc9645ef38aca67b6f2852992a34e35e9f907f58" +- `MAX_REDIRECT_DEPTH`: The number of [redirects](./redirects.md) the service worker will follow + before stopping. +- `SITE_NAMES`: Hard coded `name: objectID` mappings, to override the SuiNS names. For development + only. diff --git a/docs/walrus-sites/redirects.md b/docs/walrus-sites/redirects.md new file mode 100644 index 00000000..4ed1f2df --- /dev/null +++ b/docs/walrus-sites/redirects.md @@ -0,0 +1,41 @@ +# Redirecting objects to Walrus Sites + +We have seen in the [overview](./overview.md) how a Walrus Site object on Sui looks like. We will +discuss now how you can create ensure that a _set of arbitrary objects_ can all be tied to a +specific, and possibly unique, Walrus site. + +## The goal + +Consider a collection of NFTs, such as the one published by . As we +show there, each minted NFT has its own Walrus Site, that can be personalized based on the contents +(e.g., the color) of the NFT itself. How can we achieve this? + +## Redirect links + +The solution is simple: We add a "redirect" in the NFT's +[`Display`](https://docs.sui.io/standards/display#sui-utility-objects) property. Each time an NFT's +object ID is browsed through a Portal, the Portal will check the `Display` of the NFT and, if it +encounters the `walrus site address` key, it will go fetch the Walrus Site that is at the +corresponding object ID. + +### Redirects in Move + +Practically speaking, when creating the `Display` of the NFT, you can include the key-value pair +that points to the Walrus site that is to beused. + +``` move +... +const VISUALIZATION_SITE: address = @0x901fb0...; +display.add(b"walrus site address".to_string(), VISUALIZATION_SITE.to_string()); +... +``` + +### How to personalize based on the NFT? + +The code above will only open the specified Walrus Site when browsing the object ID of the NFT. How +do we ensure that the properties of the NFT can be used to personalize the site? + +This needs to be done in the `VISUALIZATION_SITE`: Since the subdomain is still pointing to the +NFT's object ID, the Walrus Site that is loaded can check its `origin` in Javascript, and use the +subdomain to determine the NFT, fetch it from chain, and use its internal fields to modify the +displayed site. diff --git a/docs/walrus-sites/site-builder.md b/docs/walrus-sites/site-builder.md index f54a2592..8d4e45ad 100644 --- a/docs/walrus-sites/site-builder.md +++ b/docs/walrus-sites/site-builder.md @@ -1 +1,26 @@ # The site builder + +To facilitate the creation of Walrus Sites, we provide the "site builder" tool. The site builder +takes care of creating Walrus Sites object on Sui, with the correct structure, and stores the site +resources to Walrus. + +## Site builder commands + +The site builder tool exposes the following commands: + +- `publish`: Allows to publish a specific directory to Walrus. The directory must contain files that + can be served with HTTP, and have an `index.html` file. This command will return a Sui object ID + for the newly created site. +- `update`: After creating a site, you can update it with this command. It takes as input a + directory, as above, with the new or updated files, and the object ID of the site to update. Note + that the wallet you are using must be the _owner_ of the Walrus Site object to be able to update + it. This command will remove and create resources as required, to ensure that the Walrus Sites + object on Sui matches the local directory. If run with `--watch`, this command re-updates the site + _every time a file in the directory changes_. This is useful during development, but pay attention + to costs! +- `convert`: A utility tool. Given a Sui object ID in Hex format, it converts it to Base36. This is + useful if you know the Sui object ID of a site, and want to find the URL. +- `sitemap`: A utility tool. For a give Walrus Sites Sui object ID, prints out all the resources + that compose the site and their object ID. + +Check the commands' `--help` for more information.