From 04ae76ba108b2382a2eb80a6131b2218a7cd411f Mon Sep 17 00:00:00 2001 From: giac-mysten Date: Wed, 12 Jun 2024 16:10:52 +0100 Subject: [PATCH 1/3] feat: add technical overview and walrus site diagram --- _typos.toml | 2 + docs/assets/walrus-site-diagram.svg | 1 + docs/walrus-sites/overview.md | 126 +++++++++++++++++++++++++++- 3 files changed, 128 insertions(+), 1 deletion(-) create mode 100644 _typos.toml create mode 100644 docs/assets/walrus-site-diagram.svg diff --git a/_typos.toml b/_typos.toml new file mode 100644 index 00000000..01f876a5 --- /dev/null +++ b/_typos.toml @@ -0,0 +1,2 @@ +[files] +extend-exclude = ["*.svg"] diff --git a/docs/assets/walrus-site-diagram.svg b/docs/assets/walrus-site-diagram.svg new file mode 100644 index 00000000..37ec7746 --- /dev/null +++ b/docs/assets/walrus-site-diagram.svg @@ -0,0 +1 @@ +Web BrowserService WorkerBrowser TabSuiFull NodePortal providerPortalbsite.netLoading PageWorker LogicObject 0x1234...Metadata -> Blob ID(asdf...)Metadata -> Blob ID(ghjk...)User Website DevThe browser requests & processes dataWalrusStorage Node 1Blob ID(asdf...)dapp HTMLBlob ID(ghjk...)dapp ImagesPublisherStorage Node 2CacheAggregatorCache 13a/ The Aggregator reconstructs the blob from the slivers  0b/ The publisher encodes the blob into slivers and reserves space for it on Sui(11b/ Request Walrus blob with blob ID(ghjk...) - same as 11a/)HitMiss8/ Return ObjID "0x1234..."12b/ Return sliver12b/ Return sliver3/ Return loading page Return response6/ GET dapp.walrus.site10/ Return object (BCS bytes)13b/ Cache blob0a/ Push website files to Walrus0c/15/ prepareresponse0e/ Publish site metadata(blob IDs) on Sui16/ Display dapp.walrus.site14/ Return cached blob2/ GET dapp.walrus.site12a/ Request ID(asdf...)5/ Reload0d/ Return blob IDs (asdf...) (ghjk...)4/ Install generic service workerfor *.walrus.site0c/ Slivers to every storage node11a/ Request Walrus blob with blob ID(asdf...)7/ Resolve SuiNs "dapp.sui"12a/ Request ID(asdf...)1/ Request dapp.walrus.site9/ Fetch object 0x1234... diff --git a/docs/walrus-sites/overview.md b/docs/walrus-sites/overview.md index 07dd0c5c..91a103d8 100644 --- a/docs/walrus-sites/overview.md +++ b/docs/walrus-sites/overview.md @@ -1 +1,125 @@ -# Overview +# Technical Overview + +In the following sections, we delve deeper in the technical specification of Walrus Sites. + +## High-level picture + +Walrus Sites are enabled by Sui and Walrus. The sites' resources (`html`, `css`, `js`, images, +etc.) are stored on Walrus, while the main entry points to the sites are objects stored on Sui, +which contain the metadata for the site and point to the Walrus blob IDs. + +### The Walrus Sites objects on Sui + +A Walrus `Site` is represented on Sui as a very simple object: + +``` move +struct Site has key, store { + id: UID, + name: String, +} +``` + +The resources associated with this site are then added to this object as [dynamic +fields](https://docs.sui.io/concepts/dynamic-fields/) of type `Resource`: + +``` move +struct Resource has store, drop { + path: String, + content_type: String, + content_encoding: String, + // The walrus blob id containing the bytes for this resource + blob_id: u256, +} +``` + +Each resource contains: + +- The `path` of the resource, for example `/index.html` (all the paths are always represented as + starting from root `/`); +- the `content_type` of the resource, for example `text/html`; +- the `content_encoding` of the resource. At the moment the only available value is `plaintext`; and +- the `blob_id`, which is the Walrus blob ID where the resource can be found. + +These `Resource` dynamic fields are keyed with a struct of type `ResourcePath` + +``` move +struct ResourcePath has copy, store, drop { + path: String, +} +``` + +This struct just holds the string of the path (`/index.html`); having a separate type ensures that +we will not have namespace collisions with other dynamic fields, possibly added by other packages. + +To see this in action, look at [a Walrus Site in the +explorer](https://suiscan.xyz/testnet/object/0x049b6d3f34789904efcc20254400b7dca5548ee35cd7b5b145a211f85b2532fa), +and check its dynamic fields. + +### The site rendering path + +Given the Sui object ID of a Walrus Site, then, it is easy to look up the resources that compose it +by looking at the dynamic fields, and then fetch these resources from Walrus using the blob ID +contained in the `Resource` struct. + +The only outstanding question is, therefore, how to perform these lookups on the client. A few +approaches are possible: + +- Having a server that accepts requests for a Sui object ID and possibly a path, and performs this + resolution on behalf of the client, serving back the resource as a standard HTML Response. +- Using a custom application on the client, that has both a web browser and knowledge of how Walrus + Sites work, and can locally perform this resolution. +- A hybrid approach based on service workers, where a service worker that is able to perform the + resolution is installed in the browser from a Portal. + +All of these approaches are viable (the first has been used for similar applications in IPFS +gateways, for example), and have trade offs. As an initial step, we have chosen to use the +service-worker based approach, as it is novel, light weight, and ensures that the Portal does not +have to process all the traffic from clients. In the following, therefore, we present the details of +this Portal-based approach. + +### Browsing and domain isolation + +We must ensure that, when browsing multiple sites through a Portal, for example the one hosted at +[https://walrus.site](https://walrus.site), these sites are isolated. Isolation is necessary for +security, and to ensure that the wallet connection in the browser works as expected. + +To do so, we give each Walrus Site a specific _subdomain_ of the Portal's domain. For example, the +Flatland mint dApp is hosted at [https://flatland.walrus.site](https://flatland.walrus.site), where +the subdomain `flatland` is uniquely associated to the object ID of the Walrus Site through SuiNS. + +Walrus Sites also work without SuiNS: a site can _always_ be browsed by using as subdomain the +Base36 encoding of the Sui object ID of the site. For the Flatland dApp, this URL is: + . + +Base36 was chosen for two reasons, forced by the subdomain standards: + +1. A subdomain can have at most 63, while a Hex-encoded Sui object ID requires 64. +1. A subdomain is case _insensitive_, ruling out other popular encodings, e.g., Base64 or Base58. + +## The end-to-end resolution of a Walrus Site + +We now show in greater detail how a Walrus Site is rendered in a client's browser with the service +worker approach. The steps below all reference the following figure: + +![Walrus Site resolution](../assets/walrus-site-diagram.svg) + +- **Site publishing** (step 0) The site developer publishes the Walrus Site using the + [`site-builder`](./site-builder.md), or making use of a publisher. Assume the developer uses the + SuiNS name `dapp.sui` to point to the object ID of the created Walrus Site. +- **Browsing starts** (step 1) A client browses `dapp.walrus.site/index.html` in their browser. +- **Service worker installation** (steps 2-6) The browser connects to the Portal hosted at + `walrus.site`, which responds with a page that installs the service worker for + `dapp.walrus.site`. The page is refreshed to activate the service worker. +- **Site resolution** (steps 7-10) The service worker, which is now installed, interprets its + _origin_ `dapp.walrus.site`, and makes a SuiNS resolution for `dapp.sui`, obtaining the relative + object ID. Using the object ID, it then fetches the dynamic fields of the object (also checking + [redirects](./portal.md)). From the dynamic fields, it selects the one for `/index.html`, and + extracts its Walrus blob ID and content type. +- **Blob fetch** (steps 11-14) Given the blob ID, the service worker queries a Walrus aggregator for + the blob. +- **Returning the response** (steps 15-16) Now that the service worker has the bytes for + `/index.html`, and its `content_type`, it can craft a response that is then rendered by the + browser. + +These steps are executed for all resources the browser may query thereafter (for example, if +`/index.html` points to `assets/cat.png`). From 1dc0ce9d5e3fdbaea62ca9ad77ce842f983f6e76 Mon Sep 17 00:00:00 2001 From: giac-mysten Date: Fri, 14 Jun 2024 15:03:01 +0100 Subject: [PATCH 2/3] feat: complete portal, site builder, links, and redirects --- docs/SUMMARY.md | 2 ++ docs/walrus-sites/linking.md | 42 +++++++++++++++++++++++++++ docs/walrus-sites/portal.md | 47 +++++++++++++++++++++++++++++++ docs/walrus-sites/redirects.md | 41 +++++++++++++++++++++++++++ docs/walrus-sites/site-builder.md | 25 ++++++++++++++++ 5 files changed, 157 insertions(+) create mode 100644 docs/walrus-sites/linking.md create mode 100644 docs/walrus-sites/redirects.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 30360cc0..c3cc99d1 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -39,6 +39,8 @@ - [Bonus: Set a SuiNS name](./walrus-sites/tutorial-suins.md) - [Advanced configuration](./walrus-sites/tutorial-config.md) - [Technical overview](./walrus-sites/overview.md) + - [Linking from and to Walrus Sites](./walrus-sites/linking.md) + - [Redirecting objects to Walrus Sites](./walrus-sites/redirects.md) - [The site builder](./walrus-sites/site-builder.md) - [The Walrus Sites Portal](./walrus-sites/portal.md) - [Known restrictions](./walrus-sites/restrictions.md) diff --git a/docs/walrus-sites/linking.md b/docs/walrus-sites/linking.md new file mode 100644 index 00000000..4a15c83b --- /dev/null +++ b/docs/walrus-sites/linking.md @@ -0,0 +1,42 @@ +# Linking from and to Walrus Sites + +Links in Walrus Sites work _almost_ as you would expect in a regular website. We specify here a few +of the details. + +## Linking to resources within the same site + +Relative and absolute links (`href="/path/to/resource.html"`) work as usual. + +## Linking to resources on the Web + +Linking to a resource on the web (`href="https://some.cdn.example.com/stylesheet.css"`) also work as +usual. + +## Linking to resources in other Walrus Sites + +Here is the part that is a bit different. Assume there is some image that you can browse at +`https://gallery.walrus.site/walrus_arctic.webp`, and you want to link it from your own Walrus Site. + +Recall that, however, `https://walrus.site` is just one of the possibly many Portals. I.e., the same +resource is browsable from a local portal (`http://gallery.localhost:8080/walrus_arctic.webp`), or +from any other portal (e.g., `https://gallery.myotherportal.com/walrus_arctic.webp`). Therefore, how +can you link the resource in a _Portal independent way_? This is important for interoperability, +availability, and respecting the user's choice of portal. + +### The solution: Walrus Sites links + +We solve this problem by having the Portals interpret special links, that are normally invalid on +the Web, and redirect to the corresponding Walrus Sites resource in the portal itself. + +Consider the example above, where the resource `/walrus_arctic.webp` is browsed from the Walrus Site +with SuiNS name `gallery`, which points to the object ID `abcd123…` (in Base36 encoding). Then, +the Portal-independent link is: `https://gallery.suiobj/walrus_arctic.webp`. To fix the object ID +instead of the SuiNS name, you can use `https://abcd123….suiobj/walrus_arctic.webp`. + +Another possibility is to directly point to the Walrus _blob ID_ of the resource, and have the +browser "sniff" the content type. This works for images, for example, but not for script or +stylesheets. For example to point to the blob ID (e.g., containing an image) `qwer5678…`, use the +URL `https://blobid.walrus/qwer5678…`. + +With such a link, the Portal will extract the blob ID and redirect the request to the aggregator it +is using to fetch blobs. diff --git a/docs/walrus-sites/portal.md b/docs/walrus-sites/portal.md index 728c8ca9..20018bab 100644 --- a/docs/walrus-sites/portal.md +++ b/docs/walrus-sites/portal.md @@ -1 +1,48 @@ # The Walrus Sites Portal + +We use the term "Portal" to indicate any technology that is used to access an browse Walrus Sites. +As mentioned in the [overview](./overview.md#the-site-rendering-path), we foresee three kinds of +Portals: + +1. Server-side Portals; +1. custom local apps; and +1. service-worker based Portals in the browser. + +Currently, only the service-worker based Portal is available. + +## Running the Portal locally + +You can run a service worker Portal locally: + +- To browse Walrus Sites without accessing external Portals; or +- for development purposes. + +This requires having the [`pnpm`](https://pnpm.io/) tool installed. To start, clone the repo and +enter the `portal` directory. Here, run: + +``` sh +pnpm install +``` + +to install the dependencies, and + +``` sh +pnpm serve +``` + +to serve the Portal. Typically, you will find it served at `localhost:8080` (but check the output of +the serve command). + +## Configuring the Portal + +The most important configuration parameters for the Portal are in `constants.ts`. + +- `NETWORK`: The Sui network to be used for fetching the Walrus Sites objects. Currently, we + use Sui `testnet`. +- `AGGREGATOR`: The URL of the [aggregator](../usage/web-api.md) from which the service worker will + fetch the Walrus blobs. +- `SITE_PACKAGE`: "0x514cf7ce2df33b9e2ca69e75bc9645ef38aca67b6f2852992a34e35e9f907f58" +- `MAX_REDIRECT_DEPTH`: The number of [redirects](./redirects.md) the service worker will follow + before stopping. +- `SITE_NAMES`: Hard coded `name: objectID` mappings, to override the SuiNS names. For development + only. diff --git a/docs/walrus-sites/redirects.md b/docs/walrus-sites/redirects.md new file mode 100644 index 00000000..4ed1f2df --- /dev/null +++ b/docs/walrus-sites/redirects.md @@ -0,0 +1,41 @@ +# Redirecting objects to Walrus Sites + +We have seen in the [overview](./overview.md) how a Walrus Site object on Sui looks like. We will +discuss now how you can create ensure that a _set of arbitrary objects_ can all be tied to a +specific, and possibly unique, Walrus site. + +## The goal + +Consider a collection of NFTs, such as the one published by . As we +show there, each minted NFT has its own Walrus Site, that can be personalized based on the contents +(e.g., the color) of the NFT itself. How can we achieve this? + +## Redirect links + +The solution is simple: We add a "redirect" in the NFT's +[`Display`](https://docs.sui.io/standards/display#sui-utility-objects) property. Each time an NFT's +object ID is browsed through a Portal, the Portal will check the `Display` of the NFT and, if it +encounters the `walrus site address` key, it will go fetch the Walrus Site that is at the +corresponding object ID. + +### Redirects in Move + +Practically speaking, when creating the `Display` of the NFT, you can include the key-value pair +that points to the Walrus site that is to beused. + +``` move +... +const VISUALIZATION_SITE: address = @0x901fb0...; +display.add(b"walrus site address".to_string(), VISUALIZATION_SITE.to_string()); +... +``` + +### How to personalize based on the NFT? + +The code above will only open the specified Walrus Site when browsing the object ID of the NFT. How +do we ensure that the properties of the NFT can be used to personalize the site? + +This needs to be done in the `VISUALIZATION_SITE`: Since the subdomain is still pointing to the +NFT's object ID, the Walrus Site that is loaded can check its `origin` in Javascript, and use the +subdomain to determine the NFT, fetch it from chain, and use its internal fields to modify the +displayed site. diff --git a/docs/walrus-sites/site-builder.md b/docs/walrus-sites/site-builder.md index f54a2592..8d4e45ad 100644 --- a/docs/walrus-sites/site-builder.md +++ b/docs/walrus-sites/site-builder.md @@ -1 +1,26 @@ # The site builder + +To facilitate the creation of Walrus Sites, we provide the "site builder" tool. The site builder +takes care of creating Walrus Sites object on Sui, with the correct structure, and stores the site +resources to Walrus. + +## Site builder commands + +The site builder tool exposes the following commands: + +- `publish`: Allows to publish a specific directory to Walrus. The directory must contain files that + can be served with HTTP, and have an `index.html` file. This command will return a Sui object ID + for the newly created site. +- `update`: After creating a site, you can update it with this command. It takes as input a + directory, as above, with the new or updated files, and the object ID of the site to update. Note + that the wallet you are using must be the _owner_ of the Walrus Site object to be able to update + it. This command will remove and create resources as required, to ensure that the Walrus Sites + object on Sui matches the local directory. If run with `--watch`, this command re-updates the site + _every time a file in the directory changes_. This is useful during development, but pay attention + to costs! +- `convert`: A utility tool. Given a Sui object ID in Hex format, it converts it to Base36. This is + useful if you know the Sui object ID of a site, and want to find the URL. +- `sitemap`: A utility tool. For a give Walrus Sites Sui object ID, prints out all the resources + that compose the site and their object ID. + +Check the commands' `--help` for more information. From edd59d51535f3e22b2bd9719d135ef1c45ac4382 Mon Sep 17 00:00:00 2001 From: giac-mysten Date: Fri, 14 Jun 2024 15:49:25 +0100 Subject: [PATCH 3/3] fix: markus' comments --- docs/walrus-sites/overview.md | 14 +++++++------- docs/walrus-sites/portal.md | 4 ++-- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/walrus-sites/overview.md b/docs/walrus-sites/overview.md index 91a103d8..a79be320 100644 --- a/docs/walrus-sites/overview.md +++ b/docs/walrus-sites/overview.md @@ -73,19 +73,19 @@ approaches are possible: All of these approaches are viable (the first has been used for similar applications in IPFS gateways, for example), and have trade offs. As an initial step, we have chosen to use the -service-worker based approach, as it is novel, light weight, and ensures that the Portal does not -have to process all the traffic from clients. In the following, therefore, we present the details of -this Portal-based approach. +service-worker based approach, as it is light weight and ensures that the Portal does not have to +process all the traffic from clients. In the following, therefore, we present the details of this +Portal-based approach. ### Browsing and domain isolation We must ensure that, when browsing multiple sites through a Portal, for example the one hosted at -[https://walrus.site](https://walrus.site), these sites are isolated. Isolation is necessary for -security, and to ensure that the wallet connection in the browser works as expected. +, these sites are isolated. Isolation is necessary for security, and to ensure +that the wallet connection in the browser works as expected. To do so, we give each Walrus Site a specific _subdomain_ of the Portal's domain. For example, the -Flatland mint dApp is hosted at [https://flatland.walrus.site](https://flatland.walrus.site), where -the subdomain `flatland` is uniquely associated to the object ID of the Walrus Site through SuiNS. +Flatland mint dApp is hosted at , where the subdomain `flatland` is +uniquely associated to the object ID of the Walrus Site through SuiNS. Walrus Sites also work without SuiNS: a site can _always_ be browsed by using as subdomain the Base36 encoding of the Sui object ID of the site. For the Flatland dApp, this URL is: diff --git a/docs/walrus-sites/portal.md b/docs/walrus-sites/portal.md index 20018bab..c1b92598 100644 --- a/docs/walrus-sites/portal.md +++ b/docs/walrus-sites/portal.md @@ -17,8 +17,8 @@ You can run a service worker Portal locally: - To browse Walrus Sites without accessing external Portals; or - for development purposes. -This requires having the [`pnpm`](https://pnpm.io/) tool installed. To start, clone the repo and -enter the `portal` directory. Here, run: +This requires having the [`pnpm`](https://pnpm.io/) tool installed. To start, clone the +`walrus-sites` repo and enter the `portal` directory. Here, run: ``` sh pnpm install