docs: start adding docs on walrus sites (#19)

- [x] Introduction material - [x] Walk-through tutorial - [x] Install instructions - [x] Fix configuration details - [x] CLI use instructions - [x] SuiNS instructions For followups - Technical overview - Portal - site builder Contributes to #5 --------- Co-authored-by: Karl Wuest <[email protected]>
MystenLabs · Jun 12, 2024 · 9718ea1 · 9718ea1
1 parent 189c82f
commit 9718ea1
Show file tree

Hide file tree

Showing 12 changed files with 446 additions and 0 deletions.
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -31,6 +31,17 @@
 
 # Walrus sites
 
+- [Introduction to Walrus Sites](./walrus-sites/intro.md)
+- [Your first Walrus Site](./walrus-sites/tutorial.md)
+  - [Installing the site builder](./walrus-sites/tutorial-install.md)
+  - [Publishing a Walrus Site](./walrus-sites/tutorial-publish.md)
+  - [Bonus: Set a SuiNS name](./walrus-sites/tutorial-suins.md)
+  - [Advanced configuration](./walrus-sites/tutorial-config.md)
+- [Technical overview](./walrus-sites/overview.md)
+  - [The site builder](./walrus-sites/site-builder.md)
+  - [The Walrus Sites Portal](./walrus-sites/portal.md)
+  - [Known restrictions](./walrus-sites/restrictions.md)
+
 ---
 
 [Glossary](./glossary.md)
diff --git a/docs/assets/suins-asset.png b/docs/assets/suins-asset.png
diff --git a/docs/walrus-sites/intro.md b/docs/walrus-sites/intro.md
@@ -0,0 +1,62 @@
+# Introduction to Walrus Sites
+
+_Walrus Sites_ are "web"-sites that use Sui and Walrus as their underlying technology. They are a
+prime example of how Walrus can be used to build new and exciting decentralized applications. Anyone
+can build and deploy a Walrus Site, and make it accessible to the World!
+
+At a high level, the most exciting features include:
+
+- Publishing a site does not require managing servers or complex configurations; just provide the
+  source files (produced by your favorite web framework), publish them to Walrus Sites using the
+  [site builder tool](./site-builder.md), and you are done!
+- Sites can be linked to from ordinary Sui objects. This feature enables, for example, creating a
+  NFT collection in which _every single NFT_ has a _personalized website dedicated to it_.
+- Walrus Sites are owned by addresses on Sui, and can be exchanged, shared, and updated thanks to
+  Sui's flexible programming model. This means, among other things, that Walrus Sites can leverage
+  the [SuiNS](https://suins.io/) naming system to have human readable names. No more messing around
+  with DNS!
+- Thanks to Walrus' decentralization and extremely high data availability, there is no risk of
+  having your site wiped for no reason.
+- Since they live on Walrus, these sites cannot have a backend in the traditional sense, and can be
+  therefore considered "static" sites. However, the developer can harness Sui's programmability
+  to add backend functionality to Walrus Sites!
+
+## Show me
+
+To give you a very high level intuition of how Walrus Sites work, let's look at an example: A simple
+NFT collection on Sui, that has a frontend dApp to mint the NFTs hosted on Walrus Sites, and in
+which _each NFT_ has a _specific, personalized Walrus Site_.
+
+You can check out the mint page at
+[https://flatland.walrus.site/](https://flatland.walrus.site/). This site is serve to your browser
+through the Walrus Site _Portal_ [https://walrus.site](https://walrus.site). While the Portal's
+operation is explained in a [later section](./portal.md), consider for now that there
+can be many Portals (hosted by whomever wants to have their own, and even on `localhost`). Further,
+the only function of the Portal is to provide the browser with some code (specifically, a service
+worker) that allows it to fetch the Walrus Site from Sui and Walrus.
+
+If you have a Sui wallet with some Testnet SUI, you can try and "mint a new Flatlander" from the
+site. This creates an NFT from the collection, and shows you two links, one to the explorer, and one
+to the "Flatlander site". This latter site is a special Walrus Site that exists only for that NFT,
+and has special characteristics (the background color, the image...) that are based on the contents
+of the NFT.
+
+The URL to this per-NFT site look something like this
+`https://4egmmrw9izzjn0dm2lkd3k0l8phk386z60ub1tpdc1jswbb5dr.walrus.site/`.  You'll notice that the
+domain remains `walrus.site`, but the subdomain is a long and random-looking string.  This string is
+actually the [Base36](https://en.wikipedia.org/wiki/Base36) encoding of the object ID of the NFT,
+which is
+[0xb09b312b...](https://suiscan.xyz/testnet/object/0xb09b312b28049467dd6173b6cebb60ed5fe3046883e248632bf9fb20b7dbdaff).
+
+In summary:
+
+- Walrus Sites are served through a Portal; in this case, `https://walrus.site`. There can be many
+  Portals, and anyone can host one.
+- The subdomain on the URL points to a specific object on Sui, that allows the browser to fetch and
+  render the site resources. This pointer can be:
+  - A SuiNS name, such as `flatland` in `https://flatland.walrus.site`; or
+  - the Base36 encoding of a the Sui object ID, such as `0xb09b312b...` in the example above.
+
+Curious to know how this magic is possible? Read the [technical
+overview](./overview.md)! If you just want to get started trying Walrus Sites out, check the
+[tutorial](./tutorial.md).
diff --git a/docs/walrus-sites/overview.md b/docs/walrus-sites/overview.md
@@ -0,0 +1 @@
+# Overview
diff --git a/docs/walrus-sites/portal.md b/docs/walrus-sites/portal.md
@@ -0,0 +1 @@
+# The Walrus Sites Portal
diff --git a/docs/walrus-sites/restrictions.md b/docs/walrus-sites/restrictions.md
@@ -0,0 +1,51 @@
+# Known restrictions
+
+Walrus Sites can be used to deploy almost any form of traditional static "Web 2" website build for
+modern browsers. There are, however, a number of restrictions that a developer should keep in mind
+when creating or porting a website to Walrus Sites.
+
+## No secret values
+
+Walrus Sites are fully publicly accessible, as the metadata is stored on Sui and the site content is
+stored on Walrus. Therefore, developers _must_ not store secret values within the sites.
+
+We emphasize again that any such backend-specific operations (storing secret values, authentication,
+etc.) are achievable by leveraging the integration with Sui blockchain and the Sui wallet.
+
+## There is a maximum redirect depth
+
+The number of consecutive redirects a Walrus Site can perform is capped by the
+Portal (see [Portal configuration](./portal.md)).  This measure ensures that loading a Walrus Site
+does not result in an infinite loading loop.
+
+Different Portals can set this limit as they desire. The limit for the Portal hosted at
+[walrus.site](http://walrus.site) has a maximum redirect depth of 3.
+
+## Service workers are not available
+
+**WARNING**: This limitation only applies to Portal based on service workers. A web Portal will not
+have this limitation.
+
+Walrus Sites leverage service workers in the clients' browsers to perform these essential
+operations:
+
+1. reading the site metadata from Sui;
+1. fetching the page content from Walrus; and
+1. serving the content to the browser.
+
+Therefore, a site deployed on Walrus Sites cannot use service workers. Installing a service worker
+from within a Walrus Site will result in a dysfunctional site and a poor experience for the user.
+
+## The iOS Sui Wallet Mobile does not work with Walrus Sites
+
+**WARNING**: This limitation only applies to Portal based on service workers. A web Portal will not
+have this limitation.
+
+Service workers cannot be loaded inside an in-app browser on iOS, because of a limitation of the
+WebKit engine. As a consequence, Walrus Sites cannot be used within the [Sui Wallet
+Mobile](https://apps.apple.com/us/app/sui-wallet-mobile/id6476572140) app on iOS, and therefore
+the Sui wallet can currently not be used on a Walrus Site on iOS. Note, however, that _browsing_ a
+Walrus Site is still possible on iOS through any browser. Only the connection to the wallet is
+impacted.
+
+The connection with the Sui Wallet Mobile app works on Android devices.
diff --git a/docs/walrus-sites/site-builder.md b/docs/walrus-sites/site-builder.md
@@ -0,0 +1 @@
+# The site builder
diff --git a/docs/walrus-sites/tutorial-config.md b/docs/walrus-sites/tutorial-config.md
@@ -0,0 +1,36 @@
+# Configuring the site builder
+
+Configuring the `site-builder` tool is straightforward, but care is required to ensure that
+everything works correctly.
+
+The `site-builder` tool requires a configuration file to know which package to use on Sui, which
+wallet to use, the gas budget, and other operational details. Most of these are abstracted away
+through sensible defaults, so you should not need to touch them. Yet, for completeness, we provide
+here the details for all the configuration options.
+
+## Minimal configuration
+
+The config file is expected to be in `./builder.yaml`, and it is possible to point elsewhere with
+the `--config` flag.  For your first run, it should be sufficient to call the `site-builder` with
+`--config assets/builder-example.yaml`, which is already configured appropriately.
+
+## Advanced configuration
+
+If you want to have more control over the behavior of the site builder, you can customize the
+following variables in the config file:
+
+- `package`: the object ID of the Walrus Sites package on Sui. This must always be specified in the
+  config, and is already appropriately configured in `assets/example-config.yaml`.
+- `portal`: the name of the Portal through which the site will be viewed; this only affects the
+  output of the CLI, and nothing else (default: `walrus.site`).
+  All Walrus Sites are accessible through any Portal independent of this setting.
+- `general`: these are general options, that can be configured both through the CLI and the config:
+  - `rpc_url`: The URL of the Sui RPC node to use. If not set, the `site-builder` will infer it from
+    the wallet.
+  - `wallet`: Pointer to the Sui wallet to be used. By default, it uses the system-wide wallet (the
+    one from `sui client addresses`).
+  - `walrus_binary`: Pointer to the `walrus` binary. By default, this is expected to be run from
+    `$PATH`.
+  - `walrus_config`: The configuration for the `walrus` client binary. See the relative
+    documentation (TODO: link).
+  - `gas_budget`: The maximum amount of gas to be spent for transactions (default: 500M MIST).
diff --git a/docs/walrus-sites/tutorial-install.md b/docs/walrus-sites/tutorial-install.md
@@ -0,0 +1,122 @@
+# Installing the site builder
+
+We describe here the steps necessary to setup the Walrus Sites' `site-builder` tool, and prepare
+your environment for development.
+
+## Prerequisites
+
+Before you start, make sure you:
+
+- Have a recent version of [Rust](https://www.rust-lang.org/tools/install) installed; and
+- have the [Sui CLI](https://docs.sui.io/guides/developer/getting-started/sui-install) installed.
+
+Then, follow these additional setup steps.
+
+## Get the `walrus` binary and install it
+
+Download the latest `walrus` binary for your architecture from
+`https://storage.googleapis.com/mysten-walrus-binaries/walrus-v0.1.0-a0fb8c9-<arch>`, where `<arch>`
+is your architecture. The available options are listed in this table:
+
+| OS     | CPU           | Architecture    |
+|--------|---------------|-----------------|
+| MacOS  | Apple Silicon | `macos-arm64`   |
+| MacOS  | Intel 64bit   | `macos-x86_64`  |
+| Ubuntu | Intel 64bit   | `ubuntu-x86_64` |
+
+Then, add it to your `$PATH`. For example, on MacOS you can copy it to
+`/Users/myusername/.local/bin/` (check what directories are in your `$PATH` by running `echo
+$PATH`).
+
+Once this is done, you should be able to type `walrus` in your terminal and see:
+
+``` txt
+Walrus client
+
+Usage: walrus [OPTIONS] <COMMAND>
+
+⋮
+```
+
+If, for any reason, you don't want to add `walrus` to `$PATH`, place the binary in your preferred
+directory, and remember to configure a pointer to the binary in the `site-builder` config (more on
+this [later](./tutorial-config.md)).
+
+## Point your Sui CLI to testnet, and get some SUI
+
+Walrus is currently deployed on Sui Testnet. Therefore, you have to ensure that your Sui CLI is
+configured accordingly:
+
+``` txt
+sui client envs
+╭──────────┬──────────────────────────────────────┬────────╮
+│ alias    │ url                                  │ active │
+├──────────┼──────────────────────────────────────┼────────┤
+│ devnet   │ https://fullnode.devnet.sui.io:443   │        │
+│ local    │ http://127.0.0.1:9000                │        │
+│ testnet  │ https://fullnode.testnet.sui.io:443/ │ *      │
+│ mainnet  │ https://fullnode.mainnet.sui.io:443  │        │
+╰──────────┴──────────────────────────────────────┴────────╯
+```
+
+If the `active` network is not `testnet`, switch to `testnet` by running:
+
+``` sh
+sui client switch --envs testnet
+```
+
+Further, make sure you have at least 2 separate gas coins, with at least 1 SUI each, by running `sui
+client gas`.  If you don't have enough SUI, you can hit the testnet faucet by running.
+
+``` sh
+sui client faucet --url https://faucet.testnet.sui.io/v1/gas
+```
+
+After some seconds, running again `sui client gas` should show the newly-minted coins in your
+wallet.
+
+## Clone the Walrus Sites repo, and build the `site-builder` tool
+
+First clone and enter the Walrus Sites repo from
+`https://github.com/MystenLabs/blocksite-poc`). (TODO: change link to public repo when available).
+
+``` sh
+git clone [email protected]:MystenLabs/blocksite-poc.git
+cd blocksite-poc
+cd site-builder
+```
+
+Build the release version of the site builder.
+
+``` sh
+cargo build --release
+```
+
+After the build process completes, it should be possible to run:
+
+``` sh
+./target/release/site-builder
+```
+
+And output should look like the following:
+
+``` txt
+Usage: site-builder [OPTIONS] <COMMAND>
+
+Commands:
+  publish  Publish a new site on Sui
+  update   Update an existing site
+  convert  Convert an object ID in hex format to the equivalent Base36
+               format
+  sitemap  Show the pages composing the Walrus site at the given object ID
+  help     Print this message or the help of the given subcommand(s)
+
+⋮
+```
+
+## Get the latest `walrus` client configuration
+
+First,
+[download](https://storage.googleapis.com/mysten-walrus-binaries/walrus-configs/client_config.yaml)
+the `walrus` client config.  Then, copy it to `~/.walrus/config.yaml`. This ensures that the
+`walrus` binary can connect to the correct Walrus object on Sui.
diff --git a/docs/walrus-sites/tutorial-publish.md b/docs/walrus-sites/tutorial-publish.md
@@ -0,0 +1,96 @@
+# Publishing a Walrus Site
+
+Now that everything is installed and configured, you should be able to start publishing
+your first Walrus Site!
+
+## Select the source material for the site
+
+The `site-builder` works by uploading a directory of files produced by any web framework to Walrus,
+and adding the relevant metadata to Sui. This directory should have a file called `index.html` in
+its root, which will be the entry point to the Walrus Site.
+
+For the rest of the tutorial, we will use as an example the simple site contained in
+`./examples/snake`.
+
+## Publish the site
+
+Since we have placed the `walrus` binary and configuration in their default locations, publishing
+the `./examples/snake` site is simple:
+
+- Ensure that you are in the `site-builder` directory;
+- Run the publishing command:
+
+  ``` sh
+  ./target/release/site-builder --config assets/builder-example.yaml publish ../examples/snake
+  ```
+
+The output should look like the following:
+
+``` txt
+Operations performed:
+- created resource /Oi-Regular.ttf with blob ID 2YLU3Usb-WoJAgoNSZUNAFnmyo8cfV8hJYt2YdHL2Hs
+- created resource /file.png with blob ID R584P82qm4Dn8LoQMlzkGZS9IAkU0lNZTVlruOsUyOs
+- created resource /index.html with blob ID SSzbpPfO2Tqk6xNyF1i-NG9I9CjUjuWnhUATVSs5nic
+- created resource /walrus.png with blob ID SGrrw5NQyFWtqtxzLAQ1tLpcChGc0VNbtFRhfsQPuiM
+
+Created new site: test site
+New site object ID: 0x5ac988828a0c9842d91e6d5bdd9552ec9fcdddf11c56bf82dff6d5566685a31e
+
+Browse the resulting site at: https://29gjzk8yjl1v7zm2etee1siyzaqfj9jaru5ufs6yyh1yqsgun2.walrus.site
+```
+
+This output tells you that, for each file in the folder, a new Walrus blob was created, and the
+respective blob ID.  Further it prints the object ID of the Walrus Site object on Sui (so you can
+have a look in the explorer, and use it to set the SuiNS name), and, finally, the URL at which you
+can browse the site.
+
+Note here that we are passing the example config `assets/builder-example.yaml` as the config for the
+site builder. The configuration file is necessary to ensure that the `site-builder` knows the
+correct Sui package for the Walrus Sites logic.
+
+More details on the configuration of the `site-builder` can be found under the [advanced
+configuration](tutorial-config.md) section.
+
+## Update the site
+
+Let's say now you want to update the content of the site, for example by changing the title from
+"eat all the blobs!" to "Glob all the Blobs!".
+
+First, make this edit on in the `../examples/snake/index.html` file.
+
+Then, you can update the existing site by running the `update` command, and providing the directory
+where to find the updated files (still `../example/snake`), and the object ID of the existing site
+(`0x5ac988...`):
+
+``` sh
+./target/release/site-builder --config assets/builder-example.yaml update ../examples/snake 0x5ac9888...
+```
+
+The output this time should be:
+
+``` txt
+Operations performed:
+  - deleted resource /index.html with blob ID SSzbpPfO2Tqk6xNyF1i-NG9I9CjUjuWnhUATVSs5nic
+  - created resource /index.html with blob ID LXtY0VdY5kM-3Ph7gLvj8URdz5yiRa5DUy3ZxYqDView
+
+Updated site at object ID: 0x5ac988828a0c9842d91e6d5bdd9552ec9fcdddf11c56bf82dff6d5566685a31e
+
+Browse the resulting site at: https://29gjzk8yjl1v7zm2etee1siyzaqfj9jaru5ufs6yyh1yqsgun2.walrus.site
+```
+
+Compared to the `publish` action, we can see that now the only actions performed were to delete the
+old `index.html`, and update it with the newer one.
+
+Browsing to the provided URL should reflect the change. You've updated the site!
+
+## Additional commands
+
+The `site-builder` tool provides two additional utilities:
+
+- the `convert` command, which converts an object ID in hex format to the equivalent Base36
+  format. This command is useful if you have the Sui object ID of a Walrus Site, and want to know
+  the subdomain where you can browse it.
+- the `sitemap` command, which shows the resources that compose the Walrus Site at the given object
+  ID.
+
+In general, the `--help` flag is your friend!