Quartz tooling for deployment

[ebuchman]

Thinking about the quartz dev experience, especially around deployment. Here's a short writeup on some things we discussed today with Shoaib and Thane.

With Dan's recent refactor of the quartz core enclave code, the app-specific logic now lives on its own in its own directory.

So a quartz app moving forward should have the following directory structure:

/
.. contracts/
.. enclave/
.. frontend/

And it can be in its own repo. For now our two quartz apps are living in the cycles-quartz repo, but in the future they could be in their own repos.

This is basically what we have now, but we also have this additional set of janky bash scripts for deployment. Instead of having those scripts, we discussed creating a quartz tool that would run on the enclave's host machine with the following commands:

$ quartz init                   # create base quartz app directory from template
$ quartz build                  # build the contract and enclave binaries
$ quartz start                  # gramine config (including light client), sign, start the enclave binary
$ quartz deploy $BINARY         # deploy the wasm binary to the chain and call instantiate
$ quartz handshake $CONTRACT    # run the handshake between contract and enclave (could be combined with `deploy`)
$ quartz listen $CONTRACT       # listen for events and run the enclave handler, and expose public query server for users

Maybe listen should be called run but I'll leave it as listen for now.

We could also add a quartz init to run on the dev machine to initialize the repo with template contract and enclave code to help devs get started. But the tool is mostly about easing deployment and getting rid of the scripts.

The scripts are currently a mix of bash code (init.sh, handshake.sh, etc.) and rust code called from bash (tm-prover).

The quartz tool should be written in rust (obviously), but for ease of getting started we can have it just include parts of the bash scripts as strings in its binary that it can execute directly. Might not be possible for everything but can probably help us get started quickly. Over time we can move more and more of the functionaltiy from the bash scripts into rust proper.

One thing to note is that some deployment stuff will actually be application specific, and we want this quartz tool to be general purpose. The way we achieve this is by putting more burden on the enclave code to handle things.

For instance, quartz deploy includes a call to the enclave's Instantiate method. This may need to return application specific data.

More significantly is quartz listen, which starts the main daemon to listen for blockchain events and trigger execution on the enclave. This may need to be customized in arbitrary ways. But in general it should be possible to boil this down to having the host query for data over the network and run functions on the enclave. Thus quartz listen will get everything it needs from the enclave. The enclave can tell it what events to subscribe to. And when events are triggered, it can ask the enclave what queries to make, before returning those results to the enclave for processing.

Note this does mean it can trigger some execution on the enclave that is not strictly controlled by the blockchain (ie. asking it what queries to make of the blockchain in the first place). In principle this should be ok, just something to note.

Additionally, we can have quartz listen expose a public query server so users can query enclave data. Making queries possible requires a lot more work (specifically adding ORAM), but it makes sense for the single quartz listen command to handle all requests for execution from the enclave whether from users or the blockchain itself.

So, a short summary of some flows for quartz listen:

    - ask the enclave what to subscribe to on the blockchain
    - listen for those events from the blockchain
            - when event comes in, ask the enclave what to query
            - return results to enclave
            - submit result from enclave back to chain
    - listen on public address for queries from users 
            - have some basic spam prevention
            - forward queries to enclave
            - enclave requests host to query data from blockchain
            - host forwards blockchain data to enclave
            - enclave responds with response for user
            - host returns response to user

Tasks

Preview Give feedback

1. feat(cli): setup skeleton for quartz tool #105
   +0
2. Quartz CLI: Implement init command #113
   quartz-cli +1
   
3. Quartz CLI: Implement enclave build #114
   quartz-cli +1
   
4. Quartz CLI: Implement enclave start #115
   quartz-cli +1
   
5. Quartz CLI: Implement contract build #116
   quartz-cli +1
   
6. Quartz CLI: Implement contract deploy #117
   quartz-cli +1
   
7. Quartz CLI: Implement handshake #118
   quartz-cli +1
   
8. Quartz CLI: Implement dev #120
   quartz-cli +1
   

[ebuchman]

For more context, each of the commands (except init) corresponds directly to an existing script in the transfers/scripts: https://github.com/informalsystems/cycles-quartz/tree/main/apps/transfers/scripts.

The deploy.sh and handshake.sh use another common underlying script, the relay.sh that's part of quartz core: https://github.com/informalsystems/cycles-quartz/blob/main/relayer/scripts/relay.sh. This is the thing that actually communicates with the enclave binary using grpcurl.

My expectation is that init, build, start, and handshake commands should be pretty easy and straight forward since they should be generic for all apps. For starters we can more or less just copy in the bash scripts or do whatever's easiest to call the functionality of those scripts from rust. I wouldn't spend any time yet rewriting things to use rust-based clients. We should just do whatever the simplest thing is to get the bash scripts to run from rust. But for instance trying to replace calls to wasmd with actually using a rust based cosmos-sdk client is definitely not worth it at this stage!

The same goes for the underlying relay.sh script. Presumably it's fine to just call the bash somehow. But if there's a strong reason to rewrite that script in rust, it's probably more straightforward (but I imagine the 50 lines of bash will be hundreds of lines of rust? I'd be curious actually ...)

The deploy and listen command will be a bit more complicated, since they need to be app-specific, and the way to do that is get the enclave binary to expose more endpoints and return more data.

deploy is the easier of the two. We just need to allow the app dev to easily extend the core quartz's Instantiate method to return app-specific data from the enclave. For instance what we do at the end of this line where we return the denom: https://github.com/informalsystems/cycles-quartz/blob/main/apps/transfers/scripts/deploy.sh#L11, should instead just be part of what's returned by the enclave when the relay.sh script is called.

listen will be more complicated. The enclave will have to be able to return what to subscribe to (eg. https://github.com/informalsystems/cycles-quartz/blob/main/apps/transfers/scripts/listen.sh#L19) and it will have to be able to instruct the proxy host what queries to make and light client proofs to get.

For now, the simplest thing might just be to keep the listen.sh script as an app specific requirement, and expect the app dev to write one, and have quartz listen expect it. At least for a first version. Then in the future we could generalize it properly.

[thanethomson]

A note here that, as of #105, we're preferring a slightly modified CLI due to the fact that start, deploy, run and listen overlap a lot in meaning and it wouldn't be clear to a new user what the difference is.

I'd recommend the following CLI for now:

quartz init            - Create a new Quartz app skeleton from a template
quartz build           - A wrapper for "quartz contract build && quartz enclave build"
quartz enclave build   - Build the enclave
quartz enclave start   - Start the enclave
quartz contract build  - Build the contract
quartz contract deploy - Deploy the contract
quartz handshake       - Execute the handshake between the contract and the enclave

[davekaj]

closed by #153 , as it was the last one to get merged