From a066d5093b0a00cc54206fb966cc33742f97fdac Mon Sep 17 00:00:00 2001
From: David Chu <davidchuyaya@gmail.com>
Date: Mon, 15 May 2023 17:42:54 -0700
Subject: [PATCH] More detail on benchmarks

---
 benchmarks/vldb24/README.md | 39 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)

diff --git a/benchmarks/vldb24/README.md b/benchmarks/vldb24/README.md
index f8139869..3fa770f5 100644
--- a/benchmarks/vldb24/README.md
+++ b/benchmarks/vldb24/README.md
@@ -38,4 +38,41 @@ To run a specific benchmark, run the corresponding `*_run.sh` file. Note that th
         }
     }
 }
-```
\ No newline at end of file
+```
+
+## How the benchmarks work
+Our benchmarks are derived from Michael Whittaker's, which are described [here](https://github.com/rithvikp/autocomp/tree/master/benchmarks#readme) in quite some detail. Roughly, the following files are involved in each benchmark, using the Paxos benchmarks as an example:
+- Implementation: [Scala](https://github.com/rithvikp/autocomp/tree/master/shared/src/main/scala/frankenpaxos/multipaxos) or [rust](https://github.com/rithvikp/autocomp/tree/master/rust/examples/multipaxos)
+- [Drivers](https://github.com/rithvikp/autocomp/tree/master/jvm/src/main/scala/frankenpaxos/multipaxos)
+- [Benchmark setup](https://github.com/rithvikp/autocomp/blob/master/benchmarks/multipaxos/dedalus_multipaxos.py)
+- [Benchmark parameters](https://github.com/rithvikp/autocomp/blob/master/benchmarks/vldb24/dedalus_multipaxos_benchmark.py)
+- [Run script](https://github.com/rithvikp/autocomp/blob/master/benchmarks/vldb24/dedalus_multipaxos_run.sh)
+
+### Implementation
+This is the code that represents the "protocol", processing client input and returning some outputs.
+Since the Scala implementations were written by Michael, we will mainly discuss the rust implementations.
+
+Each protocol has its own directory in `rust/examples`, which always includes `main.rs`.
+Configuration parameters can be provided at [the top of each file](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L17) and incorporated into Dedalus by [unwrapping](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L143) and [embedding as Dedalus inputs](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L173).
+The component responsible with communicating with the Scala client must specify custom [serialization and deserialization](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L33) functions, based on the [protobuf files](https://github.com/rithvikp/autocomp/blob/master/shared/src/main/scala/frankenpaxos/multipaxos/MultiPaxos.proto), which are [used by the Dedalus code](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L197).
+Note that any modification to the protobuf requires recompiling Scala and copying the `.jar` to the correct location.
+
+Communication channels between components must be [explicit](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L89).
+All channels are one-way.
+Send channels must follow [this format](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L89) and receive channels must follow [this format](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L98).
+Note that pthe ports for each channel](https://github.com/rithvikp/autocomp/blob/master/rust/examples/multipaxos/leader.rs#L90) have the format `send_to$dest$num` or `receive_from$src$num`, where `num` is unique per source and destination.
+
+### Drivers
+This determines what data will be generated by the client, [triggering client functions](https://github.com/rithvikp/autocomp/blob/master/jvm/src/main/scala/frankenpaxos/multipaxos/ClientMain.scala#LL237C26-L237C26), along with some logging and prometheus setup. Only `ClientMain.scala` is relevant to us.
+
+### Benchmark setup
+This defines the [global config parameters](https://github.com/rithvikp/autocomp/blob/master/benchmarks/multipaxos/dedalus_multipaxos.py#L33), which can be reused across components.
+IP addresses are [assigned](https://github.com/rithvikp/autocomp/blob/master/benchmarks/multipaxos/dedalus_multipaxos.py#L169) to components, which can be modified to allow colocation.
+Configuration parameters are [fed into](https://github.com/rithvikp/autocomp/blob/master/benchmarks/multipaxos/dedalus_multipaxos.py#L212) each rust executable and ports are [linked](https://github.com/rithvikp/autocomp/blob/master/benchmarks/multipaxos/dedalus_multipaxos.py#L254) between components.
+
+### Benchmark parameters
+These are the parameters that are changed between each experimental run.
+This determines the [number of machines](https://github.com/rithvikp/autocomp/blob/master/benchmarks/vldb24/dedalus_multipaxos_benchmark.py#L12) to be allocated to each component, of which some can participate in [each test](https://github.com/rithvikp/autocomp/blob/master/benchmarks/vldb24/dedalus_multipaxos_benchmark.py#L108).
+
+### Run script
+This is the shell script that executes the benchmark.