Skip to content

Commit

Permalink
Add docs for detailed add new benchmark in vSwarm
Browse files Browse the repository at this point in the history
Signed-off-by: lrq619 <[email protected]>
  • Loading branch information
lrq619 committed Aug 14, 2023
1 parent aaa178d commit 233f2bd
Showing 1 changed file with 66 additions and 0 deletions.
66 changes: 66 additions & 0 deletions docs/adding_new_benchmark.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
This is a detailed document of how to add a new benchmark in vSwarm.
# Add Standalone Benchmarks in vSwarm
## Python
### 1. Add protobuf
The first thing we need to do is to add protobuf in [vSwarm-proto](https://github.com/vhive-serverless/vSwarm-proto).

First, clone [vSwarm-proto](https://github.com/vhive-serverless/vSwarm-proto), operations below should be taken in your local `vSwarm-proto/` folder.

Add a new folder to `proto/` named by your benchmark `{bench}`, add file `{bench}.proto` to this `proto/{bench}/` folder, which should define the protobuf of your benchmark.

In `{bench}.proto` file, define the needed grpc service. Note that invoker and most of vHive work with the simple [HelloWorld](https://grpc.io/docs/what-is-grpc/introduction/) service. However, you can implement arbitrary services for your function as the relay, sitting in between the invoker and your function, can be used to make translations. Refer to [aes.proto](https://github.com/vhive-serverless/vSwarm-proto/blob/main/proto/aes/aes.proto) and the corresponding [translation](https://github.com/vhive-serverless/vSwarm-proto/blob/main/grpcclient/aes_client.go) as an example of a custom service.
```
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
}
```
After defining your `HelloRequest` and `HelloReply`, do the following to generate protobuf:
1. Repeat the steps in `vSwarm-proto/.github/.workflows/build-tests.yml` to install `pipenv` and `protoc`.
2. Add a line in `go.mod` file's `replace` module:
```
github.com/vhive-serverless/vSwarm-proto/proto/{bench} => ./proto/{bench}
```
3. Run the following two commands to generate the `protocbuf` for your benchmark:
```
cd vSwarm-proto/proto/{bench}
pipenv install
pipenv run python3 -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. ./proto/bert/*.proto
```
This generates protobuf files for python
```
protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative ./proto/bert/*.proto
```
This generates protobuf files for go.

Alternatively, run `make proto-all` in the root of vSwarm-proto, which does both steps for all protocols automatically.

Then, in `vSwarm-proto/grpcclient` folder, add a file named `{bench}_client.go`. This is the client that the relay will select to invoke your function. Each client must implement [GrpcClient](https://github.com/vhive-serverless/vSwarm-proto/blob/70781f2339083f8c6d298ce0be13e7b7869c1409/grpcclient/grpcclient.go#L79) interface.
`Init` and `Close` are used to set up and close the connection to the function.
`GetGenerator` will must return an generator for creating new input values for your function.
`Request` is the actual invocation of the function.
Refer to other `*_client.go` clients for more details about how to implement those functions.

Finally, in `vSwarm-proto/grpcclient/getclient.go`, add your `{bench}` to `func FindServiceName` and `func FindGrpcClient`

After these are done, commit and push to the upstream branch.

Below operations should be taken under `vSwarm` instead of `vSwarm-proto`
### 2.Modify relay
Assume you have changed `vSwarm-proto` but the change hasn't been merged into main branch, if you want to test its functionality locally, you should build a local relay docker image to replace the one on the cloud, so:
```
cd tools/bin/relay
# commit id is the commit you have made to vSwarm-proto repo
go get github.com/vhive-serverless/vSwarm-proto@{commit id}
make all
```

Now, you have a local relay image that uses your modified version of protoc.
Remember **DO NOT** commit changes in `tools/relay` to the remote repo

### 3.Dockerize your benchmark
This is a case-by-case process, guidelines on how to do this will be given in the future.

### 4.Adding CI
Guidelines on how to do this will be given in the future

0 comments on commit 233f2bd

Please sign in to comment.