Merge pull request #12 from keep-network/local-setup-vm

Deploy local-setup on virtual machine using Packer & Vagrant

Here we introduce some provisioning scripts and Packer & Vagrant configs 
which can be used to run the local-setup repo on a separate virtual machine. 
This is a better alternative for manual installation on the local development 
machine because:
- It provides a separate environment managed by simple commands which 
don't collide with the host environment
- It always operates on the same OS and one can avoid problems with 
incompatible tools
- It always uses the same versions of tools for local-setup installation and 
configuration so one can avoid problems caused by tools updates
- It uses automated scripts for provisioning which is faster, more comfortable 
and less error-prone compared to manual setup

Provisioning scripts are divided into two directories:
- local-setup-environment: those scripts are responsible to prepare the environment 
containing all prerequisites needed to install local-setup and run the E2E test scripts. 
Additionally, this directory contains a packerfile.json which can be used by Packer to 
build the environment as a reusable Vagrant box or Docker image (this one has limited 
functionality)
- local-setup-instance: those scripts are responsible to run all auxiliary software and 
the local-setup installation script itself. Their outcome should be properly configured 
and ready to work local-setup instance. Additionally, this directory contains a Vagrantfile 
which can be used by Vagrant to run a provisioned VM automatically.

.gitignore

@@ -1,4 +1,7 @@
 .idea
 /ethereum/data/geth/
 /storage/
-node_modules
+node_modules
+vagrant-box
+.vagrant
+*.log

README.adoc

@@ -1,148 +1,35 @@
 = KEEP & tBTC local setup
 
-== Setting up development environment
-If you’re on macOS, install Homebrew and https://github.com/keep-network/keep-core/blob/master/scripts/macos-setup.sh[macos-setup.sh] script to setup all necessary tools.
+This repository provide a bunch of scripts which allow to install tBTC and
+Keep locally for development and test purposes. It also contains and
+enables execution of automated end-to-end system tests for tBTC.
 
-If you are not on macOS or you don't have Homebrew, you need to install:
+Basically, the entire setup operates on several component submodules:
 
-- Go compiler, at least 1.14
-- Geth (Ethereum client) 1.9.9-stable
-- Solidity, at least 0.5.17
-- Truffle, at least 5.1.18
-- Protobuf compiler, at least 3.11.4
-- `protoc-gen-gogoslick` toolchain
-- `jq`
+- https://github.com/keep-network/keep-core[`keep-core`]
+- https://github.com/keep-network/keep-ecdsa[`keep-ecdsa`]
+- https://github.com/keep-network/tbtc[`tbtc`]
+- https://github.com/keep-network/tbtc.js[`tbtc.js`]
+- https://github.com/keep-network/tbtc-dapp[`tbtc-dapp`]
+- https://github.com/keep-network/relays[`relays`]
 
-Regardless you use `macos-setup.sh` or not, you should also install:
+It also uses some auxiliary software needed by tBTC and Keep:
 
-- https://github.com/pyenv/pyenv[pyenv], at least 1.2.18
-- https://github.com/pypa/pipenv[pipenv], at least 2018.11.26
-- https://docs.docker.com/get-docker[Docker] & https://docs.docker.com/compose/install/[Docker Compose]
+- https://github.com/ethereum/go-ethereum[Geth]
+- https://bitcoincore.org[Bitcoin Core]
+- https://electrumx.readthedocs.io/en/latest[ElectrumX]
 
-== Node.js version notice
+== Installation
 
-Please use Node `v11.15.0` for everything but E2E test script.
-To run the test script, please use Node `v14.3.0`.
+The installation can be performed in two different ways described below:
 
-We recommend using https://github.com/nvm-sh/nvm[nvm] to manage node versions easily.
-You can install both versions by doing:
-```
-nvm install 11.15.0
-nvm install 14.3.0
-```
-And use the specific version:
-```
-nvm use 11.15.0
-nvm use 14.3.0
-```
+=== Automatic installation on separate virtual machine
 
-== Installing the system
-. Initialize local `geth` using:
-+
-```
-./initialize-geth.sh
-```
-+
-You can skip this step if your local geth is already initialized. This script clears all Ethereum client data, initilizes genesis block, and funds five accounts. All client data are kept in the working directory of this project so all the data used for other projects stay untouched.
+To install everything automatically on a separate VirtualBox machine using Packer and
+Vagrant, please follow the <<./docs/auto-setup.adoc#title, automatic setup guide>>.
 
-. Run local `geth` node using:
-+
-```
-./run-geth.sh
-```
-. Run local Bitcoin Core node and ElectrumX using:
-+
-```
-./run-bitcoin.sh
-```
-. Run Keep & tBTC installation script:
-+
-```
-./install.sh
-```
-+ 
-This script will fetch `keep-core`, `keep-ecdsa`, and `tbtc` source code, deploy contracts of `keep-core`, `keep-ecdsa`, and `tbtc`. It will also build `keep-core` and `keep-ecdsa` off-chain clients.
-+
-Later on, if you decide to update `tbtc`, you can run just `install-tbtc.sh`. 
-+
-Keep in mind, `tbtc` depends on `keep-ecdsa` and `keep-ecdsa` depends on `keep-core` so if you decide to update `keep-ecdsa`, you can run `install-keep-ecdsa.sh` followed by `install-tbtc.sh`. 
-+
-If you decide to update `keep-core`, you have to run `install-keep-core.sh` followed by `install-keep-ecdsa.sh` followed by `install-tbtc.sh` or just the entire `install.sh` again.
+=== Manual installation on your local machine
 
-== Run clients
-The above installation script will configure:
+To install everything on your local machine by hand,
+please follow the <<./docs/manual-setup.adoc#title, manual setup guide>>.
 
-- 1 `keep-core` client
-- 3 `keep-ecdsa` clients
-
-To run the `keep-core` client use:
-```
-./run-core-1.sh
-```
-
-It is enough to run one `keep-core` client to generate a group and produce relay entries. Setting up more than one client locally is possible but consumes more resources.
-
-To run `keep-ecdsa` clients use:
-```
-./run-ecdsa-1.sh
-```
-```
-./run-ecdsa-2.sh
-```
-```
-./run-ecdsa-3.sh
-```
-
-There are at least 3 `keep-ecdsa` clients needed to open a keep. Setting up more than three clients locally is possible but consumes more resources.
-
-== Beacon genesis
-
-Before the beacon is able to produce a first relay entry, genesis needs to happen. Genesis triggers the first random beacon group selection.
-
-Genesis should be triggered after `keep-core` client started with:
-```
-cd keep-core
-KEEP_ETHEREUM_PASSWORD="password" ./keep-core --config configs/config.local.1.toml relay genesis
-```
-
-Bonded ECDSA keep factory from `keep-ecdsa` contracts requests for new relay entry to reseed after each signer selection but it is also possible to request for a new relay entry manually with:
-```
-cd keep-core
-KEEP_ETHEREUM_PASSWORD="password" ./keep-core --config configs/config.local.1.toml relay request
-```
-
-= How to interact with the system
-
-You can interact with the system through the tBTC dApp or automated
-end-to-end tests. Before you start interacting, make sure you:
-
-- Installed all system components using `install.sh` script
-- Have a local Geth instance (`run-geth.sh`) working
-- Have local Bitcoin Core and ElectrumX instances (`run-bitcoin.sh`) working
-- Have 1 `keep-core` and 3 `keep-ecdsa` clients up and running
-
-== Keep Dashboard dApp
-
-To run the Keep Dashboard dApp invoke:
-```
-./run-keep-dashboard.sh
-```
-
-== tBTC dApp
-
-To run the tBTC dApp against the local Bitcoin network,
-make sure the tBTC dApp is configured to work with the local ElectrumX instance.
-You can configure this in `./tbtc-dapp/src/wrappers/web3.js` file,
-where `TBTC.withConfig` line occurs.
-Then, you can invoke:
-```
-./run-tbtc-dapp.sh
-```
-The application will be available on `http://localhost:3000`.
-
-== E2E tests
-
-To run the automated end-to-end scenario invoke:
-```
-./run-e2e-test.sh
-```

deployments/local-setup-environment/packerfile.json

@@ -0,0 +1,67 @@
+{
+  "builders": [
+    {
+      "type": "vagrant",
+      "source_path": "ubuntu/xenial64",
+      "provider": "virtualbox",
+      "communicator": "ssh",
+      "output_dir": "vagrant-box"
+    },
+    {
+      "type": "docker",
+      "image": "ubuntu:xenial",
+      "commit": true,
+      "run_command": [
+        "-d", "-i", "-t",
+        "-w=/root",
+        "--entrypoint=/bin/bash",
+        "--",
+        "{{.Image}}"
+      ],
+      "changes": [
+        "WORKDIR /root",
+        "ENTRYPOINT /bin/bash -l"
+      ]
+    }
+  ],
+  "provisioners": [
+    {
+      "type": "shell",
+      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} bash -l {{ .Path }}",
+      "inline": ["apt-get update && apt-get -y install sudo"],
+      "only": ["docker"]
+    },
+    {
+      "type": "shell",
+      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} bash -l {{ .Path }}",
+      "scripts": [
+        "./provisioning/install-commons.sh",
+        "./provisioning/install-nodejs.sh",
+        "./provisioning/install-go.sh",
+        "./provisioning/install-geth.sh",
+        "./provisioning/install-solidity.sh",
+        "./provisioning/install-protobuf.sh",
+        "./provisioning/install-docker.sh",
+        "./provisioning/install-docker-compose.sh"
+      ]
+    },
+    {
+      "type": "shell",
+      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} bash -l {{ .Path }}",
+      "inline": ["npm -g config set user root"],
+      "only": ["docker"]
+    },
+    {
+      "type": "shell",
+      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} bash -l {{ .Path }}",
+      "script": "./provisioning/install-truffle.sh"
+    }
+  ],
+  "post-processors": [
+    {
+      "type": "docker-tag",
+      "repository": "local-setup-environment",
+      "only": ["docker"]
+    }
+  ]
+}

deployments/local-setup-environment/provisioning/install-commons.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+set -e
+
+echo "Installing common tools..."
+
+sudo apt-get update
+
+sudo apt-get install -y \
+  curl \
+  wget \
+  git \
+  unzip \
+  jq \
+  python \
+  build-essential
+
+if ! [ -x "$(command -v curl)" ]; then echo "curl installation failed"; exit 1; fi
+if ! [ -x "$(command -v wget)" ]; then echo "wget installation failed"; exit 1; fi
+if ! [ -x "$(command -v git)" ]; then echo "git installation failed"; exit 1; fi
+if ! [ -x "$(command -v unzip)" ]; then echo "unzip installation failed"; exit 1; fi
+if ! [ -x "$(command -v jq)" ]; then echo "jq installation failed"; exit 1; fi
+if ! [ -x "$(command -v python)" ]; then echo "python installation failed"; exit 1; fi
+if ! [ -x "$(command -v make)" ]; then echo "build-essential installation failed"; exit 1; fi
+
+echo "Common tools have been installed successfully!"

deployments/local-setup-environment/provisioning/install-docker-compose.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+set -e
+
+echo "Installing Docker Compose..."
+
+sudo curl -L "https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m)" \
+  -o /usr/local/bin/docker-compose
+
+sudo chmod +x /usr/local/bin/docker-compose
+
+if ! [ -x "$(command -v docker-compose)" ]; then echo "Docker compose installation failed"; exit 1; fi
+
+echo "Docker Compose has been installed successfully!"

deployments/local-setup-environment/provisioning/install-docker.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+set -e
+
+echo "Installing Docker..."
+
+sudo apt-get install -y \
+    apt-transport-https \
+    ca-certificates \
+    curl \
+    gnupg-agent \
+    software-properties-common
+
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+
+sudo add-apt-repository \
+  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
+  $(lsb_release -cs) \
+  stable"
+
+sudo apt-get update
+sudo apt-get install -y docker-ce docker-ce-cli containerd.io
+
+if ! [ -x "$(command -v docker)" ]; then echo "Docker installation failed"; exit 1; fi
+
+echo "Docker has been installed successfully!"

deployments/local-setup-environment/provisioning/install-geth.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -e
+
+echo "Installing go-ethereum..."
+
+GETH_PACKAGE=geth-alltools-linux-amd64-1.9.9-01744997.tar.gz
+
+curl -O https://gethstore.blob.core.windows.net/builds/$GETH_PACKAGE
+
+tar -xvf $GETH_PACKAGE
+mkdir ./go-ethereum && tar -xzf $GETH_PACKAGE -C ./go-ethereum --strip-components=1
+sudo chown -R root:root ./go-ethereum
+sudo mv ./go-ethereum/* /usr/local/bin
+
+if ! [ -x "$(command -v geth)" ]; then echo "go-ethereum installation failed"; exit 1; fi
+
+echo "go-ethereum has been installed successfully!"

deployments/local-setup-environment/provisioning/install-go.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+set -e
+
+echo "Installing Go..."
+
+GOLANG_PACKAGE=go1.13.4.linux-amd64.tar.gz
+
+curl -O https://storage.googleapis.com/golang/$GOLANG_PACKAGE
+
+tar -xvf $GOLANG_PACKAGE
+sudo chown -R root:root ./go
+sudo mv go /usr/local
+
+echo 'GOPATH="$HOME/go"' >> ~/.profile
+echo 'PATH="$PATH:/usr/local/go/bin:$GOPATH/bin"' >> ~/.profile
+source ~/.profile
+
+if ! [ -x "$(command -v go)" ]; then echo "Go installation failed"; exit 1; fi
+
+echo "Go has been installed successfully!"

deployments/local-setup-environment/provisioning/install-nodejs.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+set -e
+
+echo "Installing Node.js and NPM..."
+
+curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
+
+echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.profile
+echo '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"' >> ~/.profile
+source ~/.profile
+
+nvm install 14.3.0
+nvm install 11.15.0
+nvm alias default 11.15.0
+nvm use default
+
+if ! [ -x "$(command -v node)" ]; then echo "Node installation failed"; exit 1; fi
+if ! [ -x "$(command -v npm)" ]; then echo "NPM installation failed"; exit 1; fi
+
+echo "Node.js and NPM have been installed successfully!"