First pre-release (#15)

* Update repo structure

* Update install script

* Separate consensus & DV spec, rename files

* Add DV networking spec

* Update README & add figures

* Add background for images

* Unify structure with rest of ethereum repos (#2)

* Migrate consensus specs from a submodule to the pypi eth2specs

* Move from install script to Makefile & add linting

* Local slashing db (#3)

* Migrate consensus specs from a submodule to the pypi eth2specs

* Move from install script to Makefile & add linting

* All of the linting

* Move over to local slashing db & start defining own types

* Add ValidatorIdentity, fix tests, fix imports

* Revert to relative imports in src/dvspec/* module files

* Add glossary

* Update UML diagrams

* Formatting in glossary

Co-authored-by: Aditya Asgaonkar <[email protected]>

* Post #3 cleanup - remove consensus-specs submodule

* Rename bn_produce_attestation_data, describe proposal production in consensus, misc. bugfixes

* Make VC driver for signing process

* Fix linter errors

* Adds note about requesting duties & sync committees

* General refactoring

* Added tests.helpers.consensus for implementing consensus

* Fix lint errors

* Implement tests.helpers.consensus.consensus_on_block, fix lint errors

* Tests are executable! :)

* Fix lint errors

* Update README (#9)

Update README for clarity

Co-authored-by: Carl Beekhuizen <[email protected]>

* Remove debugging prints

* Update sequence diagrams

* Restructure README

* Describe DVC general operation

* Move up the glossary link

* fix typo

* Update README.md

* Add handling for sync committee duties (#14)

* Adds sync committees to dvspecs

* bump tests to import from altair

* Added sync committee methods, some bugfixes (#13)

Co-authored-by: Carl Beekhuizen <[email protected]>

Co-authored-by: Carl Beekhuizen <[email protected]>
Co-authored-by: dankrad <[email protected]>

.gitignore

@@ -1,3 +1,3 @@
 venv
-__pycache__
-consensus-specs
+*.egg-info
+__pycache__

Makefile

@@ -0,0 +1,31 @@
+VENV_NAME?=venv
+VENV_ACTIVATE=. $(VENV_NAME)/bin/activate
+
+help:
+	@echo "Makefile help. Your options are:"
+	@echo "clean - remove build and Python file artifacts"
+	@echo "install - install basic dependencies with venv"
+	@echo "venv_lint - check style with flake8 and mypy with venv"
+	@echo "venv_test - run tests with venv"
+
+clean:
+	rm -rf $(VENV_NAME)
+	rm -rf *.egg-info
+	find . -name __pycache__ -exec rm -rf {} \;
+	find . -name .mypy_cache -exec rm -rf {} \;
+
+install:
+	python3 -m venv $(VENV_NAME);
+	$(VENV_ACTIVATE)
+	${VENV_NAME}/bin/python -m pip install -r requirements.txt
+	${VENV_NAME}/bin/python -m pip install .
+	@touch $(VENV_NAME)/bin/activate
+
+venv_activate:
+	$(VENV_ACTIVATE)
+
+venv_lint: venv_activate
+	$(VENV_ACTIVATE) && flake8 --config=flake8.ini ./src ./tests && mypy --config-file mypy.ini ./src ./tests
+
+venv_test: venv_activate
+	$(VENV_ACTIVATE) && ${VENV_NAME}/bin/python -m pytest tests/test.py

README.md

@@ -1,41 +1,69 @@
 # Ethereum Distributed Validator Specification
 
-This repo is WIP. Dumping stuff here for now, will clean up soon.
+Distributed Validators (DV) are a technique for distributing the job of an Ethereum validator among a set of distributed nodes in order to improve resilience (safety, liveness, or both) as compared to running a validator client on a single machine.
 
-## General Architecture
+## Introduction
 
-`Beacon Node <--> SSV Node <--> Validator Client`
+### Motivation
 
-SSV Node sits in the middle of the BN and VC. The VC implementation is unchanged (or minimal necessary changes).
+#### Traditional Validator Client Setup
+Ethereum validators participate in the [proof-of-stake (PoS) protocol](https://github.com/ethereum/consensus-specs) by signing messages (such as blocks or attestations) using their staking private key. The staking key is accessible only by the validator client software, which schedules the creation & signing of messages according to the duties assigned to the validator. Some risks involved in a traditional validator client setup are:
+- The staking private key resides in one location. If an adversary gains access to this key, it can create conflicting messages that result in slashing of the validator's deposit.
+    - Stakers who do not operate their own validator need to hand over their staking private key to the operator. They must trust the operator for the security of their staking private key.
+- If the validator client software is unable to create timely messages to perform validator duties, the validator suffers an inactivity leak that reduces its balance.
+    - This could happen due to causes such as software crashes, loss of network connection, hardware faults, etc.
+- If the Beacon Node to which the validator client is connected has a fault, a validator may end up following a minority fork resulting it appearing to be offline to the rest of the PoS protocol.
 
-## Desired Properties
-- Safety: Validator is never slashed unless `X` fraction of the SSV-VC nodes are Byzantine, even under asynchronous network
-- No Deadlock: The protocol never ends up in a deadlock state where no progress can be made
-- Liveness: The protocol will eventually produce a new attestation/block, under partially synchronous network
+#### Distributed Validator Protocol
+The Distributed Validator protocol presents a solution to mitigate the risks & concerns associated with traditional, single Validator Client setups. In addition, this protocol can be used to enable advanced staking setups such as decentralized staking pools.
 
+### Basic Concepts
 
-## Design Rationale
-- Validity of attestation data must be checked against the slashing DB at the beginning of the consensus process
-- If the consensus process returns an attestation, then the slashing DB must allow it at that time
-- There can only be one consensus process running at any given moment
-- Slashing DB cannot be updated when a consensus process is running
-- Consensus processes for attestation duties must be spawned in increasing order of slot
+**Note**: Refer to the [glossary](glossary.md) for an explanation of new terms introduced in the Distributed Validator specifications.
 
-## Spec
+The two fundamental concepts behind Distributed Validators are:
+- **consensus**: the responsibilities of a single validator are split among several co-validators, who must work together to reach agreement on how to vote before signing any message.
+- ***M-of-N* threshold signatures**: the validator's staking key is split into *N* pieces and each of the co-validators holds a share. When at least *M* of the co-validators reach consensus on how to vote, they each sign the message with their share and a combined signature can be reconstructed from the shares.
 
-Assuming some round based consensus protocol with leader.
+Ethereum proof-of-stake uses the BLS signature scheme, in which the private keys can be *M-of-N* secret-shared (using Shamir secret sharing) to implement *M-of-N* threshold signatures.
 
-### Attestation Production
+By combining a suitable (safety-favouring) consensus algorithm with an *M-of-N* threshold signature scheme, the DV protocol ensures that agreement is backed up by cryptography and at least *M* co-validators agree about any decision.
 
-Leader:
-- Get attestation data from BN
-- Propose attestation in consensus protocol
+### Resources
 
-Node:
-- Follow consensus protocol: wait for proposal, make pre-vote, etc.
-- To determine validity of the proposal, check against for unslashability in the slashing DB.
-- When consensus returns a value, submit value to VC for threshold signing
-- Broadcast & combine threshold signed values to generate complete signed value
-- Send complete signed value to BN for p2p gossip
+#### Implementations
 
-Disregard voting for the correct fork for now. 
+The following are existing implementations of Distributed Validator technology (but not necessarily implementations of this specification).
+
+- [`python-ssv`](https://github.com/dankrad/python-ssv): A proof-of-concept implementation of the distributed validator protocol in Python that interacts with the [Prysm Ethereum client](https://github.com/prysmaticlabs/prysm).
+- [`ssv`](https://github.com/bloxapp/ssv): An implementation of the distributed validator protocol in Go that interacts with the [Prysm Ethereum client](https://github.com/prysmaticlabs/prysm).
+
+#### Documents
+- [Distributed Validator Architecture Video Introduction](https://www.youtube.com/watch?v=awBX1SrXOhk)
+
+### General Architecture
+
+![General Architecture](figures/general-architecture.png)
+
+This specification presents a way to implement Distributed Validator Client software as middleware between the Beacon Node (BN) and Validator Client (VC):
+- all communication between the BN and VC is intercepted by the DVC in order for it to provide the additional DV functionality.
+- the BN & VC are unaware of the presence of the DVC, i.e., they think they are connected to each other as usual.
+
+### Assumptions
+- We assume *N* total nodes and an *M-of-N* threshold signature scheme.
+    - For general compatibility with BFT consensus protocols, we assume that `M = (2 * N / 3) + 1`.
+- This specification assumes [some leader-based safety-favoring consensus protocol](src/dvspec/consensus.py) for the Co-Validators to decide on signing upon the same attestation/block. We assume that the consensus protocol runs successfully with *M* correct nodes out of *N* total nodes.
+- We assume the usual prerequisites for safe operation of the Validator Client, such as an up-to-date anti-slashing database, correct system clock, etc.
+- We disregard the voting on the "correct" Ethereum fork for now - this functionality will be added in a future update.
+
+### Desired Guarantees
+- **Safety (against key theft)**:
+    - The Validator's staking private key is secure unless security is compromised at more than *M* of the *N* Co-Validators.
+- **Safety (against slashing)**: 
+    - Under the assumption of an asynchronous network, the Validator is never slashed unless more than 1/3rd of the Co-Validators are Byzantine.
+    - Under the assumption of a synchronous network, the Validator is never slashed unless more than 2/3rds of the Co-Validators are Byzantine.
+- **Liveness**: The protocol will eventually produce a new attestation/block under partially synchronous network unless more than 1/3rd of the Co-Validators are Byzantine.
+
+## Specification
+
+Technical details about the specification are described in [`src/dvspec/`](src/dvspec/).

flake8.ini

@@ -0,0 +1,2 @@
+[flake8]
+max-line-length= 120

glossary.md

@@ -0,0 +1,21 @@
+# Glossary
+
+## Ethereum Concepts
+
+- **Validator**: A public key participating in proof-of-stake validation of Ethereum. In Phase 0, validators are expected to perform attestation & block proposal duties for beacon chain blocks.
+- **Validator Client (VC)**: Software to perform the duties of Validators. The VC has access to the private key of the Validators.
+
+## Distributed Validator Concepts
+
+- **Distributed Validator (DV)**: A group of participants collboratively performing the duties of a Validator. The Validator's private key is secret-shared among the participants so that a complete signature cannot be formed without some majority threshold of participants.
+- **Co-Validator**: A threshold BLS public key participating in the DV protocol for a *particular* Validator.
+- **Distributed Validator Client (DVC)**: Software to participate as a Co-Validator by running the DV protocol (or, to participate as multiple Co-Validators that are each associated with a different Validator). The DVC has access to the private key(s) of the Co-Validator(s), which is(are) the secret-shared threshold private key of the respective Validator(s).
+
+## Example
+
+An illustrative example for usage of terms described above:
+- Ethereum Validator with pubkey `0xa5c91...` is operated as a Distributed Validator.
+- 4 Co-Validators are participating in the Distributed Validator for Validator `0xa5c91...`. 
+- The private key associated with `0xa5c91...` is split using *3-of-4* secret-sharing among the 4 Co-Validators such that a *3-of-4* threshold signature scheme is setup.
+    - In simpler terms, the private key for `0xa5c91...` is split into 4 pieces, each in the custody of one of the Co-Validators such that at least 3 of them have to collaborate to produce a signature from `0xa5c91...`.
+- Each Co-Validator is running the Distributed Validator Client software to participate the in the Distributed Validator.

mypy.ini

@@ -0,0 +1,12 @@
+[mypy]
+warn_unused_ignores = True
+ignore_missing_imports = True
+strict_optional = False
+check_untyped_defs = True
+disallow_incomplete_defs = True
+disallow_untyped_defs = True
+disallow_any_generics = True
+disallow_untyped_calls = True
+warn_redundant_casts = True
+warn_unused_configs = True
+strict_equality = True

requirements.txt

@@ -0,0 +1,9 @@
+# execution requirements
+eth2spec==1.1.2
+
+# dev requirements
+mypy
+flake8
+
+# test requirements
+pytest

setup.py

@@ -0,0 +1,13 @@
+#!/usr/bin/env python3
+
+from setuptools import find_packages, setup
+
+setup(name='dvspec',
+      version='0.1',
+      description='Ethereum Distributed Validator Specification',
+      author='Aditya Asgaonkar',
+      author_email='[email protected]',
+      url='https://github.com/ethereum/distributed-validator-specs',
+      package_dir={'':'src'},
+      packages=find_packages(where='src'),
+)