Feature/grpc execution api (#1)

* add buf config and generated grpc code

* poc e2e grpc communication

* comment out panicking code

* logging help

* now reads cli args. now stops grpc server on shutdown.

* add mutex to GRPCServerHandler

update readme

containerize w/ github action to build and push to ghcr (#3)

fix gh action syntax (#4)

* containerize w/ github action to build and push to ghcr

* fix gh action yml syntax

* enable manual trigger

* fix syntax

build and push manually (#5)

* containerize w/ github action to build and push to ghcr

* fix gh action yml syntax

* enable manual trigger

* fix syntax

* dont use github action, do it ourselves

* push manually

Feature/containerize (#6)

* containerize w/ github action to build and push to ghcr

* fix gh action yml syntax

* enable manual trigger

* fix syntax

* dont use github action, do it ourselves

* push manually

* correct multi line

update SubmitTransaction to send tx to metro

ethclient: ensure returned subscription is nil on error (#26976)

core/state, trie: remove Try prefix in Trie accessors (#26975)

This change renames StateTrie methods to remove the Try* prefix.

We added the Trie methods with prefix 'Try' a long time ago, working
around the problem that most existing methods of Trie did not return the
database error. This weird naming convention has persisted until now.

Co-authored-by: Gary Rong <[email protected]>

metrics/librato: ensure resp.body closed (#26969)

This change ensures that we call Close on a http response body, in various places in the source code (mostly tests)

core/vm: use atomic.Bool (#26951)

Make use of new atomic types
---------

Co-authored-by: Felix Lange <[email protected]>
Co-authored-by: Martin Holst Swende <[email protected]>

core/bloombits: use atomic type (#26993)

core/state: use atomic.Bool (#26992)

graphql: fix data races (#26965)

Fixes multiple data races caused by the fact that resolving fields are done concurrently by the graphql library. It also enforces caching at the stateobject level for account fields.

eth/tracers/native: prevent panic for LOG edge-cases (#26848)

This PR fixes OOM panic in the callTracer as well as panicing on
opcode validation errors (e.g. stack underflow) in callTracer and
prestateTracer.

Co-authored-by: Martin Holst Swende <[email protected]>

internal/debug: add log.logfmt flag to set logging to use logfmt (#26970)

docs: update outdated DeriveSha docs comment (#26968)

add localnet genesis

update metro-transactions dep

Adding features to mempool to support our pre-ordered txs (#2)

* Adding features to mempool to support our pre-ordered txs

* Add framing for gRPC to execute blocks

* Remove public engine API call

* Fix circular dependencies

* Updated to use new Init, and fill out starting attributes

* Add clear astriaordered, cleanup

* readme fix

* add bash and jq to final docker image

* make txpool interface

* no more panics, update DoBlock to also update state + store block

* cleanup

* set post-merge at genesis

* cleanup

* doc update

* remove txpool interface changes

* cleanup

* cleanup

* build and push images wih tags defined by git tags

* build for multiple architectures. use docker-metadata action for semver

* add push: true

* only build arm for git tags/releases

---------

Co-authored-by: Jesse Snyder <[email protected]>
Co-authored-by: elizabeth <[email protected]>

tag image with latest for builds from astria branch (#8)

rename state_root to block_hash

use FROM --platform=$BUILDPLATFORM to build arm images correctly (#9)

use v4 of build and push action (#10)

* use FROM --platform=$BUILDPLATFORM to build arm images correctly

* use v4 of docker build and push

fix arm builds. only build arm for tags and merges to default branch. (#12)

* fix arm builds. only build arm for tags and merges to default branch.

* fix gh action function syntax

* more correct comment

* build for semver tags

Features and fixes needed for contract deployment test (#13)

* add and remove tx from geth mempool so forge can deploy a contract

* pass metro addr and port by flag

* fixes from pr feedback

* formatting

implement FinalizeBlock gRPC call

bump metro-transactions dep

Update Protos (#16)

* uses new protos

* remove old protos

update deps

update deps

no more unknown/unknown image (#19)

* no more unknown/unknown image

* hardcode condition

Changing from prev_state_root to prev_block_hash

remove metro

integrate submission to sequencer

log updates

use env vars for chain id/tendermint endpoint

rename to cometbft

Make the environment variables actually work

cleaner

simple logging change

removed sleep before getPayload by removing goroutine in buildPayload

removed unneeded comment

removed for loop from buildPayload

Remove direct sequencer submission

cleanup go.mod

Migrate to the v1alpha1 API

Missed pieces

set genesis block as head/safe/final

remove all mentions of metro

Initial implementation of Execution v1alpha2 api

Integrate updates

remove need for consensus api

Updates

small updates

Update grpc/execution/server.go

Co-authored-by: noot <[email protected]>

logging

remove unused dependency

Mark the chain 'finalized' on startup

.github/workflows/astria-build-and-publish-image.yml

@@ -0,0 +1,68 @@
+name: Build and Publish Docker image
+
+# Trigger on pushes to astria branch, new semantic version tags, and pull request updates
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - "astria"
+    tags:
+      - "v[0-9]+.[0-9]+.[0-9]+"
+      - "v[0-9]+.[0-9]+.[0-9]+-alpha.[0-9]+"
+      - "v[0-9]+.[0-9]+.[0-9]+-beta.[0-9]+"
+      - "v[0-9]+.[0-9]+.[0-9]+-rc[0-9]+"
+  # trigger on pull request updates when target is `astria` branch
+  pull_request:
+    branches:
+      - "astria"
+
+jobs:
+  build-and-publish-latest:
+    runs-on: ubuntu-latest
+    steps:
+      # Checking out the repo
+      - uses: actions/checkout@v3
+      # Setting up Go
+      - uses: actions/setup-go@v4
+        with:
+          go-version: "^1.20.x" # The Go version to download (if necessary) and use.
+      - run: go version
+
+      # https://github.com/docker/setup-qemu-action
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+      # https://github.com/docker/setup-buildx-action
+      - name: Set up Docker Buildx
+        id: buildx
+        uses: docker/setup-buildx-action@v2
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v2
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      # Generate correct tabs and labels
+      - name: Docker metadata
+        id: metadata
+        uses: docker/metadata-action@v4
+        with:
+          images: |
+            ghcr.io/astriaorg/go-ethereum
+          tags: |
+            type=ref,event=pr
+            type=semver,pattern={{major}}.{{minor}}.{{patch}}
+            type=sha
+            # set latest tag for `astria` branch
+            type=raw,value=latest,enable=${{ github.ref == format('refs/heads/{0}', 'astria') }}
+      - name: Build and push
+        uses: docker/build-push-action@v4
+        with:
+          # this gets rid of the unknown/unknown image that is created without this setting
+          # https://github.com/docker/build-push-action/issues/820#issuecomment-1455687416
+          provenance: false
+          context: .
+          # It takes a long time to build the arm image right now, so we only build it on tags which is what we use for releases, or on merges to the default branch.
+          platforms: ${{ (contains(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/astria') && 'linux/amd64,linux/arm64' || 'linux/amd64' }}
+          push: true
+          tags: ${{ steps.metadata.outputs.tags }}
+          labels: ${{ steps.metadata.outputs.labels }}

Dockerfile

@@ -20,9 +20,14 @@ RUN cd /go-ethereum && go run build/ci.go install -static ./cmd/geth
 FROM alpine:latest
 
 RUN apk add --no-cache ca-certificates
+# Astria - add bash and jq to support start-geth.sh in conductor
+RUN apk add bash jq
+# Astria - copy genesis.json so it can be used in start-geth.sh
+COPY genesis.json /genesis.json
 COPY --from=builder /go-ethereum/build/bin/geth /usr/local/bin/
 
-EXPOSE 8545 8546 30303 30303/udp
+# Astria - add 50051 for GRPC
+EXPOSE 8545 8546 30303 30303/udp 50051
 ENTRYPOINT ["geth"]
 
 # Add some metadata labels to help programmatic image consumption

Dockerfile.alltools

@@ -22,7 +22,8 @@ FROM alpine:latest
 RUN apk add --no-cache ca-certificates
 COPY --from=builder /go-ethereum/build/bin/* /usr/local/bin/
 
-EXPOSE 8545 8546 30303 30303/udp
+# Astria - add 50051 for GRPC
+EXPOSE 8545 8546 30303 30303/udp 50051
 
 # Add some metadata labels to help programmatic image consumption
 ARG COMMIT=""

cmd/geth/config.go

@@ -37,6 +37,7 @@ import (
 	"github.com/ethereum/go-ethereum/common/hexutil"
 	"github.com/ethereum/go-ethereum/eth/catalyst"
 	"github.com/ethereum/go-ethereum/eth/ethconfig"
+	"github.com/ethereum/go-ethereum/grpc/execution"
 	"github.com/ethereum/go-ethereum/internal/flags"
 	"github.com/ethereum/go-ethereum/internal/version"
 	"github.com/ethereum/go-ethereum/log"
@@ -202,6 +203,14 @@ func makeFullNode(ctx *cli.Context) *node.Node {
 	if ctx.IsSet(utils.GraphQLEnabledFlag.Name) {
 		utils.RegisterGraphQLService(stack, backend, filterSystem, &cfg.Node)
 	}
+
+	// Configure gRPC if requested.
+	if ctx.IsSet(utils.GRPCEnabledFlag.Name) {
+		serviceV1a1 := execution.NewExecutionServiceServerV1Alpha1(eth)
+		serviceV1a2 := execution.NewExecutionServiceServerV1Alpha2(eth)
+		utils.RegisterGRPCExecutionService(stack, serviceV1a1, serviceV1a2, &cfg.Node)
+	}
+
 	// Add the Ethereum Stats daemon if requested.
 	if cfg.Ethstats.URL != "" {
 		utils.RegisterEthStatsService(stack, backend, cfg.Ethstats.URL)

cmd/geth/main.go

@@ -188,6 +188,9 @@ var (
 		utils.AllowUnprotectedTxs,
 		utils.BatchRequestLimit,
 		utils.BatchResponseMaxSize,
+		utils.GRPCEnabledFlag,
+		utils.GRPCHostFlag,
+		utils.GRPCPortFlag,
 	}
 
 	metricsFlags = []cli.Flag{

cmd/utils/flags.go

@@ -57,6 +57,8 @@ import (
 	"github.com/ethereum/go-ethereum/ethdb/remotedb"
 	"github.com/ethereum/go-ethereum/ethstats"
 	"github.com/ethereum/go-ethereum/graphql"
+	executionv1a1 "github.com/ethereum/go-ethereum/grpc/gen/astria/execution/v1alpha1"
+	executionv1a2 "github.com/ethereum/go-ethereum/grpc/gen/astria/execution/v1alpha2"
 	"github.com/ethereum/go-ethereum/internal/ethapi"
 	"github.com/ethereum/go-ethereum/internal/flags"
 	"github.com/ethereum/go-ethereum/log"
@@ -749,6 +751,24 @@ var (
 		Usage:    "Enables the (deprecated) personal namespace",
 		Category: flags.APICategory,
 	}
+	// grpc
+	GRPCEnabledFlag = &cli.BoolFlag{
+		Name:     "grpc",
+		Usage:    "Enable the gRPC server",
+		Category: flags.APICategory,
+	}
+	GRPCHostFlag = &cli.StringFlag{
+		Name:     "grpc.addr",
+		Usage:    "gRPC server listening interface",
+		Value:    node.DefaultGRPCHost,
+		Category: flags.APICategory,
+	}
+	GRPCPortFlag = &cli.IntFlag{
+		Name:     "grpc.port",
+		Usage:    "gRPC server listening port",
+		Value:    node.DefaultGRPCPort,
+		Category: flags.APICategory,
+	}
 
 	// Network Settings
 	MaxPeersFlag = &cli.IntFlag{
@@ -1187,6 +1207,19 @@ func setHTTP(ctx *cli.Context, cfg *node.Config) {
 	}
 }
 
+// setGRPC creates the gRPC RPC listener interface string from the set command
+// line flags, returning empty if the gRPC endpoint is disabled.
+func setGRPC(ctx *cli.Context, cfg *node.Config) {
+	if ctx.Bool(GRPCEnabledFlag.Name) {
+		if ctx.IsSet(GRPCHostFlag.Name) {
+			cfg.GRPCHost = ctx.String(GRPCHostFlag.Name)
+		}
+		if ctx.IsSet(GRPCPortFlag.Name) {
+			cfg.GRPCPort = ctx.Int(GRPCPortFlag.Name)
+		}
+	}
+}
+
 // setGraphQL creates the GraphQL listener interface string from the set
 // command line flags, returning empty if the GraphQL endpoint is disabled.
 func setGraphQL(ctx *cli.Context, cfg *node.Config) {
@@ -1399,6 +1432,7 @@ func SetNodeConfig(ctx *cli.Context, cfg *node.Config) {
 	SetP2PConfig(ctx, &cfg.P2P)
 	setIPC(ctx, cfg)
 	setHTTP(ctx, cfg)
+	setGRPC(ctx, cfg)
 	setGraphQL(ctx, cfg)
 	setWS(ctx, cfg)
 	setNodeUserIdent(ctx, cfg)
@@ -1954,6 +1988,14 @@ func RegisterGraphQLService(stack *node.Node, backend ethapi.Backend, filterSyst
 	}
 }
 
+// RegisterGRPCExecutionService adds the gRPC API to the node.
+// It was done this way so that our grpc execution server can access the ethapi.Backend
+func RegisterGRPCExecutionService(stack *node.Node, execServerV1a1 executionv1a1.ExecutionServiceServer, execServerV1a2 executionv1a2.ExecutionServiceServer, cfg *node.Config) {
+	if err := node.NewGRPCServerHandler(stack, execServerV1a1, execServerV1a2, cfg); err != nil {
+		Fatalf("Failed to register the gRPC service: %v", err)
+	}
+}
+
 // RegisterFilterAPI adds the eth log filtering RPC API to the node.
 func RegisterFilterAPI(stack *node.Node, backend ethapi.Backend, ethcfg *ethconfig.Config) *filters.FilterSystem {
 	filterSystem := filters.NewFilterSystem(backend, filters.Config{

consensus/ethash/consensus.go

@@ -236,7 +236,6 @@ func (ethash *Ethash) verifyHeader(chain consensus.ChainHeaderReader, header, pa
 	}
 	// Verify the block's difficulty based on its timestamp and parent's difficulty
 	expected := ethash.CalcDifficulty(chain, header.Time, parent)
-
 	if expected.Cmp(header.Difficulty) != 0 {
 		return fmt.Errorf("invalid difficulty: have %v, want %v", header.Difficulty, expected)
 	}

core/blockchain.go

@@ -319,10 +319,10 @@ func NewBlockChain(db ethdb.Database, cacheConfig *CacheConfig, genesis *Genesis
 		return nil, ErrNoGenesis
 	}
 
-	bc.currentBlock.Store(nil)
-	bc.currentSnapBlock.Store(nil)
-	bc.currentFinalBlock.Store(nil)
-	bc.currentSafeBlock.Store(nil)
+	bc.currentBlock.Store(bc.genesisBlock.Header())
+	bc.currentSnapBlock.Store(bc.genesisBlock.Header())
+	bc.currentFinalBlock.Store(bc.genesisBlock.Header())
+	bc.currentSafeBlock.Store(bc.genesisBlock.Header())
 
 	// Update chain info data metrics
 	chainInfoGauge.Update(metrics.GaugeInfoValue{"chain_id": bc.chainConfig.ChainID.String()})

core/txpool/txpool.go

@@ -75,6 +75,8 @@ type TxPool struct {
 	term chan struct{}           // Termination channel to detect a closed pool
 
 	sync chan chan error // Testing / simulator channel to block until internal reset is done
+
+	astria *astriaOrdered
 }
 
 // New creates a new transaction pool to gather, sort and filter inbound
@@ -404,6 +406,37 @@ func (p *TxPool) Stats() (int, int) {
 	return runnable, blocked
 }
 
+func (pool *TxPool) SetAstriaOrdered(rawTxs [][]byte) {
+	txs := []*types.Transaction{}
+	for idx, rawTx := range rawTxs {
+		tx := new(types.Transaction)
+		err := tx.UnmarshalBinary(rawTx)
+		if err != nil {
+			log.Warn("failed to unmarshal raw astria tx bytes", rawTx, "at index", idx, "error:", err)
+			continue
+		}
+
+		txs = append(txs, tx)
+	}
+
+	pool.astria = newAstriaOrdered(types.Transactions(txs))
+}
+
+func (pool *TxPool) ClearAstriaOrdered() {
+	if pool.astria == nil {
+		return
+	}
+	pool.astria.clear()
+}
+
+func (pool *TxPool) AstriaOrdered() *types.Transactions {
+	// sus but whatever
+	if pool.astria == nil {
+		return &types.Transactions{}
+	}
+	return &pool.astria.txs
+}
+
 // Content retrieves the data content of the transaction pool, returning all the
 // pending as well as queued transactions, grouped by account and sorted by nonce.
 func (p *TxPool) Content() (map[common.Address][]*types.Transaction, map[common.Address][]*types.Transaction) {
@@ -480,3 +513,17 @@ func (p *TxPool) Sync() error {
 		return errors.New("pool already terminated")
 	}
 }
+
+type astriaOrdered struct {
+	txs types.Transactions
+}
+
+func newAstriaOrdered(txs types.Transactions) *astriaOrdered {
+	return &astriaOrdered{
+		txs: txs,
+	}
+}
+
+func (ao *astriaOrdered) clear() {
+	ao.txs = *&types.Transactions{}
+}

eth/catalyst/api.go

@@ -176,6 +176,7 @@ func newConsensusAPIWithoutHeartbeat(eth *eth.Ethereum) *ConsensusAPI {
 // If there are payloadAttributes: we try to assemble a block with the payloadAttributes
 // and return its payloadID.
 func (api *ConsensusAPI) ForkchoiceUpdatedV1(update engine.ForkchoiceStateV1, payloadAttributes *engine.PayloadAttributes) (engine.ForkChoiceResponse, error) {
+	log.Info("ForkchoiceUpdatedV1 called")
 	if payloadAttributes != nil {
 		if payloadAttributes.Withdrawals != nil || payloadAttributes.BeaconRoot != nil {
 			return engine.STATUS_INVALID, engine.InvalidParams.With(errors.New("withdrawals and beacon root not supported in V1"))

genesis.json

@@ -0,0 +1,22 @@
+{
+    "config": {
+        "chainId": 1337,
+        "homesteadBlock": 0,
+        "eip150Block": 0,
+        "eip155Block": 0,
+        "eip158Block": 0,
+        "byzantiumBlock": 0,
+        "constantinopleBlock": 0,
+        "petersburgBlock": 0,
+        "istanbulBlock": 0,
+        "berlinBlock": 0,
+        "londonBlock": 0,
+        "terminalTotalDifficulty": 0,
+        "ethash": {}
+    },
+    "difficulty": "10000000",
+    "gasLimit": "8000000",
+    "alloc": {
+        "0x46B77EFDFB20979E1C29ec98DcE73e3eCbF64102": { "balance": "300000000000000000000" }
+    }
+}

go.mod

@@ -35,7 +35,7 @@ require (
 	github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb
 	github.com/google/gofuzz v1.2.0
 	github.com/google/uuid v1.3.0
-	github.com/gorilla/websocket v1.4.2
+	github.com/gorilla/websocket v1.5.0
 	github.com/graph-gophers/graphql-go v1.3.0
 	github.com/hashicorp/go-bexpr v0.1.10
 	github.com/holiman/billy v0.0.0-20240216141850-2abb0c79d3c4
@@ -67,12 +67,14 @@ require (
 	github.com/tyler-smith/go-bip39 v1.1.0
 	github.com/urfave/cli/v2 v2.25.7
 	go.uber.org/automaxprocs v1.5.2
-	golang.org/x/crypto v0.22.0
-	golang.org/x/sync v0.7.0
-	golang.org/x/sys v0.19.0
+	golang.org/x/crypto v0.15.0
+	golang.org/x/sync v0.5.0
+	golang.org/x/sys v0.17.0
 	golang.org/x/text v0.14.0
-	golang.org/x/time v0.5.0
-	golang.org/x/tools v0.20.0
+	golang.org/x/time v0.3.0
+	golang.org/x/tools v0.15.0
+	google.golang.org/grpc v1.53.0
+	google.golang.org/protobuf v1.33.0
 	gopkg.in/natefinch/lumberjack.v2 v2.0.0
 	gopkg.in/yaml.v3 v3.0.1
 )
@@ -126,7 +128,8 @@ require (
 	github.com/mitchellh/pointerstructure v1.2.0 // indirect
 	github.com/mmcloughlin/addchain v0.4.0 // indirect
 	github.com/naoina/go-stringutil v0.1.0 // indirect
-	github.com/opentracing/opentracing-go v1.1.0 // indirect
+	github.com/onsi/ginkgo v1.16.4 // indirect
+	github.com/opentracing/opentracing-go v1.2.0 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/prometheus/client_golang v1.12.0 // indirect
@@ -140,9 +143,9 @@ require (
 	github.com/tklauser/numcpus v0.6.1 // indirect
 	github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
 	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
-	golang.org/x/mod v0.17.0 // indirect
-	golang.org/x/net v0.24.0 // indirect
-	google.golang.org/protobuf v1.33.0 // indirect
+	golang.org/x/mod v0.14.0 // indirect
+	golang.org/x/net v0.18.0 // indirect
+	google.golang.org/genproto v0.0.0-20230227214838-9b19f0bdc514 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
 	rsc.io/tmplfunc v0.0.3 // indirect
 )