Merge remote-tracking branch 'crypto/v0.38.x' into backport-unsync-cl…

…ient

.changelog/unreleased/features/3008-mempool-async-update.md

@@ -0,0 +1,3 @@
+- `[consensus]` Make mempool updates asynchronous from consensus Commit's,
+  reducing latency for reaching consensus timeouts.
+  ([#3008](https://github.com/cometbft/cometbft/pull/3008))

.changelog/unreleased/improvements/4-async-fire-events.md

@@ -0,0 +1,3 @@
+- `[consensus]` run fireEvents in goroutine.
+  ([#4](https://github.com/crypto-org-chain/cometbft/pull/4))
+

.changelog/v0.38.1/improvements/1558-experimental-gossip-limiting.md

@@ -4,6 +4,6 @@
   ([\#1584](https://github.com/cometbft/cometbft/pull/1584))
 - `[config]` Add mempool parameters `experimental_max_gossip_connections_to_persistent_peers` and
   `experimental_max_gossip_connections_to_non_persistent_peers` for limiting the number of peers to
-  which the node gossip transactions. 
+  which the node gossip transactions.
   ([\#1558](https://github.com/cometbft/cometbft/pull/1558))
   ([\#1584](https://github.com/cometbft/cometbft/pull/1584))

.changelog/v0.38.13/bug-fixes/4295-copy-value-state-indexer.md

@@ -0,0 +1,3 @@
+- `[state/indexer]` Fix the tx_search results not returning all results by changing the logic in the indexer to copy the key and values instead of reusing an iterator. This issue only arises when upgrading to cometbft-db v0.13 or later.
+  ([\#4295](https://github.com/cometbft/cometbft/issues/4295)). Special thanks to @faddat for reporting the issue.
+

.changelog/unreleased/dependencies/4059-update-cometbft-db.md → .changelog/v0.38.13/dependencies/4059-update-cometbft-db.md

@@ -1,4 +1,2 @@
 - `[go/runtime]` Bump Go version to 1.22
   ([\#4073](https://github.com/cometbft/cometbft/pull/4073))
-- Bump cometbft-db version to v0.12.0
-  ([\#4073](https://github.com/cometbft/cometbft/pull/4073))

.changelog/v0.38.13/dependencies/4321-update-cometbft-db.md

@@ -0,0 +1,2 @@
+- Bump cometbft-db version to v0.14.1
+  ([\#4321](https://github.com/cometbft/cometbft/pull/4321))

.changelog/v0.38.13/features/4294-remove-secp256k1-wrapper.md

@@ -0,0 +1 @@
+- `[crypto]` use decred secp256k1 directly ([#4294](https://github.com/cometbft/cometbft/pull/4294))

.changelog/v0.38.13/improvements/4123-mempool-is-full-log.md

@@ -0,0 +1,2 @@
+- `[log]` Change "mempool is full" log to debug level
+  ([\#4123](https://github.com/cometbft/cometbft/pull/4123)) Special thanks to @yihuang.

.changelog/v0.38.13/summary.md

@@ -0,0 +1,7 @@
+*October 24, 2024*
+
+This patch release addresses the issue where tx_search was not returning all results, which only arises when upgrading
+to CometBFT-DB version 0.13 or later. It includes a fix in the state indexer to resolve this problem. We recommend
+upgrading to this patch release if you are affected by this issue.
+
+

.golangci.yml

@@ -76,6 +76,7 @@ linters-settings:
           - github.com/spf13
           - github.com/stretchr/testify/require
           - github.com/syndtr/goleveldb
+          - github.com/decred/dcrd/dcrec/secp256k1/v4
       test:
         files:
           - "$test"
@@ -96,3 +97,53 @@ linters-settings:
           - github.com/prometheus/client_golang/prometheus/promhttp
           - github.com/spf13
           - github.com/stretchr/testify
+          - github.com/decred/dcrd/dcrec/secp256k1/v4
+
+  revive:
+    enable-all-rules: true
+    rules:
+      - name: comment-spacings # temporarily disabled
+        disabled: true
+      - name: max-public-structs
+        disabled: true
+      - name: cognitive-complexity
+        disabled: true
+      - name: argument-limit
+        disabled: true
+      - name: cyclomatic
+        disabled: true
+      - name: deep-exit
+        disabled: true
+      - name: file-header
+        disabled: true
+      - name: function-length
+        disabled: true
+      - name: function-result-limit
+        disabled: true
+      - name: line-length-limit
+        disabled: true
+      - name: flag-parameter
+        disabled: true
+      - name: add-constant
+        disabled: true
+      - name: empty-lines
+        disabled: true
+      - name: import-shadowing
+        disabled: true
+      - name: modifies-value-receiver
+        disabled: true
+      - name: confusing-naming
+        disabled: true
+      - name: defer
+        disabled: true
+      - name: unchecked-type-assertion
+        disabled: true
+      - name: unhandled-error
+        disabled: true
+        arguments:
+          - "fmt.Printf"
+          - "fmt.Print"
+          - "fmt.Println"
+  gosec:
+    excludes:
+      - G115

CHANGELOG.md

@@ -1,5 +1,38 @@
 # CHANGELOG
 
+## v0.38.13
+
+*October 24, 2024*
+
+This patch release addresses the issue where tx_search was not returning all results, which only arises when upgrading
+to CometBFT-DB version 0.13 or later. It includes a fix in the state indexer to resolve this problem. We recommend
+upgrading to this patch release if you are affected by this issue.
+
+### BUG FIXES
+
+- `[metrics]` Call unused `rejected_txs` metric in mempool
+  ([\#4019](https://github.com/cometbft/cometbft/pull/4019))
+- `[state/indexer]` Fix the tx_search results not returning all results by changing the logic in the indexer to copy the key and values instead of reusing an iterator. This issue only arises when upgrading to cometbft-db v0.13 or later.
+  ([\#4295](https://github.com/cometbft/cometbft/issues/4295)). Special thanks to @faddat for reporting the issue.
+
+### DEPENDENCIES
+
+- `[go/runtime]` Bump Go version to 1.22
+  ([\#4073](https://github.com/cometbft/cometbft/pull/4073))
+- Bump cometbft-db version to v0.14.1
+  ([\#4321](https://github.com/cometbft/cometbft/pull/4321))
+
+### FEATURES
+
+- `[crypto]` use decred secp256k1 directly ([#4294](https://github.com/cometbft/cometbft/pull/4294))
+
+### IMPROVEMENTS
+
+- `[metrics]` Add `evicted_txs` metric to mempool
+  ([\#4019](https://github.com/cometbft/cometbft/pull/4019))
+- `[log]` Change "mempool is full" log to debug level
+  ([\#4123](https://github.com/cometbft/cometbft/pull/4123)) Special thanks to @yihuang.
+
 ## v0.38.12
 
 *September 3, 2024*

consensus/state.go

@@ -42,7 +42,10 @@ var (
 	errPubKeyIsNotSet = errors.New("pubkey is not set. Look for \"Can't get private validator pubkey\" errors")
 )
 
-var msgQueueSize = 1000
+var (
+	msgQueueSize  = 1000
+	taskQueueSize = 128
+)
 
 // msgs from the reactor which may update the state
 type msgInfo struct {
@@ -160,6 +163,7 @@ func NewState(
 	evpool evidencePool,
 	options ...StateOption,
 ) *State {
+	blockExec.SetTaskRunner(spawnTaskRunner(taskQueueSize))
 	cs := &State{
 		config:           config,
 		blockExec:        blockExec,
@@ -2585,3 +2589,16 @@ func repairWalFile(src, dst string) error {
 
 	return nil
 }
+
+// spawnTaskRunner spawn a single goroutine to run tasks in FIFO order.
+func spawnTaskRunner(buf int) func(func()) {
+	taskCh := make(chan func(), buf)
+	go func() {
+		for f := range taskCh {
+			f()
+		}
+	}()
+	return func(f func()) {
+		taskCh <- f
+	}
+}

crypto/secp256k1/secp256k1.go

@@ -8,9 +8,9 @@ import (
 	"io"
 	"math/big"
 
-	secp256k1 "github.com/btcsuite/btcd/btcec/v2"
-	"github.com/btcsuite/btcd/btcec/v2/ecdsa"
-	"golang.org/x/crypto/ripemd160" //nolint: staticcheck // necessary for Bitcoin address format
+	"github.com/decred/dcrd/dcrec/secp256k1/v4"
+	"github.com/decred/dcrd/dcrec/secp256k1/v4/ecdsa"
+	"golang.org/x/crypto/ripemd160" //nolint: gosec,staticcheck // necessary for Bitcoin address format
 
 	"github.com/cometbft/cometbft/crypto"
 	cmtjson "github.com/cometbft/cometbft/libs/json"
@@ -43,9 +43,9 @@ func (privKey PrivKey) Bytes() []byte {
 // PubKey performs the point-scalar multiplication from the privKey on the
 // generator point to get the pubkey.
 func (privKey PrivKey) PubKey() crypto.PubKey {
-	_, pubkeyObject := secp256k1.PrivKeyFromBytes(privKey)
+	secpPrivKey := secp256k1.PrivKeyFromBytes(privKey)
 
-	pk := pubkeyObject.SerializeCompressed()
+	pk := secpPrivKey.PubKey().SerializeCompressed()
 
 	return PubKey(pk)
 }
@@ -126,7 +126,7 @@ func GenPrivKeySecp256k1(secret []byte) PrivKey {
 // Sign creates an ECDSA signature on curve Secp256k1, using SHA256 on the msg.
 // The returned signature will be of the form R || S (in lower-S form).
 func (privKey PrivKey) Sign(msg []byte) ([]byte, error) {
-	priv, _ := secp256k1.PrivKeyFromBytes(privKey)
+	priv := secp256k1.PrivKeyFromBytes(privKey)
 
 	sum := sha256.Sum256(msg)
 	sig := ecdsa.SignCompact(priv, sum[:], false)
@@ -199,7 +199,7 @@ func (pubKey PubKey) VerifySignature(msg []byte, sigStr []byte) bool {
 
 	// parse the signature:
 	signature := signatureFromBytes(sigStr)
-	// Reject malleable signatures. libsecp256k1 does this check but btcec doesn't.
+	// Reject malleable signatures. libsecp256k1 does this check but decred doesn't.
 	// see: https://github.com/ethereum/go-ethereum/blob/f9401ae011ddf7f8d2d95020b7446c17f8d98dc1/crypto/signature_nocgo.go#L90-L93
 	// Serialize() would negate S value if it is over half order.
 	// Hence, if the signature is different after Serialize() if should be rejected.

crypto/secp256k1/secp256k1_internal_test.go

@@ -7,7 +7,7 @@ import (
 
 	"github.com/stretchr/testify/require"
 
-	secp256k1 "github.com/btcsuite/btcd/btcec/v2"
+	"github.com/decred/dcrd/dcrec/secp256k1/v4"
 )
 
 func Test_genPrivKey(t *testing.T) {

crypto/secp256k1/secp256k1_test.go

@@ -6,13 +6,12 @@ import (
 	"testing"
 
 	"github.com/btcsuite/btcd/btcutil/base58"
+	underlyingsecp256k1 "github.com/decred/dcrd/dcrec/secp256k1/v4"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
 	"github.com/cometbft/cometbft/crypto"
 	"github.com/cometbft/cometbft/crypto/secp256k1"
-
-	underlyingSecp256k1 "github.com/btcsuite/btcd/btcec/v2"
 )
 
 type keyData struct {
@@ -75,7 +74,7 @@ func TestSecp256k1LoadPrivkeyAndSerializeIsIdentity(t *testing.T) {
 
 		// This function creates a private and public key in the underlying libraries format.
 		// The private key is basically calling new(big.Int).SetBytes(pk), which removes leading zero bytes
-		priv, _ := underlyingSecp256k1.PrivKeyFromBytes(privKeyBytes[:])
+		priv := underlyingsecp256k1.PrivKeyFromBytes(privKeyBytes[:])
 		// this takes the bytes returned by `(big int).Bytes()`, and if the length is less than 32 bytes,
 		// pads the bytes from the left with zero bytes. Therefore these two functions composed
 		// result in the identity function on privKeyBytes, hence the following equality check
@@ -87,7 +86,7 @@ func TestSecp256k1LoadPrivkeyAndSerializeIsIdentity(t *testing.T) {
 
 func TestGenPrivKeySecp256k1(t *testing.T) {
 	// curve oder N
-	N := underlyingSecp256k1.S256().N
+	N := underlyingsecp256k1.S256().N
 	tests := []struct {
 		name   string
 		secret []byte

go.mod

@@ -1,13 +1,13 @@
 module github.com/cometbft/cometbft
 
-go 1.22.2
-
-toolchain go1.22.7
+go 1.22.7
 
 require (
 	github.com/BurntSushi/toml v1.4.0
 	github.com/adlio/schema v1.3.6
+	github.com/btcsuite/btcd/btcutil v1.1.6
 	github.com/cenkalti/backoff v2.2.1+incompatible // indirect
+	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1
 	github.com/fortytw2/leaktest v1.3.0
 	github.com/go-kit/kit v0.13.0
 	github.com/go-kit/log v0.2.1
@@ -20,7 +20,7 @@ require (
 	github.com/minio/highwayhash v1.0.3
 	github.com/ory/dockertest v3.3.5+incompatible
 	github.com/pkg/errors v0.9.1
-	github.com/prometheus/client_golang v1.20.4
+	github.com/prometheus/client_golang v1.20.5
 	github.com/prometheus/client_model v0.6.1
 	github.com/prometheus/common v0.59.1
 	github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475
@@ -30,18 +30,16 @@ require (
 	github.com/spf13/cobra v1.8.1
 	github.com/spf13/viper v1.19.0
 	github.com/stretchr/testify v1.9.0
-	golang.org/x/crypto v0.27.0
+	golang.org/x/crypto v0.28.0
 	golang.org/x/net v0.29.0
-	google.golang.org/grpc v1.67.0
+	google.golang.org/grpc v1.67.1
 )
 
 require github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
 
 require (
 	github.com/Masterminds/semver/v3 v3.3.0
-	github.com/btcsuite/btcd/btcec/v2 v2.3.4
-	github.com/btcsuite/btcd/btcutil v1.1.6
-	github.com/cometbft/cometbft-db v0.12.0
+	github.com/cometbft/cometbft-db v0.14.1
 	github.com/cosmos/gogoproto v1.7.0
 	github.com/go-git/go-git/v5 v5.12.0
 	github.com/gofrs/uuid v4.4.0+incompatible
@@ -72,11 +70,9 @@ require (
 	github.com/containerd/continuity v0.3.0 // indirect
 	github.com/cyphar/filepath-securejoin v0.2.4 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
-	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 // indirect
 	github.com/dgraph-io/badger/v4 v4.2.0 // indirect
 	github.com/dgraph-io/ristretto v0.1.1 // indirect
-	github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
-	github.com/docker/cli v23.0.1+incompatible // indirect
+	github.com/docker/cli v24.0.7+incompatible // indirect
 	github.com/docker/go-connections v0.4.0 // indirect
 	github.com/docker/go-units v0.5.0 // indirect
 	github.com/dustin/go-humanize v1.0.1 // indirect
@@ -133,8 +129,8 @@ require (
 	go.uber.org/multierr v1.10.0 // indirect
 	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
 	golang.org/x/mod v0.17.0 // indirect
-	golang.org/x/sys v0.25.0 // indirect
-	golang.org/x/text v0.18.0 // indirect
+	golang.org/x/sys v0.26.0 // indirect
+	golang.org/x/text v0.19.0 // indirect
 	golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20240814211410-ddb44dafa142 // indirect
 	gopkg.in/ini.v1 v1.67.0 // indirect