Upstream merge v1.14.12 #432

This pull request removes the `fsync` of index files in freezer.ModifyAncients function for performance gain. Originally, fsync is added after each freezer write operation to ensure the written data is truly transferred into disk. Unfortunately, it turns out `fsync` can be relatively slow, especially on macOS (see ethereum/go-ethereum#28754 for more information). In this pull request, fsync for index file is removed as it turns out index file can be recovered even after a unclean shutdown. But fsync for data file is still kept, as we have no meaningful way to validate the data correctness after unclean shutdown. --- **But why do we need the `fsync` in the first place?** As it's necessary for freezer to survive/recover after the machine crash (e.g. power failure). In linux, whenever the file write is performed, the file metadata update and data update are not necessarily performed at the same time. Typically, the metadata will be flushed/journalled ahead of the file data. Therefore, we make the pessimistic assumption that the file is first extended with invalid "garbage" data (normally zero bytes) and that afterwards the correct data replaces the garbage. We have observed that the index file of the freezer often contain garbage entry with zero value (filenumber = 0, offset = 0) after a machine power failure. It proves that the index file is extended without the data being flushed. And this corruption can destroy the whole freezer data eventually. Performing fsync after each write operation can reduce the time window for data to be transferred to the disk and ensure the correctness of the data in the disk to the greatest extent. --- **How can we maintain this guarantee without relying on fsync?** Because the items in the index file are strictly in order, we can leverage this characteristic to detect the corruption and truncate them when freezer is opened. Specifically these validation rules are performed for each index file: For two consecutive index items: - If their file numbers are the same, then the offset of the latter one MUST not be less than that of the former. - If the file number of the latter one is equal to that of the former plus one, then the offset of the latter one MUST not be 0. - If their file numbers are not equal, and the latter's file number is not equal to the former plus 1, the latter one is valid And also, for the first non-head item, it must refer to the earliest data file, or the next file if the earliest file is not sufficient to place the first item(very special case, only theoretical possible in tests) With these validation rules, we can detect the invalid item in index file with greatest possibility. --- But unfortunately, these scenarios are not covered and could still lead to a freezer corruption if it occurs: **All items in index file are in zero value** It's impossible to distinguish if they are truly zero (e.g. all the data entries maintained in freezer are zero size) or just the garbage left by OS. In this case, these index items will be kept by truncating the entire data file, namely the freezer is corrupted. However, we can consider that the probability of this situation occurring is quite low, and even if it occurs, the freezer can be considered to be close to an empty state. Rerun the state sync should be acceptable. **Index file is integral while relative data file is corrupted** It might be possible the data file is corrupted whose file size is extended correctly with garbage filled (e.g. zero bytes). In this case, it's impossible to detect the corruption by index validation. We can either choose to `fsync` the data file, or blindly believe that if index file is integral then the data file could be integral with very high chance. In this pull request, the first option is taken.

Remove console extensions for already deleted API namespaces (les, vflux and ethash).

@lightclient

The bulk of this PR is authored by @lightclient , in the original EOF-work. More recently, the code has been picked up and reworked for the new EOF specification, by @MariusVanDerWijden , in ethereum/go-ethereum#29518, and also @shemnon has contributed with fixes. This PR is an attempt to start eating the elephant one small bite at a time, by selecting only the eof-validation as a standalone piece which can be merged without interfering too much in the core stuff. In this PR: - [x] Validation of eof containers, lifted from #29518, along with test-vectors from consensus-tests and fuzzing, to ensure that the move did not lose any functionality. - [x] Definition of eof opcodes, which is a prerequisite for validation - [x] Addition of `undefined` to a jumptable entry item. I'm not super-happy with this, but for the moment it seems the least invasive way to do it. A better way might be to go back and allowing nil-items or nil execute-functions to denote "undefined". - [x] benchmarks of eof validation speed --------- Co-authored-by: lightclient <[email protected]> Co-authored-by: Marius van der Wijden <[email protected]> Co-authored-by: Danno Ferrin <[email protected]>

minimizes the time when the lock is held

New release: https://groups.google.com/g/golang-announce/c/NKEc8VT7Fz0

This implements recent changes to EIP-7685, EIP-6110, and execution-apis. --------- Co-authored-by: lightclient <[email protected]> Co-authored-by: Shude Li <[email protected]>

…(#30520) This fixes `debug_traceBlock` methods for JS tracers in that it correctly applies the beacon block root processing to the state.

A couple of tests set the debug level to `TRACE` on stdout, and all subsequent tests in the same package are also affected by that, resulting in outputs of tens of megabytes. This PR removes such calls from two packages where it was prevalent. This makes getting a summary of failing tests simpler, and possibly reduces some strain from the CI pipeline.

Block no longer has Requests. This PR just removes some code that wasn't removed in #30425.

Allows live custom tracers to access contract transient storage through the StateDB interface.

This is a redo of #29052 based on newer specs. Here we implement EIPs scheduled for the Prague fork: - EIP-7002: Execution layer triggerable withdrawals - EIP-7251: Increase the MAX_EFFECTIVE_BALANCE Co-authored-by: lightclient <[email protected]>

This fixes a few issues missed in #29052: * `requests` must be hex encoded, so added a helper to marshal. * The statedb was committed too early and so the result of the system calls was lost. * For devnet-4 we need to pull off the type byte prefix from the request data.

This change makes the trie commit operation concurrent, if the number of changes exceed 100. Co-authored-by: stevemilk <[email protected]> Co-authored-by: Gary Rong <[email protected]>

Changelog: https://golangci-lint.run/product/changelog/#1610 Removes `exportloopref` (no longer needed), replaces it with `copyloopvar` which is basically the opposite. Also adds: - `durationcheck` - `gocheckcompilerdirectives` - `reassign` - `mirror` - `tenv` --------- Co-authored-by: Marius van der Wijden <[email protected]>

This change brings geth into compliance with the current engine API specification for the Prague fork. I have moved the assignment of ExecutionPayloadEnvelope.Requests into BlockToExecutableData to ensure there is a single place where the type is removed. While doing so, I noticed that handling of requests in the miner was not quite correct for the empty payload. It would return `nil` requests for the empty payload even for blocks after the Prague fork. To fix this, I have added the emptyRequests field in miner.Payload.

calculating a reasonable tx blob fee cap (`max_blob_fee_per_gas * total_blob_gas`) only depends on the excess blob gas of the parent header. The parent header is assumed to be correct, so the method should not be able to fail and return an error.

Use `github.com/decred/dcrd/dcrec/secp256k1/v4` directly rather than `github.com/btcsuite/btcd/btcec/v2` which is just a wrapper around the underlying decred library. Inspired by cosmos/cosmos-sdk#15018 `github.com/btcsuite/btcd/btcec/v2` has a very annoying breaking change when upgrading from `v2.3.3` to `v2.3.4`. The easiest way to workaround this is to just remove the wrapper. Would be very nice if you could backport this to the release branches. References: - btcsuite/btcd#2221 - cometbft/cometbft#4294 - cometbft/cometbft#3728 - zeta-chain/node#2934

## Description Omit null `witness` field from payload envelope. ## Motivation Currently, JSON encoded payload types always include `"witness": null`, which, I believe, is not intentional.

Implements ethereum/go-ethereum#29819

~~Opening this as a draft to have a discussion.~~ Pressed the wrong button I had [a previous PR ](ethereum/go-ethereum#24616 long time ago which reduced the peak memory used during reorgs by not accumulating all transactions and logs. This PR reduces the peak memory further by not storing the blocks in memory. However this means we need to pull the blocks back up from storage multiple times during the reorg. I collected the following numbers on peak memory usage: // Master: BenchmarkReorg-8 10000 899591 ns/op 820154 B/op 1440 allocs/op 1549443072 bytes of heap used // WithoutOldChain: BenchmarkReorg-8 10000 1147281 ns/op 943163 B/op 1564 allocs/op 1163870208 bytes of heap used // WithoutNewChain: BenchmarkReorg-8 10000 1018922 ns/op 943580 B/op 1564 allocs/op 1171890176 bytes of heap used Each block contains a transaction with ~50k bytes and we're doing a 10k block reorg, so the chain should be ~500MB in size --------- Co-authored-by: Péter Szilágyi <[email protected]>

Breaking changes: - The ChainConfig was exposed to tracers via VMContext passed in `OnTxStart`. This is unnecessary specially looking through the lens of live tracers as chain config remains the same throughout the lifetime of the program. It was there so that native API-invoked tracers could access it. So instead we moved it to the constructor of API tracers. Non-breaking: - Change the default config of the tracers to be `{}` instead of nil. This way an extra nil check can be avoided. Refactoring: - Rename `supply` struct to `supplyTracer`. - Un-export some hook definitions.

Fixes an issue missed in #30576 where we send empty requests for a full payload being resolved, causing hash mismatch later on when we get the payload back via `NewPayload`.

Co-authored-by: Felix Lange <[email protected]>

…#30537)

Swarm moved out more than 5 years ago, time to let it go.

Clean out some ancient stuff from git ignore.

This change ports some non-important changes from ethereum/go-ethereum#30159, including interface renaming and some trivial refactorings.

Co-authored-by: lightclient <[email protected]>

…638) This PR tries to break the ci.go to common dependency by moving the version number out of params.

While looking at some mem profiles from `evm` runs, I noticed that `goja` compilation of the bigint library was present. The bigint library compilation happens in a package `init`, whenever the package `eth/tracers/js` is loaded. This PR changes it to load lazily when needed. It becomes slightly faster with this change, and slightly less alloc:y. Non-scientific benchmark with 100 executions: ``` time for i in {1..100}; do ./evm --code 6040 run; done; ``` current `master`: ``` real 0m6.634s user 0m5.213s sys 0m2.277s ``` Without compiling bigint ``` real 0m5.802s user 0m4.191s sys 0m1.965s ```

Clique currently depends on the `accounts` package. This was a bit of a big cannon even in the past, just to pass a signer "account" to the Clique block producer. Either way, nowadays Geth does not support clique mining any more, so by removing that bit of functionality from our code, we can also break this dependency. Clique should ideally be further torn out, but this at least gets us one step closer to cleanups.

Way back we've added `common.math.BigMin` and `common.math.BigMax`. These were kind of cute helpers, but unfortunate ones, because package all over out codebase added dependencies to this package just to avoid having to write out 3 lines of code. Because of this, we've also started having package name clashes with the stdlib `math`, which got solves even more badly by moving some helpers over ***from*** the stdlib into our custom lib (e.g. MaxUint64). The latter ones were nuked out in a previous PR and this PR nukes out BigMin and BigMax, inlining them at all call sites. As we're transitioning to uint256, if need be, we can add a min and max to that.

This PR moves the logging/tracing-facilities out of `*state.StateDB`, in to a wrapping struct which implements `vm.StateDB` instead. In most places, it is a pretty straight-forward change: - First, hoisting the invocations from state objects up to the statedb. - Then making the mutation-methods simply return the previous value, so that the external logging layer could log everything. Some internal code uses the direct object-accessors to mutate the state, particularly in testing and in setting up state overrides, which means that these changes are unobservable for the hooked layer. Thus, configuring the overrides are not necessarily part of the API we want to publish. The trickiest part about the layering is that when the selfdestructs are finally deleted during `Finalise`, there's the possibility that someone sent some ether to it, which is burnt at that point, and thus needs to be logged. The hooked layer reaches into the inner layer to figure out these events. In package `vm`, the conversion from `state.StateDB + hooks` into a hooked `vm.StateDB` is performed where needed. --------- Co-authored-by: Gary Rong <[email protected]>

… (#30585) Makes the gnark precompile methods more amenable to fuzzing

rebased ethereum/go-ethereum#29766 . The downstream branch appears to have been deleted and I don't have perms to push to that fork. `TerminalTotalDifficultyPassed` is removed. `TerminalTotalDifficulty` must now be non-nil, and it is expected that networks are already merged: we can only import PoW/Clique chains, not produce blocks on them. --------- Co-authored-by: stevemilk <[email protected]>

…#30652) Fixes a configuration issue in a test-helper, so that we can do call tracing-tests post-merge

previous key expired 2023-07-27, the new one expires 2026-02-22: pub rsa4096 2016-11-11 [SC] [expires: 2026-02-22] AE96ED969E479B0084F3E17FE88D3334FA5F6A0A uid Ethereum Foundation Security Team <[email protected]> uid Ethereum Foundation Bug Bounty <[email protected]> sub rsa4096 2016-11-11 [E] [expires: 2026-02-22]

This change makes it so that the wrapped statedb with tracing-hooks is passed to the system call processing Fixes #30658

This PR adds the beacon chain config for the holesky testnet. It also updates beacon checkpoints for Mainnet and Sepolia.

This PR adds `DeleteRange` to `ethdb.KeyValueWriter`. While range deletion using an iterator can be really slow, `DeleteRange` is natively supported by pebble and apparently runs in O(1) time (typically 20-30ms in my tests for removing hundreds of millions of keys and gigabytes of data). For leveldb and memorydb an iterator based fallback is implemented. Note that since the iterator method can be slow and a database function should not unexpectedly block for a very long time, the number of deleted keys is limited at 10000 which should ensure that it does not block for more than a second. ErrTooManyKeys is returned if the range has only been partially deleted. In this case the caller can repeat the call until it finally succeeds.

When using the prestateTracer, in some cases users are only concerned with balances or nonce information, and are not interested in the lengthy contract code or storage data. Therefore, this PR introduces two new configuration options in the `prestateTracerConfig` structure: - `disableCode` - `disableStorage` These options allow users to control whether the tracer returns contract code and storage data during execution tracing. By setting these options, users can more flexibly customize their needs and focus on obtaining information that is more critical and relevant to their specific use cases. These options work with the default mode as well as `diffMode: true`. --------- Signed-off-by: jsvisa <[email protected]> Co-authored-by: Sina M <[email protected]>

…30696)

This field is a duplicate of UpdateScore.SignerCount and never referenced.

Looking at the cpu profile of a burntpix benchmark, I noticed that a lot of time was spent in gas-used, in the interpreter loop. It's an actual call (not inlined), which explicitly wants to be ignored by tracing ("tracing.GasChangeIgnored"), so it can be safely and simply inlined. The other change is in `pushX`. These also do a call to `common.RightPadBytes`. I replaced that by a doing a corresponding `Lsh` on the `u256` if needed. Note: it's needed only to make the stack output look right, for fuzzers. It technically doesn't matter what we put there: if code ends on a pushdata immediate, nothing will consume the stack element. We could just as well just ignore it, if we didn't care about fuzzers (which I do). Seems quite a lot faster on burntpix, according to my runs. This PR: ``` EVM gas used: 5642735088 execution time: 34.84609475s allocations: 915683 allocated bytes: 175334088 ``` ``` EVM gas used: 5642735088 execution time: 36.671958278s allocations: 915701 allocated bytes: 175340528 ``` Master ``` EVM gas used: 5642735088 execution time: 49.349209526s allocations: 915684 allocated bytes: 175333368 ``` ``` EVM gas used: 5642735088 execution time: 46.581006598s allocations: 915681 allocated bytes: 175330728 ``` --------- Co-authored-by: Sina M <[email protected]> Co-authored-by: Felix Lange <[email protected]>

I think the core code should generally be agnostic about the witness and the statedb layer should determine what elements need to be included in the witness. Because code is accessed via `GetCode`, and `GetCodeLength`, the statedb will always know when it needs to add that code into the witness. The edge case is block hashes, so we continue to add them manually in the implementation of `BLOCKHASH`. It probably makes sense to refactor statedb so we have a wrapped implementation that accumulates the witness, but this is a simpler change that makes #30078 less aggressive.

This is a flag leftover from the swarm era. No need to deprecate it, it's been useless/dead forever now.

This is a noop change to not have custom code for stdlib functionality.

@V3

Currently we have a custom TextMarshalerFlag. It's a nice idea, allowing anything implementing text marshaller to be used as a flag. That said, we only ever used it in one place because it's not that obvious how to use and it needs some boilerplate on the type itself too, apart of the heavy boilerplate got the custom flag. All in all there's no *need* to drop this feature just now, but while porting the cmds over to cli @V3, all other custom flags worker perfectly, whereas this one started crashing deep inside the cli package. The flag handling in v3 got rebuild on generics and there are a number of new methods needed; and my guess is that maybe one of them doesn't work like this flag currently is designed too. We could definitely try and redesign this flag for cli v3... but all that effort and boilerplate just to use it for 1 flag in 1 location, seems not worth it. So for now I'm suggesting removing it and maybe reconsider a similar feature in cli v3 with however it will work.

This PR is a first step towards removing account management from geth, and contains a lot of the user-facing changes. With this PR, the `personal` namespace disappears. **Note**: `personal` namespace has been deprecated for quite some time (since ethereum/go-ethereum#26390 1 year and 8 months ago), and users who have wanted to use it has been forced to used the flag `--rpc.enabledeprecatedpersonal`. So I think it's fairly non-controversial to drop it at this point. Specifically, this means: - Account/wallet listing -`personal.getListAccounts` -`personal.listAccounts` -`personal.getListWallets` -`personal.listWallets` - Lock/unlock -`personal.lockAccount` -`personal.openWallet` -`personal.unlockAccount` - Sign ops -`personal.sign` -`personal.sendTransaction` -`personal.signTransaction` - Imports / inits -`personal.deriveAccount` -`personal.importRawKey` -`personal.initializeWallet` -`personal.newAccount` -`personal.unpair` - Other: -`personal.ecRecover` The underlying keystores and account managent code is still in place, which means that `geth --dev` still works as expected, so that e.g. the example below still works: ``` > eth.sendTransaction({data:"0x6060", value: 1, from:eth.accounts[0]}) ``` Also, `ethkey` and `clef` are untouched. With the removal of `personal`, as far as I know we have no more API methods which contain credentials, and if we want to implement logging-capabilities of RPC ingress payload, it would be possible after this. --------- Co-authored-by: Felix Lange <[email protected]>

Compatible error message in the flat call tracer with parity-style endpoints. Signed-off-by: jsvisa <[email protected]>

This PR updates the fuzzing verbosity a bit, in case of mismatches

This test depends on a 100ms timer, which fails quite often, messing up our pipelines. Hook directly into the internal version of getPayload which has the capacity to wait for the full payload before returning. This might not be absolutely correct from a test perspective, but it beats failing ci. The alternative would be to expose the full build hook into the outside, but it might be a bit overkill for this scenario.

When using the `core/vm/runtime` helpers to execute code, callbacks for the tx end were not invoked. This change fixes it by invoking them.

Tests that are crucial to for verifying the verkle testnet functions properly. --------- Signed-off-by: Guillaume Ballet <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> Co-authored-by: Gary Rong <[email protected]> Co-authored-by: Martin HS <[email protected]>

This PR fixes some issues with benchmarks - [x] Removes log output from a log-test - [x] Avoids a `nil`-defer in `triedb/pathdb` - [x] Fixes some crashes re tracers - [x] Refactors a very resource-expensive benchmark for blobpol. **NOTE**: this rewrite touches live production code (a little bit), as it makes the validator-function used by the blobpool configurable. - [x] Switch some benches over to use pebble over leveldb - [x] reduce mem overhead in the setup-phase of some tests - [x] Marks some tests with a long setup-phase to be skipped if `-short` is specified (where long is on the order of tens of seconds). Ideally, in my opinion, one should be able to run with `-benchtime 10ms -short` and sanity-check all tests very quickly. - [x] Drops some metrics-bechmark which times the speed of `copy`. --------- Co-authored-by: Sina Mahmoodi <[email protected]>

`flatCallTracer` will now specify the type of a create in the action via the `creationMethod` field. --------- Signed-off-by: jsvisa <[email protected]> Co-authored-by: Sina Mahmoodi <[email protected]>

fixes a very tiny bug

closes ethereum/go-ethereum#30304 We already use `DirectoryFlag` for `authrpc.jwtsecret` which expands the tilde, so this should work out of the box

…30728) Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.5.0 to 4.5.1. Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Here I'm adding a new helper function that extracts the revert reason of a contract call. Unfortunately, this aspect of the API is underspecified. See these spec issues for more detail: - ethereum/execution-apis#232 - ethereum/execution-apis#463 - ethereum/execution-apis#523 The function added here only works with Geth-like servers that return error code `3`. We will not be able to support all possible servers. However, if there is a specific server implementation that makes it possible to extract the same info, we could add it in the same function as well. --------- Co-authored-by: Marius van der Wijden <[email protected]>

… (#30194) Adds testcase for createAccessList when user requested gasPrice is less than baseFee, also makes the tests tabledriven --------- Co-authored-by: Martin Holst Swende <[email protected]>

…ot chain tip (#30538)

Adds support non-legacy transaction-signing using ledger --------- Co-authored-by: Martin Holst Swende <[email protected]>

This change updates the EIP-712 implementation to resolve [#30619](ethereum/go-ethereum#30619). The test cases have been repurposed from the ethers.js [repository](https://github.com/ethers-io/ethers.js/blob/main/testcases/typed-data.json.gz), but have been updated to remove tests that don't have a valid domain separator; EIP-712 messages without a domain separator are not supported by geth. --------- Co-authored-by: Martin Holst Swende <[email protected]>

… and fork (#30442) When `evm statetest --bench` is specified, benchmark the execution similarly to `evm run`. Also adds the ability to filter tests by name, index and fork. --------- Co-authored-by: Martin Holst Swende <[email protected]>

This PR moves chain config related code (config file processing, fork logic, network defaults) from `beacon/types` and `beacon/blsync` into `beacon/params` while the command line flag logic of the chain config is moved into `cmd/utils`, thereby removing the cli dependencies from package `beacon` and its sub-packages.

This change invokes the OnCodeChange hook when selfdestruct operation is performed, and a contract is removed. This is an event which can be consumed by tracers.

@barnabasbusa

Requested by @barnabasbusa

Co-authored-by: Martin HS <[email protected]>

Adds a protocol handler fuzzer to fuzz the ETH68 protocol handlers

Fix panic in tests

add unit tests for `p2p/addrutil` --------- Co-authored-by: Martin HS <[email protected]>

fixes a typo on one of the postmortems

Fixes an error in the binary iterator, adds additional testcases --------- Co-authored-by: Gary Rong <[email protected]>

This is one further step towards removing account management from `geth`. This PR deprecates the flag `unlock`, and makes the flag moot: unlock via geth is no longer possible.

New release: https://groups.google.com/g/golang-announce/c/X5KodEJYuqI

The [kilic](https://github.com/kilic/bls12-381) bls12381 implementation has been archived. It shouldn't be necessary to include it as a fuzzing target any longer. This also adds fuzzers for G1/G2 mul that use inputs that are guaranteed to be valid. Previously, we just did random input fuzzing for these precompiles.

This adds an API method `DropTransactions` to legacy pool, blob pool and txpool interface. This method removes all txs currently tracked in the pools. It modifies the simulated beacon to use the new method in `Rollback` which removes previous hacky implementation that also erroneously reset the gas tip to 1 gwei. --------- Co-authored-by: Felix Lange <[email protected]>

Continuation of ethereum/go-ethereum#30381

Commits on Nov 20, 2024

Merge tag 'v1.14.12' into optimism

roberto-bayardo committed Nov 20, 2024

Configuration menu

View commit details

Copy full SHA for fdfd720

Browse repository at this point

Copy the full SHA

fdfd720 View commit details

Browse the repository at this point in the history

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upstream merge v1.14.12 #432

Upstream merge v1.14.12 #432

Commits on Nov 19, 2024

Commits on Nov 20, 2024

Commits on Nov 21, 2024

Upstream merge v1.14.12 #432

Are you sure you want to change the base?

Upstream merge v1.14.12 #432

Commits on Nov 19, 2024

Commits on Nov 20, 2024

Commits on Nov 21, 2024