Releases · ethereum/tests

31 Aug 10:59

winsvega

9201075

Latest

The recent tests changes:

From now on there would be no new test vectors in this repository and existing ones will slowly be converted into https://github.com/ethereum/execution-spec-tests. NOTE: Pyspecs doesn't cover all the vectors from this repo, but has many new vectors that are not covered in ethereum/tests
Eventually there will be only ethereum/legacytests repo with all .json tests from Frontier to Cancun.
And ethereum/execution-spec-tests will handle all the official ethereum testing vectors for all the froks.

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v14.0

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

Documentation

TESTS

Remove TXCREATE from EOF tests
PR #1378
Update EOF Tests top level container rules
PR #1381

Converted to python Tests

Test all opcodes
PR #1399
A few eip1153 tests
PR #1383
EOF return contract tests removed
PR #1391
EIP eof tests
PR #1395
PR #1397
PR #1400

Cosmetic

Remove retesteth expect section tests from legacy folder
PR #1392
Correct filenames
PR #1393

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Assets 2

01 Jul 09:12

winsvega

v14.0

e46e1db

Tests update 14 Legacy Tests

The recent tests changes:

Make all existing tests snapshot into LegacyTests folder Cancun
Make all recent test vectors >=Cancun only
Add PYSPEC style exceptions to develop tests
Begin convertion of json fillers to .py fillers (migrate to pyspecs repo)

Announcement

Test format

Tests Fillers

New folder LegacyTests/Cancun

GeneralStateTests/BlockchainTests Filled

New folder LegacyTests/src/LegacyTests/Cancun

Test Regeneration

Update all:
PR #1377 (make all tests snapshot)
PR #1380 (refill only cancun vectors)
PR #1382 (update pyspec tests)

Documentation
PR #1376 (list KEVM in readme)

TESTS

Rename exceptions to PYSPEC style
PR #1380
Replace addresses in tests 0x00..0100 to 0x00..01000
PR #1372
Enable ecparing tests for cancun and beyond
PR #1375

EIPTests/EOFTests

Reintroduce BLS precompile tests
PR #1366
Remove TXCREATE from EOF tests
PR #1378

Converted to python Tests
The following tests will now be managed by pyspecs test generator
PR #1383 (Cancun/stEIP1153)

Cosmetic

Fix invalid json fields from pyspecs
PR #1385
Mark bigInt in test ValueOverflow when generated from state test
PR #1387
Fix pyspec test filled with bug
PR #1388

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Assets 2

06 May 08:42

winsvega

v13.3

faf33b4

Tests update 13.3 Cancun tests

The recent tests changes:

Fix the issue in blockchain tests with beacon root precompile, cosmetic updates
EIP7610 legacy tests fix
EIP tests for BLS

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

Documentation
PR #1356 (fix typos)

TESTS

Fix legacy tests to revert on create collision with non empty storage dest (EIP7610)
PR #1367
**Fix the blockchain tests logic with beacon root in genesis **
PR #1357
Add more ecpairing test vectors
PR #1358
Update bad opcode tests for Cancun
PR #1362
Update pyspec tests, rename Merge to Paris
PR #1363

EIPTests/EOFTests

Reintroduce BLS precompile tests
PR #1366
Fix and update EOF tests
PR #1349
More EOF validation tests from evmone
PR #1364

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Assets 2

13 Mar 14:04

winsvega

v13.2

066a587

Tests update 13.2 Cancun tests

The recent tests changes:

Cosmetic update
Insert beacon precompile into bc tests's pre >= Cancun
Fix genesis excessBlobGasUsed for first block excessBlobGasUsed to be = 0 or x calculation
Remove empty storage records from pre state

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Insert beacon precompile into tests pre if not exist for BlockchainTests >= Cancun

"0x000f3df6d732807ef1319fb7b8bb8522d0beac02" : {
                "balance" : "0x00",
                "code" : "0x3373fffffffffffffffffffffffffffffffffffffffe14604d57602036146024575f5ffd5b5f35801560495762001fff810690815414603c575f5ffd5b62001fff01545f5260205ff35b5f5ffd5b62001fff42064281555f359062001fff015500",
                "nonce" : "0x01",
                "storage" : {
                }
            },

Test Regeneration

Update all:

Pyspec update: remove beacon contract from tests, remove empty storage records
PR #1345
Refill all blockchain tests, add beacon root on cancun
PR #1351
Documentation

TESTS

--
PR --

EIPTests/EOFTests

update for relaxed stack validation
PR #1334
**Move EOF creation tests into EOF test format **
PR #1337

Cosmetic

Move random blockhash calling state tests to bc format
PR #1354
Change exception name in PYSPEC
PR #1342
Remove empty storage keys (0x00) in pre state
PR #1347

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Assets 2

14 Feb 14:07

winsvega

v13.1

853b1e0

Tests update 13.1 Cancun tests

The recent tests changes:

Rename Merge to Paris (note: Pyspecs tests still have Merge keyword)
Remove empty accounts from post Paris
Update cancun rules, state test format, update pyspecs vectors
Update filled state test format

Announcement

Test format

Tests Fillers

Pyspec tests folder moved from /src/BlockchainTestsFiller/Pyspecs to /src/GeneralStateTestsFiller/Pyspecs
and the generated tests moved accordingly. as .py tests now could have state and blockchain tests.
--filltests flag will generate all state tests out of .py files
--filltests --fillchain flag will generate all blockchain tests out of .py files.

GeneralStateTests/BlockchainTests Filled

rename forkname in all of the tests: "Merge" => "Paris"
empty accounts (zero balance, no code, zero nonce, with or without storage) in pre state are not allowed post Paris.

In state tests remove blockchain related fields:

 "currentBeaconRoot" : "0x0000000000000000000000000000000000000000000000000000000000000000",
 "currentWithdrawalsRoot" : "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",`
 "previousHash" : "0x5e20a0453cecd065ea59c37ac63e079ee08998b6045136a8ce6635c7912ec0b6"

currentBeaconRoot although not set, the value could still be found in pre state in beacon root contract.
(ethereum/execution-spec-tests#413)

In blockchain tests if invalid block rlp is unreadable, or supposed to fail during rlp parsing, rlp_decoded field no longer provided

Test Regeneration

Update all:

All tests regeneration with Cancun rules
PR #1332
PR #1333 (PYSPEC UPDATE)
Adapt tests and remove empty accounts from state/bc tests post Paris/Merge
PR #1335
PR #1336
Documentation
PR #1323

TESTS

Update the TestEachOpcode template for Cancun
PR #1320
Fix genesis blobgasused params
PR #1328

EIPTests/EOFTests

Update EOF tests to not contain unreachable code sections
PR #1319

Cosmetic

---
PR ---

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Assets 2

02 Nov 09:44

winsvega

v13

428f218

Tests update 13 Cancun tests

The recent tests changes:

Regenerate all tests using cancun rules
Move Cancun eip tests from EIPTests folder into normal test folders
Test diagram updated: https://tinyurl.com/y525ezs7

Announcement

Test format

New test folder GeneralStateTests/Cancun
New test folder /src/GeneralStateTestsFiller/Cancun

Tests Fillers

defining solidity yul now allows to switch optimisations and select evm version:
--evm-version shanghai, use default solc optimisations

      code: |
        :yul shanghai optimise {
          sstore(0, 1)
      }

--evm-version london, no optimisations (default)

      code: |
        :yul london {
          sstore(0, 1)
      }

This is because each new version of solc introduce default optimisations that change the compiled code bytes. to avoid this affecting the tests by default solc optimisations in tests are disabled

GeneralStateTests/BlockchainTests Filled

Invalid blocks are now transcripted. new format of invalid bc block:

{
                "blocknumber" : "3",
                "chainname" : "default",
                "expectException" : "InvalidGasLimit2",
                "rlp" : "0x...",
                "rlp_decoded" : {
                    "blockHeader" : {
                             ...
                    },
                    "transactions" : [
                        {
                               ...
                        }
                    ],
                    "uncleHeaders" : [
                        {
                               ...
                        }
                    ],
                    "withdrawals" : [
                    ]
                }
}

Keep in mind that this are debug info fields: blocknumber, chainname, expectException

Test Regeneration

Update all:

All tests regeneration with Cancun rules
PR #1297
PR #1312
Documentation
PR ---

TESTS

Tests for contract creation
PR #1306
Regenerate transaction sequence into invalid blocks
PR #1316
Move Cancun tests from EIP Tests
PR #1313

EIPTests/EOFTests

Pyspecs Cancun tests
PR #1299 Update Pyspecs to 1.0.5

Cosmetic

Fix fillers json with missing commas
PR #1310

Upcoming changes:

Support pyspec .py test filling with different clients using retesteth
Nimbus --vmtrace support

Assets 2

19 Sep 08:12

winsvega

v12.4

6613563

Tests update 12.4 Preparations to fill Cancun

The recent tests changes:

Add Cancun blob fields and transactions to json tests
Update pyspec tests

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: http://tinyurl.com/2ctak8yn
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v12.3

Test format

Tests Fillers

new cancun fields in env section

    parentExcessBlobGas: 0x1000
    parentBlobGasUsed: 0x2000
    currentBeaconRoot: 0x45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8

withdrawals root explicitly exported in state tests

    "currentWithdrawalsRoot" : "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",

new lllc instruction

             [[0]] (BLOBHASH 0)

new transaction type (blob transaction)

  transaction:
    data:
        - data: :label declaredKeyWrite :raw 0x00
          accessList:
          - address: 0x095e7baea6a6c7c4c2dfeb977efac326af552d87
            storageKeys:
            - 0x00
            - 0x01

    maxFeePerGas: '0x12A05F200'
    maxPriorityFeePerGas: '2'
    gasLimit:
    - '4000000'
    nonce: '0'
    to: 095e7baea6a6c7c4c2dfeb977efac326af552d87
    value:
    - '100000'
    secretKey: "45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8"
    maxFeePerBlobGas: '10'
    blobVersionedHashes:
        - "0x01a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8"

GeneralStateTests/BlockchainTests Filled

new block header fields

                    "blobGasUsed" : "0x00",
                    "excessBlobGas" : "0x00",
                    "parentBeaconBlockRoot" : "0x01a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8"

Test Regeneration

Update all:

Refill solc/yul tests using solc evm version flag
PR #1280 PR #1279 PR #1278 PR #1277
PR #1276 PR #1275 PR #1274 PR #1273
PR #1272 PR #1271 PR #1270 PR #1269
PR #1268 PR #1267 PR #1266 PR #1265
PR #1264 PR #1263 PR #1262 PR #1261
PR #1260 PR #1259 PR #1258 PR #1257
PR #1256 PR #1255 PR #1254 PR #1253
PR #1252 PR #1251 PR #1250
Documentation
PR #1283
Update Pyspecs tests
PR #1288
PR #1292
State tests transactions from non existing account
PR #1290
PR #1291

EIPTests/EOFTests

Pyspecs Cancun tests
PR #1282
PR #1288
Refill EIPTests with new blob names
PR #1224
Update tests according to Mega-EOF spec
PR #1238
PR #1285

Cosmetic
PR #1247
PR #1248
PR #1281
PR #1289

Upcoming changes:

Fill all tests with cancun rules
Support pyspec .py test filling with different clients using retesteth
Nimbus --vmtrace support

Assets 2

30 Jul 12:56

winsvega

v12.3

06e2767

Tests update 12.3 EIP tests and Pyspecs support

The recent tests changes:

Update .py tests to the latest version (but not Cancun)
Update EIPTests to new specs and add more test cases

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

Documentation
PR #1216
PR #1223
PR #1241
Refill invalid tx state tests into invalid block bc tests
PR #1225
Update Pyspecs tests
PR #1240
PR #1245

EIPTests/EOFTests

More EIP1153 (Transient storage) tests
PR #1220
PR #1235
PR #1237
CALLF Stack Overflow test
PR #1214
Update EOF tests
PR #1219 (Forbid DELEGATECALL EOF → legacy)
MCOPY tests from evmone
PR #1229
PR #1234
BLOBHASH tests from nimbus
PR #1226
PR #1243

Cosmetic

Fix nimbus test generation exception
PR #1242
Mark bigint in bc withdrawals tests
PR #1213
PR #1232
More test exception names fix
PR #1230
PR #1225
Nimbus version string to valid json
PR #1231

Upcoming changes:

Support pyspec .py test filling with different clients using retesteth
Nimbus --vmtrace support

Assets 2

27 Apr 19:52

winsvega

v12.2

b25623d

Tests update 12.2: EOF Tests

The recent tests changes:

Update EOFTests (EIPTests)

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

Documentation

EIPTests/EOFTests

EIP-5450: State Tests And EOF Tests
PR #1163 (EOF Stack validation)
EIP-4750: State Tests And EOF Tests
PR #1115 (EOF Functions)
EIP4200: EOF - Static relative jumps
(3540-3670-4200
PR #1103 (EOF Jumps)
EIP3540: EOF - Ori's tests
PR #1206 (change to Cancun frokword)

Cosmetic

Change ttWrongRLP tx exception
PR #1211

Upcoming changes:

SSZ tests
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

13 Apr 18:16

winsvega

v12.1

04f338a

Tests update 12.1: EIPTests separate, Pyspecs

The recent tests changes:

Move EIPTests into a separate folder so not to confuse
Introduce Pyspecs test vectors and support .py fillers

Announcement

Test format

Tests Fillers

new folder /src/BlockchainTestsFiller/Pyspecs
new folder /src/EIPTestsFiller/StateTests
new folder /src/EIPTestsFiller/BlockchainTests

GeneralStateTests/BlockchainTests Filled

new folder /BlockchainTests/Pyspecs
new folder /EIPTests/StateTests
new folder /EIPTests/BlockchainTests/StateTests (EIP state tests generated in blockchain form)

Test Regeneration

Update all:
PR #1197 (Update Transaction and Difficulty tests)
PR #1203 (Update Legacy Tests)

Documentation
Pyspec test vectors
PR #1209
Move EIPTests into separate folder from all of the common tests
PR #1192
EIP1153 - Transient Storage Tests
PR #1196
More withdrawals tests
PR #1198
PR #1199 (Merge era comes block with withdrawals body)
Fix typos
PR #1201
PR #1202

EIPTests

Update EIP-3540 and EIP-3670 Tests to Cancun
PR #1193 (EIPTests)
EOFTests add exception checks
PR #1208

Upcoming changes:

Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

Releases: ethereum/tests

Tests update 14.1 Legacy Tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 14 Legacy Tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 13.3 Cancun tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 13.2 Cancun tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 13.1 Cancun tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 13 Cancun tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 12.4 Preparations to fill Cancun

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 12.3 EIP tests and Pyspecs support

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 12.2: EOF Tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 12.1: EIPTests separate, Pyspecs

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration