Releases · ethereum/tests

15 Mar 12:14

winsvega

291118c

The recent tests changes:

Regenerate all test vectors with Shanghai rules
Introducing a shortened vectors for EOF code validation
Added nimbus to web interface http://retesteth.ethdevops.io/web

Announcement

The test process diagram: http://tinyurl.com/2ctak8yn
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v11.3

Test format

Tests Fillers

new folder /src/EOFTestsFiller
new folder /src/GeneralStateTestsFiller/Shanghai

GeneralStateTests/BlockchainTests Filled

new folder EOFTests
new folder GeneralStateTests/Shanghai

Test Regeneration

Update all:
PR #1164 (generate all test vectors with Shanghai rules)
PR #1180

Documentation
PR #1156 (fix typos)
PR #1149 (warning about test validity)
PR #1150 (EOFTests suite docs)
PR #1167
Fix test bugs/typos
PR #1155 (bcShanghaiExample)
PR #1158 (0 withdrawals hash)
PR #1161 (transaction tests src path)
PR #1168
PR #1191 (lower gas limit in underflow test)
Move Shanghai tests from experimental folder to Shanghai tests
PR #1180
EOF Test suite
PR #1159
EIP-3651 tests (warm coinbase)
PR #1082
update ORI'S badOpcode and diffPlaces tests
PR #1166
Fill all remaining tests to Shanghai
PR #1188
PR #1187
PR #1186
PR #1190
Double call contract that has suicide after initial suicide is called
PR #1169
PR #1181
PR #1182 (upgrade the test)

Upcoming changes:

Support pyspec .py fillers
Organize EIPTests
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

24 Jan 10:16

winsvega

v11.3

69c4c2a

Tests update 11.3: EIP tests, prepare to Shanghai

The recent tests changes:

Introduces new folder EIPTests in GeneralStateTests that have tests for potential EIPS to be included in Shanghai
Tests in EIPTests folders are not mandatory to pass and can contain errors or outdated specs. Those are early development stage tests that will be transformed into proper tests once the specs are finalized.
This allows us to have coverage for future EIPs that not necesseraly implemented by all clients just yet.

Announcement

Test format

New test ID tracker system: /src/README.MD contains id tags that can be found in tests that cover the provided test vectors

Tests Fillers

new folder /src/GeneralStateTestsFiller/EIPTests
new folder /src/GeneralStateTestsFiller/EIPTests/stEOF
new folder /src/BlockchainTestsFiller/EIPTests

GeneralStateTests/BlockchainTests Filled

new folder GeneralStateTests/EIPTests
new folder GeneralStateTests/EIPTests/stEOF
new folder BlockchainTests/EIPTests
new folder BlockchainTests/GeneralStateTests/EIPTests
new folder BlockchainTests/GeneralStateTests/EIPTests/stEOF

Test Regeneration

Update all:
Regenerate basefee bug in EIPTests
PR #1141
Update EIPTests to latests specs
PR #1147
PR #1152

Documentation
PR #1121 (Example of new debug options with retesteth)
PR #1128
Shanghai transition tests
PR #1123
PR #1134 (switch to timestamp based)
**Shanghai Withdrawals tests **
PR #1118 (example)
PR #1124
PR #1129
PR #1137 (basefee fix)
Rare bug with refund on Create OOG during deployment
PR #1111
Move fork+xxx EIP tests to a separate folder
PR #1110
EIP-3855 PUSH0 instruction tests
PR #1033
EIP-3860 Update tests to new specs
PR #1105 (create/2 init code size)
PR #1012
PR #1127
PR #1125
EIP-3860 Transaction tests
PR #1131
Remove JUMP tests from EIP3540
PR #1107
PR #1132 (fix 3540 tests to updated specs)
PR #1126 (EOF code deployment)
EOF contract deployment tests from ORI
PR #1136
EIP3670 tests
PR #1135

Upcoming changes:

EOFTest suite
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

21 Nov 16:15

winsvega

v11.2

a16217c

Tests update 11.2: Potential EIPs

The recent tests changes:

Introduce first EIP specific tests (fork + 3540 + 3670) (fork + 3860)
More test coverage around recently discovered consensus bugs
A few cosmetic changes to the test fillers (expect sections and tx type)

Announcement

Test format

Tests Fillers

new folder stEIP3540
new folder stEIP3860

GeneralStateTests/BlockchainTests Filled

new folder BlockchainTests/GeneralStateTests/stEIP3540
new folder BlockchainTests/GeneralStateTests/stEIP3860
new folder GeneralStateTests/stEIP3540
new folder GeneralStateTests/stEIP3860

Test Regeneration

Update all:
no all tests regeneration

Documentation
PR #1081
PR #1096
Tests for EIP-3860 Limit and meter initcode
PR #990
Tests for EIP-3540: EVM Object Format V1
PR #847
PR #1095
PR more tests around 0 touches
PR #1097
Fix 0 reward account touch in state tests in pre eip158
(Clean more expect sections that check coinbase or sender balance that could be affected by many EIPs)
PR #1092
Change transaction type in stBadOpcode generated tests
PR #1099
PR #1102
Account that makes itself eipty during exectime
PR #1088
Check address warm up on different cases with CREATE
PR #1077
More ecRecover precompile Tests
PR #1080
Unit tests for gtests filler expect section
PR #1089
Gas leak test, int boundaries verification
PR #1085
Fix test fillers expect checks that rely on gas (where possible)
PR #1098

Upcoming changes:

EIP tests support
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

27 Aug 09:05

winsvega

v11.1

c76c5d2

Tests update 11.1: Pre Merge

The recent tests changes:

More Merge tests, more edge case coverage contribution

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

Documentation
PR #1054
PR #1064 (bigint)
PR #1065 (custom eip test implementation)
PR #1070
PR #1071
PR #1072 (tutorial for t8ntool)
PR #1074 (custom compilers in retesteth)
#1076 (vmtrace and poststate export)
Cosmetic changes
PR #1075
Update legacy tests link
PR #1059
Timestamp/tips on Merge checks
PR #1060
PR #1061
Merge transition difficulty check
PR #1066
EIP 2929 check the account is hot
PR #1068
ECRecover on empty message test
PR #1069
Modexp test vector
Input to the modExp precompile that overflowed the gas calculation on Nethermind
PR #1057
More edge cases just to be sure:
PR #1056 (Transaction with max nonce being rejected)
PR #1051 (Touch contract deleted from the state)
PR #1055 (Precompile 09 existance check)
PR #1062 (Sender balance check with EIP1559 rules)
PR #1063 (Double selfdestruct touches different beneficiaries)

Upcoming changes:

Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

13 Jul 13:16

winsvega

v11

95a8490

Tests update 11: Merge PoS

The recent tests changes:

Regenerate all tests using PoS evm rules
Gray Glacier (EIP-5133) Difficulty Bomb tests
PoW to PoS transition tests (BlockchainTests/TransitionTests/bcArrowGlacierToMerge)

Announcement

Test format

Support "Merge" forkname for PoS block headers.
Support "ArrowGlacierToMergeAtDiffC0000" forkname for bc transition tests.
Old blocks headers/env are auto-corrected to PoS format when filling old tests in Merge mode.
(difficulty becomes 0x00, mixHash becomes the existed difficulty value)

Tests Fillers

        "currentRandom: 0x1500000000000000000000000000000000000000000000000000000000000000"

New field support in state tests env section, indicating random for PoS block. (both fillers and filled state tests)

GeneralStateTests/BlockchainTests Filled

            "difficulty" : "0x00"

must be defined to 0x00 in blockHeader, so that block is recognized as PoS block.

Test Regeneration

Update all:
PR #1045
PR #1030

Documentation
PR #1036
PR #1038
PR #1046
PR #1048
PR #1053
Gray Glacier (EIP-5133) Difficulty Bomb tests
PR #1043
Merge tests example
PR #1019
Quick fix for test names
PR #1035
Unit tests for env section correction (retesteth only)
PR #1034
Update blockchain tests to Merge PoS
PR #1049
PR #1047
PR #1045 (regenerate all)
PR #1052 (allow incorrect timestamp)
PR #1058 (revoke incorrect timestamp)

Upcoming changes:

Implement more PoS/Merge tests
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

18 May 08:50

winsvega

v10.4

a380655

Tests update 10.4: winter

The recent tests changes:

Tests development throug out the winter. A few more cases here and there.
Unit tests for typed transactions.

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

Documentation
PR #1025
PR #1026
RETURNDATACOPY boundaries
PR #1010
Signextend(31, +/- 1) opcode case
PR #1013
CREATE/2 return buffer check
PR #1015
EC Recover bondary tests from TransactionTests
PR #1018
Clarify transaction types validity on different forks
PR #1023
Unit tests for typed transactions
PR #1031

Upcoming changes:

Implement PoS/Merge tests
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

09 Jan 12:43

winsvega

v10.3

005931a

Tests update 10.3: Test Headers

The recent tests changes:

A few more edge test cases around jump, return, revert, maxnonce, maxcodesize, sar/shr/shl.
Updated the test headers to include solidity version.
Transactions include sender field.

Announcement

Test format

Tests Fillers

"indexes" section can be entirely/partially omitted. the defult value is -1 to all fields.
individual fields in "indexes" can be omitted. the default value is -1.

        "expect" : [
            {
                "indexes" : {  #the entire section can be omitted
                    "data" : -1,
                    "//gas" : -1,  #this field can be omitted.
                    "value" : -1
                },

GeneralStateTests/BlockchainTests Filled

Changes in the "_info" section:
Optional field "labels" with debug info on transaction index labels does not include : label prefix to save space
Added solidity solc version string info

        "_info" : {
            ...
            "labels" : {
                "0" : "transaction1"
            },
            "solidity" : "Version: 0.8.5+commit.a4f2e591.Linux.g++"
            ...
        }

Changes in the "transaction" section:
Added "sender" field to recover the public address from secretKey

        "transaction" : {
            "secretKey" : "0x45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8",
            "sender" : "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b"
            ...
        },
        "transaction" : {
             "r" : "0x8bb6e17e660cdefc2f6f8124e8f11cd74ac6d91ee05d3a1aa6f513335da0e4a7",
             "s" : "0x1d356c9ff34a96d10ec2e0f9d708eea9da2801a37de491c5af43df7c77d56c0c",
             "sender" : "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b",
             "v" : "0x1b",
            ...
        }

Test Regeneration

Update all:
PR #1009 (regenerate test headers)

Documentation
PR #999
PR #1005
More codesize limit tests around contract creation
PR #986 (tests for account with max code size)
Max account nonce tests
PR #934 (check that create/create2 with account max nonce does not work)
PR #989 (delegate call + create with max nonce)
Additional tests around return buffer
PR #994 (check return buffer, revert buffer)
Additional test vectors around SAR/SHR/SHL
PR #1004 (shift operations test vectors)
Stress test for max contract code size creation/revert
PR #992 (create maxCodeSize x 250, OOG/revert, selfdestruct)
Correctnes of revert operation (refunds) upon oog/revert inside create/2 and transaction create subcall
PR #1003 (correctness of revert operation (refunds) in subcalls)
Random test generator script for retesteth from ORI
PR #949 (random test generator script for retesteth)
More tests around JUPM/JUMPDEST
PR #1006
Random test failing on besu
PR #1007

Upcoming changes:

Assets 2

18 Nov 11:46

winsvega

v10.2

6401889

Tests update 10.2: Difficulty Tests

The recent tests changes:

Old difficulty tests removed from BasicTests folder
Introduce new difficulty formula test vectors for bomb delay check in DifficultyTests folder

Announcement

Test format

New test folder for new test suite DifficultyTestsFiller in /src
and DifficultyTests in root

Tests Fillers

GeneralStateTests Filled

Test Regeneration

Update all:
#984

Documentation
PR #981
invalid EC signature recovery test
PR #973
remove outdated difficulty tests
PR #976
PR ethereum/legacytests#1
introduce reworked DifficultyTests
PR #977
difficulty tests for ArrowGlacier
PR #972
more transaction tests
PR #979
PR #969
PR #982
PR #984
PR #988
Travis validator script for new test formats
PR #980

Upcoming changes:

Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

23 Oct 14:11

winsvega

v10.1

fbff6fe

Tests update 10.1: Transaction Tests

The recent tests changes:

Legacy tests moved into a separate repository.
Use git submodule update --init to fetch it
Transaction tests reworked to be supported now with geth t9n
Transaction test format changed

Announcement

Test format

New test folder stEIP3607 in StateTests and BCGeneralStateTests

Tests Fillers

GeneralStateTests Filled

Test Regeneration

Update all:

Documentation
Blake precompile bug coverage
PR #948
Move legacy tests into a separate repo
PR #950
Transaction from account with code is prohibited
PR #963
Covert old transaction tests to new transaction tests
with geth t9n transaction test support
PR #953
PR #954
PR #957
PR #958
PR #960
PR #955
PR #959
PR #956
PR #961
PR #962
PR #964
PR #967
PR #968
PR #966

Upcoming changes:

==>Difficulty Tests suite for difficulty formula and ice age calculation (just before PoS :lol)
Implement malicious rlp tests in StateTest or BlockchainTest format (need more debugging with t8n)
Besu retesteth RPC support (web vmtracing support)
Random state test script

Assets 2

15 Sep 19:35

winsvega

v10.0

52cb3b3

Tests update 10.0: Regeneration

The recent tests changes:

Regenerate all the tests, change the sorting for pretty looks
Filler hash calculation algorithm is changed
Filled tests _info now has hash to detect changes of the filled test

Announcement

Test format

Tests Fillers

GeneralStateTests Filled

_info section now has

"generatedTestHash" : "f3fd299513374b390d8ee5aae37cd3ece229fd069866b01d45e404b700e24064",

A hash field of the filled test.

Test Regeneration

Update all:
PR #931

Documentation
PR #929
Test for overlapping memory in a call to precomp 4 (identity)
PR #938 (Issue #935)
The effect of CREATE[2] on the RETURNDATA buffer
PR #943 (Issue #942)
Rework modexp tests
PR #945
Remove archaic difficulty formula tests for test networks
PR #946

Upcoming changes:

Besu retesteth RPC support (web vmtracing support)
Move legacy tests into separate repo (too much space used)

Assets 2

Releases: ethereum/tests

Tests update 12: Shanghai tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 11.3: EIP tests, prepare to Shanghai

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 11.2: Potential EIPs

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 11.1: Pre Merge

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 11: Merge PoS

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 10.4: winter

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 10.3: Test Headers

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Upcoming changes:

Tests update 10.2: Difficulty Tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests Filled

Test Regeneration

Upcoming changes:

Tests update 10.1: Transaction Tests

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests Filled

Test Regeneration

Upcoming changes:

Tests update 10.0: Regeneration

The recent tests changes:

Announcement

Test format

Tests Fillers

GeneralStateTests Filled

Test Regeneration