feat(levm): add legacy tests to the suite #1870

tomip01 · 2025-02-03T21:11:20Z

Motivation

This pull request introduces changes to the Ethereum Foundation (EF) test framework, specifically enhancing the handling of forks within the test reports. The changes include modifications to the deserialization process, test report structure, and test runner logic to support multiple forks.

Description

cmd/ef_tests/levm/report.rs: Refactored EFTestReport to include fork_results instead of a single fork field. Introduced EFTestReportForkResult to encapsulate fork-specific results. Updated methods to handle the new structure and ensure correct reporting of test results per fork.
cmd/ef_tests/levm/runner/levm_runner.rs: Updated the run_ef_test and run_ef_test_tx functions to iterate over forks and handle fork-specific test execution. Adjusted prepare_vm_for_tx and ensure_post_state functions to accept a fork parameter.
cmd/ef_tests/levm/runner/mod.rs: Handled cases where test hashes might be missing to prevent panics.
cmd/ef_tests/levm/runner/revm_runner.rs: Updated imports to include EFTestReportForkResult.

Closes #1701

github-actions · 2025-02-03T21:14:19Z

| File                                                                    | Lines | Diff |
+-------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/ethrex/ef_tests.rs         | 3     | +3   |
+-------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/ethrex/network.rs          | 79    | +79  |
+-------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/ethrex/test_runner.rs      | 137   | +137 |
+-------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/ethrex/types.rs            | 384   | +384 |
+-------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/levm/deserialize.rs        | 405   | +34  |
+-------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/levm/report.rs             | 911   | +64  |
+-------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/levm/runner/levm_runner.rs | 418   | +20  |
+-------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/levm/runner/revm_runner.rs | 513   | +36  |
+-------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/cmd/ef_tests/levm/types.rs              | 268   | -64  |
+-------------------------------------------------------------------------+-------+------+

Total lines added: +757
Total lines removed: 64
Total lines changed: 821

github-actions · 2025-02-03T21:25:56Z

Benchmark Results Comparison

PR Results

Benchmark Results: Factorial

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Factorial`	232.9 ± 1.1	232.0	235.7	1.00
`levm_Factorial`	900.6 ± 10.0	889.3	919.8	3.87 ± 0.05

Benchmark Results: Factorial - Recursive

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_FactorialRecursive`	1.428 ± 0.081	1.333	1.526	1.00
`levm_FactorialRecursive`	15.616 ± 0.058	15.567	15.770	10.94 ± 0.62

Benchmark Results: Fibonacci

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Fibonacci`	201.8 ± 1.0	201.2	204.6	1.00
`levm_Fibonacci`	894.3 ± 7.1	881.8	904.2	4.43 ± 0.04

Benchmark Results: ManyHashes

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ManyHashes`	8.9 ± 0.6	8.6	10.4	1.00
`levm_ManyHashes`	18.6 ± 0.5	18.3	19.9	2.09 ± 0.14

Benchmark Results: BubbleSort

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_BubbleSort`	3.225 ± 0.030	3.201	3.295	1.00
`levm_BubbleSort`	6.095 ± 0.032	6.058	6.143	1.89 ± 0.02

Benchmark Results: ERC20 - Transfer

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Transfer`	247.7 ± 1.3	246.6	250.5	1.00
`levm_ERC20Transfer`	537.6 ± 3.1	533.0	542.8	2.17 ± 0.02

Benchmark Results: ERC20 - Mint

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Mint`	143.1 ± 0.9	141.7	144.4	1.00
`levm_ERC20Mint`	348.6 ± 1.9	345.5	351.0	2.44 ± 0.02

Benchmark Results: ERC20 - Approval

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_ERC20Approval`	1.037 ± 0.007	1.029	1.053	1.00
`levm_ERC20Approval`	2.025 ± 0.008	2.013	2.037	1.95 ± 0.02

Main Results

Benchmark Results: Factorial

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Factorial`	235.8 ± 1.2	233.6	237.5	1.00
`levm_Factorial`	899.4 ± 7.5	890.2	912.9	3.81 ± 0.04

Benchmark Results: Factorial - Recursive

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_FactorialRecursive`	1.486 ± 0.075	1.356	1.570	1.00
`levm_FactorialRecursive`	15.622 ± 0.059	15.579	15.784	10.51 ± 0.53

Benchmark Results: Fibonacci

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_Fibonacci`	208.9 ± 3.8	206.0	216.3	1.00
`levm_Fibonacci`	893.2 ± 11.2	881.4	914.8	4.28 ± 0.09

Benchmark Results: ManyHashes

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ManyHashes`	8.6 ± 0.0	8.5	8.7	1.00
`levm_ManyHashes`	18.0 ± 0.2	17.9	18.5	2.09 ± 0.02

Benchmark Results: BubbleSort

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_BubbleSort`	3.235 ± 0.016	3.219	3.268	1.00
`levm_BubbleSort`	6.092 ± 0.072	6.040	6.290	1.88 ± 0.02

Benchmark Results: ERC20 - Transfer

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Transfer`	248.9 ± 4.5	246.3	261.4	1.00
`levm_ERC20Transfer`	539.2 ± 7.5	533.2	557.8	2.17 ± 0.05

Benchmark Results: ERC20 - Mint

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`revm_ERC20Mint`	142.3 ± 0.9	140.9	143.8	1.00
`levm_ERC20Mint`	347.8 ± 2.7	345.1	354.3	2.44 ± 0.02

Benchmark Results: ERC20 - Approval

Command	Mean [s]	Min [s]	Max [s]	Relative
`revm_ERC20Approval`	1.036 ± 0.003	1.031	1.040	1.00
`levm_ERC20Approval`	2.026 ± 0.061	1.990	2.197	1.96 ± 0.06

github-actions · 2025-02-04T19:50:54Z

Summary: 9449/14408 (65.58%)

Prague: 2373/2373 (100.00%)
Cancun: 3579/3579 (100.00%)
Shanghai: 221/221 (100.00%)
Paris: 62/62 (100.00%)
London: 39/39 (100.00%)
Berlin: 35/35 (100.00%)
Istanbul: 34/35 (97.14%)
Constantinople: 1040/2439 (42.64%)
Byzantium: 1000/2330 (42.92%)
Homestead: 585/1324 (44.18%)
Frontier: 141/742 (19.00%)

tomip01 · 2025-02-05T13:58:28Z

cmd/ef_tests/levm/types.rs

-impl EFTest {
-    pub fn fork(&self) -> Fork {
-        match &self.post {
-            EFTestPost::Prague(_) => Fork::Prague,
-            EFTestPost::Cancun(_) => Fork::Cancun,
-            EFTestPost::Shanghai(_) => Fork::Shanghai,
-            EFTestPost::Homestead(_) => Fork::Homestead,
-            EFTestPost::Istanbul(_) => Fork::Istanbul,
-            EFTestPost::London(_) => Fork::London,
-            EFTestPost::Byzantium(_) => Fork::Byzantium,
-            EFTestPost::Berlin(_) => Fork::Berlin,
-            EFTestPost::Constantinople(_) | EFTestPost::ConstantinopleFix(_) => {
-                Fork::Constantinople
-            }
-            EFTestPost::Paris(_) => Fork::Paris,
-            EFTestPost::Frontier(_) => Fork::Frontier,
-        }
-    }
-}
-


I removed this because we no longer have a single fork for a test. Now a test consists of a single pre state and expected post state for different forks

.github/workflows/ci_levm.yaml

crates/vm/levm/Makefile

fborello-lambda

LGTM

crates/vm/levm/Makefile

Oppen · 2025-02-05T22:15:40Z

crates/vm/levm/Makefile

+	git clone --recurse-submodules $(ETH_TEST_URL) $(TESTS_REPO) 
+	cd $(TESTS_REPO) && git checkout tags/$(ETH_TEST_TAG)


This can be simplified to:

Suggested change

git clone --recurse-submodules $(ETH_TEST_URL) $(TESTS_REPO)

cd $(TESTS_REPO) && git checkout tags/$(ETH_TEST_TAG)

git clone --recurse-submodules -b tags/$(ETH_TEST_TAG) $(ETH_TEST_URL) $(TESTS_REPO)

I believe that would also cover the following line, but check.

Thanks! I tried that but it didn't work out. I stick to:
git clone --recurse-submodules --depth 1 --branch $(ETH_TEST_TAG) $(ETH_TEST_URL) $(TESTS_REPO)
Done here 16a6725

cmd/ef_tests/levm/report.rs

download legacy tests

97fb6a6

tomip01 changed the title ~~download legacy tests~~ feat(levm): download legacy tests Feb 3, 2025

tomip01 added 8 commits February 4, 2025 09:40

remove cancun download

e4f7063

deserialize post values with forks

6526da5

deserialize post values with forks

eda99d8

progress on parsing and running tests

f219e19

resolve errors

822421e

parse succesfully

56561ac

solve parsing

b32c5cb

fix none unwrap

35ab175

tomip01 added 2 commits February 4, 2025 17:02

pass all

e901d9f

all working

243ad39

tomip01 self-assigned this Feb 4, 2025

tomip01 added ef-tests Hive tests, execution-spec-tests levm Lambda EVM implementation labels Feb 4, 2025

tomip01 changed the title ~~feat(levm): download legacy tests~~ feat(levm): add legacy tests to the suite Feb 4, 2025

tomip01 added 2 commits February 5, 2025 10:01

remove comment

4283c2c

rename variables

6ff5f50

tomip01 commented Feb 5, 2025

View reviewed changes

tomip01 added 5 commits February 5, 2025 10:59

refactor

5ededc6

remove comments

c6e0f4b

remove comments

6d2b5e8

make re run report handle forks

632e5e7

simplify makefile

e3ac34d

tomip01 marked this pull request as ready for review February 5, 2025 15:28

tomip01 requested a review from a team as a code owner February 5, 2025 15:29

change ci

66d4ffd

tomip01 added 5 commits February 5, 2025 13:05

change ci

b9dec74

update ci

05d4f63

remove empty line

c4ee0e5

first approach test count

d7880b6

fix report

9849e83

Oppen reviewed Feb 5, 2025

View reviewed changes

.github/workflows/ci_levm.yaml Outdated Show resolved Hide resolved

Oppen reviewed Feb 5, 2025

View reviewed changes

crates/vm/levm/Makefile Outdated Show resolved Hide resolved

tomip01 added 3 commits February 5, 2025 17:42

update comment

9ccad69

refactor makefile

4165bcb

remove withespace

1a8dbff

lambdaclass deleted a comment from github-actions bot Feb 5, 2025

fborello-lambda approved these changes Feb 5, 2025

View reviewed changes

crates/vm/levm/Makefile Outdated Show resolved Hide resolved

remove test func

0b4cef2

Oppen reviewed Feb 5, 2025

View reviewed changes

lambdaclass deleted a comment from github-actions bot Feb 6, 2025

tomip01 and others added 2 commits February 6, 2025 09:40

make better clone efts

16a6725

Merge branch 'main' into levm/add-legacy-tests

be43535

fkrause98 approved these changes Feb 6, 2025

View reviewed changes

cmd/ef_tests/levm/report.rs Outdated Show resolved Hide resolved

remove old comment

4dd4e36

tomip01 enabled auto-merge February 6, 2025 18:08

tomip01 added this pull request to the merge queue Feb 6, 2025

Merged via the queue into main with commit 00a3488 Feb 6, 2025
40 checks passed

tomip01 deleted the levm/add-legacy-tests branch February 6, 2025 18:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(levm): add legacy tests to the suite #1870

feat(levm): add legacy tests to the suite #1870

tomip01 commented Feb 3, 2025 •

edited

Loading

github-actions bot commented Feb 3, 2025 •

edited

Loading

github-actions bot commented Feb 3, 2025 •

edited

Loading

github-actions bot commented Feb 4, 2025 •

edited

Loading

tomip01 Feb 5, 2025

fborello-lambda left a comment

Oppen Feb 5, 2025

tomip01 Feb 6, 2025

		git clone --recurse-submodules $(ETH_TEST_URL) $(TESTS_REPO)
		cd $(TESTS_REPO) && git checkout tags/$(ETH_TEST_TAG)

	git clone --recurse-submodules $(ETH_TEST_URL) $(TESTS_REPO)
	cd $(TESTS_REPO) && git checkout tags/$(ETH_TEST_TAG)
	git clone --recurse-submodules -b tags/$(ETH_TEST_TAG) $(ETH_TEST_URL) $(TESTS_REPO)

feat(levm): add legacy tests to the suite #1870

feat(levm): add legacy tests to the suite #1870

Conversation

tomip01 commented Feb 3, 2025 • edited Loading

github-actions bot commented Feb 3, 2025 • edited Loading

github-actions bot commented Feb 3, 2025 • edited Loading

Benchmark Results Comparison

PR Results

Benchmark Results: Factorial

Benchmark Results: Factorial - Recursive

Benchmark Results: Fibonacci

Benchmark Results: ManyHashes

Benchmark Results: BubbleSort

Benchmark Results: ERC20 - Transfer

Benchmark Results: ERC20 - Mint

Benchmark Results: ERC20 - Approval

Main Results

Benchmark Results: Factorial

Benchmark Results: Factorial - Recursive

Benchmark Results: Fibonacci

Benchmark Results: ManyHashes

Benchmark Results: BubbleSort

Benchmark Results: ERC20 - Transfer

Benchmark Results: ERC20 - Mint

Benchmark Results: ERC20 - Approval

github-actions bot commented Feb 4, 2025 • edited Loading

tomip01 Feb 5, 2025

Choose a reason for hiding this comment

fborello-lambda left a comment

Choose a reason for hiding this comment

Oppen Feb 5, 2025

Choose a reason for hiding this comment

tomip01 Feb 6, 2025

Choose a reason for hiding this comment

tomip01 commented Feb 3, 2025 •

edited

Loading

github-actions bot commented Feb 3, 2025 •

edited

Loading

github-actions bot commented Feb 3, 2025 •

edited

Loading

github-actions bot commented Feb 4, 2025 •

edited

Loading