feat: nat; per-validator configuration.

[scharissis]

Description

• Introduces a way to add per-validator configuration (just tests for now)
• Introduces some tests, which check the above and more
• Note that there is not yet support for reading these values from a file (or elsewhere)

Repercussions

1. Each can now define its own config struct and also default values

(tx_fuzz.go as an example)

type TxFuzzParams struct {
	NSlotsToRunFor     int
	TxPerAccount       uint64
	GenerateAccessList bool
}

var TxFuzz = nat.Test{
	ID: "tx-fuzz",
	DefaultParams: TxFuzzParams{
		NSlotsToRunFor:     120, // Duration of the fuzzing
		TxPerAccount:       3,
		GenerateAccessList: false,
	},
	Fn: func(ctx context.Context, log log.Logger, cfg nat.Config, params interface{}) (bool, error) {
		p := params.(TxFuzzParams)
		err := runBasicSpam(cfg, p)
		if err != nil {
			return false, err
		}
		return true, nil
	},
}

2. This can be specified, for example, when defining our suite:

var LoadTest = nat.Suite{
	ID: "load-test",
	Tests: []nat.Test{
		tests.TxFuzz,
	},
	TestsParams: map[string]interface{}{
		"tx-fuzz": tests.TxFuzzParams{
			NSlotsToRunFor:     2400,
			TxPerAccount:       9,
			GenerateAccessList: false,
		},
	},
}

Pros

• Each test (and validator, although for now we've just implemented support for tests) can have variable config/params
• The params are somewhat typed (runtime checks) [could potentially be improved with generics but would complicate things]
• The params have support for defaults so they don't need to be specified everywhere or everytime
• Quite easy and intuitive

Cons

• If you wish to change just a single param of a test you must nonetheless provide ALL params (we could 'fix' this, but would likely make the code much messier; full of pointers to distinguish the difference between a zero value and a missing value. perhaps something to explore in the future if we like this approach)
• We aren't validating params (we could fix this by forcing the testParam structs to satisfy an interface with a Validate function, but then EVERY test would need to implement this)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 47.02%. Comparing base (cee8001) to head (04ba094).

❗ There is a different number of reports uploaded between BASE (cee8001) and HEAD (04ba094). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (cee8001)	HEAD (04ba094)
contracts-bedrock-tests	4	2

Additional details and impacted files

@@             Coverage Diff             @@
##              nat   #13805       +/-   ##
===========================================
- Coverage   79.10%   47.02%   -32.08%     
===========================================
  Files         166      961      +795     
  Lines        9343    80337    +70994     
  Branches      753      753               
===========================================
+ Hits         7391    37782    +30391     
- Misses       1693    39655    +37962     
- Partials      259     2900     +2641     

Flag	Coverage Δ
cannon-go-tests-32	62.26% <ø> (ø)
cannon-go-tests-64	57.35% <ø> (ø)
contracts-bedrock-tests	91.72% <ø> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 797 files with indirect coverage changes

[jelias2]

Nice gwei

[jelias2]

I think this provides a fair way to override the test defaults however how would this be configured at execution time? wouldn't there need to be a flag or something to override the default params

[scharissis]

I think this provides a fair way to override the test defaults however how would this be configured at execution time? wouldn't there need to be a flag or something to override the default params

As-is we'd have to recompile and specify overrides it in the suite/gate:

var LoadTest = nat.Suite{
	ID: "load-test",
	Tests: []nat.Test{
		tests.TxFuzz,
	},
	TestsParams: map[string]interface{}{
		"tx-fuzz": tests.TxFuzzParams{
			NSlotsToRunFor:     2400,
			TxPerAccount:       9,
			GenerateAccessList: false,
		},
	},
}

I didn't want to overload this PR with handling of os.Args or reading Files, so we could focus on the params abstractions themselves.

Does that answer your question? Thoughts?

[jelias2]

Overall I like the way to specify test parameters, and it fits in well.
Good call on splitting up the args parsing and the parameter structuring