Best way to structure benchmarks that need clean state before each benchmark but take less than 100ms?

[grofit]

So I incorrectly assumed that [GlobalSetup] was run once before each Benchmark, and while I am kinda right, unfortunately it seems that the benchmark is run multiple times before it actually gets run for real which means the state is all messed up, for example here is a simple test trying to check a built in .net type against a custom type and compare results:

[SimpleJob(RuntimeMoniker.Net90, warmupCount: 1, invocationCount: 1, iterationCount: 1)]
[MemoryDiagnoser]
public class KeyedCollectionVsDictionaryBenchmarks
{
    [Params(100,10000,100000)]
    public int Amount;
    
    public TestKeyedCollection TestKeyedCollection = new TestKeyedCollection();
    public Dictionary<int,int> TestDictionary = new Dictionary<int,int>();

    [GlobalSetup]
    public void Setup()
    {
        TestKeyedCollection.Clear();
        TestDictionary.Clear();
    }

    [GlobalSetup(Targets =
    [
        nameof(LookupElements_Dictionary), nameof(LookupElements_Lookup), 
        nameof(RemoveElements_Dictionary), nameof(RemoveElements_Lookup)
    ])]
    public void PrePopulatedIterationSetup()
    {        
        for(var i=0;i<Amount;i++)
        { 
            TestKeyedCollection.Add(i);
            TestDictionary.Add(i,i);
        }
    }
    
    [Benchmark]
    public void AddElements_Dictionary()
    {
        for(var i=0;i<Amount;i++)
        { TestDictionary.Add(i,i); }
    }
    
    [Benchmark]
    public void AddElements_Lookup()
    {
        for(var i=0;i<Amount;i++)
        { TestKeyedCollection.Add(i); }
    }

    [Benchmark]
    public void LookupElements_Dictionary()
    {
        for(var i=0;i<Amount;i++)
        { TestDictionary.TryGetValue(i, out _); }
    }

    [Benchmark]
    public void LookupElements_Lookup()
    {
        for(var i=0;i<Amount;i++)
        { TestKeyedCollection.TryGetValue(i, out _); }
    }
    
    [Benchmark]
    public void RemoveElements_Dictionary()
    {
        for(var i=0;i<Amount;i++)
        { TestDictionary.Remove(i); }
    }

    [Benchmark]
    public void RemoveElements_Lookup()
    {
        for(var i=0;i<Amount;i++)
        { TestKeyedCollection.Remove(i); }
    }
}

The tests never run as it will blow up saying the key has already been entered, if I were to move to an IterationSetup approach that works but as warned in the docs these tests are done in under 1ms so the iteration step would skew the results.

I am fine refactoring the benchmark into another format but I dont want to have to new up and add elements to the objects as part of the benchmark logic as it will skew results with allocations and time spent, I just want to be able to go in with an empty object for first 2 benchmarks then a pre-populated object for last 4 ones.

Any advice?

[timcassell]

#2600 (comment)

[adamsitnik]

[SimpleJob(RuntimeMoniker.Net90, warmupCount: 1, invocationCount: 1, iterationCount: 1)]

You should not need to set the number of invocations or iterations unless you have a good understanding of what they mean:

https://benchmarkdotnet.org/articles/guides/how-it-works.html

    for(var i=0;i<Amount;i++)
   { TestDictionary.TryGetValue(i, out _); }

https://github.com/dotnet/performance/blob/main/docs/microbenchmark-design-guidelines.md#loops

    for(var i=0;i<Amount;i++)
   { TestDictionary.Add(i,i); }

Just create a new collection, add all the items and return the collection. If you want the results to get scaled, you can use https://github.com/dotnet/performance/blob/main/docs/microbenchmark-design-guidelines.md#operationsperinvoke

You can also take a look at the collection benchmarks used by the .NET Team: https://github.com/dotnet/performance/tree/main/src/benchmarks/micro/libraries/System.Collections