Performance degradation in v7.0.0 compared to v6.8.1?

[edenbd500]

Hi all,
I'm working on upgrading my .NET8 application Rabbit nuget to your latest v7 version.
I did some benchmark tests to verify that there isn't a performance degradation while upgrading.
I created a small Console App to run the benchmark using Microsoft's Benchmark.Dotnet.
My environment is:
RabbitMQ Server 3.12.4 (running locally via docker)
Erlang 25.3.2.5
.NET version - 8.0.0
System.Text.Json - 9.0.0

Result for RabbitMQ.Client v6.8.1:

| Method    | Mean     | Error   | StdDev  | Gen0   | Gen1   | Allocated |
|---------- |---------:|--------:|--------:|-------:|-------:|----------:|
| PublishV6 | 461.3 us | 6.60 us | 6.17 us | 9.7656 | 8.7891 | 130.19 KB |

Result for RabbitMQ.Client v7.0.0:

| Method    | Mean     | Error     | StdDev    | Gen0    | Allocated |
|---------- |---------:|----------:|----------:|--------:|----------:|
| PublishV7 | 8.871 ms | 0.1711 ms | 0.1429 ms | 15.6250 | 216.35 KB |

This is the v7 benchmark:

public class RabbitV7Benchmark
{
    private IChannel _channel;
    private BenchmarkMsg _message;
    private const string QueueName = "MemoryTest.JsonSerializer";

    [GlobalSetup]
    public void Setup()
    {
        var factory = new ConnectionFactory() { HostName = "localhost" };
        var connection =  factory.CreateConnectionAsync().Result;
        _channel =  connection.CreateChannelAsync().Result;
        _channel.QueueDeclareAsync(queue: QueueName, durable: false, exclusive: false, autoDelete: true, arguments: null).Wait();

        _message = new BenchmarkMsg() { Prop1 = "1", Prop2 = "2" };
    }

    [Benchmark]
    public async Task PublishV7()
    {
        var basicProperties = new BasicProperties();
        var body = System.Text.Json.JsonSerializer.SerializeToUtf8Bytes(_message);
        for (int i = 0; i < 1000; i++)
        {
            await _channel.BasicPublishAsync<BasicProperties>(exchange: "", routingKey: QueueName, basicProperties: basicProperties, body: body, mandatory: false);
        }
    }
}

v6 benchmark:

[MemoryDiagnoser]
public class RabbitV6Benchmark
{
    private IModel _channel;
    private BenchmarkMsg _message;
    private const string QueueName = "MemoryTest.JsonSerializer";

    [GlobalSetup]
    public void Setup()
    {
        var factory = new ConnectionFactory() { HostName = "localhost" };
        var connection =  factory.CreateConnection();
        _channel = connection.CreateModel();
        _channel.QueueDeclare(queue: QueueName, durable: false, exclusive: false, autoDelete: true, arguments: null);

        _message = new BenchmarkMsg() { Prop1 = "1", Prop2 = "2" };
    }

    [Benchmark]
    public void PublishV6()
    {
        var basicProperties = _channel.CreateBasicProperties();
        var body = System.Text.Json.JsonSerializer.SerializeToUtf8Bytes(_message);
        for (int i = 0; i < 1000; i++)
        {
            _channel.BasicPublish(exchange: "", routingKey: QueueName, basicProperties: basicProperties, body: body, mandatory: false);
        }
    }
}

Would you happen to have any thoughts on what I am doing wrong?
I can share the entire sample project if it's needed.

[lukebakken]

Please share a complete project I can clone, compile, and run. Provide instructions for how to observe these results. I'm sure you're doing something wrong with your code, because other users have migrated without issue.

[edenbd500]

Please see the code here:
https://github.com/edenbd500/rabbitmq-client-benchmark
When you run it, it will run v7 by default. To run v6, change the property "RabbitNugetVersion" to "V6" in the csproj file.
Results can be observed on the Console, as they printed the same as I shared above.

[lukebakken]

Thank you very much for taking the time to provide a runnable project.

Environment:

• Benchmarks run on my laptop, 64GiB RAM, i9 processor
• RabbitMQ 4 (main branch) running on my Arch Linux system. 32 GiB RAM, i7 processor, Erlang 27.2 built with my patch here - Make poll() test work with gcc 14 erlang/otp#9234
• Gigabit ethernet connectivity between systems.
• Using a fork of your code here: https://github.com/lukebakken/rabbitmq-dotnet-client-1769/tree/lukebakken/misc

Please note the Message Rate stats in the following screenshots.

RabbitMQ management UI stats from a run of the V6 benchmark:

From a run of the V7 version of the client. Note the much improved publish rate, without any variance:

I have no idea why BenchmarkDotNet is reporting the numbers that it does, but the stats from RabbitMQ clearly show that V7 of this library performs better.

[edenbd500]

@michaelklishin, I agree with you. Therefore, I used Benchmark.Dotnet in the first place. @lukebakken said "I have no idea why BenchmarkDotNet is reporting the numbers that it does", so I also did the same benchmark using Stopwatch to prove that the issue is not in the Benchmark library.

[lukebakken]

@edenbd500 - what does the RabbitMQ management interface report in your environment? Please ensure your screenshots show the "Message rates" graph while running your test using v6 and v7 of this library.

When I have time, I can re-run this test, but it's pretty clear to me there is no issue. Nobody else has reported it, either. In fact, they've reported the opposite:

#1721 (comment)

[edenbd500]

@lukebakken, here you go:
V7:

V6:

Throughput seems the same in my environment, even though the throughput is not my concern in the workloads we use Rabbit for.

[michaelklishin]

Note the very choppy message rates in this environment. These could be GC pauses. Luke's example has a very smooth publishing rate curve.

[lukebakken]

I modified your test to use publisher confirmations (code), because they are a requirement for reliable publishing. To publish 4 million messages in my envirnoment, yep, V6 is a little faster. I'm not inclined to investigate further unless a paying customer requests it. I will gladly review pull requests to improve the V7 performance.

Unless someone submits a pull request specifically to address this issue, I consider this discussion to be completed. Thanks.

V6: 25014ms

V7: 26696ms