redis-benchmark results show no performance gains over Redis

[binjamil]

Hi, so I ran a quick benchmark for both Dragonfly and Redis using redis-benchmark utility on my personal machine whose specs are listed below. The benchmark made 10 million requests using 10,000 parallel clients. Since Dragonfly using multi-threading, I expected it to perform significantly better than Redis on my 16-core machine. However, the results are strikingly similar. What am I missing?

Command that I ran on my machine:

redis-benchmark -n 10000000 -c 10000 -q -t get,set

Dragonfly

SET: 37895.73 requests per second, p50=131.455 msec                     
GET: 38179.45 requests per second, p50=130.879 msec

Redis

SET: 38965.39 requests per second, p50=130.431 msec
GET: 39210.15 requests per second, p50=128.767 msec

My machine specs are:

OS: openSUSE Tumbleweed x86_64 
Host: 82JD Legion 5 Pro 16ITH6H 
Kernel: 6.2.12-1-default
CPU: 11th Gen Intel i7-11800H (16) @ 4.600GHz 
GPU: NVIDIA GeForce RTX 3060 Mobile / Max-Q 
GPU: Intel TigerLake-H GT1 [UHD Graphics] 
Memory: 31868MiB 

[romange]

on your laptop you run with -c 10000?
use threads on redis-benchmark (--threads) and reduce -c to 50.

[romange]

Also you probably run the loadtest vs the same process twice by mistake.

[binjamil]

Yes, I ran with -c 10000 to have 10K parallel clients. Why is that a problem? I thought more clients would mean better throughput for a multi-threaded system, no?

And I didn't run the benchmark on same process as the Redis one is about a week or two earlier. And I ran against Dragonfly today and the Redis server was not up during this run.

[romange]

I did not say it's a problem for Dragonfly or even Redis but it's definitely a problem for redis-benchmark. I gave you a hint on how to proceed but in general benchmarking is a domain expertise on its own that requires performing lots of runs with lots of configurations. And for dragonfly specifically, I advise not to perform it on laptops when both Dragonfly and load test program are running on the same machine.

[binjamil]

Totally agree that benchmarking is not that simple and certainly running both redis-benchmark and the KV server on same machine impacts the performance. I'm actively learning more about databases and scalability, and as a part of it I'm building a toy in-memory KV store like Redis. It performed better than Redis and Dragonfly using the benchmark setting I provided above so I was certain something's fishy 😂.

Anyway, I rerun the benchmark with --threads 16 option (on my laptop again as that is only what I can do right now) and thankfully I can see the performance of Dragonfly's multi-threaded architecture:

Dragonfly

SET: 105385.18 requests per second, p50=10.143 msec
GET: 171244.61 requests per second, p50=11.879 msec

Radahn

SET: 71110.10 requests per second, p50=139.903 msec                            
GET: 73074.84 requests per second, p50=135.935 msec                            

Redis

SET: 44430.43 requests per second, p50=223.999 msec                            
GET: 46842.36 requests per second, p50=212.607 msec