Poor multithreaded performance

[pikrzysztof]

I have an application which was reading capnproto structures from a network and feeding it to ~600 goroutines which would try analyzing it (read-only) in parallel. This caused a massive lock contention, where according to perf, ReadLimiter's spinlock only was responsible for 40% of cumulative CPU usage, please see flamegraph.html.gz (compressed to avoid github from stripping embedded JS)

Once we adapted 1b552dd application's CPU usage dropped more than 10x, from 60 cores to <6 cores.

I think this library might benefit from replacing spinlocks with sleeping locks. Spinlocks are generally not the best choice for userspace where you can't guarantee having the CPU while holding the lock. In case your goroutine gets the lock and gets descheduled other application threads will be stupidly burning CPU cycles until both kernel and golang scheduler agree to give CPU to the goroutine holding the lock. Another disadvantage is that CPU usage poorly correlates with the amount of actual work achieved and can go up unbounded.

[lthibault]

@pikrzysztof You have impeccable timing. @zenhack and I are scheduled to pair up today and dig into performance-tuning for the RPC system. Would you be willing to share your application/benchmark code?

In the meantime, can you confirm which code revision you are using?

[pikrzysztof]

@lthibault Thank you for a prompt response. I can try writing some synthetic benchmark, but I cannot share the original code, but this may take a while.

I was using latest v2 release, i.e. zombiezen.com/go/capnproto2 v2.18.2+incompatible.

[ocdtrekkie]

I think this was when Kenton hit this on the C++ implementation: https://twitter.com/kentonvarda/status/1553120336018870272

[lthibault]

@pikrzysztof I can't recall if this specific issue has evolved between v2 and v3, but I would strongly encourage you to switch to v3. Any improvements are going to happen there. It's currently labeled as alpha, but I'm currently running it in production and haven't had any major issues. YMMV, of course.

[pikrzysztof]

@lthibault the spinlock is still there, please see https://github.com/capnproto/go-capnproto2/blob/0a7a48ce93a8cdf8c932689d71edc812a9e1e617/message.go#L152-L163

[lthibault]

So it is. Thanks!

We're focusing our initial perf-tuning efforts on memory allocations in the rpc package, but this spinlock issue definitely seems like something worth investigating afterwards. I don't have an ETA, unfortunately, but I am more than happy to provide support if you want to take a stab at a PR. That would likely be the fastest way of getting this change incorporated.

A synthetic benchmark would indeed be helpful in the meantime.

[zenhack]

That's not a spinlock; the compare & swap isn't implementing a lock it's just retrying if the value was modified from underneath it. We could probably make this a better by using atomic.AddInt64 instead of having a loop that retries (and changing the limit to be signed, and checking for negatives).

...but it is quite sad that we even need to do atomic operations for reads in the first place. The more invasive change that I think might make sense would be to instead of having a single global limit attached to the message, have a limit that is tracked separately, and can then just be per-goroutine. Maybe store a pointer to it in types like Ptr, Struct, etc. and allocate it when you do Message.Root() and such. There are a few API design questions to be worked out. But I think this would be worth doing before we tag 3.0 so we have the option of making breaking changes.

[pikrzysztof]

Here's a quick & dirty benchmark that hits the hotpath: https://github.com/pikrzysztof/gocapnp/blob/master/main_test.go

Please note that this is a benchmark on a laptop CPU. Scaling across sockets can be even worse.

-> % go test -bench=. -cpuprofile profile.out
goos: linux
goarch: amd64
pkg: github.com/pikrzysztof/capnp-benchmark
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
BenchmarkOne-8    	16024563	      139.0 ns/op
BenchmarkFour-8   	5696427	      322.9 ns/op
BenchmarkTwo-8    	10138142	      213.8 ns/op
PASS
ok  	github.com/pikrzysztof/capnp-benchmark	6.830s

-> % go tool pprof profile.out
(pprof) top
Showing nodes accounting for 10360ms, 68.75% of 15070ms total
Dropped 150 nodes (cum <= 75.35ms)
Showing top 10 nodes out of 72
      flat  flat%   sum%        cum   cum%
    3460ms 22.96% 22.96%     3490ms 23.16%  capnproto.org/go/capnp/v3.(*Message).canRead
    1350ms  8.96% 31.92%     3830ms 25.41%  runtime.mallocgc
    1070ms  7.10% 39.02%    13020ms 86.40%  capnproto.org/go/capnp/v3.Struct.Ptr
     980ms  6.50% 45.52%     1370ms  9.09%  fmt.(*pp).doPrintf
     780ms  5.18% 50.70%    11780ms 78.17%  capnproto.org/go/capnp/v3.(*Segment).readPtr
     710ms  4.71% 55.41%      710ms  4.71%  runtime.nextFreeFast (inline)
     610ms  4.05% 59.46%      670ms  4.45%  capnproto.org/go/capnp/v3.Ptr.text
     550ms  3.65% 63.11%      800ms  5.31%  runtime.heapBitsSetType
     510ms  3.38% 66.49%    14290ms 94.82%  github.com/pikrzysztof/capnp-benchmark.Book.TitleBytes
     340ms  2.26% 68.75%      340ms  2.26%  runtime.memmove

You can see the more goroutines we create, the slower the benchmark score, showing ReadLimiter as the main source of our pain.

[ocdtrekkie]

@pikrzysztof Is your benchmark repo set to private? (I am not going to run it, but since it 404s out, I figured I'd mention it to see if that was intentional or not.)

[pikrzysztof]

@ocdtrekkie thanks for noticing, fixed

[zenhack]

You can probably work around this for the moment by creating a separate message with the same backing arena:

msg2 := &capnp.Message{Arena: msg1.Arena, CapTable: msg1.CapTable}

...which will then have its own read limit and thus not have to contend with the other threads.

[pikrzysztof]

To illustrate scaling characteristics, I ran the benchmark on my server. I limited the benchmark to 14 threads only to avoid cross-socket chatter and hyperthreading from kicking in.

Here's the data fitted into universal scalability law curve:

• x axis is the number of goroutines concurrently accessing capnp structures
• y is the the benchmark throughput, i.e. number of accesses per second (note 1e6 multiplier)

The specific values are not very important, it's more about showing that this particular benchmark does not scale well beyond 4 cores, even hurting performance if you throw more than 12 cores at it.

and here's the same curve, extrapolated for the case if we scaled to 28 threads but without taking multiplesockets and hyperthreading into consideration.

Optimized parameters: [gamma = 4.68456920e+06, alpha = 4.68893905e-01, beta = 3.84146869e-03]
Estimated covariance: [[ 4.62262018e+10 8.83418242e+03 -2.47746327e+02]
[ 8.83418242e+03 2.04440998e-03 -8.13796353e-05]
[-2.47746327e+02 -8.13796353e-05 4.96395899e-06]]