pageserver: 2x ingest performance #8452

jcsp · 2024-07-22T08:40:20Z

Ingest performance can be improved a lot compared to where it is today. This ticket has an initial achievable goal of doubling the measured ingest performance in MB/s as measured by test_bulk_insert.

The bottleneck is currently pageserver CPU/memory:

We issue a huge number of calls to TimelineWriter::put, one for each delta. In a typical WAL stream that means 10MB/s is 100,000 calls per second.
InMemoryLayerInner::index is a btree map of Key to modifications of that key. It grows huge, and also grows large because the keys are at least 18 bytes each. When profiling, a significant fraction of runtime is spent in btreemap::entry (and Key::cmp).
We repeatedly call Timeline::get during ingest to check the size of relations (in case they need extending), even though we're the only thing that can modify that size, so we can trust our own latest value.

Tasks

Give feedback

pageserver: add bench_ingest #7409

a/tech_debt c/storage/pageserver
tests: reinstate test_bulk_insert #8683

a/tech_debt a/test run-benchmarks
Change TimelineWriter::put_batch to avoid calling into put separately for each key: hold locks across the whole operation. Also replace InMemoryLayer::put_value with a batched variant to avoid repeatedly locking the inner layer.
bypass PageCache for L0 flush #7418

18 of 18

a/tech_debt c/storage/pageserver
A more efficient implementation of InMemoryLayerInner::index: this might be as simple as adding a layer of indirection, where we store a HashMap of RelTags, and for each RelTag we store a BTreeMap of u32 (blkno).
Cache latest relation sizes in DatadirModification so that we read the sizes at most one per batch.
Options

The branch where I spent a few hours investigating is https://github.com/neondatabase/neon/commits/jcsp/faster-ingest-2/. This branch achieved a ~25% throughput improvement. I anticipate that the next biggest thing to do is the index efficiency improvement.

This is not an exhaustive list of ways to improve performance. Other stuff that will help but probably isn't necessary for an initial jump in performance:

Make DataDirModification build its own serialized vector that can be dumped straight into an ephemeral layer, rather than our current multi-hop behavior of first storing Value in a map, then later serializing.
Pipelining I/O in ephemeral layer's buffered writer, so that we don't block waiting for the last buffer to finish writing before we start on the next.
Reading relation sizes even more rarely by maintain some state about latest relation sizes between batches.

The text was updated successfully, but these errors were encountered:

## Problem We lack a rust bench for the inmemory layer and delta layer write paths: it is useful to benchmark these components independent of postgres & WAL decoding. Related: #8452 ## Summary of changes - Refactor DeltaLayerWriter to avoid carrying a Timeline, so that it can be cleanly tested + benched without a Tenant/Timeline test harness. It only needed the Timeline for building `Layer`, so this can be done in a separate step. - Add `bench_ingest`, which exercises a variety of workload "shapes" (big values, small values, sequential keys, random keys) - Include a small uncontroversial optimization: in `freeze`, only exhaustively walk values to assert ordering relative to end_lsn in debug mode. These benches are limited by drive performance on a lot of machines, but still useful as a local tool for iterating on CPU/memory improvements around this code path. Anecdotal measurements on Hetzner AX102 (Ryzen 7950xd): ``` ingest-small-values/ingest 128MB/100b seq time: [1.1160 s 1.1230 s 1.1289 s] thrpt: [113.38 MiB/s 113.98 MiB/s 114.70 MiB/s] Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) low mild Benchmarking ingest-small-values/ingest 128MB/100b rand: Warming up for 3.0000 s Warning: Unable to complete 10 samples in 10.0s. You may wish to increase target time to 18.9s. ingest-small-values/ingest 128MB/100b rand time: [1.9001 s 1.9056 s 1.9110 s] thrpt: [66.982 MiB/s 67.171 MiB/s 67.365 MiB/s] Benchmarking ingest-small-values/ingest 128MB/100b rand-1024keys: Warming up for 3.0000 s Warning: Unable to complete 10 samples in 10.0s. You may wish to increase target time to 11.0s. ingest-small-values/ingest 128MB/100b rand-1024keys time: [1.0715 s 1.0828 s 1.0937 s] thrpt: [117.04 MiB/s 118.21 MiB/s 119.46 MiB/s] ingest-small-values/ingest 128MB/100b seq, no delta time: [425.49 ms 429.07 ms 432.04 ms] thrpt: [296.27 MiB/s 298.32 MiB/s 300.83 MiB/s] Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) low mild ingest-big-values/ingest 128MB/8k seq time: [373.03 ms 375.84 ms 379.17 ms] thrpt: [337.58 MiB/s 340.57 MiB/s 343.13 MiB/s] Found 1 outliers among 10 measurements (10.00%) 1 (10.00%) high mild ingest-big-values/ingest 128MB/8k seq, no delta time: [81.534 ms 82.811 ms 83.364 ms] thrpt: [1.4994 GiB/s 1.5095 GiB/s 1.5331 GiB/s] Found 1 outliers among 10 measurements (10.00%) ```

## Problem This follows a PR that insists all input keys are representable in 16 bytes: - #8648 & a PR that prevents postgres from sending us keys that use the high bits of field2: - #8657 Motivation for this change: 1. Ingest is bottlenecked on CPU 2. InMemoryLayer can create huge (~1M value) BTreeMap<Key,_> for its index. 3. Maps over i128 are much faster than maps over an arbitrary 18 byte struct. It may still be worthwhile to make the index two-tier to optimize for the case where only the last 4 bytes (blkno) of the key vary frequently, but simply using the i128 representation of keys has a big impact for very little effort. Related: #8452 ## Summary of changes - Introduce `CompactKey` type which contains an i128 - Use this instead of Key in InMemoryLayer's index, converting back and forth as needed. ## Performance All the small-value `bench_ingest` cases show improved throughput. The one that exercises this index most directly shows a 35% throughput increase: ``` ingest-small-values/ingest 128MB/100b seq, no delta time: [374.29 ms 378.56 ms 383.38 ms] thrpt: [333.88 MiB/s 338.13 MiB/s 341.98 MiB/s] change: time: [-26.993% -26.117% -25.111%] (p = 0.00 < 0.05) thrpt: [+33.531% +35.349% +36.974%] Performance has improved. ```

…LSN during ingest (#8591) ## Problem/Solution TimelineWriter::put_batch is simply a loop over individual puts. Each put acquires and releases locks, and checks for potentially starting a new layer. Batching these is more efficient, but more importantly unlocks future changes where we can pre-build serialized buffers much earlier in the ingest process, potentially even on the safekeeper (imagine a future model where some variant of DatadirModification lives on the safekeeper). Ensuring that the values in put_batch are written to one layer also enables a simplification upstream, where we no longer need to write values in LSN-order. This saves us a sort, but also simplifies follow-on refactors to DatadirModification: we can store metadata keys and data keys separately at that level without needing to zip them together in LSN order later. ## Why? In this PR, these changes are simplify optimizations, but they are motivated by evolving the ingest path in the direction of disentangling extracting DatadirModification from Timeline. It may not obvious how right now, but the general idea is that we'll end up with three phases of ingest: - A) Decode walrecords and build a datadirmodification with all the simple data contents already in a big serialized buffer ready to write to an ephemeral layer **<-- this part can be pipelined and parallelized, and done on a safekeeper!** - B) Let that datadirmodification see a Timeline, so that it can also generate all the metadata updates that require a read-modify-write of existing pages - C) Dump the results of B into an ephemeral layer. Related: #8452 ## Caveats Doing a big monolithic buffer of values to write to disk is ordinarily an anti-pattern: we prefer nice streaming I/O. However: - In future, when we do this first decode stage on the safekeeper, it would be inefficient to serialize a Vec of Value, and then later deserialize it just to add blob size headers while writing into the ephemeral layer format. The idea is that for bulk write data, we will serialize exactly once. - The monolithic buffer is a stepping stone to pipelining more of this: by seriailizing earlier (rather than at the final put_value), we will be able to parallelize the wal decoding and bulk serialization of data page writes. - The ephemeral layer's buffered writer already stalls writes while it waits to flush: so while yes we'll stall for a couple milliseconds to write a couple megabytes, we already have stalls like this, just distributed across smaller writes. ## Benchmarks This PR is primarily a stepping stone to safekeeper ingest filtering, but also provides a modest efficiency improvement to the `wal_recovery` part of `test_bulk_ingest`. test_bulk_ingest: ``` test_bulk_insert[neon-release-pg16].insert: 23.659 s test_bulk_insert[neon-release-pg16].pageserver_writes: 5,428 MB test_bulk_insert[neon-release-pg16].peak_mem: 626 MB test_bulk_insert[neon-release-pg16].size: 0 MB test_bulk_insert[neon-release-pg16].data_uploaded: 1,922 MB test_bulk_insert[neon-release-pg16].num_files_uploaded: 8 test_bulk_insert[neon-release-pg16].wal_written: 1,382 MB test_bulk_insert[neon-release-pg16].wal_recovery: 18.981 s test_bulk_insert[neon-release-pg16].compaction: 0.055 s vs. tip of main: test_bulk_insert[neon-release-pg16].insert: 24.001 s test_bulk_insert[neon-release-pg16].pageserver_writes: 5,428 MB test_bulk_insert[neon-release-pg16].peak_mem: 604 MB test_bulk_insert[neon-release-pg16].size: 0 MB test_bulk_insert[neon-release-pg16].data_uploaded: 1,922 MB test_bulk_insert[neon-release-pg16].num_files_uploaded: 8 test_bulk_insert[neon-release-pg16].wal_written: 1,382 MB test_bulk_insert[neon-release-pg16].wal_recovery: 23.586 s test_bulk_insert[neon-release-pg16].compaction: 0.054 s ```

jcsp · 2024-10-25T11:34:03Z

This ticket is redundant now that #9329 is well underway.

jcsp added t/feature Issue type: feature, for new features or requests c/storage/pageserver Component: storage: pageserver labels Jul 22, 2024

skyzh mentioned this issue Jul 23, 2024

pageserver: faster and simpler inmem layer vec read #8469

Merged

5 tasks

This was referenced Aug 1, 2024

pageserver: add bench_ingest #7409

Merged

pageserver: batch InMemoryLayer puts, remove need to sort items by LSN during ingest #8591

Merged

jcsp mentioned this issue Aug 8, 2024

pageserver: add CompactKey, use it in InMemoryLayer #8652

Merged

5 tasks

jcsp closed this as completed Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pageserver: 2x ingest performance #8452

pageserver: 2x ingest performance #8452

jcsp commented Jul 22, 2024 •

edited

Loading

Tasks

jcsp commented Oct 25, 2024

pageserver: 2x ingest performance #8452

pageserver: 2x ingest performance #8452

Comments

jcsp commented Jul 22, 2024 • edited Loading

Tasks

jcsp commented Oct 25, 2024

jcsp commented Jul 22, 2024 •

edited

Loading