pageserver: issue concurrent IO on the read path

[VladLazar]

Refs

• Epic: pageserver: concurrent IO on the get page read path #9378

Co-authored-by: Vlad Lazar [email protected]
Co-authored-by: Christian Schwarz [email protected]

Problem

The read path does its IOs sequentially.
This means that if N values need to be read to reconstruct a page,
we will do N IOs and getpage latency is O(N*IoLatency).

Solution

With this PR we gain the ability to issue IO concurrently within one
layer visit and to move on to the next layer without waiting for IOs
from the previous visit to complete.

This is an evolved version of the work done at the Lisbon hackathon,
cf #9002.

Design

will_init now sourced from disk btree index keys

On the algorithmic level, the only change is that the get_values_reconstruct_data
now sources will_init from the disk btree index key (which is PS-page_cache'd), instead
of from the Value, which is only available after the IO completes.

Concurrent IOs, Submission & Completion

To separate IO submission from waiting for its completion, while simultaneously
feature-gating the change, we introduce the notion of an IoConcurrency struct
through which IO futures are "spawned".

An IO is an opaque future, and waiting for completions is handled through
tokio::sync::oneshot channels.
The oneshot Receiver's take the place of the img and records fields
inside VectoredValueReconstructState.

When we're done visiting all the layers and submitting all the IOs along the way
we concurrently collect_pending_ios for each value, which means
for each value there is a future that awaits all the oneshot receivers
and then calls into walredo to reconstruct the page image.
Walredo is now invoked concurrently for each value instead of sequentially.
Walredo itself remains unchanged.

The spawned IO futures are driven to completion by a sidecar tokio task that
is separate from the task that performs all the layer visiting and spawning of IOs.
That tasks receives the IO futures via an unbounded mpsc channel and
drives them to completion inside a FuturedUnordered.

(The behavior from before this PR is available through IoConcurrency::Sequential,
which awaits the IO futures in place, without "spawning" or "submitting" them
anywhere.)

Alternatives Explored

A few words on the rationale behind having a sidecar task and what
alternatives were considered.

One option is to queue up all IO futures in a FuturesUnordered that is polled
the first time when we collect_pending_ios.

Firstly, the IO futures are opaque, compiler-generated futures that need
to be polled at least once to submit their IO. "At least once" because
tokio-epoll-uring may not be able to submit the IO to the kernel on first
poll right away.

Second, there are deadlocks if we don't drive the IO futures to completion
independently of the spawning task.
The reason is that both the IO futures and the spawning task may hold some
and try to acquire more shared limited resources.
For example, both spawning task and IO future may try to acquire

• a VirtualFile file descriptor cache slot async mutex (observed during impl)
• a tokio-epoll-uring submission slot (observed during impl)
• a PageCache slot (currently this is not the case but we may move more code into the IO futures in the future)

Another option is to spawn a short-lived tokio::task for each IO future.
We implemented and benchmarked it during development, but found little
throughput improvement and moderate mean & tail latency degradation.
Concerns about pressure on the tokio scheduler made us discard this variant.

The sidecar task could be obsoleted if the IOs were not arbitrary code but a well-defined struct.
However,

1. the opaque futures approach taken in this PR allows leaving the existing
   code unchanged, which
2. allows us to implement the IoConcurrency::Sequential mode for feature-gating
   the change.

Once the new mode sidecar task implementation is rolled out everywhere,
and ::Sequential removed, we can think about a descriptive submission & completion interface.
The problems around deadlocks pointed out earlier will need to be solved then.
For example, we could eliminate VirtualFile file descriptor cache and tokio-epoll-uring slots.
The latter has been drafted in neondatabase/tokio-epoll-uring#63.

See the lengthy doc comment on spawn_io() for more details.

Error handling

There are two error classes during reconstruct data retrieval:

• traversal errors: index lookup, move to next layer, and the like
• value read IO errors

A traversal error fails the entire get_vectored request, as before this PR.
A value read error only fails that value.

In any case, we preserve the existing behavior that once
get_vectored returns, all IOs are done. Panics and failing
to poll get_vectored to completion will leave the IOs dangling,
which is safe but shouldn't happen, and so, a rate-limited
log statement will be emitted at warning level.
There is a doc comment on collect_pending_ios giving more code-level
details and rationale.

Feature Gating

The new behavior is opt-in via pageserver config.
The Sequential mode is the default.
The only significant change in Sequential mode compared to before
this PR is the buffering of results in the oneshots.

Code-Level Changes

Prep work:

• Make GateGuard clonable.

Core Feature:

• Traversal code: track will_init in BlobMeta and source it from
  the Delta/Image/InMemory layer index, instead of determining will_init
  after we've read the value. This avoids having to read the value to
  determine whether traversal can stop.
• Introduce IoConcurrency & its sidecar task.
  • IoConcurrency is the clonable handle.
  • It connects to the sidecar task via an mpsc.
• Plumb through IoConcurrency from high level code to the
  individual layer implementations' get_values_reconstruct_data.
  We piggy-back on the ValuesReconstructState for this.
  • The sidecar task should be long-lived, so, IoConcurrency needs
    to be rooted up "high" in the call stack.
  • Roots as of this PR:
    • page_service: outside of pagestream loop
    • create_image_layers: when it is called
    • basebackup(only auxfiles + replorigin + SLRU segments)
  • Code with no roots that uses IoConcurrency::sequential
    • any Timeline::get call
      • collect_keyspace is a good example
      • follow-up: move IoConcurrency plumbing to RequestContext & integrate with TenantHarness #10460
    • TimelineAdaptor code used by the compaction simulator, unused in practive
    • ingest_xlog_dbase_create
• Transform Delta/Image/InMemoryLayer to
  • do their values IO in a distinct async {} block
  • extend the residence of the Delta/Image layer until the IO is done
  • buffer their results in a oneshot channel instead of straight
    in ValuesReconstructState
  • the oneshot channel is wrapped in OnDiskValueIo / OnDiskValueIoWaiter
    types that aid in expressiveness and are used to keep track of
    in-flight IOs so we can print warnings if we leave them dangling.
• Change ValuesReconstructState to hold the receiving end of the
  oneshot channel aka OnDiskValueIoWaiter.
• Change get_vectored_impl to collect_pending_ios and issue walredo concurrently, in a FuturesUnordered.

Testing / Benchmarking:

• Support queue-depth in pagebench for manual benchmarkinng.
• Add test suite support for setting concurrency mode ps config
  field via a) an env var and b) via NeonEnvBuilder.
• Hacky helper to have sidecar-based IoConcurrency in tests.
  This will be cleaned up later.

More benchmarking will happen post-merge in nightly benchmarks, plus in staging/pre-prod.
Some intermediate helpers for manual benchmarking have been preserved in #10466 and will be landed in later PRs.
(L0 layer stack generator!)

Drive-By:

• test suite actually didn't enable batching by default because
  config.compatibility_neon_binpath is always Truthy in our CI environment
  => https://neondb.slack.com/archives/C059ZC138NR/p1737490501941309
• initial logical size calculation wasn't always polled to completion, which was
  surfaced through the added WARN logs emitted when dropping a
  ValuesReconstructState that still has inflight IOs.
• remove the timing histograms pageserver_getpage_get_reconstruct_data_seconds
  and pageserver_getpage_reconstruct_seconds because with planning, value read
  IO, and walredo happening concurrently, one can no longer attribute latency
  to any one of them; we'll revisit this when Vlad's work on tracing/sampling
  through RequestContext lands.
• remove code related to get_cached_lsn().
  The logic around this has been dead at runtime for a long time,
  ever since the removal of the materialized page cache in remove materialized page cache #8105.

Testing

Unit tests use the sidecar task by default and run both modes in CI.
Python regression tests and benchmarks also use the sidecar task by default.
We'll test more in staging and possibly preprod.

Future Work

Please refer to the parent epic for the full plan.

The next step will be to fold the plumbing of IoConcurrency
into RequestContext so that the function signatures get cleaned up.

Once Sequential isn't used anymore, we can take the next
big leap which is replacing the opaque IOs with structs
that have well-defined semantics.

[github-actions]

7370 tests run: 6985 passed, 0 failed, 385 skipped (full report)

---

Flaky tests (7)

Postgres 17

• test_isolation[None]: release-x86-64
• test_scrubber_physical_gc_ancestors[None]: debug-x86-64
• test_scrubber_tenant_snapshot[4]: debug-x86-64

Postgres 16

• test_metrics_normal_work: release-x86-64

Postgres 15

• test_metrics_normal_work: release-x86-64, release-arm64

Postgres 14

• test_metrics_normal_work: release-x86-64

Code coverage* (full report)

• functions: 33.5% (8493 of 25334 functions)
• lines: 49.3% (71391 of 144677 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
b0b9206 at 2025-01-22T14:39:32.695Z :recycle:}

[erikgrinaker]

As of this commit, one IO failure does not stop any other IO requests. When awaiting for the IOs to complete, we stop waiting on the first failure, but we do not signal any other pending IOs to complete and they will just fail silently.

Is this true? It really depends on how the IO futures are implemented, but in general, dropping a future should cancel the in-flight operation and stop polling it. Assuming they're implemented that way, it should be sufficient to ensure that the caller receives the error as soon as it happens and then drops the in-flight futures by returning the error. I don't think we need any synchronization beyond that, or am I missing something?

[erikgrinaker]

Still reviewing, just flushing some comments for now. All nits, take them or leave them.

[VladLazar]

As of this commit, one IO failure does not stop any other IO requests. When awaiting for the IOs to complete, we stop waiting on the first failure, but we do not signal any other pending IOs to complete and they will just fail silently.

Is this true? It really depends on how the IO futures are implemented, but in general, dropping a future should cancel the in-flight operation and stop polling it. Assuming they're implemented that way, it should be sufficient to ensure that the caller receives the error as soon as it happens and then drops the in-flight futures by returning the error. I don't think we need any synchronization beyond that, or am I missing something?

I missed this question.

All "IO futures" are collected in VectoredValueReconstructState::on_disk_values. With this PR, the read path does not wait for the outcome of one IO before issuing the next one. Hence, at the end of the layer traversal, we will have created "IO futures" for all required values. Each "IO future" is independent. If the first one fails, the others are still present.

Let's also consider what happens when all the "IO futures" after the failed one are not complete. We bail out in collect_pending_ios, but the tasks for all the incomplete IOs still run since we don't call abort on the JoinHandle or have cancellation wired in.

[erikgrinaker]

I've done another pass to check that there aren't any issues with ordering or races, and I can't see any -- even though we dispatch IOs concurrently, we always access the results in a predetermined order.

I think this should be good to go, once we resolve the tasks vs. futures discussion above.

[problame]

I remain irritated by the CI logs now showing =SidecarTask.

Asked devprod team on Slack: https://neondb.slack.com/archives/C059ZC138NR/p1737490501941309

[VladLazar]

🚢

[Bodobolero]

I verified that the two additional rust tests only add 1 minute to build time which should be ok.