pageserver: handle decompression outside vectored read_blobs

[yliang412]

Part of #8130.

Problem

Currently, decompression is performed within the read_blobs implementation and the decompressed blob will be appended to the end of the BytesMut buffer. We will lose this flexibility of extending the buffer when we switch to using our own dio-aligned buffer (WIP in #8730). To facilitate the adoption of aligned buffer, we need to refactor the code to perform decompression outside read_blobs.

Summary of changes

• VectoredBlobReader::read_blobs will return VectoredBlob without performing decompression and appending decompressed blob. It becomes the caller's responsibility to decompress the buffer.
• Added a new BufView type that functions as Cow<Bytes, &[u8]>.
• Perform decompression within VectoredBlob::read so that people don't have to explicitly thinking about compression when using the reader interface.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

4986 tests run: 4822 passed, 0 failed, 164 skipped (full report)

---

Flaky tests (6)

Postgres 17

• test_pageserver_compaction_smoke: release-arm64
• test_subscriber_synchronous_commit: release-arm64
• test_pg_regress[4]: debug-x86-64
• test_readonly_node_gc: debug-x86-64

Postgres 16

• test_ondemand_wal_download_in_replication_slot_funcs: release-x86-64

Postgres 15

• test_ondemand_wal_download_in_replication_slot_funcs: release-x86-64

Code coverage* (full report)

• functions: 32.1% (7475 of 23255 functions)
• lines: 50.0% (60227 of 120574 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
67bd614 at 2024-09-24T16:56:24.080Z :recycle:}

[problame]

Regarding @koivunej's #8942 (comment): I think if decompress took a Cow<Bytes> instead of the Option<Bytes> it would make the consuming code paths more concise.

[arpad-m]

I chose appending to the buffer in read_blobs because I didn't want to make compression someone has to explicitly think about when using the reader interface.

Because if it is something someone has to explicitly think about, mistakes can happen, and people can forget about it, resulting in the code working on compressed data. And it is more cumbersome to work with the API.

Therefore, I agree with Christian's suggestion to add a function to VectoredBlob to allow reading of the blobs, and returning something like Cow<Bytes, &[u8]> (or a custom enum if that doesn't work).

Ideally one would make the members of VectoredBlob private so that one can't mistakenly forget about calling the decompress function on it.

[orca-security-us]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca

[arpad-m]

Much better now, thanks for adjusting it. It's well done work.