diff --git a/ipip-template.md b/ipip-template.md index 11e761f92..2a4b15786 100644 --- a/ipip-template.md +++ b/ipip-template.md @@ -36,13 +36,6 @@ interoperable implementations. When modifying an existing specification file, this section should provide a summary of changes. When adding new specification files, list all of them. -## Test fixtures - -List relevant CIDs. Describe how implementations can use them to determine -specification compliance. This section can be skipped if IPIP does not deal -with the way IPFS handles content-addressed data, or the modified specification -file already includes this information. - ## Design rationale The rationale fleshes out the specification by describing what motivated @@ -67,6 +60,13 @@ Explain the security implications/considerations relevant to the proposed change Describe alternate designs that were considered and related work. +## Test fixtures + +List relevant CIDs. Describe how implementations can use them to determine +specification compliance. This section can be skipped if IPIP does not deal +with the way IPFS handles content-addressed data, or the modified specification +file already includes this information. + ### Copyright Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). diff --git a/src/http-gateways/path-gateway.md b/src/http-gateways/path-gateway.md index c3ee9e440..e23061ddb 100644 --- a/src/http-gateways/path-gateway.md +++ b/src/http-gateways/path-gateway.md @@ -21,6 +21,7 @@ editors: url: https://hacdias.com/ xref: - url + - trustless-gateway tags: ['httpGateways', 'lowLevelHttpGateways'] order: 0 --- @@ -214,11 +215,13 @@ These are the equivalents: - `format=cbor` → `Accept: application/cbor` - `format=ipns-record` → `Accept: application/vnd.ipfs.ipns-record` - +### `dag-scope` (request query parameter) + +Only used on CAR requests, same as :ref[dag-scope] from :cite[trustless-gateway]. + +### `entity-bytes` (request query parameter) + +Only used on CAR requests, same as :ref[entity-bytes] from :cite[trustless-gateway]. # HTTP Response @@ -592,7 +595,11 @@ The following response types require an explicit opt-in, can only be requested w - Raw Block (`?format=raw`) - Opaque bytes, see [application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw). - CAR (`?format=car`) - - Arbitrary DAG as a verifiable CAR file or a stream, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car). + - A CAR file or a stream that contains all blocks required to trustlessly verify the requested content path query, see [application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) and :cite[trustless-gateway]. + - **Note:** by default, block order in CAR response is not deterministic, + blocks can be returned in different order, depending on implementation + choices (traversal, speed at which blocks arrive from the network, etc). + An opt-in ordered CAR responses MAY be introduced in a future IPIP. - TAR (`?format=tar`) - Deserialized UnixFS files and directories as a TAR file or a stream, see :cite[ipip-0288]. - IPNS Record diff --git a/src/http-gateways/trustless-gateway.md b/src/http-gateways/trustless-gateway.md index 1a9c18ee5..4a8632898 100644 --- a/src/http-gateways/trustless-gateway.md +++ b/src/http-gateways/trustless-gateway.md @@ -4,7 +4,7 @@ description: > Trustless Gateways are a minimal subset of Path Gateways that allow light IPFS clients to retrieve data behind a CID and verify its integrity without delegating any trust to the gateway itself. -date: 2023-03-30 +date: 2023-06-20 maturity: reliable editors: - name: Marcin Rataj @@ -17,25 +17,33 @@ tags: ['httpGateways', 'lowLevelHttpGateways'] order: 1 --- -Trustless Gateway is a minimal _subset_ of :cite[path-gateway] +Trustless Gateway is a _subset_ of :cite[path-gateway] that allows light IPFS clients to retrieve data behind a CID and verify its integrity without delegating any trust to the gateway itself. The minimal implementation means: -- data is requested by CID, only supported path is `/ipfs/{cid}` -- no path traversal or recursive resolution, no UnixFS/IPLD decoding server-side - response type is always fully verifiable: client can decide between a raw block or a CAR stream +- no UnixFS/IPLD deserialization +- for CAR files: + - the behavior is identical to :cite[path-gateway] +- for raw blocks: + - data is requested by CID, only supported path is `/ipfs/{cid}` + - no path traversal or recursive resolution # HTTP API A subset of "HTTP API" of :cite[path-gateway]. -## `GET /ipfs/{cid}[?{params}]` +## `GET /ipfs/{cid}[/{path}][?{params}]` -Downloads data at specified CID. +Downloads verifiable data for the specified **immutable** content path. -## `HEAD /ipfs/{cid}[?{params}]` +Optional `path` is permitted for requests that specify CAR format (`application/vnd.ipld.car`). + +For RAW requests, only `GET /ipfs/{cid}[?{params}]` is supported. + +## `HEAD /ipfs/{cid}[/{path}][?{params}]` Same as GET, but does not return any payload. @@ -45,13 +53,13 @@ Downloads data at specified IPNS Key. Verifiable :cite[ipns-record] can be reque ## `HEAD /ipns/{key}[?{params}]` -same as GET, but does not return any payload. +Same as GET, but does not return any payload. # HTTP Request Same as in :cite[path-gateway], but with limited number of supported response types. -## HTTP Request Headers +## Request Headers ### `Accept` (request header) @@ -67,12 +75,174 @@ Below response types SHOULD to be supported: Gateway SHOULD return HTTP 400 Bad Request when running in strict trustless mode (no deserialized responses) and `Accept` header is missing. +## Request Query Parameters + +### :dfn[dag-scope] (request query parameter) + +Optional, `dag-scope=(block|entity|all)` with default value `all`, only available for CAR requests. + +Describes the shape of the DAG fetched the terminus of the specified path whose blocks +are included in the returned CAR file after the blocks required to traverse +path segments. + +- `block` - Only the root block at the end of the path is returned after blocks + required to verify the specified path segments. + +- `entity` - For queries that traverse UnixFS data, `entity` roughly means return + blocks needed to verify the terminating element of the requested content path. + For UnixFS, all the blocks needed to read an entire UnixFS file, or enumerate a UnixFS directory. + For all queries that reference non-UnixFS data, `entity` is equivalent to `block` + +- `all` - Transmit the entire contiguous DAG that begins at the end of the path + query, after blocks required to verify path segments + +When present, returned `Etag` must include unique prefix based on the passed scope type. + +### :dfn[entity-bytes] (request query parameter) + +The optional `entity-bytes=from:to` parameter is available only for CAR +requests. + +It implies `dag-scope=entity` and serves as a trustless equivalent of an HTTP +Range Request. + +When the terminating entity at the end of the specified content path: + +- can be interpreted as a continuous array of bytes (such as a UnixFS file), a + Gateway MUST return only the minimal set of blocks necessary to verify the + specified byte range of that entity. + + - When dealing with a sharded UnixFS file (`dag-pb`, `0x70`) and a non-zero + `from` value, the UnixFS data and `blocksizes` determine the + corresponding starting block for a given `from` offset. + +- cannot be interpreted as a continuous array of bytes (such as a DAG-CBOR/JSON + map or UnixFS directory), the parameter MUST be ignored, and the request is + equivalent to `dag-scope=entity`. + +Allowed values for `from` and `to` follow a subset of section 14.1.2 from +:cite[rfc9110], where they are defined as offset integers that limit the +returned blocks to only those necessary to satisfy the range `[from,to]`: + +- `from` value gives the byte-offset of the first byte in a range. +- `to` value gives the byte-offset of the last byte in the range; + that is, the byte positions specified are inclusive. + +The following additional values are supported: + +- `*` can be substituted for end-of-file + - `entity-bytes=0:*` is the entire file (a verifiable version of HTTP request for `Range: 0-`) +- Negative numbers can be used for referring to bytes from the end of a file + - `entity-bytes=-1024:*` is the last 1024 bytes of a file + (verifiable version of HTTP request for `Range: -1024`) + - It is also permissible (unlike with HTTP Range Requests) to ask for the + range of 500 bytes from the beginning of the file to 1000 bytes from the + end: `entity-bytes=499:-1000` + +A Gateway MUST augment the returned `Etag` based on the passed `entity-bytes`. + +A Gateway SHOULD return an HTTP 400 Bad Request error when the requested range +cannot be parsed as valid offset positions. + +In more nuanced error scenarios, a Gateway MUST return a valid CAR response +that includes enough blocks for the client to understand why the requested +`entity-bytes` was incorrect or why only a part of the requested byte range was +returned: + +- If the requested `entity-bytes` resolves to a range that partially falls + outside of the entity's byte range, the response MUST include the subset of + blocks within the entity's bytes. + - This allows clients to request valid ranges of the entity without needing + to know its total size beforehand, and it does not require the Gateway to + buffer the entire entity before returning the response. + +- If the requested `entity-bytes` resolves to a zero-length range or falls + fully outside of the entity's bytes, the response is equivalent to + `dag-scope=block`. + - This allows client to produce a meaningful error (e.g, in case of UnixFS, + leverage `Data.blocksizes` information present in the root `dag-pb` block). + +- In streaming scenarios, if a Gateway is capable of returning the root block + but lacks prior knowledge of the final component of the requested content + path being invalid or absent in the DAG, a Gateway SHOULD respond with HTTP 200. + - This behavior is a consequence of HTTP streaming limitations: blocks are + not buffered, by the time a related parent block is being parsed and + returned to the client, the HTTP status code has already been sent to the + client. + # HTTP Response Below MUST be implemented **in addition** to "HTTP Response" of :cite[path-gateway]. -## HTTP Response Headers +## Response Headers + +### `Content-Type` (response header) + +MUST be returned and include additional format-specific parameters when possible. + +If a CAR stream was requested, the response MUST include the parameter specifying CAR version. +For example: `Content-Type: application/vnd.ipld.car; version=1` ### `Content-Disposition` (response header) MUST be returned and set to `attachment` to ensure requested bytes are not rendered by a web browser. + +## Response Payload + +### Block Response + +An opaque bytes matching the requested block CID +([application/vnd.ipld.raw](https://www.iana.org/assignments/media-types/application/vnd.ipld.raw)). + +The Body hash MUST match the Multihash from the requested CID. + +### CAR Response + +A CAR stream for the requested +[application/vnd.ipld.car](https://www.iana.org/assignments/media-types/application/vnd.ipld.car) +content type, path and optional `dag-scope` and `entity-bytes` URL parameters. + +#### CAR version + +Value returned in +[`CarV1Header.version`](https://ipld.io/specs/transport/car/carv1/#header) +field MUST match the `version` parameter returned in `Content-Type` header. + +#### CAR roots + +The behavior associated with the +[`CarV1Header.roots`](https://ipld.io/specs/transport/car/carv1/#header) field +is not currently specified. + +Clients MAY ignore it. + +:::issue + +As of 2023-06-20, the behavior of the `roots` CAR field remains an [unresolved item within the CARv1 specification](https://web.archive.org/web/20230328013837/https://ipld.io/specs/transport/car/carv1/#unresolved-items). + +::: + +#### CAR determinism + +The default CAR header and block order in a CAR response is not specified and is non-deterministic. + +Clients MUST NOT assume that CAR responses are deterministic (byte-for-byte identical) across different gateways. + +Clients MUST NOT assume that CAR includes CIDs and their blocks in the same order across different gateways. + +:::issue + +In controlled environments, clients MAY choose to rely on undocumented CAR determinism, +subject to the agreement of the following conditions between the client and the +gateway: +- CAR version +- content of [`CarV1Header.roots`](https://ipld.io/specs/transport/car/carv1/#header) field +- order of blocks +- status of duplicate blocks + +In the future, there may be an introduction of a convention to indicate aspects +of determinism in CAR responses. Please refer to +[IPIP-412](https://github.com/ipfs/specs/pull/412) for potential developments +in this area. + +::: diff --git a/src/ipips/ipip-0402.md b/src/ipips/ipip-0402.md new file mode 100644 index 000000000..6164c02bb --- /dev/null +++ b/src/ipips/ipip-0402.md @@ -0,0 +1,395 @@ +--- +title: "IPIP-0402: Partial CAR Support on Trustless Gateways" +date: 2023-04-17 +ipip: proposal +editors: + - name: Hannah Howard + github: hannahhoward + - name: Adin Schmahmann + github: aschmahmann + - name: Rod Vagg + github: rvagg + - name: Marcin Rataj + github: lidel + url: https://lidel.org/ +relatedIssues: + - https://github.com/ipfs/specs/issues/348 + - https://github.com/ipfs/kubo/issues/8769 +order: 402 +tags: ['ipips'] +--- + +## Summary + +Add path and partial CAR response support to the :cite[trustless-gateway]. + +## Motivation + +:cite[trustless-gateway] solves the verifiability problem in HTTP contexts, +and allows for inexpensive retrieval and caching in a way that is not tied to +a specific service, library or IPFS implementation. + +This IPIP improves the performance related to trustless HTTP gateways by adding +additional capabilities to requests for CAR files. + +The goal is to enable a client capable of translating/decoding CAR files to +make a single request to a trustless gateway that in most case allows them to +render the same output generated via a request to a trusted gateway (or, if not +in a single request, as few requests as possible). + +Save round-trips, allow more efficient resume and parallel downloads. + +## Detailed design + +The solution is to allow the :cite[trustless-gateway] to support partial +responses by: + +- allowing for requesting sub-paths within a DAG, and getting blocks necessary + for traversing all path segments for end-to-end verification + +- opt-in `dag-scope` parameter that allows for narrowing down returned blocks + to a `block`, `entity` (a logical IPLD entity, such as a file, directory, + CBOR document), or `all` (default) + +- opt-in `entity-bytes` parameter that allows for returning only a subset of blocks + within a logical IPLD entity + +Details are in :cite[trustless-gateway]. + +## Design rationale + +The approach here is pragmatic: we add a minimum set of predefined CAR export +scopes based on real world needs, as a product of the rough consensus across +key stakeholders: gateway operators, Project Saturn, Project Rhea (`ipfs.io` +and `dweb.link`), light clients such as Capyloon and IPFS in Chromium, and +gateway implementation in `boxo/gateway` library (Go). + +Terse rationale for each feature: + +- Including blocks necessary for traversing parents when a sub-path is present + makes a better default, as it produces verifiable archive that does not + require any follow-up requests. The response is always enough to verify + end-to-end and reject any unexpected / invalid blocks. + +- The ability to narrow down CAR response based on logical scope or specific byte + range within an entity comes directly from the types of requests existing + path gateways need to handle. + - `dag-scope=block` allows for resolving content paths to the final CID, and + learn its type (unixfs file/directory, or a custom codec) + - `dag-scope=entity` covers the majority of website hosting needs (returning a + file, enumerating directory contents, or any other IPLD entity) + - `dag-scope=all` returns all blocks in a DAG: was the existing behavior and + remains the implicit default + - `entity-bytes=from:to` enables efficient, verifiable analog to HTTP Range Requests + (resuming downloads or seeking within bigger files, such as videos) + - `from` and `to` match the behavior of HTTP Range Requests. + +### User benefit + +- Trustless HTTP Clients will be able to fetch a CAR with a file, byte range, + or a directory enumeration using a way lower number of HTTP requests, which + will translate to improved resource utilization, longer battery time on + mobile, and lower latency due to lower number of round trips. + +- CAR files downloaded from HTTP Gateways will always be end-to-end verifiable. + In the past, user had to manually ensure they have blocks for all path + segments. With this IPIP, the CAR will always include parent blocks when a + file located on a sub-path is requested. + +- Creating a standard way of fetching partial CAR over HTTP enables a diverse set of clients and gateway services to interoperate, and reuse libraries: + - [Service Worker Gateway based on Helia](https://github.com/ipfs-shipyard/helia-service-worker-gateway) + - [IPFS in Chromium](https://github.com/ipfs/ipfs-blog/pull/560/files) + - [boxo/gateway](https://github.com/ipfs/boxo/tree/main/gateway) provides HTTP Gateway implementation for + - [Kubo](https://github.com/ipfs/kubo) + - [HTTP retrieval in Boost](https://boost.filecoin.io/http-retrieval) + - [bifrost-gateway](https://github.com/ipfs/bifrost-gateway) + +- Trustless Gateway is solidified as the ecosystem wide standard. + + - IPIP tests added to + [gateway-conformance](https://github.com/ipfs/gateway-conformance) test + suite and fixtures listed at the end of this IPIP make it easier to + implement or operate a conformant gateway, and reduce maintenance costs. + + - End users are empowered with primitives and tools that reduce retrieval + cost, encourage self-hosting, or make validation of conformance claims of + free or commercial gateways possible. + +### Compatibility + +#### CAR responses with blocks for parent path segments + +In order to serve CAR requests for content paths other than just a CID root in +a trustless manner, we are requiring the gateway to return intermediate blocks +from the CID root to the path terminus as part of the returned CAR file. + +HTTP Gateway implementations are currently only returning blocks starting at +the end of the content path, which means an implementation of this IPIP will +introduce additional blocks required for verifying. + +As long the client was written in a trustless manner, and follows ring and was discarding +unexpected blocks, this will be a backward-compatible change. + +#### CAR format with `entity-bytes` and `dag-scope` parameters + +These parameters are opt-in, which means no breaking changes. + +Gateways ignore unknown URL parameters. A client sending them to a +gateway that does not implement this IPIP will get all blocks for the requested +DAG. + +#### CAR roots and determinism + +As of 2023-06-20, the behavior of the `roots` CAR field remains an [unresolved item within the CARv1 specification](https://web.archive.org/web/20230328013837/https://ipld.io/specs/transport/car/carv1/#unresolved-items): + +> Regarding the roots property of the Header block: +> +> - The current Go implementation assumes at least one CID when creating a CAR +> - The current Go implementation requires at least one CID when reading a CAR +> - The current JavaScript implementation allows for zero or more roots +> - Current usage of the CAR format in Filecoin requires exactly one CID +> +> [..] +> +> It is unresolved how the roots array should be constrained. It is recommended +> that only a single root CID be used in this version of the CAR format. +> +> A work-around for use-cases where the inclusion of a root CID is difficult +> but needing to be safely within the "at least one" recommendation is to use +> an empty CID: `\x01\x55\x00\x00` (zero-length "identity" multihash with "raw" +> codec). Since current implementations for this version of the CAR +> specification don't check for the existence of root CIDs +> (see [Root CID block existence](https://web.archive.org/web/20230328013837/https://ipld.io/specs/transport/car/carv1/#root-cid-block-existence)), +> this will be safe as far as CAR implementations are concerned. However, there +> is no guarantee that applications that use CAR files will correctly consume +> (ignore) this empty root CID. + +Due to the inconsistent and non-deterministic nature of CAR implementations, +the gateway specification faces limitations in providing specific +recommendations. Nevertheless, it is crucial for implementations to refrain +from making implicit assumptions based on the legacy behavior of individual CAR +implementations. + +Due to this, gateway specification changes introduced in this IPIP clarify that: +- The CAR `roots` behavior is out of scope and flags that clients MAY ignore it. +- CAR determinism is not present by default, responses may differ across + requests and gateways. +- Opt-in determinism is possible, but standardized signaling mechanism does not + exist until we have IPIP-412 or similar. + +### Security + +This IPIP allows clients to narrow down the amount of data returned as a CAR, +and introduces a need for defensive programming when the feature set of the +remote gateway is unknown. + +To avoid denial of service, and resource starvation, clients should probe if +the gateway supports features described in this IPIP before requesting data, to +avoid fetching big DAGs when only a small subset of blocks is expected. + +Following the robustness principle, invalid, duplicate or unexpected blocks +should be discarded. + +### Alternatives + +Below are alternate designs that were considered, but decided to be out of scope for this IPIP. + +#### Arbitrary IPLD Selectors + +Passing arbitrary selectors was rejected due to the implementation complexity, +risks, and weak value proposition, as [discussed during IPFS Thing 2022](https://github.com/ipfs/specs/issues/348#issuecomment-1326869509) + +#### Additional "Web" Scope + +A request for +`/ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze/wiki/?format=car&dag-scope=entity` +returns all blocks required for enumeration of the big HAMT `/wiki` directory, +and then an additional request for `index.html` needs to be issued. + +Website hosting use case could be made more efficient if gateway returned a CAR +with `index.html` instead of all blocks for directory enumeration. The server +already did the work: it knows the entity is a directory, already parsed it, it +knows it has child entity named `index.html`, and everyone would pay a lower cost due +to lower number of blocks being returned in a single round-trip, instead of two. + +Rhea/Saturn projects requested this to be out of scope for now, but this "web" +entity scope could be added in the future, as a follow-up optimization IPIP. + +#### Requesting specific DAG depth + +Blindly requesting specific DAG depth did not translate to any type of +requests web gateways like `ipfs.io` or one in Brave browser have to handle. + +It is impossible to know if some entity on a sub-path is a file or a directory, +without sending a probe for the root block, which introduces one round-trip overhead +per entity. + +This problem is not present in the case of `dag-scope=entity`, which shifts the +decision to the server, and allows for fetching unknown UnixFS entity with a +single request. + +## Test fixtures + +Relevant tests were added to +[gateway-conformance](https://github.com/ipfs/gateway-conformance) test suite +in [#56](https://github.com/ipfs/gateway-conformance/pull/56) and +[#85](https://github.com/ipfs/gateway-conformance/issues/85). +A detailed list of compliance checks for `dag-scope` and `entity-bytes` can be found in +[`v0.2.0/trustless_gateway_car_test.go`](https://github.com/ipfs/gateway-conformance/blob/v0.2.0/tests/trustless_gateway_car_test.go) or later. + +Below are CIDs, CARs, and short summary of each fixture. + +### Testing pathing + +The main utility of this scope is saving round-trips when retrieving a specific +entity as a member of a bigger DAG. + +To test, request a small file that fits in a single block from a sub-path. The +returned CAR MUST include both the block with the file data and all blocks +necessary for traversing from the root CID to the terminating element (all +parents, root CID and a subdirectory below it). + +:::example + +Sample fixtures: + +- `bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu` + from [`subdir-with-two-single-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-two-single-block-files.car) + for testing `/ipfs/dag-pb-cid/subdir/ascii.txt?format=car` + (UnixFS file in a subdirectory) + +- `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i` + from [`single-layer-hamt-with-multi-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/single-layer-hamt-with-multi-block-files.car) + for testing `/ipfs/dag-pb-hamt-cid/686.txt?format=car` + (UnixFS file on a path within HAMT-sharded parent directory) + +- `bafybeia264q44a3kmfc2otctzu4egp2k235o3t7mslz2yjraymp4nv6asi` + from [`dir-with-dag-cbor-with-links.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/dir-with-dag-cbor-with-links.car) + for testing `/ipfs/dag-cbor-cid/document?format=car` + (UnixFS file on a path with DAG-CBOR root CID) + +::: + +### Testing `dag-scope=block` + +The main utility of this scope is resolving content paths. This means a CAR +response with blocks related to path traversal, and the root block of the +terminating entity. + +To test real world use, request UnixFS `file` or a `directory` from a sub-path. +The returned CAR MUST include blocks required for path traversal and ONLY the +root block of the terminating entity. + +:::example + +Sample fixtures: + +- `bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu` + from [`subdir-with-two-single-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-two-single-block-files.car) + for testing + - `/ipfs/dag-pb-cid/subdir/ascii.txt?format=car&dag-scope=block` (UnixFS file in a subdirectory) + - `/ipfs/dag-pb-cid?format=car&dag-scope=block` (UnixFS directory) + +- `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i` + from [`single-layer-hamt-with-multi-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/single-layer-hamt-with-multi-block-files.car) + for testing: + - `/ipfs/dag-pb-hamt-cid/1.txt?format=car&dag-scope=block` + (UnixFS multi-block file on a path within HAMT-sharded parent directory) + - `/ipfs/dag-pb-hamt-cid?format=car&dag-scope=block` + (UnixFS HAMT-sharded directory) + +::: + +### Testing `dag-scope=entity` + +The main utility of this scope is retrieving all blocks related to a meaningful +IPLD entity. Currently, the most popular entity types are: + +- UnixFS `file` + (blocks for all chunks with file data) + +- UnixFS `directory` + (blocks for the directory node, allowing its enumeration; + no root blocks for any of the child entities). + +- `raw` / `dag-cbor` + (block with raw data or DAG-CBOR document, potentially linking to other CIDs) + +:::example + +Sample fixtures: + +- `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu` + from [`subdir-with-mixed-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-mixed-block-files.car) + for testing: + - `/ipfs/dag-pb-cid/subdir/multiblock.txt?format=car&dag-scope=entity` (UnixFS multi-block file in a subdirectory) + - `/ipfs/dag-pb-cid/subdir?format=car&dag-scope=entity` (UnixFS directory) + +- `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i` + from [`single-layer-hamt-with-multi-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/single-layer-hamt-with-multi-block-files.car) + for testing: + - `/ipfs/dag-pb-hamt-cid/1.txt?format=car&dag-scope=entity` + (UnixFS multi-block file on a path within HAMT-sharded parent directory, returned blocks MUST be enough to deserialize the file) + - `/ipfs/dag-pb-hamt-cid?format=car&dag-scope=entity` + (UnixFS HAMT-sharded directory, response MUST include the minimal set of blocks required for enumeration of directory contents, and no blocks that belong to child entities) + +- `bafybeia264q44a3kmfc2otctzu4egp2k235o3t7mslz2yjraymp4nv6asi` + from [`dir-with-dag-cbor-with-links.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/dir-with-dag-cbor-with-links.car) + for testing `/ipfs/dag-cbor-cid/document?format=car&dag-scope=entity` + (DAG-CBOR document with IPLD Links must return all necessary blocks to verify the path, the document itself, but not the content behind any of the child entity IPLD Links) + +::: + +### Testing `dag-scope=all` + +This is the implicit default used when `dag-scope` is not present, +and used in the context of deserialized UnixFS TAR responses from :cite[ipip-0288]. + +:::example + +Sample fixtures: + +- `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu` + from [`subdir-with-mixed-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-mixed-block-files.car) + for testing: + - `/ipfs/dag-pb-cid/subdir?format=car&dag-scope=all` (path parents and the entire UnixFS subdirectory, returned recursively) + - `/ipfs/dag-pb-cid/subdir/multiblock.txt?format=car&dag-scope=all` (path parents and the entire UnixFS multi-block file) + +::: + +### Testing `entity-bytes=from:to` + +This type of CAR response is used for facilitating HTTP Range Requests and +byte seek within bigger entities. + +:::warning + +Properly testing this type of response requires synthetic DAG that is only +partially retrievable. This ensures systems that perform internal caching +won't pass the test due to the entire DAG being precached, or fetched in full. + +::: + +Use of the below fixture is highly recommended: + +:::example + +- `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu` + from [`subdir-with-mixed-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/subdir-with-mixed-block-files.car) + for testing: + - `/ipfs/dag-pb-cid/subdir/multiblock.txt?format=car&dag-scope=entity&entity-bytes=0:*` (path blocks and all the blocks for the multi-block UnixFS file) + - `multiblock.txt?format=car&dag-scope=entity&entity-bytes=512:1023` (path blocks and all the blocks for the the range request within multi-block UnixFS file) + - `multiblock.txt?format=car&dag-scope=entity&entity-bytes=512:-256` (path blocks and all the blocks for the the range request within multi-block UnixFS file) + - `/ipfs/dag-pb-cid/subdir?format=car&dag-scope=entity&entity-bytes=0:*` (path blocks and all the blocks to enumerate UnixFS directory) + +- `QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk` + from [`file-3k-and-3-blocks-missing-block.car`](https://github.com/ipfs/gateway-conformance/raw/v0.2.0/fixtures/trustless_gateway_car/file-3k-and-3-blocks-missing-block.car) + for testing: + - `/ipfs/dag-pb-cid?format=car&dag-scope=entity&entity-bytes=0:1000` (only the blocks needed to fulfill the request, MUST succeed despite the fact that a block after the range is not retrievable) + - `/ipfs/dag-pb-cid?format=car&dag-scope=entity&entity-bytes=2200:*` (only the blocks needed to fulfill the request, MUST succeed despite the fact that a block before the range is not retrievable) + +::: + +### Copyright + +Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).