Test HEAD requests before GET

[hsanjuan]

The Trustless Gateway Spec requires supporting HEAD requests.

I have observed providers with HTTP-endpoint that return 500 on HEAD requests, while returning content with GET requests for the same URL.

It is important that HEAD requests are supported as it allows opportunistic content discovery without causing unnecessary load on the providers.

For example, in general we can use HEAD to test if multiple providers have the content they are announcing. We can then use GET to fetch according to latency measurements, or any scoring mechanism.

In the context of Spark, we can expand tests by checking the presence of many more CIDs and randomly picking which ones to fully retrieve.

[bajtos]

Thank you for the suggestion, @hsanjuan.

As I understand it, you would like to add a third check to the existing two we have in place:

1. The CID advertised to IPNI.
2. [NEW] If the protocol is HTTP, the provider responds with 200 for a HEAD request.
3. The root block of the content identified by CID can be retrieved and matches the hash in CID.

Is that correct?

[bajtos]

To maintain backward compatibility, it may be better to run the HEAD check only if the retrieval check passes and also to report the outcome of HEAD checks as a new metric distinct from RSR.