SPARK M2 Lassie Retrievals

[juliangruber]

eta: 2023-06-30

Replace the code making HTTP requests to IPFS Gateway with a retrieval client like Lassie.

Important: Retrieval requests from this module should be indistinguishable from “legit” requests made by other actors in the network (e.g. Saturn). Otherwise SPs can prioritise checker requests over regular traffic.

Children:

• Write a Go/FFI shim to start & stop the Lassie HTTP server. Write Rust bindings for this shim and make sure we can use this from Rust! We will need to configure tooling for CI/CD and also write tests! [5 days]
  → feat: proof of concept rusty-lassie#1
• Integrate Lassie into Zinnia (both zinnia and zinniad). (Start Lassie on start, stop it on exit.) [2 day]
  • We want to configure Lassie to use Module’s CACHE_DIR as the temp dir. Fortunately, there is already such an option exposed.
    → feat: IPFS retrieval client zinnia#243
• Implement Zinnia JS API for retrieving content for a given CID, optionally allowing the caller to specify the address and protocol to use. [3 days]
  • E.g. we can add a small fetch wrapper that will translate URLs like ipfs://bafy... into Lassie HTTP GET requests.
  • Maybe it’s better to create our own API that will allow us to support additional request parameters like the peer address and the protocol to use. This new API can return fetch response under the hood.
    → feat: IPFS retrieval client zinnia#243
• Lassie improvements [2 days]
  • Implement authentication - only Zinnia requests should be handled by Lassie. Requests from other clients must be rejected.
  • We want to configure Lassie HTTP mode to not cache blocks between retrieval requests.
• Update SPARK to use the new API Lassie spark#5
• On zinniad start, delete all files in Lassie’s temp dir. These files may be created if Station exits Station while a CID retrieval request is in progress.
  → Delete Lassie temp dir when zinniad starts zinnia#245

[bajtos]

@juliangruber I think we forgot to add one more task to this milestone - update SPARK to use the new Zinnia/Lassie API.

In Lassie's HTTP spec, I can see that we can ask to retrieve from a specific provider(s) using a specific protocol(s).

I think the SPARK request URL will become something like this:

fetch(`ipfs://${cid}?protocols=bitswap&providers=/ip4/1.2.3.4/tcp/1234/tls/p2p/QmFoo`)

Do you remember if SPARK API includes the protocol and provider multi address when creating a new job? Do we have that data in our CID list?

[juliangruber]

Yes!

https://github.com/filecoin-station/spark-api/blob/2a288f4a1355f25ac7c76d30c50fa79ac87129e1/scripts/top-cids.js#L33

We have protocol and provider multi address available.

[bajtos]

@juliangruber adding you as an assignee for the SPARK related work 🙇🏻‍♂️

[bajtos]

Lassie improvements [2 days]

Implement authentication - only Zinnia requests should be handled by Lassie. Requests from other clients must be rejected.

Moved to CheckerNetwork/zinnia#268

We want to configure Lassie HTTP mode to not cache blocks between retrieval requests.

Moved to CheckerNetwork/zinnia#269

Closing this milestone as done.

[juliangruber]

@bajtos I've removed the newlines between the items, so that starmaps can properly parse the children. See https://filecoinproject.slack.com/archives/C0418TBQ9CH/p1687850671862689?thread_ts=1687165457.518709&cid=C0418TBQ9CH