Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initial proposal for graphsync support in ipfs #78

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

initial proposal for graphsync support in ipfs #78

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions proposals/ipfs-graphsync.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# IPFS nodes speak Graphsync

Authors: @willscott

Initial PR: TBD <!-- Reference the PR first proposing this document. Oooh, self-reference! -->

<!--
This template is for a proposal/brief/pitch for a significant project to be undertaken by a Web3 Dev project team.
The goal of project proposals is to help us decide which work to take on, which things are more valuable than other things.
-->
<!--
A proposal should contain enough detail for others to understand how this project contributes to our team’s mission of product-market fit
for our unified stack of protocols, what is included in scope of the project, where to get started if a project team were to take this on,
and any other information relevant for prioritizing this project against others.
It does not need to describe the work in much detail. Most technical design and planning would take place after a proposal is adopted.
Good project scope aims for ~3-5 engineers for 1-3 months (though feel free to suggest larger-scoped projects anyway).
Projects do not include regular day-to-day maintenance and improvement work, e.g. on testing, tooling, validation, code clarity, refactors for future capability, etc.
-->
<!--
For ease of discussion in PRs, consider breaking lines after every sentence or long phrase.
-->

## Purpose &amp; impact
#### Background &amp; intent

Currently, ipfs nodes speak [bitswap](https://docs.ipfs.io/concepts/bitswap/) for exchanging blocks of data. bitswap works at a block layer where you ask for a CID, and get back the block that hashes to that cid.

[graphsync](https://github.com/ipfs/go-graphsync) is a protocol for synchronizing graphs of data across peers. It allows a host to make a single request to a remote peer for all of the results of traversing an IPLD selector on the remote peer's local IPLD graph. It is what filecoin uses.

#### Assumptions &amp; hypotheses

* There are cases where users will want to retrieve a piece of structured data they can efficiently describe with a selector, but which would be expensive to retrieve with bitswap.
* There are cases where it would be valuable for an IPFS node and a filecoin node to exchange data.

#### User workflow example

* A user would ask for data using a selector in e.g. the [go-fetcher](https://github.com/ipfs/go-fetcher) interface within IPFS (from a plugin)
* An API is built such that dag exploration of ipld nodes from a client built on IPFS is translated into selectors
* a client builds a car of data to store, creates a filecoin deal for it, and the miner fetches the data from an IPFS client with graphsync

#### Impact

High. we invision cases where we want IPFS and filecoin nodes to be able to communicate
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is the main goal, I'm not sure this proposal is right.

I'm trying to think through this use case: "a client builds a car of data to store, creates a filecoin deal for it, and the miner fetches the data from an IPFS client with graphsync"

Can you explain in more detail what this means? So I have all the data, I make a car file, and then I make a filecoin deal for it -- why don't I just send to the miner with data transfer and graphsync?

What's the use case for the miner fetching with graphsync via an IPFS client? Moreover, why would a miner run a full instance of IPFS when IPFS already supports retrieving via graphsync as an experimental feature -- it seems to me all of the software then is just built into the filecoin mining software-- they don't run a whole IPFS node.

Again, this is a fine project, but as specified the impact is to IPFS data transfer speeds. it really doesn't unlock much on its own for filecoin miners getting data from IPFS.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, Filecoin doesn't speak Graphsync: It speaks data transfer on top of graphsync. Just supporting graphsync as a protocol is not enough on its own to unlock Filecoin transfer. IPFS needs to have an integrated data transfer instance to do what I think you want to do here. But again, I am not sure I understand the use case.


#### Leverage

High. we have other projects that are enabled by IPFS and filecoin nodes being able to communicate

#### Confidence

Two. We haven't heard direct requests for this flow, but it's our best guess at what's practical.


## Project definition
#### Brief plan of attack

[go-merkledag](https://github.com/ipfs/go-merkledag) currently takes a [blockservice](https://github.com/ipfs/go-merkledag/blob/bf51443272bb98cff071eb44ed9ce6c940e82f1f/merkledag.go#L32). It should also be able to be created from a graphsync provider, or both. We can initially enable graphsync as an opt-in option, and need to design a transition (e.g. do we have merkledag make requests to both graphsync and to blockservice when present? do we make this code somewhat peer-aware, so we know based on provider which one to use?)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR: In trying to make GraphSync usable in go-ipfs having an interface that handles both bitswap and graphsync backed DAGServices is the easy part. Content discovery, spreading requests across peers, prioritizing peers + protocols, etc. are the hard parts.

IIUC the main areas GraphSync is lacking in order to be a drop in replacement for Bitswap are:

  • We have no code (or AFAIK proposal yet) for distributing data across multiple peers using GraphSync
  • Graphsync is not on its own suitable for peer discovery in the way Bitswap is (e.g. it may be less efficient due to being unable to ask for HAVEs)

From what I see this limits our options, and some remaining ones are:

  • Make GraphSync able to be a drop-in replacement:
    • Extract sessions code from Bitswap/create a way for sessions to be managed across Bitswap and GraphSync
    • Make sessions smart enough to split queries across peers with GraphSync, or something simpler like trying to use GraphSync only if we fail at using Bitswap for long enough
    • Use Bitswap (or some other protocol, potentially including GraphSync) for discovery of content from peers we're already connected to
  • Have options like ipfs cat <cid> [--from=<multiaddr>] that will use GraphSync for direct point-to-point transfers

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are actually proposals that enable local swarm content discovery in graphsync similar to bitswap. See: ipld/specs#355 (and potentially more useful cause it would be full selector discovery). However note that actually implementing this in a way that is performant is a non-trivial task and may involve building the beginnings of a dag store to record what content we actually have locally not just at the block level but at the DAG level.

More importantly, this proposal allows an additional type of request that sits between bitswap's WANT-HAVE and WANT-BLOCK request -- since Graphsync deals with requests that span multiple CIDs, another potentially useful piece of information a graphsync peer can provide is the list of CIDs that will satisfy a selector request, without the blocks attached to those CIDs. This list of CIDs will have to be untrusted, but it makes for a potentially fruitful handoff to Bitswap:

  1. graphsync tells you what you are fetching, and you don't really need to split graphsync requests given you're just getting a list of CIDs (which is small)
  2. Bitswap handles the actual block fetching. (Note: This is a non-trivial implementation cause the CID list will be untrusted, meaning fetched data will need to go in a temporary store till a local selector traversal can verify it, but it's still feels doable to me.)

However: @aschmahmann is totally correct that all of this is difficult to make much use of if we don't provide access to bitswap's session peer list, and ideally pull out session peer tracking from bitswap and make it a more general service. That's a significant though IMHO necessary and timely refactor.

Moreover, getting bitswap to support putting blocks in a temporary store before they go into a permanent store on a per request basis is another significant refactor.

Basically, to do all this, there will be refactoring of bitswap.

I am generally not in favor of moving the protocol selector up to the user a.l.a. ipfs cat <cid> [--from=<multiaddr>] cause it strikes me as something the user should never have to think about.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go-merkledag is IMHO a legacy service for retrieving data. go-fetcher is the replacement -- the top level interface is selectors specifically so we can do this kind of request mixing. I would prefer we do this work inside go-fetcher, and slowly deprecate go-merkledag for this sort of this.

Also, go-merkeldag's DAGService APIs are block level, so it's not even clear how it could talk to graphsync at all.

In short, we should replace the reference to go-merkledag here with go-fetcher.


#### What does done look like?

* IPFS nodes advertise support for the graphsync multiprotocol
* When an IPFS node at the go-merkledag level receives a request including a selector, it attempts a graphsync session if the other node also advertises support for graphsync.

#### What does success look like?

* observation of increased use of graphsync

#### Counterpoints &amp; pre-mortem

* graphsync wasn't as ready as we expected
* merkledag / blockservice needed a re-factor to become peer-aware before we added graphsync

#### Alternatives

* we live with bitswap
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hate to trashtalk my own library, but I'm not sure we can't just live with bitswap. I'm not clear ultimately that bitswap is a huge bottleneck. I really would like to explore what the heck is going on in the BeyondBitswap testbed that's resulting in such huge diffs between full-stack IPFS transfer and Bitswap only transfer before we embark on a huge optimization project with go-graphsync. While all the refactors I mentioned above to bitswap, along with request mixing with graphsync, represent my personal dream project in terms of my own interests and competencies, I'm not sure if it's actually our highest priority here. Is data transfer speed truly a big pain point for IPFS right now? I'm not sure. What's the product team say?


#### Dependencies/prerequisites

* [ipld in ipfs](https://github.com/protocol/web3-dev-team/pull/25)

#### Future opportunities

moves us a step closer to ipfs and filecoin nodes being able to exchange data
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IPFS nodes already have graphsync as an experimental feature for responding to requests -- maybe we just turn it on?


## Required resources

#### Effort estimate
Medium

#### Roles / skills needed

* Understanding of bitswap
* Understanding of IPFS
* Understanding of merkledag
* Understanding of graphsync