Capability to skip deals for Indexing

[honghaoq]

Some SPs have reported the need for a more granular level control to not index certain deals, since some deals have private data that should not be made public to the network, while indexer considers all indexed content public (all indexed data are considered retrievable).

We believe this is a market issue instead of an indexing issue, because the content needs to be made not available for retrieval at the same time that it is not published to the index. If a content not indexed but available for retrieval, then we are misrepresenting its state to providers.

We'd appreciate SP input on this quick survey:

• should these deals be marked not retrievable and not indexable at the same time?
• or there are scenarios that deals are retrievable but should not be indexed?

[f8-ptrk]

if a deal is retrievable is none of the indexers business for deals it shouldn't index

a white list for what can be indexed is the way to go - a blacklist will most likely leak

[TorfinnOlsen]

⚠️ This discussion is being relegated in favor of a committed plan to move forward represented by this github issue in the boost repo:

• Capability to skip deals for indexing: need config to mark content as not retrievable in market filecoin-project/boost#689 (comment)

We perceive this as the simplest change we can make in order to enable SP's to set the announce status, without impacting retrieval processes. Future efforts will address the broader goal of an access list service.

⚡ tl;dr Request to add a flag to boost to allow storage clients to elect to not announce deal data to IPNI. Default behavior will be to announce just as it is today.

🕐 When is a deal announced to the indexer?

• When a deal is made, if AnnounceToIPNI=True or not set at all, CID’s announced to indexer.

• If AnnounceToIPNI=False, no announcement.

🔧 How will this be implemented?

• Currently, we are on Storage Deal Protocol v1.2 (/fil/storage/mk/1.2.0) where every deal is announced.

• This new proposal is for Storage Deal Protocol v1.2.1 (/fil/storage/mk/1.2.1)

• Boost will be updated to recognize and store as metadata this new flag.

🥡 How does this impact whether the data may be retrieved?

• It doesn’t. No impact. ACL’s must be set separately. This just relates to IPNI.

🛃 How can a client change the announce status?

• No change currently permitted. The decision is made at the time the deal is made.

👴 What about legacy deals?

• They stay announced as they are today.

🍴 Related Github Discussion

• Ability for SP/client to mark deal as private (i.e. not publicly retrievable) filecoin-project/boost#728

👋 Relegated github issues/discussions being closed as a result of this update

• Scope default indexing to only FIL+ deals with FastRetrieval enabled filecoin-project/notary-governance#666

• Capability to skip deals for Indexing #635

⌛ Timeline for implementation

• Current scope of effort is in the process of being analyzed and assessed by the boost team. Work not yet scheduled, but expected outcome is work completed early Q123.