Design interface for SPARK as a datasource

[coreymjames]

User interviews has surfaced that some retrieval incentive programs (namely ReputationDAO, FIL+, and others) will need to access SPARK data, meaning that we need:

1. a simple interface through which this data can be pulled
2. a clear db solution, with an understanding of where and for how long SPARK data will be stored

[coreymjames]

Or is this within /spark-api repo?

[juliangruber]

With ReputationDAO, wouldn't we insert the data into their DB?

[coreymjames]

I am curious of others' recommendation of how this should be structured. Miro mentioned here that *spark-publisher *"takes measurements collected by spark-api and submits them to IPFS & MERidian smart contract": https://filecoinproject.slack.com/archives/C03PQG6UT2B/p1695037674160909?thread_ts=1695030437.152979&cid=C03PQG6UT2B My questions are: 1. Is there a simple/easy way for us to create an interface/API for RepDAO, FIL+, or any other potential project to access SPARK data in the future? 2. Is it possible/easy for this to be paywalled? The context for this question is w/r/t SPARK biz/funding model work being done here: https://github.com/orgs/filecoin-station/projects/1/views/9?pane=issue&itemId=36789356

[MF416]

I think the clear next steps are to understand how these various groups would use the data more tactically, which we could accomplish by following up with RG / Torfinn / Caro and digging into details / offering a few options

ReputationDAO --> @coreymjames do they have a technical lead here? we should get answers to questions like:

• Have they defined their report format / how they generate analytics?
• Do they have a DB set up that we can look at interfacing with?
• How are they planning on maintaining this?

FIL+ --> likely that this gets fed into the reports that RG generates and adds to github threads, need to understand how the bot / script he's using would ingest the data

Could quickly outline the options (below) and show them a few formats to get feedback

• API (have a sample JSON with datapoints / format)
• Data dump into csv
• Proposed time and feature list

[coreymjames]

The technical lead on RepDAO is Caro. She has told me via slack dm that she can provide RepDAO docs within 48 hrs. Will relay to the team once more clarity is reached.

[bajtos]

We are discussing integration with Reputation DAO in space-meridian/roadmap#21, you can find a lot of details there.

SPARK does submit measurements to IPFS, but it's raw data before fraud detection.

What we need instead is only measurements that we consider as valid (honest).

To get that, we need to change the Evaluate service to publish the data + fraud assessment to a new datastore.

Is there a simple/easy way for us to create an interface/API for RepDAO,
FIL+, or any other potential project to access SPARK data in the future?

1. Reputation DAO is push-based - we are pushing data to their DB, no API is needed on our side.
2. Creating a new API for consuming our measurements is relatively simple. Depending on the required features, of course. The simplest possible option is to publish our data to a SaaS offering DB + API out of the box, e.g. https://supabase.com

Is it possible/easy for this to be paywalled?

It depends on how what we want to charge for.

If the paywall is a simple "has full read access" vs "access not allowed", then it's very easy to implement.

That may not be enough though, we will soon need to add rate limiting mechanism to keep our DB and egress costs manageable.