WIP: First very dirty prototype of SP1 distributed proof

[Champii]

Still a WIP

[petarvujovic98]

@Champii Aside from updating this branch with main, I've left some comments concering some code quality.

As far as I see from the code, this makes every Raiko instance a potential distributed sp1 worker. Meaning every instance can handle a shard of execution. Maybe it would make more sense to have a service outside of Raiko handle just the sp1 distributed proving part so that it is less coupled with the host. This would allow us to change and/or scale the distributed part of sp1 proofs independently of the host which should not care about how a single block is proven (whereas now it distinguishes between a full proof and a shard proof).

[Champii]

@Champii Aside from updating this branch with main, I've left some comments concering some code quality.

As far as I see from the code, this makes every Raiko instance a potential distributed sp1 worker. Meaning every instance can handle a shard of execution. Maybe it would make more sense to have a service outside of Raiko handle just the sp1 distributed proving part so that it is less coupled with the host. This would allow us to change and/or scale the distributed part of sp1 proofs independently of the host which should not care about how a single block is proven (whereas now it distinguishes between a full proof and a shard proof).

Awesome, thanks for looking into it :)

I can probably create a new CLI command/argument that starts a Raiko node as a distributed worker specifically. I'll look into it.

As for the code quality, this is only a first draft of a POC, so I will definitely improve it before making a final PR proposal.

[Brechtpd]

Not sure how to best handle this. Having a separate service also seems to be not so ideal because a worker ideally can still be used to process different kinds of things (not only SP1 shards, but also risc zero segments, and even full proofs of smaller programs). If different binaries/endpoints need to be used to start this work on the workers then maybe it makes things more complicated then just having every type of work pass through a single interface within a single binary. Once we create a general prover interface that allows anybody to create their own custom prover backend, it will also be harder to differentiate between the multiple prover backend types, so keeping the host interface general may also pay off for this.

I guess we can wait and see because not a lot of code and easy to move stuff around, at least for now.

[Champii]

Closing in favor of #302