Technical Concerns and Questions on the Protocol

[twdragon]

27.09.2024 we had a very inspiring and intense discussion with Philipp Koellinger. Please find the video here: 

...and the Fireflies.ai transcript and summary here.

We have discussed many aspects of decentralised PID from conceptual, philosophical and technical points of view, referring to our new Concept document and CODEX library. After the discussion, we highlighted the following questions that, as we all agreed, any team working on the concept of the decentralised PID should address:

1. Should the minting and provenance of the PID-associated data be based on the transaction-based model, as the blockchain-based technologies suggest? Is there a way to provide persistence, versioning and provenance without spending resources on blockchain maintenance? The other concern is the reliance on the third-party blockchain, like ETH we have right now.
2. Do we need to store all the operations and intention descriptions? Would creating a network of nodes exchanging non-anonymous open messages with a limited lifetime, but strictly standardised (and identified with highest-level PIDs) format and structure be more useful? This approach is highly inspired by ROS and SWIFT systems, where a message is always trusted due to its format identified with PID, and it is a subject for cross-control by every node the message passes. The idea here is to preserve persistence and versioning without keeping a log of all actions the involved Agents perform, as the ownership concept is already implemented in the persistent storage infrastructure.
3. Is the Agent's identity equal to his private key? Should we use the unified internal representation of an Agent to perform 2FA and persistent login of the single agent over many third-party identity token providers?
4. The synonymic model proposed in the CODEX library works like direct aliasing, and Philipp placed an example about dPID aliased by DOI. Could side-effectors be implemented to provide functionality for UNIRESOLVER? Ideally, here we should think about the decentralised system working over any other, as we may be the last ones having valid lookup tables available.
5. The users urge over the sovereignty implemented via prefixes. As the limit, every internal Agent's ID should be a) a PID, and b) the prefix. What do you think the best way is to implement this?
6. Is there a way to control and partially localize indexing, to keep it reusable?
7. The deployment costs need to be addressed. The technologies we use may be quite computationally expensive, and the users should have a very thorough estimation of costs and efforts. The deployment packages are also extremely useful, so is there a way to provide them?
8. Do we have now a central bootstrap point when the CODEX endpoint is deployed? How would we avoid this?
9. Do we consider the possibility of having different implementations of the same functionality of the key components of the PID system, like in the $\LaTeX$ ecosystem we have?

[m0ar]

Hi @twdragon ! 🐉 Good idea to use the discussions platform, I have activated it and converted your issue :)

Lots of good thoughts and questions here! Happy to hopefully provide some clarity:

1. Blockchain transactions are indeed a bit of a nuisance, in particular because it increases the technical threshold and brings a plethora of new UX issues along with it. These are the main reason why we have moved away from them. The basis of CODEX is using Ceramic streams for tracking history. There is no need for gas, and no long transaction times.

   The only current dependency on any particular blockchain is indirect. Ceramic uses it to stash merkle proofs to timestamp events, otherwise there wouldn't be any (verifiable) notion of time. Alias registries for PIDs can use a blockchain for implementation, and it's likely one of the best ways to do it. Both of these dependencies have immutability in common, meaning it's easy to migrate it to a new home if the need arises, and and easy to prove that the migration is correct.

2. It's possible that I'm not able to visualize exactly what you mean here, but to me it sounds a lot like Ceramic. There are fundamental PIDs for both streams (versioned objects) and each individual commits (update events), support for community-defined schemas, all operations are signed, and announced p2p through the network agents. It is pseudonymous, but you can make yourself known. Ownership is implemented by indexing the information under the public key that signed it. Is there something from your vision that is missing here?

3. Agents (as in data publishers, which are not necessarily node runners) are publicly known by their public key, and prove their identity by signing data with their private key. Not sure what you mean by this, care to elaborate?

   unified internal representation of an Agent to perform 2FA and persistent login of the single agent over many third-party identity token providers

   Meanwhile, I'll share some general thoughts. I know of no simpler way to trustlessly verify identity than private-key derived DID's. Signing is simple, verification is simple. It's battle tested, well understood, and has a low cognitive load. The main challenge if course, is key management. This is a complicated UX problem that goes way back. IMO, crypto wallets have solved this part nicely, and the remaining issues are related to the technology you interface with. The UX flow of signing a plain message on request of an app is a single click, and completely ignores all gas/transaction/blockchain things that we associate with sending a blockchain transaction.

   If you do not want to require the user to bring their own wallet/keys, there are ways of securely deriving them from other user secrets. DeSci Publish, for example, defaults to creating a wallet underneath the hood for the user. We use secret material returned from the ORCiD auth handshake, effectively managing a crypto wallet on the client-side, in a process that looks and feels a lot like traditional single sign on.

   It's not a perfect system, but I'd say it's a good take on pub/priv key management to pop the abstraction level up to what our (DeSci Labs) users have expressed the wish to use (ORCiD). For others, you can bring your own keys and/or wallets. The reason it's much more difficult than "regular" SSO is that anyone that gets access to the key can make irrevokable harm to your content. In federated/centralised systems, this can be handled by just.. Undoing it.

4. A fundamental requirement of the substrate used to implement CODEX is solid PIDs for identity, objects, and individual object updates. In practice, these are DID's, and the Ceramic stream ID and commit ID. Here is an example stream ID: globally unique, persistent, and universally resolvable: kjzl6kcym7w8y9xujdr5plbzz73c3zr819dne0bbqbtetc9o99zushin08tg9lz. Note this is a third-party network indexer we don't have a relation with.

   Now, CODEX welcomes use-cases for application-specific PIDs. Say you need a shorter format (the integer alias registry we run at @desci-labs), or a gated registry for a large organisation/publisher (random ideas: nasa/dept/asdf or nature/2024/12/whatever). Implementation of anything is super simple, as it only needs to map one persistent ID to another one. Assigning PIDs over a namespace of PIDs is a lot simpler, as it just needs to implement one step, and then use shared implementations/definitions of the hairier stuff like version identifiers, file paths, location specifiers, schemas, etc.

   A while back I made a light-hearted diagram over this pattern, calling it PIÑA (Persistent Identifier Namespaces):
   

5. Excellent question, and the seed of the PIÑA idea. It's really hard to find a solid path to protected namespaces at the abstraction level of unfederated PIDs. Surely, not impossible, but it turns out it's really simple on the aliasing level. Anyone can make a PID for anything, but your can hoist anything up from this soup and tie it to an alias in a registry which fulfills your needs.

   Somewhat related hot take of mine: the apparent need for prefixes is in large caused by a lack of reliable, transparent indexing. There is no way to find associated content by just looking at a PID, and there is no way to identify the author by just looking at the PID. But, if that PID already has an associated public key, you can easily and trustlessly identify all other PIDs created by that identity. Super personal opinion, but I think we find it comfortable to "read" PIDs, but we can already attach solid cryptographic trace to them, yielding unparalleled trustworthiness and indexability (not a word, but it should be). I haven't yet heard a convincing argument to why we need fundamental support for protected prefixes, but if you do, I'd be super interested to hear it!

6. Not sure what you refer to here, feel free to elaborate. Right now, indexing can be done somewhat automatically if you're okay with the scope being a set of entities/schemas/models. Down the line, there is a really cool event aggregator functionality currently being built over at 3Box Labs (the creators of Ceramic). This will allow defining basically arbitrary indexing rules and transformations over only the data your care about.

7. Assuming you are referring to infrastructure requirements to run nodes. I think the worst offender is probably IPFS in this regard, but this is an issue that has already been addressed. Instead of trying to (ab)use Kubo primitives to perform a pairwise state sync protocol, there is now a purpose-built libp2p protocol called RECON. It's a huge bump in efficiency, with way fewer back-and-forths required to sync state. It's also less affected by general congestion, as you only communicate about the specific streams you care about. The communicated data is still IPLD though, and the shape doesn't change.

   What deployment packages are you looking for. The entities/models are public in this repo. As for infrastructure, it's currently a bit mixed between @desci-labs/nodes (ceramic infra), @desci-labs/ipfs-s3 (the backing IPFS node), @desci-labs/dpid-resolver (resolution service). I absolutely agree this is a mess and near impossible to figure out from the outside at the moment 🙏

8. Not entirely sure what "endpoint" means in this case, but basically there a two pieces of public infrastructure: the node keeping track of objects and their updates, and the resolver that understands paths and queries. We would warmly welcome you to run either, or both, to help distribute the responsibility. Running the resolver is lightweight, running a "full node" is a bit more involved.

9. Having alternative implementations of the resolver would be neat. It's possible to do over indexing as well, for example by using OrbisDB as an alternative way to explore the data. The "core" is more difficult, as the uniqueness and resolution depends on the Ceramic consensus algorithm. Any other implementation would be completely isolated and hence lose some of the fundamental guarantees that are protected by network consensus.

   Worth to keep in mind that maintaining separate implementations on critical components can lead to significant overhead in terms of coordination, something which at this point in time feels more like a burden than the lack of alternative implementations. I absolutely agree with the long-term need to reach that top slice of reliability and availability, but personally I don't think this is the right time for that.

[pkoellinger]

Just adding to what @m0ar wrote above - in cases where his answers to your questions @twdragon deviate from what I said on our call, please assume @m0ar response is correct (not mine). He is the expert on what's under the hood of the CODEX protocol. :-)