Shared global key-value storage

[evgenykuzyakov]

To be able to reuse contracts code across multiple accounts, we suggest to dedicate a shared global key-value storage that is available for reads from every shard.

Simple version:

This storage has the following properties:

• the total cost to write into the storage is one time payment of PRICE_PER_BYTE_PER_SHARD * NUM_SHARD * (NUM_BYTES_VALUES + NUM_BYTES_KEY)
• once the value is stored, it can't be removed
• to write into the storage, you need to issue a transaction from the account, that creates a custom receipt.
• once written, every shard should be able to read any value from this storage

This will allow the following use-cases:

• reuse contract code. Code can be deployed once into the global storage and then accounts can just link to the hash by paying only about 32 bytes of storage, instead of the size of the storage cost.
• Have multiple contracts on one account
• Have some precomputed data deployed accessible from multiple contracts
• Contract Modules?
• Shared config per version (needs write permissions)

[bowenwang1996]

I think this will create some issues long term because we cannot reshard this global storage. Even if it is more expensive to write to, because values cannot be removed, this storage will eventually grow so large that nodes will not be able to track it anymore.

[evgenykuzyakov]

I think this will create some issues long term because we cannot reshard this global storage. Even if it is more expensive to write to, because values cannot be removed, this storage will eventually grow so large that nodes will not be able to track it anymore.

The price of this storage is linked to the total_supply + inflation. Let's say we dedicate 1TB for it and map it to the total supply of 1B NEAR. Then we define the price of global storage will be 1 NEAR = 1Kb. Let's say the global storage grows similarly to the inflation at 5% per year. Then the maximum storage requirement is 1TB + 5% grows per year.

[mattlockyer]

Big fan of this idea! Why would all developers deploy the same code for common design patterns? Why not leverage existing code and extend it by wrapping it in a higher order contract?

I think every account where the code is referenced having it's own storage space is key here too.

Personally I think there will be many more use cases where someone will want to deploy, instantiate and provide their own arguments to a contract that's already written. The examples from Ethereum in ERC20/721s, DeFi tokens, etc... are many.

[bowenwang1996]

@evgenykuzyakov I don't see how that answers my concern. Given that this storage is shared globally, even 100GB feels too much. It will decrease the speed of syncing significantly.

[evgenykuzyakov]

Current proposal is for permanent storage, so there are no issue for syncing it. Downloading 100GB seems like a trivial problem, so long as you don't have to worry about it changing over time.

[bowenwang1996]

@evgenykuzyakov are we not going to make it verifiable?

[evgenykuzyakov]

Are we not going to make it verifiable?

If the key is the hash of the value, then we need to merkalize keys only.

Rollout idea for global contracts.

Let's say we have a global storage that all nodes have access to. And each key is the hash of the corresponding value.

Automatic state clean-up

Let's say I deploy a fungible contract to a global storage. If the contract was already used by a lot of existing accounts, then these accounts can now use it for free and we should be able to release the occupied trie space (by deleting a copy for an account) and refund the locked storage value for the account.

Ideally we do this the next time we touch this account. We can do this without modifying Account structure, but it will always require a few extra key checks. When we see a contract hash on the account, we check it against global key-value store and if it's present then we may need to cleanup account state. If the account has a contract copy in the trie under TrieKey::Contract { account_id } then we delete this copy and decrease the storage bytes accordingly. If the account doesn't have a copy, then we don't need to do anything.

Or we can have upgradable Account record that can include flexible contract hash, e.g. either global or local. If we see local and the global is available, then we upgrade it same way as above.

Pros

Developers can create contracts and first deploy them on accounts. If contract becomes popular then someone can create a DAO to publish it globally. Once the DAO reaches the limit, it will submit the contract to the global key-value store and the contract becomes free to deploy.

[evgenykuzyakov]

As discussed during a townhall session. The current proposal for implementation is to introduce a new shard with a single ephemeral account (e.g. global_contracts) that accepts contract code deployments.

All nodes will have to track this shard. In order to deploy there, you'll have to pass enough token amount to cover the storage of contract through a transfer action. The deployment may look as:

receiver_id: "global_contracts",
actions: [
  (Transfer amount: COST),
  (Deploy code: CODE),
]

If this action succeeds then the contract code will be hashed and stored in the state.

We want to achieve the following:

• State sync works and sync global contracts
• It's deterministic when a global contract is deployed
• There are little changes needed to the protocol
• This action of deploying is composable with current contracts. E.g. we can have a DAO to deploy a permanent contract.

[bowenwang1996]

@evgenykuzyakov @Longarithm @abacabadabacaba I wonder whether it is better to do this per shard vs. having a global shard that everyone needs to track. This reduces the complexity on the protocol level since we don't need to worry about how it interacts with sharding, state sync, etc. However, it likely requires changing the deploy contract action to allow users to specify how the contract should be deployed (burning tokens vs. staking tokens).

[MaksymZavershynskyi]

• @bowenwang1996 @Longarithm what is the decision that we are currently implementing?
• @mikedotexe reported that there we noticeable feedback from community on storage complexity. I suggest we collect the feedback from the dev community and attach it somewhere as an evidence that this change is needed. Currently, there appears to be only singular anecdotal testimonies that this is needed, mostly motivated by complex use-cases like ref-finance.
• @abacabadabacaba proposed that we don't need generic global storage but we only need global storage for contract binaries. I support this idea;
• I also support @bowenwang1996 's idea of avoiding having global shard, since it will greatly increase the complexity of the system.

[bowenwang1996]

@bowenwang1996 @Longarithm what is the decision that we are currently implementing?

We have not decided on this yet. @abacabadabacaba correctly pointed out that my proposal does not work well if we decide to implement dynamic resharding one day (contract deployment would be underpaid because it now replicates to another shard). Currently @Longarithm is discussing with @abacabadabacaba on possible solutions.

[bowenwang1996]

@abacabadabacaba actually we should not consider the storage cost to be constant. If we increase the number of shards, theoretically the amount of storage also increases and therefore the storage price should go down proportionally. Let's say we have dynamic resharding and there is some contract deployed in one out of n shards. If, after some time, the contract is replicated to m shards, the total number of shards should also be expected to increased m times, making the storage price on each shard 1/m of what it was before. Of course you could argue that this can be exploited by an attacker, but I think the cost of forcing a shard to split into two is pretty high and for any meaningful exploit, you have to force such splits multiple times.

[ryancwalsh]

https://www.pagoda.co/blog/protocol-roadmap-2023-4-the-next-2-years-of-near linked here, but the comments in this discussion are very old.

Are there more updated discussions published somewhere else? Thanks!

[vgrichina]

@abacabadabacaba proposed that we don't need generic global storage but we only need global storage for contract binaries. I support this idea;

There are other use cases like http://web4.near.page/ and http://near.social/ where you want to be able to store some common code (which isn't smart contract) and assets on chain.

I think it makes sense if we essentially have arweave-like storage built in, so that you can smoothly pay for it with NEAR token.