Start with ~10GB genesis.json consume ~150GB of RAM at peaks

[alpha-omega-labs]

Hello,
Start with ~10GB genesis.json consume ~150GB of RAM at peaks and about 80GB after initialization. Is there a way to improve that, maybe cut some checks?
It takes 30min to 10h to finish process.
Thank you

[yihuang]

you need to open this to cosmos-sdk, I think there's some streaming json support in newer version.

[alpha-omega-labs]

Yes, they seems to be working on it for some time with genesis.json streaming.
cosmos/cosmos-sdk#11601
At the moment start of chain from big (not so big, just ~10GB) genesis.json seems to be not possible because or resource usage - ram and CPU. e.g 10GB genesis.json cause up to 200GB ram spikes during initialization and attempts to sign blocks.
Keeping in mind that cronos also have very big state - you might benefit from streaming genesis.json in future too.

[yihuang]

I think there's already some streaming support with the collections module: https://github.com/cosmos/cosmos-sdk/blob/main/collections/genesis.go

[alpha-omega-labs]

Looks like there is no actual way to optimize this, chain, resource wide, cannot be started with .json of big size.

We will rollback on v0.44 and after try to make inplace upgrade to cronos.
Few questions:
Is v0.47 of cronos available? are you going towards v0.5?

Do you have in-place upgrade code or examples for v0.44 to v0.46.15 jump?

Thank you!

[leejw51crypto]

changing to capnproto and flatbuffer would be helpful (not plain text json)
or bson