Feat: Reduce time when fetching masp

[pnam1609]

Issue

Link: Fetching shielded balance via SDK nor extension working

Solution

• First, Make fetches by block ranges instead of all blocks.
• Second, Use Promise pool from @supercharge/promise-pool library to process multiple fetch block ranges in parallel. It significantly reduced time when fetching block.
• Then save the context of the rage block to a new temporary Indexed DB with key as startBlock-EndBlock and value as context after serializing this ranges block.It void loss data if the process is down. Only need fetching again missed ranges block.
• Once finished fetching and saving all ranges blocks to temporary Indexed DB . Will call a function to get all the data from temporary Indexed DB and load it into the context.
• The last step is saved to the root db.
• Also the solution help get the progress of process and display in to user.

Image

• Button to naviagte shielded sync page
  
• Shielded sync page
  

Note:

• I have another change for Namada source. And change depency in cargo.toml to my change. PR: Feature: Create a new function load and save to temp indexed DB support for Reduce time Fetching MASP namada#2898

[phy-chain]

Amazing @pnam1609 !
Can i test this locally or is it still too early ?
I was working on different paths to improve masp context for the web app/extension, but you're way ahead of me !
I have a spending key that's fully synced through the CLI, that would be could to see if I can get it sync in the extension too

[phy-chain]

Is that necessary ? It drags an old version babel/runtime with it, which combine are quite huge, and an open door for potentail security issues through babel runtime : GHSA-67hx-6x53-jw92 (just an example)

[phy-chain]

Would be nice too if we could load a shielded.dat file generated on a node instance for instance (maybe only in dev ?), that way, we can speed up the process of syncing if we already have a synced context via CLI

[phy-chain]

Ping me on Discord so we can chat more easily, I want to help ! "phychain"

[mateuszjasiuk]

@pnam1609 I really appriciate the effort. I think it will be hard to merge this soon. The reason is you worked under some assumptions about how things should work and look like. The proper flow would be:

• create an issue
• discuss solution in the created issue
• settle on requirements
• work on functionality
• create PR

We plan to create contribution guide soon. This way it will be easier for both you and us to cooperate :)

[HadesGuard]

Cool, thank you for your hard work. We need it ASAP

[pnam1609]

Hi @mateuszjasiuk , Thanks for your response. I hope it's available as soon as possible and i can contribute more for project.

[nodeopsdev]

@pnam1609 Really nice ! I appreciate your comprehensive solution, which outlines a systematic approach to optimize block fetching by implementing block ranges, utilizing Promise pools for parallel processing, and ensuring data integrity through temporary IndexedDB storage. Your solution not only enhances efficiency by reducing fetch times but also provides a robust mechanism for handling interruptions and displaying progress to the user. Awesome valuable contribution.

[mateuszjasiuk]

@pnam1609

Hello,
I think if we want to merge this quicker we have to chage few things :):

1. UI - We did not have a plan to create Shielded Sync button. User should not need to remember to do shielded sync every time they want to, for example, see up to date balance. So for the namada.me we call the "sync" each time we query for the shielded balance. So in general I would move all the UI changes to the separate issue so we can discuss it further :)

2. Changes to Namada repo - I don't think your changes will be merged in near future or at all. On the upside, there is a recent change in the Namada SDK(0.32.0) that allows you to specify the starting index of the fetch function, which should help with your solution.

WDYT?

[pnam1609]

Hi @mateuszjasiuk, Thanks for your reponse.

1. For this shielded sync button and UI for that. Let me remove this.
2. I will change the dependency to Namada SDK(0.32.0) and will check to logic code. When it complete, i will ping you later for checking code again.