(wip) protocol level details along with dynamic data markets

docs/Protocol/Specifications/Snapshotter/implementations.md

@@ -0,0 +1,32 @@
+---
+sidebar_position: 5
+---
+
+# Data market specific implementations
+
+![Submodule architecture of snapshotter](/images/submodule_architecture.png)
+
+The Snapshotter Peer is designed with a modular and highly configurable architecture, allowing for easy customization and seamless integration with a diverse set of data markets.
+
+## Snapshotter Core
+
+This foundational component defines all the essential interfaces and handles a wide range of tasks, from listening to epoch release events to distributing tasks and managing snapshot submissions. Read more about it in the detailed section on its [components](/docs/Protocol/Specifications/Snapshotter/components.md).
+
+
+## Data market specific
+
+Use case specific logic of generating snapshots as well as other configuration are available as [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) which can be pulled in as desired. Presently you can find diverse use cases that serve customized datapoints and track activity for the following across multiple data source chains like Ethereum Mainnet, Polygon zkEVM, Polygon PoS etc.
+
+* DEXs like Uniswap v2, Quickswap
+* Bridges like bungee.exchange, Owlto finance
+
+The architecture has been designed to facilitate the seamless interchange of configuration and modules. Adapting the system to different use cases is as straightforward as changing a Git branch.
+
+### Configuration Files
+
+Configuration files, located in the `/config` directory are linked to [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/) repo, play a pivotal role in defining project types, specifying paths for individual compute modules, and managing various project-related settings.
+
+
+### Compute Modules
+
+The heart of the system resides in the `snapshotter/modules` directory that's linked to [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/), where the actual computation logic for each project type is defined. These modules drive the snapshot generation process for specific project types.

docs/Protocol/Specifications/Snapshotter/preloading.md

@@ -12,6 +12,8 @@ Each project type within the project configuration as found in [config/projects.
 https://github.com/PowerLoom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf8070/config/projects.example.json#L3-L12
 ```
 
+![Preloading](/images/preloading.png)
+
 ## Interface
 
 The preloaders implement one of the following two generic interfaces

docs/Protocol/Specifications/Snapshotter/snapshot_build.md

@@ -4,6 +4,16 @@ sidebar_position: 2
 
 # Snapshot Generation
 
+## Bulk mode
+
+Bulk Mode is highly effective in situations where the project list is continually expanding or where snapshots don't need to be submitted in every epoch, perhaps because the data hasn't changed significantly. Example use cases include monitoring on-chain activities and tracking task or quest completion statuses on the blockchain.
+
+The system monitors all transactions and blocks without the need for predefined data sources. The `Processor Distributor` generates a `SnapshotProcessMessage` with bulk mode enabled for each project type. When snapshot workers receive this message, they leverage [preloaded transaction receipts](/docs/Protocol/Specifications/Snapshotter/preloading.md#shipped-preloaders), filtering out relevant transactions to generate snapshots for all data sources that interacted with the blockchain during that epoch. Snapshot worker then generates relevant project Ids for these snapshots and submits them for further processing.
+
+An important advantage of Bulk Mode is that, since all transaction receipts are preloaded, this approach can efficiently scale to accommodate a large number of project types with little to no increase in RPC (Remote Procedure Call) calls.
+
+Every time a new project is added for either of these two types on the protocol state smart contract by an off-chain data source-detector and signaller, a ProjectUpdated event is emitted according to the following data model
+
 ## Base snapshots
 
 ## Aggregate snapshots

docs/Protocol/data_composition.md

@@ -0,0 +1,40 @@
+---
+sidebar_position: 4
+---
+
+# Snapshot composition
+
+The protocol state collects finalized snapshots for incremental epochs against contracts configured as data sources. These snapshots can be further composed with either one or a combination of the following strategies 
+
+* primitives like map-reduce, summation, filter etc
+* a specific span of epochs to denote a time period, for eg, 24 hours
+* and anything else you can think of...!
+
+## Single project composition
+
+We can compose snapshots on top of base snapshots for a single contract.
+
+![single project composition](/images/single_project_composition.png)
+
+## Multiple projects composition
+
+As defined by the data sources configuration, the protocol state collects snapshots across multiple smart contracts which can be further composed according to the strategies outlined in the beginning of this page.
+
+![Multiple projects snapshots](/images/multi_projects_snapshots.png)
+
+## Higher order aggregations
+
+An example of this can be found in the [Uniswap V2 dashboard implementation](/docs/Build-with-Powerloom/UniswapV2%20Dashboard/index.md) where trade activity aggregation dataset is generated by 
+
+* combining individual snapshots of trade volume and feees across multiple pair contracts 
+* spanning a specific set of epochs that satisfy a time duration, for eg, 24 hours
+
+Since all snapshots are finalized on an epoch basis, you can weave together snapshots across multiple contracts and accurately derive complex data compositions without approximations.
+
+![Multiple projects composition](/images/multi_projects_composition.png)
+
+### Dependency graph
+
+All of this is controlled by the dependency graph as specified in the data market's configuration of data sources.
+
+![Data composition dependency graph](/images/DependencyDataComposition.png)

docs/Protocol/data_sources.md

@@ -5,7 +5,44 @@ title: Data markets and sources
 
 # Data markets
 
-Snapshotters generate snapshots, base as well as higher order, composable snapshots according to the data sources defined by the data markets they participate in. Every data market maintains its [specific protocol state](/docs/Protocol/Specifications/protocol_state.md) regarding submission, calculation and finalization of such snapshots.
+Snapshotters generate snapshots, base as well as higher order, composable snapshots according to the data sources _defined by the data markets they participate in_. Every data market maintains its [specific protocol state](/docs/Protocol/Specifications/protocol_state.md) regarding submission, calculation and finalization of such snapshots.
 
 The data sources defined by a market can be static as well as dynamic, depending on the use case at hand.
 
+## Static data sources
+
+This is utilized by our own implementation of a data market that serves datasets to render a live Uniswap V2 dashboard. You can find further details of the data source configuration and snapshot schema in the following section within our documentation
+
+* [Building with Powerloom -- Uniswap V2 Dashboard](/docs/Build-with-Powerloom/UniswapV2%20Dashboard/index.md)
+
+We will continue with this example and take a look at the data sources list defined as part of the Uniswap v2 specific configuration to be found in the [`snapshotter-configs`](https://github.com/PowerLoom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json#L1-L11) repo.
+
+```json reference
+https://github.com/PowerLoom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json#L1-L11
+```
+
+The `projects` field in the above configuration snippet are nothing but the Uniswap v2 pair contract addresses that are tracked and on which snapshots are generated. Jump to the section on [project types and IDs](#project-types-and-ids) to further understand the way the snapshots on these contracts are tracked.
+
+## Dynamic data sources
+
+For situations where data sources are constantly changing or numerous, making it impractical to maintain an extensive list of them, the data sources need not be defined explicitly in the configuration.
+
+Instead it is left to the snapshotter implementation to operate in ['bulk mode'](/docs/Protocol/Specifications/Snapshotter/snapshot_build.md#bulk-mode). The data source configuration merely specifies the computation modules which will utilize general purpose, [preloaded](/docs/Protocol/Specifications/Snapshotter/preloading.md) datasets to filter out transactions, event logs etc on contract addresses of interest. This is where signaling of data sources comes into picture.
+
+```json reference
+https://github.com/PowerLoom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json#L1-L28
+```
+
+### Data source signaling
+
+Data sources can be dynamically added to the contract according to the role of certain peers in the ecosystem known as 'signalers'. This is the most significant aspect of the Powerloom Protocol ecosystem apart from snapshotting and will soon be decentralized to factor in on-chain activity, and market forces and accommodate a demand-driven, dynamic data ecosystem.
+
+In the present implementation of the use case that tracks wallet activity for Quests on Polygon zkEVM, such wallets are added from a data feed supplied by Mercle that consists of wallets that signup on their platform. Only these wallet addresses are of interest to the Quest platform on Mercle for their activities to be tracked across DEXs and asset bridges.
+
+
+## Project types and IDs
+
+
+## Useful links and concepts
+
+* [Modular architecture of Use case specific snapshotter implementations](/docs/Protocol/Specifications/Snapshotter/implementations.md)

docs/Snapshotters/Overview.md

@@ -31,12 +31,6 @@ Continue along within this category to explore the different testnet based data
 
 * [Joining a network](/docs/category/joining-networks)
 
-## Setting up the snapshotter node 
-
-The snapshotter is a distributed system with multiple moving parts.
-
-If you're planning to participate as a snapshotter, refer to our specific subsections within [joining networks](/docs/Snapshotters/Joining%20Networks/) to start snapshotting.
-
 
 :::note