details of data markets, sources, project IDs

docs/Protocol/Specifications/Snapshotter/implementations.md

@@ -22,6 +22,12 @@ Use case specific logic of generating snapshots as well as other configuration a
 
 The architecture has been designed to facilitate the seamless interchange of configuration and modules. Adapting the system to different use cases is as straightforward as changing a Git branch.
 
+You can observe the corresponding branches within [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/) and [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/) repos:
+
+* `eth_uniswapv2`
+* `eth_uniswapv2_5_pairs` in `snapshotter-configs` that corresponds to `eth_uniswapv2_lite` in `snapshotter-computes`
+* `zkevm_quests`
+
 ### Configuration Files
 
 Configuration files, located in the `/config` directory are linked to [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/) repo, play a pivotal role in defining project types, specifying paths for individual compute modules, and managing various project-related settings.
@@ -30,3 +36,10 @@ Configuration files, located in the `/config` directory are linked to [snapshott
 ### Compute Modules
 
 The heart of the system resides in the `snapshotter/modules` directory that's linked to [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/), where the actual computation logic for each project type is defined. These modules drive the snapshot generation process for specific project types.
+
+
+# Useful links
+
+* [Snapshot generation specifications](/docs/Protocol/Specifications/Snapshotter/snapshot_build.md)
+* [Data markets and sources](/docs/Protocol/data_sources.md)
+* [Composition of snapshots and higher order datapoints](/docs/Protocol/data_composition.md)

docs/Protocol/Specifications/Snapshotter/snapshot_build.md

@@ -14,6 +14,73 @@ An important advantage of Bulk Mode is that, since all transaction receipts are
 
 Every time a new project is added for either of these two types on the protocol state smart contract by an off-chain data source-detector and signaller, a ProjectUpdated event is emitted according to the following data model
 
+## Snapshot computation modules
+
+As briefly introduced in the section on snapshotter implementations that [leverage Git Submodules for specific computation logic](/docs/Protocol/Specifications/Snapshotter/implementations.md), the modules are specified in the configuration for project types under the key `processor`.
+
+```json reference 
+https://github.com/PowerLoom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json#L15-L28
+```
+
+Let us take the example of the snapshot builder configured for the project type `zkevm:owlto_bridge` and locate it in the `snapshotter-computes` repo, in the `zkevm_quests` branch
+
+```python reference
+https://github.com/PowerLoom/snapshotter-computes/blob/29199feab449ad0361b5867efcaae9854992966f/owlto_bridge.py#L1-L31
+```
+
+As you can observe, it implements the `compute()` interface expected from snapshotter implementations inheriting `GenericProcessorSnapshot`.
+
+```python reference
+https://github.com/PowerLoom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/callback_helpers.py#L179-L196
+```
+
+
 ## Base snapshots
 
-## Aggregate snapshots
+Callback workers calculate base snapshots against an `epochId` which corresponds to collections of state observations and event logs between the blocks at height in the range `begin, end`. They call the use case specific computation logic as configured in the [computation modules section](#snapshot-computation-modules).
+
+The data sources are determined according to the following specification for the `projects` key in the configuration:
+
+* an empty array against the `projects` indicates no specific data source is defined, if `bulk_mode` is `False`
+  * the snapshotter node attempts to retrieve data sources corresponding to the `projects` key from the protocol state
+
+```python reference
+https://github.com/PowerLoom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L304-L332
+```
+
+* If the `projects` key is non-existent
+  * data sources can also be dynamically added on the protocol state contract which the [processor distributor](#processor-distributor) [syncs with](https://github.com/PowerLoom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/processor_distributor.py#L1107)
+
+```python reference
+https://github.com/PowerLoom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L738-L751
+```
+
+
+* Else, we can have a [static list of contracts](/docs/Protocol/data_sources.md#static-data-sources)
+
+### Format of data sources added as `projects`
+* EVM-compatible wallet address strings
+* `"<addr1>_<addr2>"` strings that denote the relationship between two EVM addresses (for eg ERC20 balance of `addr2` against a token contract `addr1`)
+
+
+### Project ID Generation
+
+```python reference
+https://github.com/PowerLoom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/snapshot_worker.py#L51-L71
+```
+
+## Aggregate snapshots
+
+Aggregate and higher order snapshots that build on computed base snapshots are configured in their specific repos like the following in our Uniswap V2 Dashboard use case.
+
+```python reference
+https://github.com/PowerLoom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/aggregator.example.json#L1-L29
+```
+
+The order and dependencies of these compositions are specified as such according to the `aggregate_on` key
+
+### `SingleProject` aggregation type
+
+
+
+### `MultiProject` aggregation type

docs/Protocol/data_sources.md

@@ -39,9 +39,15 @@ Data sources can be dynamically added to the contract according to the role of c
 
 In the present implementation of the use case that tracks wallet activity for Quests on Polygon zkEVM, such wallets are added from a data feed supplied by Mercle that consists of wallets that signup on their platform. Only these wallet addresses are of interest to the Quest platform on Mercle for their activities to be tracked across DEXs and asset bridges.
 
+Read more about it in the [snapshotter specs of bulk node](/docs/Protocol/Specifications/Snapshotter/snapshot_build.md#bulk-mode).
+
 
 ## Project types and IDs
 
+All data sources are tracked with a project ID on the protocol. Think of it as a stream of datasets, finalized by consensus against [each epoch released](/docs/Protocol/Specifications/Epoch.md#1-epoch_released) on the protocol.
+
+Find more details on this in the [specifications of snapshot generation](/docs/Protocol/Specifications/Snapshotter/snapshot_build.md).
+
 
 ## Useful links and concepts