feat: added a main runner class as the entry point for the indexing t…

…ool (#1058) * chore: updated artifacts Signed-off-by: Logan Nguyen <[email protected]> * Feat: added a main runner class as the entry point for the indexing tool Signed-off-by: Logan Nguyen <[email protected]> * docs: added README file Signed-off-by: Logan Nguyen <[email protected]> * fix: updated log message Signed-off-by: Logan Nguyen <[email protected]> * fix: added MIRROR_NODE_URL to README.md Signed-off-by: Logan Nguyen <[email protected]> --------- Signed-off-by: Logan Nguyen <[email protected]>
hashgraph · Dec 6, 2024 · 78fb1ee · 78fb1ee
1 parent f8c21fb
commit 78fb1ee
Show file tree

Hide file tree

Showing 6 changed files with 202 additions and 20 deletions.
diff --git a/tools/erc-repository-indexer/docs/registry_design.md b/tools/erc-repository-indexer/docs/registry_design.md
@@ -59,30 +59,37 @@ The following class diagram illustrates the code structure and relationships bet
 
 ```mermaid
 classDiagram
-    class ContractScanner {
-        +fetchContracts(startingPoint: String): List~Contract~
-        +fetchContractDetails(contractId: String): Contract
+    class Main {
+        +run() Promise<void>
     }
 
-    class InterfaceMatcher {
-        +isErc20(contract: Contract): Boolean
-        +isErc721(contract: Contract): Boolean
+    class ConfigService {
+        +getNetwork() string
+        +resolveStartingPoint(registryGenerator: RegistryGenerator) Promise<string>
+        +validateConfigs() void
     }
 
-    class Validator {
-        +validateErc20(contract: Contract): Boolean
-        +validateErc721(contract: Contract): Boolean
+    class ByteCodeAnalyzer {
+        +categorizeERCContracts(contractScannerService: ContractScannerService, contractObject: MirrorNodeContract[]) Promise<string>
+    }
+
+    class ContractScannerService {
+        +fetchContracts(next: string) Promise<string>
+        +fetchContractObject(contractId: string) Promise<string>
     }
 
     class RegistryGenerator {
-        +generateErc20Registry(contracts: List~Contract~): void
-        +generateErc721Registry(contracts: List~Contract~): void
+        +generateErcRegistry(erc20Contracts: string[], erc721Contracts: string[]) Promise<void>
+        +updateNextPointer(next: string) Promise<void>
+        +retrieveNextPointer() Promise<string>
     }
 
-    ContractScanner --> InterfaceMatcher : "Uses for matching"
-    InterfaceMatcher --> Validator : "Optional validation"
-    InterfaceMatcher --> RegistryGenerator : "Passes matched contracts"
-    Validator --> RegistryGenerator : "Passes validated contracts"
+    Main --> ConfigService : "Validates configuration settings"
+    Main --> ContractScannerService : "Fetches batches of contracts"
+    Main --> ByteCodeAnalyzer : "Processes contracts and analyzes bytecode"
+    ByteCodeAnalyzer --> ContractScannerService : "Fetches detailed contract objects and bytecodes"
+    ByteCodeAnalyzer --> RegistryGenerator : "Provides categorized ERC-20 and ERC-721 contracts"
+    RegistryGenerator --> ContractScannerService : "Uses to generate registries"
 
 ```
 

diff --git a/tools/erc-repository-indexer/erc-contract-indexer/README.md b/tools/erc-repository-indexer/erc-contract-indexer/README.md
@@ -0,0 +1,94 @@
+# ERC Contract Indexer for Hedera Hashgraph
+
+## Overview
+
+The ERC Contract Indexer is a tool designed to facilitate the indexing and management of ERC20 and ERC721 smart contracts on the Hedera Hashgraph network. This project provides a set of services to fetch, analyze, and store contract data efficiently, enabling developers and users to interact with smart contracts seamlessly.
+
+## Features
+
+- **Contract Fetching**: Retrieve contract details from the Hedera mirror node.
+- **Bytecode Analysis**: Analyze smart contract bytecode to categorize contracts as ERC20 or ERC721.
+- **Registry Management**: Generate and update registries for ERC20 and ERC721 contracts.
+- **Next Pointer Handling**: Manage pagination for contract indexing using a next pointer.
+
+## Getting Started
+
+### Prerequisites
+
+- Node.js (version 18 or higher)
+- npm (Node Package Manager)
+
+### Installation
+
+1. Clone the repository:
+
+   ```bash
+   git clone https://github.com/hashgraph/hedera-smart-contracts.git
+   cd tools/erc-repository-indexer/erc-contract-indexer
+   ```
+
+2. Install the dependencies:
+
+   ```bash
+   npm install
+   ```
+
+3. Create a `.env` file in the root directory and configure your environment variables:
+
+| Variable          | Description                               | Accepted Values                                                                                                                                                                                                                 |
+| ----------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `HEDERA_NETWORK`  | The network to connect to.                | `previewnet`, `testnet`, or `mainnet`                                                                                                                                                                                           |
+| `MIRROR_NODE_URL` | The URL for the Hedera mirror node API.   | A valid URL pointing to the Hedera mirror node (e.g., `https://{previewnet\|testnet\|mainnet}.mirrornode.hedera.com`)                                                                                                           |
+| `STARTING_POINT`  | The starting point for contract indexing. | A Hedera contract ID (e.g., `0.0.369`), an EVM 20-byte address (e.g., `0x0000000000000000000000000000000000000369`), or a get contract list next pointer (e.g., `/api/v1/contracts?limit=100&order=asc&contract.id=gt:0.0.369`) |
+
+Example configuration:
+
+```plaintext
+HEDERA_NETWORK=testnet
+MIRROR_NODE_URL=https://testnet.mirrornode.hedera.com
+STARTING_POINT=0.0.1013
+```
+
+> **Note:** If a `STARTING_POINT` is not set, the tool will first look for a saved `next` pointer in the registry storage to continue indexing. If no pointer is available, the tool will start from the genesis block.
+
+### Usage
+
+To start the indexing process, run the following command:
+
+```bash
+npm start
+```
+
+This will initiate the indexing process, fetching contracts from the specified starting point and categorizing them into ERC20 and ERC721 contracts. The end goal is to generate a comprehensive registry that contains JSON files for both ERC-20 and ERC-721 tokens. The expected format for these registries is as follows:
+
+- **ERC20**:
+  ```json
+  [
+    {
+      "address": "0xevm_address",
+      "contractId": "0.0.hedera_contract_id"
+    }
+  ]
+  ```
+- **ERC721**:
+  ```json
+  [
+    {
+      "address": "0xevm_address",
+      "contractId": "0.0.hedera_contract_id"
+    }
+  ]
+  ```
+  These JSON files will serve as a reliable registry of ERC tokens on the Hedera network, enhancing visibility and usability for developers and users alike.
+
+> **Note:** Future updates will enhance the JSON file format to include additional details about the ERC tokens.
+
+### Testing
+
+To run the tests for the project, use the following command:
+
+```bash
+npm test
+```
+
+This will execute the test suite and provide feedback on the functionality of the services.
diff --git a/tools/erc-repository-indexer/erc-contract-indexer/package.json b/tools/erc-repository-indexer/erc-contract-indexer/package.json
@@ -3,7 +3,7 @@
   "version": "0.1.0",
   "main": "index.js",
   "scripts": {
-    "dev": "ts-node src/index.ts",
+    "start": "ts-node src/index.ts",
     "test": "jest"
   },
   "keywords": [],

diff --git a/tools/erc-repository-indexer/erc-contract-indexer/src/index.ts b/tools/erc-repository-indexer/erc-contract-indexer/src/index.ts
@@ -0,0 +1,81 @@
+/*-
+ *
+ * Hedera Smart Contracts
+ *
+ * Copyright (C) 2024 Hedera Hashgraph, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+import dotenv from 'dotenv';
+import { ByteCodeAnalyzer } from './services/byteCodeAnalyzer';
+import { ConfigService } from './services/config';
+import { ContractScannerService } from './services/contractScanner';
+import { RegistryGenerator } from './services/registryGenerator';
+
+dotenv.config();
+
+export const main = async () => {
+  const configService = new ConfigService();
+  const registryGenerator = new RegistryGenerator();
+  const contractScannerService = new ContractScannerService(
+    configService.getMirrorNodeUrl()
+  );
+  const byteCodeAnalyzer = new ByteCodeAnalyzer();
+
+  try {
+    let next = await configService.resolveStartingPoint(registryGenerator);
+    await processContracts(
+      next,
+      contractScannerService,
+      byteCodeAnalyzer,
+      registryGenerator
+    );
+  } catch (error) {
+    console.error('Error during the indexing process:', error);
+  }
+};
+
+const processContracts = async (
+  next: string | null,
+  contractScannerService: ContractScannerService,
+  byteCodeAnalyzer: ByteCodeAnalyzer,
+  registryGenerator: RegistryGenerator
+) => {
+  do {
+    const fetchContractsResponse =
+      await contractScannerService.fetchContracts(next);
+
+    if (!fetchContractsResponse || !fetchContractsResponse.contracts.length) {
+      console.warn('No contracts found.');
+      return;
+    }
+
+    next = fetchContractsResponse.links.next;
+
+    const ercContracts = await byteCodeAnalyzer.categorizeERCContracts(
+      contractScannerService,
+      fetchContractsResponse.contracts
+    );
+
+    // let the registry update process to run asynchronously in the background
+    registryGenerator.generateErcRegistry(
+      ercContracts.erc20Contracts,
+      ercContracts.erc721Contracts
+    );
+    registryGenerator.updateNextPointer(next);
+  } while (next);
+};
+
+main().then();
diff --git a/tools/erc-repository-indexer/erc-contract-indexer/src/services/registryGenerator.ts b/tools/erc-repository-indexer/erc-contract-indexer/src/services/registryGenerator.ts
@@ -118,8 +118,6 @@ export class RegistryGenerator {
     filePath: string,
     newContracts: ERCOutputInterface[]
   ): Promise<void> {
-    console.log('Pushing new ERC token contracts to registry...');
-
     const fileContent = this.readContentsFromFile(filePath);
     const existingContracts = fileContent
       ? (JSON.parse(fileContent) as ERCOutputInterface[])
@@ -137,7 +135,9 @@ export class RegistryGenerator {
     const uniqueContracts = Array.from(contractMap.values());
 
     await this.writeContentsToFile(filePath, uniqueContracts);
-    console.log('Finish pushing new ERC token contracts to registry.');
+    console.log(
+      `Finished writing ${newContracts.length} new ERC token contracts to registry.`
+    );
   }
 
   /**

diff --git a/tools/erc-repository-indexer/erc-contract-indexer/src/utils/constants.ts b/tools/erc-repository-indexer/erc-contract-indexer/src/utils/constants.ts
@@ -19,7 +19,7 @@
  */
 
 export default {
-  RETRY_DELAY_MS: 15000,
+  RETRY_DELAY_MS: 9000,
   GET_CONTRACT_ENDPOINT: '/api/v1/contracts',
   ERC_20_JSON_FILE_NAME: 'erc-20.json',
   ERC_721_JSON_FILE_NAME: 'erc-721.json',