Refactor/v2

.vscode/settings.json

@@ -0,0 +1,5 @@
+{
+    "eslint.workingDirectories": [
+        "./airdrop-cli",
+    ]
+}

airdrop-cli/.cspell.json

@@ -4,7 +4,32 @@
   "ignorePaths": ["**/*.json", "**/*.css", "node_modules", "**/*.log"],
   "useGitignore": true,
   "language": "en",
-  "words": ["dataurl", "devpool", "outdir", "servedir"],
+  "words": [
+    "dataurl",
+    "devpool",
+    "outdir",
+    "servedir",
+    "TYPEHASH",
+    "PAIDOUT",
+    "Supabase",
+    "SUPABASE",
+    "viem",
+    "WXDAI",
+    "Populator",
+    "leaderboard",
+    "Leaderboard",
+    "publicnode",
+    "txhashes",
+    "gtxs",
+    "etxs",
+    "gnosisscan",
+    "txreceipt",
+    "ubiquibar",
+    "blockscan",
+    "ubiquibot",
+    "pavlovcik",
+    "sonarjs"
+  ],
   "dictionaries": ["typescript", "node", "software-terms"],
   "import": ["@cspell/dict-typescript/cspell-ext.json", "@cspell/dict-node/cspell-ext.json", "@cspell/dict-software-terms"],
   "ignoreRegExpList": ["[0-9a-fA-F]{6}"]

airdrop-cli/.gitignore

@@ -9,5 +9,4 @@ node_modules
 static/dist
 .env
 dist
-*.csv
-debug/repos/*
+/src/scripts/data/*

airdrop-cli/.yarnrc.yml

@@ -1 +1 @@
-nodeLinker: node-modules
+nodeLinker: node-modules

airdrop-cli/README.md

@@ -1,112 +1,46 @@
 # UBQ Airdrop Tally Tool
 
 ## Overview
-This CLI tool tallies UBQ airdrop amounts for contributors. It does this by parsing issue comments for payout links from the UBQ bot using the GitHub GraphQL API.
 
-## Setup
-1. **GitHub Token**
-    - Add the token to ``.env`` as `GITHUB_TOKEN`.
-2. **Install Dependencies**
-    - Install the required dependencies using `npm` or `yarn`.
+This tool is designed to tally up Ubiquity contributor permits from across all issues and create a verifiable leaderboard based on earnings from completed tasks, ensuring a high level of data integrity and transparency.
 
-    ```bash
-    # npm install
-    ```
-3. **Build the CLI**
-    - Build the CLI using the available commands.
+A huge improvement from the first iteration of the airdrop tally tool, this is expected to store the underlying data into the `Permits` table in the database, which would render the need for future use of this tool obsolete.
 
-    ```bash
-    # npm run cli:build
-    ```
+## Usage
 
-4. **Run the CLI**
-    - Run the CLI using the available commands.
+1. Install dependencies
 
-    ```bash
-    # npm run cli:start
-    ```
+```bash
+yarn install
+```
 
-## Commands
-1. **start**
-    - Display information about the available commands and their usage.
+2. Open three terminals and run one parser in each terminal
 
-    ```bash
-    # yarn cli:start
-    ```
+```bash
+yarn dune
+```
 
-2. **single**
-    - Tally UBQ airdrop for a specific repository or shortcode from the beginning of 2023.
+```bash
+yarn issue
+```
 
-    ```bash
-    # yarn cli:single [shortcode/repo-name]
-    ```
+```bash
+yarn userTx
+```
 
-3. **tally**
-    - Tally UBQ airdrop for all indexable repositories since the start of 2023.
+##### Note: This will take around 10-15 minutes to complete the process.
 
-    ```bash
-    # yarn cli:tally
-    ```
+3. Close all but one terminal and run the following command
 
-4. **tally-from**
-    - Tally UBQ airdrop for all indexable repositories since a specified date (YYYY-MM-DD).
+```bash
+yarn all
+```
 
-    ```bash
-    # yarn cli:tally-from [date]
-    ```
+## Changes Made
 
-5. **tally-multi-csv**
-    - Tally UBQ airdrop for all indexable repositories since the start of 2023 and output repository-specific CSV files.
-
-    ```bash
-    # yarn cli:tally-multi-csv
-    ```
-
-6. **help**
-    - Display a list of indexable repositories and their shortcodes.
-
-    ```bash
-    # yarn run cli:help
-    ```
-
-## Output
-The CLI outputs three CSV files:
-
-1. [**All Payments**](all_repos_all_payments.csv)
-    - Includes payments with or without an assignee. Manual checking required for entries without an assignee which is often due to issues having been reopened or manual payouts because of issues with the bot.
-
-2. [**Contributors**](all_repos_contributors.csv)
-    - Provides a username-to-UBQ mapping for total UBQ earned from all payments across all repositories since the chosen time.
-
-3. [**No Payments**](all_repos_no_payments.csv)
-    - Lists repositories that have been indexed as having no payments released, including archived and inactive repositories. Manual checking may be required.
-
-
-## Usage Examples
-1. Tally UBQ airdrop for a specific repository or shortcode:
-
-    ```bash
-    # npm run cli:single dollar || npm single ubiquity-dollar
-    ```
-
-2. Tally UBQ airdrop for all indexable repositories since the start of 2023:
-
-    ```bash
-    # npm run cli:tally
-    ```
-
-3. Tally UBQ airdrop for all indexable repositories since a specified date:
-
-    ```bash
-    # npm run cli:tally-from 2023-01-01
-    ```
-4. Tally UBQ airdrop for all indexable repositories since the start of 2023 and output repository-specific CSV files:
-
-    ```bash
-    # npm run cli:tally-multi-csv
-    ```
-5. Display a list of indexable repositories and their shortcodes:
-
-    ```bash
-    # npm run cli:help
-    ```
+- Optimized and refactored the core `tally` function into a far more readable codebase.
+- Created a parser for both blockscan APIs and Dune Analytics.
+- Improved data integrity and cohesion of data from all sources as opposed to the previous version, through multiple parsers, checks and rechecks as well as outputting debug data/verification data that is much more workable.
+- Combined all data available for a given permit into a single source of truth, handy for attributing permits to the respective issues (should the DB be extended to support that)
+- Geared the tool towards seeding the database as opposed to outputting CSV files.
+- Removed the unnecessary CLIME and CLI code, as it was never used in the first place.

airdrop-cli/package.json

@@ -16,11 +16,10 @@
     "format:prettier": "prettier --write .",
     "format:cspell": "cspell **/*",
     "prepare": "husky install",
-    "cli:debug": "npx tsx src/debug/cli-entry.ts",
-    "cli:tally": "npx tsx src/tally/cli-entry.ts",
-    "cli:single": "npx tsx src/single/cli-entry.ts",
-    "cli:unspent": "npx tsx src/unspent/cli-entry.ts",
-    "cli:help": "npx tsx src/help/cli-entry.ts",
+    "dune": "npx tsx src/scripts/dune-data-parser.ts",
+    "issue": "npx tsx src/scripts/paid-issue-parser.ts",
+    "userTx": "npx tsx src/scripts/user-tx-parser.ts",
+    "all": "npx tsx src/scripts/data-controller.ts",
     "test": "npx jest"
   },
   "keywords": [
@@ -31,8 +30,8 @@
     "open-source"
   ],
   "dependencies": {
-    "axios": "^1.6.7",
-    "clime": "^0.5.16",
+    "@octokit/rest": "^20.1.1",
+    "@supabase/supabase-js": "^2.43.2",
     "dotenv": "^16.4.2",
     "ethers": "^5.7.2",
     "graphql": "^16.8.1",
@@ -50,7 +49,6 @@
     "@typescript-eslint/eslint-plugin": "^6.13.1",
     "@typescript-eslint/parser": "^6.13.1",
     "cspell": "^8.3.2",
-    "esbuild": "^0.19.8",
     "eslint": "^8.54.0",
     "eslint-config-prettier": "^9.0.0",
     "eslint-plugin-prettier": "^5.0.1",
@@ -80,4 +78,4 @@
       "@commitlint/config-conventional"
     ]
   }
-}
+}