chore: move notebooks

davidgasquez · Dec 14, 2023 · fbbef6d · fbbef6d
1 parent e0ccbc9
commit fbbef6d
Show file tree

Hide file tree

Showing 3 changed files with 122 additions and 22 deletions.
diff --git a/Makefile b/Makefile
@@ -13,7 +13,7 @@ render:
 	@quarto render
 
 publish:
-	@quarto publish gh-pages --no-prompt
+	@quarto publish gh-pages notebooks/sandbox.ipynb --no-prompt
 
 clean:
 	@rm -rf portal/.quarto data/*.parquet data/*.duckdb

diff --git a/README.md b/README.md
@@ -13,6 +13,20 @@ Access the [latest Gitcoin Grants Data tables here](https://ipfs.filebase.io/ipf
 
 ![img](https://user-images.githubusercontent.com/1682202/268236925-d44915ab-d46b-49ff-85ec-2ad06bcfe5e0.png)
 
+## 📂 Gitcoin Grants Data
+
+You can get the latest data a Parquet files from IPFS. The data lives under the [IPFS CID](https://raw.githubusercontent.com/davidgasquez/gitcoin-grants-data-portal/main/data/IPFS_CID) pointer available in this repository.
+
+The following command will give you a working URL to explore the available tables.
+
+```bash
+# Get the latest IPFS CID
+LATEST_IPFS_CID=$(curl https://raw.githubusercontent.com/davidgasquez/gitcoin-grants-data-portal/main/data/IPFS_CID)
+
+# Download round_votes.parquet from a public IPFS gateway
+echo https://ipfs.filebase.io/ipfs/$LATEST_IPFS_CID/
+```
+
 ## 📖 Overview
 
 The repository contains code and artifacts to help process Gitcoin Grants data from the [Allo Indexer Data API](https://indexer-production.fly.dev/data/). It is an instance of [Datadex](https://github.com/davidgasquez/datadex) allowing you and everyone else to:

diff --git a/reports/sandbox.ipynb → notebooks/sandbox.ipynb b/reports/sandbox.ipynb → notebooks/sandbox.ipynb
@@ -20,28 +20,114 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Welcome! This is the result of a PoC for a Gitcoin Grants Data Portal.\n",
+    "Welcome to the Gitcoin Grants Data Portal. Your open source, serverless, and local-first Data Platform for Gitcoin Grants Data. With this data hub, we aim to improve data access and empower data scientists to conduct research and guide community driven analysis and decisions!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Gitcoin Grants Data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To get the latest Gitcoin Grants data, you can run the following cells."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import requests\n",
     "\n",
-    "You can access the output files from here:\n",
+    "LATEST_IPFS_CID = requests.get(\n",
+    "    \"https://raw.githubusercontent.com/davidgasquez/gitcoin-grants-data-portal/main/data/IPFS_CID\"\n",
+    ").text.strip()\n",
     "\n",
-    "- [Passport Scores](/passport_scores.parquet)\n",
-    "- [Votes](/round_votes.parquet)"
+    "GATEWAY_URL = f\"https://ipfs.filebase.io/ipfs/{LATEST_IPFS_CID}/\""
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "::: {.callout-warning} \n",
-    "Data and models shown here are probably not accurate!\n",
-    ":::"
+    "Once you have the latest URL, you can remotely access the data from your favorite data science tools like pandas, polars, DuckDB.\n",
+    "\n",
+    "For example, let's see the number of votes per round."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "3a4c3206cb0b4c65947a4136f3b1b49c",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "FloatProgress(value=0.0, layout=Layout(width='auto'), style=ProgressStyle(bar_color='black'))"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "┌────────────────────────────────────────────┬────────────────────┐\n",
+       "│                  round_id                  │ count(DISTINCT id) │\n",
+       "│                  varchar                   │       int64        │\n",
+       "├────────────────────────────────────────────┼────────────────────┤\n",
+       "│ 0x8de918f0163b2021839a8d84954dd7e8e151326d │             202232 │\n",
+       "│ 0xd4cc0dd193c7dc1d665ae244ce12d7fab337a008 │             133859 │\n",
+       "│ 0xa1d52f9b5339792651861329a046dd912761e9a9 │              82364 │\n",
+       "│ 0x12bb5bbbfe596dbc489d209299b8302c3300fa40 │              81811 │\n",
+       "│ 0x222ea76664ed77d18d4416d2b2e77937b76f0a35 │              68476 │\n",
+       "│ 0x984e29dcb4286c2d9cbaa2c238afdd8a191eefbc │              57366 │\n",
+       "│ 0x98720dd1925d34a2453ebc1f91c9d48e7e89ec29 │              52290 │\n",
+       "│ 0x2871742b184633f8dc8546c6301cbc209945033e │              29180 │\n",
+       "│ 0x5eb890e41c8d2cff75ea942085e406bb90016561 │              14559 │\n",
+       "│ 0xb6be0ecafdb66dd848b0480db40056ff94a9465d │              14089 │\n",
+       "├────────────────────────────────────────────┴────────────────────┤\n",
+       "│ 10 rows                                               2 columns │\n",
+       "└─────────────────────────────────────────────────────────────────┘"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "import duckdb\n",
+    "\n",
+    "duckdb.sql(\n",
+    "    f\"\"\"\n",
+    "    select\n",
+    "        round_id,\n",
+    "        count(distinct id)\n",
+    "    from '{GATEWAY_URL}/round_votes.parquet'\n",
+    "    group by round_id\n",
+    "    order by 2 desc\n",
+    "    limit 10;\n",
+    "    \"\"\"\n",
+    ")"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Data Examples"
+    "If you've executed GGDP locally, you can also rely on the local database to query the data."
    ]
   },
   {
@@ -53,7 +139,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 8,
    "metadata": {},
    "outputs": [
     {
@@ -240,15 +326,15 @@
        "9             19.728           20.00000  "
       ]
      },
-     "execution_count": 6,
+     "execution_count": 8,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
     "from ggdp.db import query\n",
     "\n",
-    "query(\"from passport_scores order by last_score_timestamp desc limit 10\")query(\"from passport_scores order by last_score_timestamp desc limit 10\")"
+    "query(\"from passport_scores order by last_score_timestamp desc limit 10\")"
    ]
   },
   {
@@ -260,7 +346,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 9,
    "metadata": {},
    "outputs": [
     {
@@ -413,7 +499,7 @@
        "9 2023-12-13 07:00:00    6672037     424  "
       ]
      },
-     "execution_count": 10,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -431,7 +517,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": 13,
    "metadata": {},
    "outputs": [
     {
@@ -549,7 +635,7 @@
        "4  [0x6E77e78D0ee74DF94dCAD5603551e1660752a456]       159319922   42161  "
       ]
      },
-     "execution_count": 11,
+     "execution_count": 13,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -567,7 +653,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": 14,
    "metadata": {},
    "outputs": [
     {
@@ -924,7 +1010,7 @@
        "9         153348089    42161  "
       ]
      },
-     "execution_count": 13,
+     "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -942,7 +1028,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
+   "execution_count": 15,
    "metadata": {},
    "outputs": [
     {
@@ -1244,7 +1330,7 @@
        "9        4.998367e+22  "
       ]
      },
-     "execution_count": 14,
+     "execution_count": 15,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1269,7 +1355,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
+   "execution_count": 17,
    "metadata": {},
    "outputs": [
     {
@@ -1366,7 +1452,7 @@
        "9  0xfc66a1f969bb77eb89a314725d657312d58f1589      2504.862395"
       ]
      },
-     "execution_count": 15,
+     "execution_count": 17,
      "metadata": {},
      "output_type": "execute_result"
     }