Fix Issue #96 by checking for unallocated origin ASN (#207)

* Remove unused tying import * Remove unsued typing import and fix comment typo * Remove unused typing import * Change MRT downloader to generic file downloader and add unit tests * Update README to use docker compose * Fix linting issues * Don't need venv for docker compose * Only download file if size has changed * Use new data directory variable name * Fix lintint issues * Add parser and stats support for unallocated ASNs, and update as part of daily cron script * Fix typo in archive name * Add support for running pytest in parallel * Download allocated ASN lists for pytest --------- Co-authored-by: James Bensley <[email protected]>
jwbensley · Nov 11, 2024 · 5fcc786 · 5fcc786
1 parent 95965a1
commit 5fcc786
Show file tree

Hide file tree

Showing 52 changed files with 2,411 additions and 1,807 deletions.
diff --git a/README.md b/README.md
@@ -23,7 +23,7 @@ There is a single DNAS container that is built and used for all stages of the pi
 ```bash
 sudo apt-get update
 
-# Install Docker (from: https://docs.docker.com/engine/install/ubuntu/):
+# Install Docker & Docker Compose (from: https://docs.docker.com/engine/install/ubuntu/):
 sudo apt-get install -y curl
 
 curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
@@ -35,42 +35,31 @@ https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
 | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
 
 sudo apt-get update
-sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
+sudo apt-get install --no-install-recommends -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
 sudo systemctl enable docker.service
 sudo systemctl enable containerd.service
 sudo groupadd docker
 sudo usermod -aG docker $USER
 # ^ Log out and in again for this to take effect
 
 # Set up base directory:
-sudo mkdir /opt/dnas/ && sudo chown $USER:$USER /opt/dnas/
+BASE_DIR="/opt/dnas"
+sudo mkdir -p "$BASE_DIR" && sudo chown $USER:$USER "$BASE_DIR"
 
 # Install git:
-sudo apt-get install -y git
+sudo apt-get install --no-install-recommends -y git
 
-# Clone this repo (first add read-only key to Deploy Keys under https://github.com/jwbensley/dfz_name_and_shame/settings/keys):
-git clone [email protected]:jwbensley/dfz_name_and_shame.git /opt/dnas
+# Clone repo to base directory:
+git clone [email protected]:jwbensley/dfz_name_and_shame.git "$DATA_DIR"
 
-# Install virtualenv:
-sudo apt-get install -y virtualenv
-
-# Install docker-compose in a venv and build containers
-cd /opt/dnas/ && virtualenv venv && source venv/bin/activate
-pip3 install --upgrade pip
-pip3 install docker-compose
-cd docker/
-docker-compose build
-
-# Create the data directory
-BASE_DIR=$(grep "BASE_DIR =" /opt/dnas/dnas/dnas/config.py | awk -F "\"" '{print $2}')
-sudo mkdir -p "${BASE_DIR}" && sudo chmod a+rwx "${BASE_DIR}"
+# Run the build script
+${BASE_DIR}/docker/build_dnas.sh
 ```
 
 After the steps above, DNAS is ready to run inside the containers. See documentation under [docker/](docker/) for more details.
 
 To run DNAS "natively", not in a container, see documentation under [dnas/](dnas/).  
 
-
 ## Credits
 
 Thanks to the following people/organisations/groups for their help:

diff --git a/dnas/README.md b/dnas/README.md
@@ -2,14 +2,33 @@
 
 ## Running and Testing DNAS
 
+```shell
+sudo apt-get install --no-install-recommends -y virtualenv
+
+DATA_DIR="/opt/dnas_data/"
+APP_DIR="/opt/dnas/"
+VENV_DIR="./venv/"
+
+cd "$APP_DIR"
+
+if [ ! -d "$VENV_DIR" ]
+then
+    python3 -m virtualenv venv
+fi
+# Shellcheck can't follow "source" imports:
+# shellcheck disable=SC1091
+source venv/bin/activate
+```
+
 ### Redis
+
 To run the code natively (not inside a container) a Redis instance is required. The best way is to spin up the existing redis container:
 
 ```shell
 cd /opt/dnas
 source venv/bin/activate
 cd docker/
-docker-compose up -d dnas_redis
+docker compose up -d dnas_redis
 ```
 
 If you really need/want to, you can use the steps below to spin up a stand-alone Redis container. Note that DNAS expects to authenticate to Redis so set a password, and update the Redis hostname in redis_auth.py:
@@ -29,7 +48,7 @@ To run the DNAS code natively in Python3 outside of the container, install the r
 cd /opt/dnas/
 source venv/bin/activate
 cd ./dnas/
-pip install -r requirements.txt
+python3 -m pip install -r requirements.txt
 ```
 
 The code is developed in Python3 but the DNAS container actually uses PyPy3 to run faster. The following commands manually install PyPy3 and the required modules in PyPy, to manually run the code outside of a container:
@@ -60,7 +79,7 @@ Tox is used to provide linting (black and isort), type checking (mypy), and run
 ```shell
 cd /opt/dnas/
 source venv/bin/activate
-pip install tox
+python3 -m pip install tox
 tox
 ```
 
@@ -71,5 +90,4 @@ Merge requests need to maintain the existing coding style[^1] for consistency:
 * Stringageddon: Currently all values from MRT files are strings so that there is a single uniform data type (AS Numbers are strings, IP prefixes are strings, AS paths are list of strings etc.).
 * All Redis keys are also strings.
 
-
 [^1]:[In the loosest sense](https://en.wikipedia.org/wiki/Infinite_monkey_theorem)
diff --git a/dnas/dnas/bogon_asn.py b/dnas/dnas/bogon_asn.py
@@ -1,8 +1,3 @@
-import typing
-
-from dnas.config import config as cfg
-
-
 class bogon_asn:
     """
     Class to check if an ASN is a bogon ASN (meaning reserved by an RFC,

diff --git a/dnas/dnas/bogon_attr.py b/dnas/dnas/bogon_attr.py
@@ -1,6 +1,3 @@
-import typing
-
-
 class bogon_attr:
     """
     Class to check if a BGP attribute is "well known"/defined.

diff --git a/dnas/dnas/bogon_ip.py b/dnas/dnas/bogon_ip.py
@@ -1,5 +1,4 @@
 import ipaddress
-import typing
 
 from dnas.config import config as cfg
 

diff --git a/dnas/dnas/config.py b/dnas/dnas/config.py
@@ -1,5 +1,4 @@
 import os
-import typing
 
 
 class config:
@@ -12,10 +11,9 @@ class config:
     #################
 
     """
-    Application root for everything (logging, downloads, tmp files, etc.)
-    with trailing slash:
+    Data root (for logging, downloads, tmp files, etc.) with trailing slash:
     """
-    BASE_DIR = "/opt/dnas_data/"
+    DATA_DIR = "/opt/dnas_data/"
 
     """
     The time format used for generating new timestamps and parsing existing
@@ -38,7 +36,7 @@ class config:
         "%(asctime)s|%(levelname)s|%(process)d|%(funcName)s|%(message)s"
     )
     # Log directory for all logs
-    LOG_DIR = os.path.join(BASE_DIR, "logs/")
+    LOG_DIR = os.path.join(DATA_DIR, "logs/")
     # Logging from script: get_mrts.py
     LOG_GETTER = os.path.join(LOG_DIR, "get_mrts.log")
     # Logging from script: git_report.py
@@ -55,6 +53,22 @@ class config:
     LOG_TESTER = os.path.join(LOG_DIR, "mrt_test.log")
     # Logging from script: tweet.py
     LOG_TWITTER = os.path.join(LOG_DIR, "tweet.log")
+    # Logging from script: update_asn_allocations.py
+    LOG_UPDATE_ASN = os.path.join(LOG_DIR, "update_asn.log")
+
+    ###########################
+    # ASN ALLOCATION SETTINGS #
+    ###########################
+
+    # URL of ASN allocation stats
+    asn_allocation_url = (
+        "https://www.iana.org/assignments/as-numbers/as-numbers-2.csv"
+    )
+    ASN_DATA = os.path.join(DATA_DIR, "asn_data/")
+    # Output file
+    asn_stats_file = os.path.join(ASN_DATA, "iana-32bit-asns.csv")
+    # Allocated ASNs list
+    unallocated_asns_file = os.path.join(ASN_DATA, "unallocated-asns.txt")
 
     ###################
     # PARSER SETTINGS #
@@ -89,7 +103,7 @@ class config:
     )
 
     # Local git repo details
-    GIT_BASE = os.path.join(BASE_DIR, "dnas_stats/")
+    GIT_BASE = os.path.join(DATA_DIR, "dnas_stats/")
     GIT_REPORT_BRANCH = "main"
 
     ####################
@@ -142,10 +156,10 @@ class config:
     ########################
 
     # Base dir to save MRT files to
-    DL_DIR = os.path.join(BASE_DIR, "downloads/")
+    DL_DIR = os.path.join(DATA_DIR, "downloads/")
 
     # Temporary directory to split MRT files into
-    SPLIT_DIR = "/tmp/"  # Set to None to disable
+    SPLIT_DIR = "/tmp/"  # Set to None to disable MRT splitting
 
     # Default interval for downloading and parsing new MRT files (seconds)
     DFT_INTERVAL = 3600
@@ -154,13 +168,13 @@ class config:
     MRT_ARCHIVES = []
     """
     If the machine running this code is in a different timezone to the MRT
-    archive, an additional offset in hous is required. A negative int means
+    archive, an additional offset in hours is required. A negative int means
     "hours in the future", a positive int means "hours in the past". These are
     the RIB_OFFSET and UPD_INTERVAL values below.
     """
 
     # DISABLED AS PART OF https://github.com/jwbensley/dfz_name_and_shame/issues/135
-    # This is an example of how to configured a local BIRD instance which is dumping MRTs
+    # This is an example of how to configure a local BIRD instance which is dumping MRTs
     """
     Transit session from @LukaszBromirski
     RIB dumps are every 1 hour. RIB dumps are disabled!
@@ -351,7 +365,7 @@ class config:
     )
 
     """
-    RRC12 Frankfurt DEC-IX
+    RRC12 Frankfurt DE-CIX
     RIB dumps are every 8 hours
     RIB dump example: https://data.ris.ripe.net/rrc12/2023.07/bview.20230702.0000.gz
     UPDATE dumps are every 5 minutes

diff --git a/dnas/dnas/git.py b/dnas/dnas/git.py
@@ -2,21 +2,20 @@
 import logging
 import os
 import subprocess
-import typing
 import urllib.parse
 
 from dnas.config import config as cfg
 
 
 class git:
     """
-    A class for commiting and pushing files to GitHub.
+    A class for committing and pushing files to GitHub
     """
 
     @staticmethod
     def add(filename: str) -> None:
         """
-        Add files to the git index, to be commited.
+        Add files to the git index, to be committed
         """
         if not filename:
             raise ValueError(
@@ -87,19 +86,19 @@ def clone() -> None:
 
         ret = subprocess.run(
             ["git", "clone", cfg.GIT_STAT_CLONE_URL],
-            cwd=cfg.BASE_DIR,
+            cwd=cfg.DATA_DIR,
             capture_output=True,
         )
         if ret.returncode != 0:
             raise ChildProcessError(
                 f"Couldn't clone git repo {cfg.GIT_STAT_CLONE_URL} to directory "
-                f"{cfg.BASE_DIR}:\n"
+                f"{cfg.DATA_DIR}:\n"
                 f"args: {ret.args}\n"
                 f"stdout: {ret.stdout.decode()}\n"
                 f"stderr: {ret.stderr.decode()}"
             )
         logging.debug(
-            f"Cloned git repo {cfg.GIT_STAT_CLONE_URL} to {cfg.BASE_DIR}"
+            f"Cloned git repo {cfg.GIT_STAT_CLONE_URL} to {cfg.DATA_DIR}"
         )
 
     @staticmethod

diff --git a/dnas/dnas/log.py b/dnas/dnas/log.py
@@ -1,6 +1,5 @@
 import logging
 import os
-import typing
 
 from dnas.config import config as cfg
 

diff --git a/dnas/dnas/mrt_archive.py b/dnas/dnas/mrt_archive.py
@@ -1,7 +1,6 @@
 import datetime
 import os
 import re
-import typing
 
 from dnas.config import config as cfg
 

diff --git a/dnas/dnas/mrt_entry.py b/dnas/dnas/mrt_entry.py
@@ -1,13 +1,12 @@
 import datetime
 import json
-import typing
 
 from dnas.config import config as cfg
 
 
 class mrt_entry:
     """
-    An MRT Entry object contains the prased BGP data which is a single data
+    An MRT Entry object contains the parsed BGP data which is a single data
     point for one of the stats in an MRT Stats object.
     """