Merge pull request #63 from freedomofpress/python-311

Move to Python 3.11 baseline; increase blocklist import verbosity

.github/workflows/ci.yml

@@ -10,7 +10,7 @@ jobs:
         run: curl -sSL https://install.python-poetry.org | python3 -
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.9' 
+          python-version: '3.11'
           cache: 'poetry'
       - name: Install Python dependencies
         run: poetry install --with=dev

.gitignore

@@ -1,4 +1,4 @@
-__pycache__/*
+**/__pycache__/
 *.db
 ttnconfig/*
 *.egg-info

README.md

@@ -4,7 +4,7 @@
 
 If you want to run your own instance of `trackthenews`, you can download and install the package, and run its built-in configuration process. It can be installed with `pip`:
 
-Python 3.9 is what we currently test against, though it may work with other versions.
+Python 3.11 is what we currently test against, though it may work with other versions.
 
 
 ```bash
@@ -60,16 +60,16 @@ All articles are recorded in a sqlite database.
 
 ### Advanced feature: blocklist
 
-In some cases, you may wish to suppress articles from being posted, even though they would otherwise match. You can do so by writing two new functions, `check_article` and `check_paragraph`, and placing them in a file named `blocklist.py` in the configuration directory:
-- `check_article` takes an Article (and so has access to its `outlet`, `title`, and `url`) and should return `True` for any article that should be skipped in its entirety.
-- `check_paragraph` takes an Article and an individual matching paragraph and should return `True` for any paragraph that should be skipped (if other paragraphs match, the article will still be posted, but without the skipped paragraphs).
+In some cases, you may wish to suppress articles or paragraphs from being posted, even though they would otherwise match. To do so, implement a CustomBlocklist class following the abstract base class template in `trackthenews/base_blocklist.py`, and drop it as a file named `blocklist.py` in your `ttnconfig` directory.
+
+You can import the `bs4` library in `blocklist.py` for advanced parsing.
 
 ## Development
 
 ### Quick Start
 
 ```bash
-poetry env use 3.9  # Necessary if you have a different default python version
+poetry env use 3.11  # Necessary if you have a different default python version
 poetry install
 poetry run trackthenews sample_project
 # Follow the setup script instructions
@@ -83,8 +83,8 @@ poetry run trackthenews sample_project
 To develop `trackthenews`, clone the repository and install the package using [poetry][] and run the CLI tool:
 
 ```bash
-# Ensure you're using Python 3.9
-poetry env use 3.9
+# Ensure you're using Python 3.11
+poetry env use 3.11
 # This will create a virtual environment and install trackthenews and its dependencies
 poetry install --with=dev
 # This will run the setup script

pyproject.toml

@@ -21,7 +21,7 @@ repository = "https://github.com/freedomofpress/trackthenews"
 include = ["trackthenews/fonts/*"]
 
 [tool.poetry.dependencies]
-python = "~3.9"
+python = "~3.11"
 feedparser = "^6.0"
 future = "^0.17.1"
 html2text = "2018.1.9"
@@ -31,6 +31,9 @@ readability-lxml = "^0.8.1"
 requests = "^2.31.0"
 tweepy = "^4.14.0"
 "Mastodon.py" = "^1.8.1"
+lxml = {extras = ["html-clean"], version = "^5.3.0"}
+# For blocklist implementers
+beautifulsoup4 = "^4.12.3"
 
 [tool.poetry.group.dev.dependencies]
 black = "^23.7.0"

trackthenews/base_blocklist.py

@@ -0,0 +1,15 @@
+from abc import ABC, abstractmethod
+
+
+class BaseBlocklist(ABC):
+    """Abstract base class for blocklists."""
+
+    @abstractmethod
+    def check_article(self, article):
+        """Check if an entire article should be blocked."""
+        pass
+
+    @abstractmethod
+    def check_paragraph(self, article, paragraph):
+        """Check if an otherwise matchign paragraph should be blocked based on its content."""
+        pass

trackthenews/core.py

@@ -74,21 +74,21 @@ def clean(self, http_session):
 
         self.plaintext = h.handle(doc.summary())
 
-    def check_for_matches(self, http_session):
+    def check_for_matches(self, http_session, blocklist=None):
         """
         Clean up an article, check it against a block list, then for matches.
         """
         self.clean(http_session)
         plaintext_grafs = self.plaintext.split("\n")
 
-        if blocklist_loaded and blocklist.check_article(self):
+        if blocklist and blocklist.check_article(self):
             pass
         else:
             for graf in plaintext_grafs:
                 if any(word.lower() in graf.lower() for word in matchwords) or any(
                     word in graf for word in matchwords_case_sensitive
                 ):
-                    if blocklist_loaded and blocklist.check_paragraph(self, graf):
+                    if blocklist and blocklist.check_paragraph(self, graf):
                         continue
                     self.matching_grafs.append(graf)
 
@@ -604,14 +604,25 @@ def main():
     sys.path.append(home)
     global blocklist_loaded
     global blocklist
-    try:
-        import blocklist as blocklist
 
-        blocklist_loaded = True
-        print("Loaded blocklist.")
-    except ImportError:
+    blocklist_path = os.path.join(home, "blocklist.py")
+
+    if os.path.exists(blocklist_path):
+        try:
+            from blocklist import CustomBlocklist
+
+            blocklist_instance = CustomBlocklist()
+            blocklist_loaded = True
+            print("Loaded blocklist.")
+        except ImportError as e:
+            blocklist_loaded = False
+            print(f"Error loading blocklist: {e}")
+        except Exception as e:
+            blocklist_loaded = False
+            print(f"Unexpected error loading blocklist: {e}")
+    else:
         blocklist_loaded = False
-        print("No blocklist to load.")
+        print("No blocklist file found to load.")
 
     if matchwords:
         print("Matching against the following words: {}".format(matchwords))
@@ -660,7 +671,7 @@ def main():
                 print("Checking {} article {}/{}".format(article.outlet, counter, len(deduped)))
 
                 try:
-                    article.check_for_matches(http_session)
+                    article.check_for_matches(http_session, blocklist=blocklist_instance)
                 except Exception as e:
                     print(e)
                     print("Having trouble with that article. Skipping for now.")