[#508] Use setuptools for plugin mechanism. (#509)

* Switch to pyproject.toml project with src-based layout. * Switch from yapsy to standard Python plugins. * Update unit tests to support built-in plugins approach. * Temporarily disable statick-md checks. * Add Python 3.12 and 3.13. * Bump versions of GitHub actions. * Add ubuntu-24.04 to os list for testing. * Drop Python 3.8. * Make new function to get version of cppcheck binary. Use that to fix unit test with different outcomes based on version of cppcheck. * Use runner.os to simplify Linux-specific action steps.
sscpac · Jan 20, 2025 · 5a0adb7 · 5a0adb7
1 parent c8dd003
commit 5a0adb7
Show file tree

Hide file tree

Showing 357 changed files with 1,166 additions and 1,840 deletions.
diff --git a/.github/workflows/publish.yaml b/.github/workflows/publish.yaml
@@ -9,53 +9,41 @@ on:  # NOLINT
 jobs:
   build-n-publish:
     name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
-    runs-on: ubuntu-22.04
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/statick
+    permissions:
+      id-token: write
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
-      - name: Set up Python 3.11
-        uses: actions/setup-python@v1
+      - name: Set up Python
+        uses: actions/setup-python@v5
         with:
           python-version: '3.11'
-
-      - uses: actions/cache@v3
-        if: startsWith(runner.os, 'Linux')
-        with:
-          path: ~/.cache/pip
-          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
-          restore-keys: |
-            ${{ runner.os }}-pip-
+          cache: 'pip'
 
       - name: Install tools
         run: |
-          python -m pip install --upgrade setuptools
-          python -m pip install --upgrade wheel
-          python -m pip install -r requirements-docs.txt
-          python -m pip install -r requirements.txt
+          pip install .[docs]
 
       - name: Build a binary wheel and a source tarball
         run: |
-          python setup.py sdist bdist_wheel
-
-      - name: Publish distribution 📦 to Test PyPI
-        uses: pypa/gh-action-pypi-publish@master
-        with:
-          password: ${{ secrets.PYPI_TEST_TOKEN }}
-          repository_url: https://test.pypi.org/legacy/
+          pip install -q build
+          python -m build
 
       - name: Publish distribution 📦 to PyPI
-        uses: pypa/gh-action-pypi-publish@master
-        with:
-          password: ${{ secrets.PYPI_TOKEN }}
+        uses: pypa/gh-action-pypi-publish@release/v1
 
       - name: Sphinx lint
         uses: ammaraskar/sphinx-action@master
         with:
           docs-folder: 'docs/'
 
       - name: Publish documentation
-        uses: peaceiris/actions-gh-pages@v3
+        uses: peaceiris/actions-gh-pages@v4
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           publish_dir: ./docs/build/html

diff --git a/.github/workflows/publish_docker_image.yaml b/.github/workflows/publish_docker_image.yaml
@@ -11,32 +11,32 @@ env:
 
 jobs:
   build-and-push-image:
-    runs-on: ubuntu-22.04
+    runs-on: ubuntu-latest
     permissions:
       contents: read
       packages: write
 
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Log in to the Container registry
-        uses: docker/login-action@v2
+        uses: docker/login-action@v3
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Extract metadata (tags, labels) for Docker
         id: meta
-        uses: docker/metadata-action@v4
+        uses: docker/metadata-action@v5
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
           tags: |
             type=semver,pattern={{version}}
 
       - name: Build and push Docker image
-        uses: docker/build-push-action@v3
+        uses: docker/build-push-action@v6
         with:
           context: docker
           push: true

diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -14,69 +14,36 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [macos-latest, ubuntu-20.04, ubuntu-22.04, windows-latest]
-        python-version: ['3.9', '3.10', '3.11']
+        os: [macos-latest, ubuntu-20.04, ubuntu-22.04, ubuntu-24.04, windows-latest]
+        python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
-      - uses: actions/setup-node@v3
+      - uses: actions/setup-node@v4
         with:
           node-version: '16'
 
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
-
-      - uses: actions/cache@v3
-        if: startsWith(runner.os, 'Linux')
-        with:
-          path: ~/.cache/pip
-          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
-          restore-keys: |
-            ${{ runner.os }}-pip-
-
-      - uses: actions/cache@v3
-        if: startsWith(runner.os, 'macOS')
-        with:
-          path: ~/Library/Caches/pip
-          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
-          restore-keys: |
-            ${{ runner.os }}-pip-
-
-      - uses: actions/cache@v3
-        if: startsWith(runner.os, 'Windows')
-        with:
-          path: ~\AppData\Local\pip\Cache
-          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
-          restore-keys: |
-            ${{ runner.os }}-pip-
+          cache: 'pip'
 
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-          python -m pip install --upgrade -r requirements.txt
-          python -m pip install --upgrade mypy
-          python -m pip install --upgrade setuptools
-          python -m pip install --upgrade tox
-          python -m pip install --upgrade tox-gh-actions
-          python -m pip install --upgrade types-deprecated
-          python -m pip install --upgrade types-PyYAML
-          python -m pip install --upgrade types-tabulate
-          python -m pip install --upgrade wheel
+          pip install --upgrade pip
+          pip install .[docs,test]
 
       # Remove apt repos that are known to break from time to time.
       # See https://github.com/actions/virtual-environments/issues/323
       - name: Remove broken apt repos [Ubuntu]
-        if: matrix.os == 'ubuntu-20.04' || matrix.os == 'ubuntu-22.04'
+        if: runner.os == 'Linux'
         run: |
           for apt_file in `grep -lr microsoft /etc/apt/sources.list.d/`; do sudo rm $apt_file; done
 
-      # Use apt-get instead of apt as apt does not have a stable CLI interface. The apt tool prints out
-      # "WARNING: apt does not have a stable CLI interface. Use with caution in scripts."
       - name: Install tools (Linux)
-        if: matrix.os == 'ubuntu-20.04' || matrix.os == 'ubuntu-22.04'
+        if: runner.os == 'Linux'
         run: |
           sudo apt-get update
           sudo apt-get install cccc
@@ -98,24 +65,25 @@ jobs:
           sudo apt-get install clang-format-14
           sudo apt-get install clang-tidy-14
 
-      # Have to install newer version from non-apt source due to SSL library compatibility issues.
-      - name: Install node
-        if: matrix.os == 'ubuntu-20.04' || matrix.os == 'ubuntu-22.04'
+      - name: Install node tools
+        if: runner.os == 'Linux'
         run: |
           npm install -g [email protected]
           npm install -g npm-groovy-lint
 
       - name: Test with mypy
         run: |
-          mypy --ignore-missing-imports --strict statick statick_tool/
+          mypy --ignore-missing-imports --strict src/statick_tool/
 
-      - name: Statick markdown
-        if: matrix.os == 'ubuntu-20.04' || matrix.os == 'ubuntu-22.04'
-        run: |
-          statick . --check --profile documentation.yaml
+      # Re-enable this step when statick-md switches from yapsy to entry points plugin approach.
+      # - name: Statick markdown
+      #   if: runner.os == 'Linux'
+      #   run: |
+      #     pip install statick-md
+      #     statick . --check --profile documentation.yaml
 
       - name: Sphinx lint
-        if: matrix.os == 'ubuntu-20.04' || matrix.os == 'ubuntu.22.04'
+        if: runner.os == 'Linux'
         uses: ammaraskar/sphinx-action@master
         with:
           docs-folder: 'docs/'
@@ -125,12 +93,13 @@ jobs:
           python -m tox
 
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@v3
+        uses: codecov/codecov-action@v4
         with:
           fail_ci_if_error: false
+          token: ${{ secrets.CODECOV_TOKEN }}
 
       - name: Self check
-        if: matrix.os == 'ubuntu-20.04' || matrix.os == 'ubuntu-22.04'
+        if: runner.os == 'Linux'
         run: |
           mkdir statick-output
-          ./statick . --output-directory statick-output --check --profile self_check.yaml --log INFO
+          statick . --output-directory statick-output --check --profile self_check.yaml --log INFO
diff --git a/.gitignore b/.gitignore
@@ -14,6 +14,7 @@ build/
 dist/
 *.egg-info/
 MANIFEST
+requirements.txt
 
 docs/_build
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,19 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## Unreleased
+
+### Added
+
+- Use of `pyproject.toml` instead of `setup.py` and `requirements.txt`.
+- Added support for Python 3.12 and 3.13.
+- Switched from yapsy to setuptools for plugin mechanism. (#508)
+
+### Fixed
+
+- Run `isort` on unit test files.
+- Handle updated warning type from `cppcheck` introduced in version 2.8.
+
 ## v0.10.0 - 2025-01-03
 
 ### Added

diff --git a/README.md b/README.md
@@ -160,7 +160,6 @@ Each _tool_ plugin provides the types of files it can analyze, then the output o
 determine the specific files that should be analyzed by each tool.
 
 The _tool_ plugin can also specify any other tools that are required to run before the current tool can act.
-For example, `cppcheck` depends on the output of the `make` tool.
 
 The _tool_ plugin then scans each package by invoking the binary associated with the tool.
 The output of the scan is parsed to generate the list of issues discovered by Statick.
@@ -418,10 +417,14 @@ $ statick . --output-directory /tmp/x --timings
 
 ## Advanced Installation
 
-To install Statick from source on your system and make it part of your `$PATH`:
+To install from source best practices is to use a virtual environment.
+These instructions are for Linux, activating a virtual environment on Mac/Windows is similar.
+To install Statick from source on your system:
 
 ```shell
-sudo python3 setup.py install
+python3 -m venv venv
+. venv/bin/activate
+pip install .
 ```
 
 ## Existing Plugins
@@ -648,34 +651,43 @@ The resource file (in your _user path_) must be named `_clang-format`.
 
 If you have the need to support any type of _discovery_, _tool_, or _reporting_ plugin that does not come built-in
 with Statick then you can write a custom plugin.
+Declare the plugin module as a [module entry point] in `pyproject.toml` and provide the path  according to plugin type
+(_discovery_, _tool_, _reporting_).
 
-Plugins consist of both a Python file and a `yapsy` file.
-For a description of how yapsy works, check out the [yapsy documentation](http://yapsy.sourceforge.net/).
-
-A _user path_ with some custom plugins may look like
+A _user path_ with custom plugins and configuration files may look like
 
 ```shell
-my_custom_config
-  setup.py
-  |- plugins
-     |- my_discovery_plugin
-        |- my_discovery_plugin.py
-        |- my_discovery_plugin.yapsy
-     |- my_tool_plugins
-        |- my_tool_plugin.py
-        |- my_tool_plugin.yapsy
-        |- my_other_tool_plugin.py
-        |- my_other_tool_plugin.yapsy
-  |- rsc
-     |- config.yaml
-     |- exceptions.yaml
+my-custom-project
+  pyproject.toml
+  |- src
+     |- statick_tool
+        |- plugins
+           |- discovery
+              |- my_discovery_plugin.py
+           |- tool
+              |- my_tool_plugin.py
+              |- my_other_tool_plugin.py
+        |- rsc
+           |- config.yaml
+           |- exceptions.yaml
+```
+
+In `pyproject.toml` declare the plugins as entry points.
+
+```toml
+[project.entry-points."statick_tool.plugins.discovery"]
+my_discovery_name = "statick_tool.plugins.discovery.my_discovery_plugin:MyDiscoveryPlugin"
+
+[project.entry-points."statick_tool.plugins.tool"]
+my_tool_name = "statick_tool.plugins.tool.my_tool_plugin:MyToolPlugin"
+my_other_tool_name = "statick_tool.plugins.tool.my_other_tool_plugin:MyOtherToolPlugin"
 ```
 
 For the actual implementation of a plugin, it is recommended to copy a suitable default plugin provided by Statick and
 modify as needed.
 
-For the contents of `setup.py`, it is recommended to copy a working external plugin.
-Some examples are [statick-fortify](https://github.com/soartech/statick-fortify) and [statick-tex](https://github.com/tdenewiler/statick-tex).
+For the contents of `pyproject.toml`, it is recommended to copy a working external plugin.
+Some examples are [statick-md] and [statick-tex].
 Those plugins are set up in such a way that they work with Statick when released on PyPI.
 
 ## Examples
@@ -834,14 +846,14 @@ If you have a unit test file at `tests/my_module/test_my_module.py` you can easi
 and save yourself a lot of time during development.
 
 ```shell
-python3 -m pytest --cov=statick_tool/ tests/my_module/test_my_module.py
+python3 -m pytest --cov=src/statick_tool/ tests/my_module/test_my_module.py
 ```
 
 To run all the tests and get a report with branch coverage specify the `tests` directory.
 Any subdirectory will run all the tests in that subdirectory.
 
 ```shell
-python3 -m pytest --cov=statick_tool/ --cov-report term-missing --cov-report html --cov-branch tests/
+python3 -m pytest --cov=src/statick_tool/ --cov-report term-missing --cov-report html --cov-branch tests/
 ```
 
 ### Mypy
@@ -893,6 +905,7 @@ His commits were scrubbed from git history upon the initial public release.
 [lizard]: https://github.com/terryyin/lizard
 [logging]: https://docs.python.org/3/howto/logging.html
 [make]: https://gcc.gnu.org/onlinedocs/libstdc++/index.html
+[module entry point]: https://setuptools.pypa.io/en/latest/userguide/entry_point.html#entry-points-syntax
 [mypy]: https://github.com/python/mypy
 [npm-groovy-lint]: https://nvuillam.github.io/npm-groovy-lint/
 [perlcritic]: http://perlcritic.com/
@@ -902,10 +915,12 @@ His commits were scrubbed from git history upon the initial public release.
 [pydocstyle]: http://www.pydocstyle.org/en/stable/
 [pyflakes]: https://github.com/PyCQA/pyflakes
 [pylint]: https://pylint.org/
-[ruff]: https://github.com/charliermarsh/ruff
 [ros]: https://www.ros.org/
+[ruff]: https://github.com/charliermarsh/ruff
 [shellcheck]: https://github.com/koalaman/shellcheck
 [spotbugs]: https://github.com/spotbugs/spotbugs
+[statick-md]: https://github.com/sscpac/statick-md
+[statick-tex]: https://github.com/tdenewiler/statick-tex
 [uncrustify]: https://github.com/uncrustify/uncrustify
 [xmllint]: http://xmlsoft.org/
 [yamllint]: https://yamllint.readthedocs.io/en/stable/
diff --git a/docker/requirements.txt b/docker/requirements.txt
@@ -26,4 +26,3 @@ statick-web
 tabulate
 xmltodict
 yamllint
-yapsy