diff --git a/_static/pyopensci-logo-package-guide.png b/_static/pyopensci-logo-package-guide.png new file mode 100644 index 00000000..cad4d0ff Binary files /dev/null and b/_static/pyopensci-logo-package-guide.png differ diff --git a/conf.py b/conf.py index 96b03ac0..f8631606 100644 --- a/conf.py +++ b/conf.py @@ -104,7 +104,6 @@ "show_toc_level": 1, # "navbar_align": "left", # [left, content, right] For testing that the navbar items align properly "github_url": "https://github.com/pyopensci/python-package-guide", - "footer_start": ["copyright"], "footer_end": [], } @@ -148,4 +147,7 @@ # Social cards ogp_site_url = "https://www.pyopensci.org/python-package-guide/" -ogp_image = "_static/pyopensci-python-package-guide.png" +ogp_social_cards = { + "line_color": "#6D597A", + "image": "_static/pyopensci-logo-package-guide.png", +} diff --git a/index.md b/index.md index 7e4160bd..483beb56 100644 --- a/index.md +++ b/index.md @@ -133,24 +133,26 @@ Learn about best practices for: :::: ::::{grid-item} -:::{card} ✨ Tests ✨ +:::{card} ✨ Tests for your Python package ✨ :class-card: left-aligned * [Intro to testing](tests/index.md) -* [Write tests](tests/write-tests.md) -* [Types of tests](tests/test-types.md) +* [Write tests](tests/write-tests) +* [Types of tests](tests/test-types) +* [Run tests locally](tests/run-tests) +* [Run tests in CI](tests/tests-ci) *We are actively working on this section. [Follow development here.](https://github.com/pyOpenSci/python-package-guide)* ::: :::: ::::{grid-item} -:::{card} ✨ Code checks & clean code ✨ +:::{card} ✨ Code style & Format ✨ :class-card: left-aligned +:link: CONTRIBUTING +:link-type: doc * [Code style](package-structure-code/code-style-linting-format.md) - -*We are actively working on this section. [Follow development here.](https://github.com/pyOpenSci/python-package-guide)* ::: :::: diff --git a/tests/index.md b/tests/index.md index e4dade25..4ba7adfa 100644 --- a/tests/index.md +++ b/tests/index.md @@ -36,6 +36,29 @@ package: unit tests, integration tests and end-to-end (or functional) tests. Lea ::: :::: +::::{grid-item} +:::{card} ✨ Run tests locally ✨ +:link: test-types +:link-type: doc +:class-card: left-aligned + +If you expect your users to use your package across different versions +of Python, then using an automation tool such as nox to run your tests is useful. Learn about the various tools that you can use to run your tests across python versions here. +::: +:::: + +::::{grid-item} +:::{card} ✨ Run tests online (using CI) ✨ +:link: test-types +:link-type: doc +:class-card: left-aligned + +Continuous integration platforms such as GitHub actions can be +useful for running your tests across both different Python versions +and different operating systems. Learn about setting up tests to run in Continuous Integration here. +::: +:::: + ::::: @@ -54,4 +77,6 @@ Graphic showing the elements of the packaging process. Intro Write tests Test types +Run tests locally +Run tests online (using CI) ``` diff --git a/tests/run-tests.md b/tests/run-tests.md new file mode 100644 index 00000000..73cd7510 --- /dev/null +++ b/tests/run-tests.md @@ -0,0 +1,268 @@ +```{eval-rst} +:og:description: Learn how to setup and run tests for your Python package locally on your computer using automation tools such as Nox. Also learn about other tools that scientific Python community members use to run tests. +:og:title: Run tests for your Python package across Python versions +``` + +# Run Python package tests + +Running your tests is important to ensure that your package +is working as expected. It's good practice to consider that tests will run on your computer and your users' computers that may be running a different Python version and operating systems. Think about the following when running your tests: + +1. Run your test suite in a matrix of environments that represent the Python versions and operating systems your users are likely to have. +2. Running your tests in an isolated environment provides confidence in the tests and their reproducibility. This ensures that tests do not pass randomly due to your computer's specific setup. For instance, you might have unexpectedly installed dependencies on your local system that are not declared in your package's dependency list. This oversight could lead to issues when others try to install or run your package on their computers. + +On this page, you will learn about the tools that you can use to both run tests in isolated environments and across +Python versions. + + + +### Tools to run your tests + +There are three categories of tools that will make is easier to setup +and run your tests in various environments: + +1. A **test framework**, is a package that provides a particular syntax and set of tools for _both writing and running your tests_. Some test frameworks also have plugins that add additional features such as evaluating how much of your code the tests cover. Below you will learn about the **pytest** framework which is one of the most commonly used Python testing frameworks in the scientific ecosystem. Testing frameworks are essential but they only serve to run your tests. These frameworks don't provide a way to easily run tests across Python versions without the aid of additional automation tools. +2. **Automation tools** allow you to automate running workflows such as tests in specific ways using user-defined commands. For instance it's useful to be able to run tests across different Python versions with a single command. Tools such as [**nox**](https://nox.thea.codes/en/stable/index.html) and [**tox**](https://tox.wiki/en/latest/index.html) also allow you to run tests across Python versions. However, it will be difficult to test your build on different operating systems using only nox and tox - this is where continuous integration (CI) comes into play. +3. **Continuous Integration (CI):** is the last tool that you'll need to run your tests. CI will not only allow you to replicate any automated builds you create using nox or tox to run your package in different Python environments. It will also allow you to run your tests on different operating systems (Windows, Mac and Linux). [We discuss using CI to run tests here](tests-ci). + +:::{list-table} Table: Testing & Automation Tool +:widths: 40 15 15 15 15 +:header-rows: 1 +:align: center +:stub-columns: 1 +:class: pyos-table + +* - Features + - Testing Framework (pytest) + - Test Runner (Tox) + - Automation Tools (Nox) + - Continuous Integration (GitHub Actions) +* - Run Tests Locally + - + - + - + - +* - Run Tests Online + - + - + - + - +* - Run Tests Across Python Versions + - + - + - + - +* - Run Tests In Isolated Environments + - + - + - + - +* - Run Tests Across Operating Systems (Windows, MacOS, Linux) + - + - + - + - +* - Use for other automation tasks (e.g. building docs) + - + - + - + - +::: + + +## What testing framework / package should I use to run tests? + +We recommend using `Pytest` to build and run your package tests. Pytest is the most common testing tool used in the Python ecosystem. + +[The Pytest package](https://docs.pytest.org/en/latest/) also has a number of +extensions that can be used to add functionality such as: + +- [pytest-cov](https://pytest-cov.readthedocs.io/en/latest/) allows you to analyze the code coverage of your package during your tests, and generates a report that you can [upload to codecov](https://codecov.io/). + +:::{todo} +[Learn more about code coverage here.](code-cov) +::: + +```{note} +Your editor or IDE may add additional convenience for running tests, setting breakpoints, and toggling the `–no-cov` flag. Check your editor's documentation for more information. +``` + +## Run tests using pytest + +If you are using **pytest**, you can run your tests locally by +calling: + +`pytest` + +Or if you want to run a specific test file - let's call this file "`test_module.py`" - you can run: + +`pytest test_module.py` + +Learn more from the [get started docs](https://docs.pytest.org/en/7.1.x/getting-started.html). + +Running pytest on your computer is going to run your tests in whatever +Python environment you currently have activated. This means that tests will be +run on a single version of Python and only on the operating system that you +are running locally. + +An automation tool can simplify the process of running tests +in various Python environments. + +:::{admonition} Tests across operating systems +If you want to run your tests across different operating systems you can [continuous integration. Learn more here](tests-ci). +::: + +### Tools to automate running your tests + +To run tests on various Python versions or in various specific environments with a single command, you can use an automation tool such as `nox` or `tox`. +Both `nox` and `tox` can create an isolated virtual environments. This allows you to easily run your tests in multiple environments and across Python versions. + +We will focus on [Nox](https://nox.thea.codes/) in this guide. `nox` is a Python-based automation tool that builds upon the features of both `make` and `tox`. `nox` is designed to simplify and streamline testing and development workflows. Everything that you do with `nox` can be implemented using a Python-based interface. + +```{admonition} Other automation tools you'll see in the wild +:class: note + +- **[Tox](https://tox.wiki/en/latest/index.html#useful-links)** is an automation tool that supports common steps such as building documentation, running tests across various versions of Python, and more. You can find [a nice overview of tox in the plasmaPy documentation](https://docs.plasmapy.org/en/stable/contributing/testing_guide.html#using-tox). + +- **[Hatch](https://github.com/ofek/hatch)** is a modern end-to-end packaging tool that works with the popular build backend called hatchling. `hatch` offers a `tox`-like setup where you can run tests locally using different Python versions. If you are using `hatch` to support your packaging workflow, you may want to also use its testing capabilities rather than using `nox`. + +* [**make:**](https://www.gnu.org/software/make/manual/make.html) Some developers use Make, which is a build automation tool, for running tests +due to its versatility; it's not tied to a specific language and can be used +to run various build processes. However, Make's unique syntax and approach can +make it more challenging to learn, particularly if you're not already familiar +with it. Make also won't manage environments for you like **nox** will do. +``` + +## Run tests across Python versions with nox + +**Nox** is a great automation tool to learn because it: + +- Is Python-based making it accessible if you already know Python and +- Will create isolated environments to run workflows. + +`nox` simplifies creating and managing testing environments. With `nox`, you can +set up virtual environments, and run tests across Python versions using the environment manager of your choice with a +single command. + +:::{note} Nox Installations + +When you install and use nox to run tests across different Python versions, nox will create and manage individual `venv` environments for each Python version that you specify in the nox function. + +Nox will manage each environment on its own. +::: + +Nox can also be used for other development tasks such as building +documentation, creating your package distribution, and testing installations +across both PyPI related environments (e.g. venv, virtualenv) and `conda` (e.g. `conda-forge`). + +To get started with nox, you create a `noxfile.py` file at the root of your +project directory. You then define commands using Python functions. +Some examples of that are below. + +## Test Environments + +By default, `nox` uses the Python built in `venv` environment manager. A virtual environment (`venv`) is a self-contained Python environment that allows you to isolate and manage dependencies for different Python projects. It helps ensure that project-specific libraries and packages do not interfere with each other, promoting a clean and organized development environment. + +An example of using nox to run tests in `venv` environments for Python versions 3.9, 3.10, 3.11 and 3.12 is below. + +```{warning} +Note that for the code below to work, you need to have all 4 versions of Python installed on your computer for `nox` to find. +``` + +### Nox with venv environments + +```{todo} +TODO: add some tests above and show what the output would look like in the examples below... +``` + +Below is an example of setting up nox to run tests using `venv` which is the built in environment manager that comes with base Python. + +Note that the example below assumes that you have [setup your `pyproject.toml` to declare test dependencies in a way that pip +can understand](../package-structure-code/declare-dependencies.md). An example +of that setup is below. + +```toml +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "pyosPackage" +version = "0.1.0" +dependencies = [ + "geopandas", + "xarray", +] + +[project.optional-dependencies] +tests = ["pytest", "pytest-cov"] +``` + +If you have the above setup, then you can use `session.install(".[tests]")` to install your test dependencies. +Notice that below one single nox session allows you to run +your tests on 4 different Python environments (Python 3.9, 3.10, 3.11, and 3.12). + +```python +# This code would live in a noxfile.py file located at the root of your project directory +import nox + +# For this to run you will need to have python3.9, python3.10 and python3.11 installed on your computer. Otherwise nox will skip running tests for whatever versions are missing + +@nox.session(python=["3.9", "3.10", "3.11", "3.12"]) +def test(session): + + # install + session.install(".[tests]") + + # Run tests + session.run("pytest") + +``` + +Above you create a nox session in the form of a function +with a `@nox.session` decorator. Notice that within the decorator you declare the versions of python that you +wish to run. + +To run the above you'd execute the following command, specifying which session +with `--session` (sometimes shortened to `-s`). Your function above +is called test, therefore the session name is test. + +``` +nox --session test +``` + +### Nox with conda / mamba + +Below is an example for setting up nox to use mamba (or conda) for your +environment manager. +Note that unlike venv, conda can automatically install +the various versions of Python that you need. You won't need to install all four Python versions if you use conda/mamba, like you do with `venv`. + +```{note} +For `conda` to work with `nox`, you will need to +ensure that either `conda` or `mamba` is installed on your computer. +``` + +```python +# This code should live in your noxfile.py file +import nox + +# The syntax below allows you to use mamba / conda as your environment manager, if you use this approach you don’t have to worry about installing different versions of Python + +@nox.session(venv_backend='mamba', python=["3.9", "3.10", "3.11", "3.12"]) +def test_mamba(session): + """Nox function that installs dev requirements and runs + tests on Python 3.9 through 3.12 + """ + + # Install dev requirements + session.conda_install(".[tests]") + # Run tests using any parameters that you need + session.run("pytest") +``` + +To run the above session you'd use: + +```bash +nox --session test_mamba +``` diff --git a/tests/tests-ci.md b/tests/tests-ci.md new file mode 100644 index 00000000..2ca0bf1d --- /dev/null +++ b/tests/tests-ci.md @@ -0,0 +1,83 @@ +# Run tests with Continuous Integration + +Running your [test suite locally](run-tests) is useful as you develop code and also test new features or changes to the code base. However, you also will want to setup Continuous Integration (CI) to run your tests online. CI allows you to run all of your tests in the cloud. While you may only be able to run tests locally on a specific operating system, using CI you can specify tests to run both on various versions of Python and across different operating systems. + +CI can also be triggered for pull requests and pushes to your repository. This means that every pull request that you, your maintainer team or a contributor submit, can be tested. In the end CI testing ensures your code continues to run as expected even as changes are made to the code base. + +::::{todo} +```{note} +[Learn more about Continuous Integration and how it can be used, here.](ci) +``` +:::: + +## CI & pull requests + +CI is invaluable if you have outside people contributing to your software. +You can setup CI to run on all pull requests submitted to your repository. +CI can make your repository more friendly to new potential contributors. +It allows users to contribute code, documentation fixes and more without +having to create development environments, run tests and build documentation +locally. + +## Example GitHub action that runs tests + +Below is an example GitHub action that runs tests using nox +across both Windows, Mac and Linux and on Python versions +3.9-3.11. + +To work properly, this file should be located in a root directory of your +GitHub repository: + +```bash +pyospackage/ +├──.github/ + └── workflows/ + └── run-tests.yml # The name of this file can be whatever you wish +``` + + +```yaml +name: Pytest unit/integration + +on: + pull_request: + push: + branches: + - main + +# Use bash by default in all jobs +defaults: + run: + shell: bash + +jobs: + build-test: + name: Test Run (${{ matrix.python-version }}, ${{ matrix.os }}) + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: ["ubuntu-latest", "macos-latest", "windows-latest"] + python-version: ["3.9", "3.10", "3.11"] + + steps: + - uses: actions/checkout@v4 + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v4 + with: + python-version: ${{ matrix.python-version }} + - name: Install dependencies + run: | + python -m pip install --upgrade pip + python -m pip install nox + - name: List installed packages + run: pip list + - name: Run tests with pytest & nox + run: | + nox -s tests-${{ matrix.python-version }} + # You only need to upload code coverage once to codecov unless you have a + # more complex build that you need coverage for. + - name: Upload coverage to Codecov + if: ${{ matrix.os == 'ubuntu-latest' && matrix.python-version == '3.10'}} + uses: codecov/codecov-action@v3 +```