-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: new page overviewing CI & code coverage #165
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
47d43f7
Fix: add initial tests content to guide
lwasser 640a07d
Fix: edits from @namurphy
lwasser 67d2627
Fix: review fixes from Trevor
lwasser 900960d
Fix: edits from @willingc to ci page
lwasser 848a483
Fix: force to main
lwasser e9b849c
Fix: Edits from @ucodery
lwasser f16a18e
'[pre-commit.ci 🤖] Apply code format tools to PR'
pre-commit-ci[bot] 8ca6bd2
fix: cleanup
lwasser 708b23c
fix: pre commit and url fixes
lwasser 044b24e
fix: add toctree to ci landing so it's less empty
lwasser File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,2 @@ | ||
aways | ||
Soler |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,84 @@ | ||
(ci-cd)= | ||
# Continuous Integration and Continuous Deployment (CI/CD) For Python Packages | ||
|
||
When you develop, work on, and contribute to software, there is more to consider than | ||
just writing code. Having tests and checks ensures that your code | ||
runs reliably and follows a consistent format is also important. You can use | ||
**Continuous Integration (CI)** and **Continuous | ||
Deployment (CD)** to run tests and checks on your code every time someone suggests a change online | ||
in a platform like GitHub or GitLab. | ||
|
||
- **Continuous Integration (CI):** Automates the process of running tests, | ||
code checks, and other workflows each time code is updated. | ||
- **Continuous Deployment (CD):** Extends CI by allowing you to automate publishing your package to PyPI, publishing your documentation, and more. | ||
|
||
CI and CD streamline software development by automating repetitive | ||
tasks and ensuring code quality and consistency. Having CI setup also makes it easier for new contributors | ||
to contribute to your code base without setting up all your test suites and | ||
other local checks. | ||
|
||
## What is continuous integration? | ||
|
||
When you’re ready to publish your code online, you can set up Continuous Integration (CI). CI is a platform that allows you to specify and run jobs or workflows you define. | ||
These workflows include: | ||
|
||
- Running your test suite | ||
- Running code checkers / linters / spellcheck | ||
- Building your documentation | ||
|
||
CI allows you to automate running workflows across a suite of environments, including: | ||
|
||
- environments containing different Python versions and | ||
- different operating systems (Mac, Linux, Windows). | ||
|
||
## What is Continuous Deployment (CD)? | ||
|
||
Continuous deployment (CD) extends the CI process by automating the deployment of code changes to production or staging environments. In the case of your open source tool, CD can be used to: | ||
|
||
- Automate publishing to PyPI | ||
- Automate publishing your documentation to GitHub Pages or Read the Docs. | ||
|
||
It is also used once your conda-forge recipe is set up to keep your package up to date on conda-forge. | ||
|
||
### Why use CI | ||
|
||
CI can be configured to run a workflow on every commit pushed to GitHub and every pull request opened. This ensures that any changes made to your package are tested across environments before merging into the main branch of your code. | ||
|
||
These checks are particularly useful if someone new is contributing to your code. Every contributor's change will be tested when pushed to your code repository. | ||
|
||
Together, CI and CD streamline the process of building, testing, and deploying code. They aim to improve software development and publication efficiency, quality, and reliability. | ||
|
||
```{note} | ||
All pyOpenSci packages must use some form of continuous integration. Even if you are not planning to go through peer review, we strongly recommend that you use continuous integration, too! | ||
``` | ||
|
||
In the case of GitHub actions (which we will focus on here), CI workflows are running on online servers that support GitHub. | ||
|
||
## CI / CD platforms | ||
|
||
There are numerous platforms available for CI/CD. Here, we will focus on GitHub Actions (GHA), built into GitHub. GitHub is the most commonly used platform to store scientific open-source software. | ||
|
||
:::{note} | ||
If you use [GitLab](https://about.gitlab.com/) CI/CD, many of the principles described here will apply. However, the workflow files may look different. | ||
::: | ||
|
||
### If you aren't sure, use GitHub Actions | ||
|
||
While you are welcome to use the continuous integration platform of your choice, | ||
we recommend GitHub Actions because it is free-to-use and integrated tightly | ||
into the GitHub user interface. There is also an entire store of GitHub action | ||
templates that you can easily use and adapt to your own needs. | ||
|
||
:::{admonition} Other platforms that you may run into | ||
:class: info | ||
|
||
- [Appveyor:](https://www.appveyor.com/): Supports running tests on Windows operating systems and predated the release of GitHub Actions. Today, AppVeyor supports operating systems beyond Windows. | ||
- [Travis CI:](https://www.travis-ci.com/) had been a common CI platform choice in our ecosystem. Usage dropped after Travis CI ended free support for open-source projects. | ||
- [CircleCI:](https://circleci.com/) CircleCI can be useful for automated builds of websites and documentation since it offers a preview of the PR changes. | ||
::: | ||
|
||
## Embrace automation | ||
|
||
By embracing CI/CD, you can ensure that your code runs as you expect it to across the diverse landscapes of user environments. Further, you can | ||
automate certain checks (and, in some cases, code fixes), including linting and code style. You can even automate spell-checking your documentation | ||
and docstrings! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
(ci-cd-intro)= | ||
# Continuous Integration (CI) and Continuous Deployment (CD) for your Python package | ||
|
||
|
||
:::{toctree} | ||
:caption: Continuous Integration | ||
|
||
|
||
What is CI? <ci.md> | ||
::: |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,66 @@ | ||
# Code coverage for your Python package test suite | ||
|
||
Code coverage measures how much of your package's code runs during testing. | ||
Achieving high coverage can help ensure the reliability of your codebase, but | ||
it’s not a guarantee of quality. Below, we outline key considerations for | ||
using code coverage effectively. | ||
|
||
## Why aim for high code coverage? | ||
|
||
A good practice is to ensure that every line of your code runs at least once | ||
during your test suite. This helps you: | ||
|
||
- Identify untested parts of your codebase. | ||
- Catch bugs that might otherwise go unnoticed. | ||
- Build confidence in your software's stability. | ||
|
||
## Limitations of code coverage | ||
|
||
While high code coverage is valuable, it has its limits: | ||
|
||
- **Difficult-to-test code:** Some parts of your code might be challenging to | ||
test, either due to complexity or limited resources. | ||
- **Missed edge cases:** Running all lines of code doesn’t guarantee that edge | ||
cases are handled correctly. | ||
|
||
Ultimately, you should focus on how your package will be used and ensure your | ||
tests cover those scenarios adequately. | ||
|
||
## Tools for analyzing Python package code coverage | ||
|
||
Some common services for analyzing code coverage are [codecov.io](https://codecov.io/) and [coveralls.io](https://coveralls.io/). These projects are free for open source tools and will provide dashboards that tell you how much of your codebase is covered during your tests. We recommend setting up an account (on either CodeCov or Coveralls) and using it to keep track of your code coverage. | ||
|
||
:::{figure} ../images/code-cov-stravalib.png | ||
:height: 450px | ||
:alt: Screenshot of the code cov service - showing test coverage for the stravalib package. This image shows a list of package modules and the associated number of lines and % lines covered by tests. At the top of the image, you can see what branch is being evaluated and the path to the repository. | ||
|
||
The CodeCov platform is a useful tool if you wish to track code coverage visually. Using it, you can not only get the same summary information that you can get with the **pytest-cov** extension. You can also see what lines are covered by your tests and which are not. Code coverage is useful for evaluating unit tests and/or how much of your package code is "covered". It, however, will not evaluate things like integration tests and end-to-end workflows. | ||
|
||
::: | ||
|
||
|
||
|
||
:::{admonition} Typing & MyPy coverage | ||
You can also create and upload typing reports to CodeCov. | ||
::: | ||
|
||
## Exporting Local Coverage Reports | ||
|
||
In addition to using services like CodeCov or Coveralls, you can generate local coverage reports directly using the **coverage.py** tool. This can be especially useful if you want to create reports in Markdown or HTML format for offline use or documentation. | ||
|
||
To generate a coverage report in **Markdown** format, run: | ||
|
||
```bash | ||
$ python -m coverage report --format=markdown | ||
``` | ||
This command will produce a Markdown-formatted coverage summary that you can easily include in project documentation or share with your team. | ||
|
||
To generate an HTML report that provides a detailed, interactive view of which lines are covered, use: | ||
|
||
```bash | ||
python -m coverage html | ||
``` | ||
|
||
The generated HTML report will be saved in a directory named htmlcov by default. Open the index.html file in your browser to explore your coverage results. | ||
|
||
These local reports are an excellent way to quickly review coverage without setting up an external service. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes space for a CI&data section in our navigation.