Skip to content

Commit

Permalink
Merge pull request #60 from daler/v1.0rc
Browse files Browse the repository at this point in the history
V1.0rc
  • Loading branch information
daler authored Apr 20, 2024
2 parents 11a1d0d + 96878b0 commit 6165bd6
Show file tree
Hide file tree
Showing 39 changed files with 2,054 additions and 1,587 deletions.
102 changes: 36 additions & 66 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,71 +2,57 @@ name: main
on: [push]

jobs:
conda-env:
test:
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10"]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2


- uses: actions/checkout@v4
- name: git setup
# Set up git and export env vars to be used in later steps.
# Note the unconventional mechanism for exporting envs by appending to
# $GITHUB_ENV.
id: git-setup
run: |
git config --global user.email "[email protected]"
git config --global user.name "GitHub Action"
echo "BRANCH=${GITHUB_REF##*/}" >> $GITHUB_ENV
echo "WORKDIR=$(pwd)" >> $GITHUB_ENV
- name: load cached conda env
# Load cached env.
# This looks for a cache based on the hash of requirements.txt and
# test-requirements.txt; if it's found the cached directory is
# restored. There's an extra "v1" on the end so it can be forced to be
# regenerated when needed.
id: cache-env
uses: actions/cache@v2
with:
path: /tmp/test-env
key: ${{ hashFiles('requirements.txt') }}-${{ hashFiles('test-requirements.txt') }}-v1


- name: build new conda env
# Build cache if needed.
# Only runs if there was a cache miss. If this is created. there's
# a "Post load cached env" job (which is automatically created, it's
# not defined here) that will load this into the cache for use next
# time.
if: steps.cache-env.outputs.cache-hit != 'true'
run: |
eval "$(conda shell.bash hook)"
conda create -p /tmp/test-env -y --file requirements.txt --file test-requirements.txt --channel conda-forge --channel bioconda
conda install -n base mamba -y --channel conda-forge
mamba create -p ./env -y python=${{ matrix.python-version }} --file test-requirements.txt --channel conda-forge --channel bioconda
conda activate ./env
pip install -e .
- name: run pytests and build docs
- name: run pytests
# pytests and doctests happen here
run: |
eval "$(conda shell.bash hook)"
source activate /tmp/test-env
python setup.py install
conda activate ./env
pytest -vv --doctest-modules trackhub
- name: build docs
run: |
# To make sure that the links in the built docs refer to the correct
# branch on the trackhub-demo repo, we replace the branch names in the
# .rst files. Note if we're on master branch then this is a no-op.
cd doc
find . -name "*.rst" | xargs sed -i "s|trackhub-demo/master|trackhub-demo/$BRANCH|g"
make doctest html
conda deactivate
eval "$(conda shell.bash hook)"
conda activate ./env
conda env export
(
cd doc
find . -name "*.rst" | xargs sed -i "s|trackhub-demo/master|trackhub-demo/$BRANCH|g"
make doctest html
)
- name: test command-line script
# Ensure that the installed command-line script can be run and creates
# the correct files
run: |
eval "$(conda shell.bash hook)"
source activate /tmp/test-env
conda activate ./env
trackhub_from_excel --template
trackhub_from_excel --create-example a.xlsx
trackhub_from_excel --excel-file a.xlsx
Expand All @@ -77,32 +63,26 @@ jobs:
err=1
fi
done
if [ err == 1 ]; then
if [ $err == 1 ]; then
exit 1
fi
- name: upload the just-built docs as an artifact
# The built docs will be uploaded as a zip file (called docs.zip).
# This file will be available on the Actions page and can be used to
# inspect the final rendered docs. This is useful when building on
# a branch and for contributors to make corrections to the docs without
# needing to set everything up locally.
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: docs
name: docs-${{matrix.python-version}}-${{ github.run_id }}
path: doc/build/html
overwrite: true # v4 requires unique names; this will delete existing before uploading


- name: commit built docs to gh-pages branch
# Commit to the gh-pages branch.
# Note that this step is not restricted to the master branch, which
# lets us better test the process. The changes aren't actually pushed
# though unless we're on the master branch (see next step).
- name: commit built docs to gh-pages
# The changes aren't actually pushed unless we're on the master branch
# (see next step).
#
# Note that cloning just the gh-pages branch to a new directory ended
# up being easier than staying in this directory and copying stuff
# around within it.
# Note that cloning just the gh-pages branch to a new directory ended
# up being easier than staying in this directory and copying stuff
# around within it.
run: |
git clone \
--single-branch \
Expand Down Expand Up @@ -145,7 +125,7 @@ jobs:
git branch
git checkout $BRANCH
eval "$(conda shell.bash hook)"
source activate /tmp/test-env
conda activate ./env
ci/build_examples.py
Expand Down Expand Up @@ -192,15 +172,15 @@ jobs:
# It's possible that the $BRANCH here in the trackhub repo does not yet
# exist over in the trackhub-demo repo. In that case, we should be good
# to push.
# to push immediately.
if [[ -z $(git ls-remote --heads origin $BRANCH) ]]; then
echo "remote branch $BRANCH does not exist"
git commit -m 'update hub'
git push origin $BRANCH --force
exit 0
fi
# Otherwise, only push if there are changes.
# Otherwise, only push to existing branch if there are changes.
if git diff origin/$BRANCH --quiet; then
echo "no changes to push to branch $BRANCH!";
else
Expand All @@ -219,7 +199,7 @@ jobs:
run: |
git checkout $BRANCH
eval "$(conda shell.bash hook)"
source activate /tmp/test-env
conda activate ./env
ci/check_hubs.py
Expand All @@ -231,13 +211,3 @@ jobs:
run: |
ssh-add -D
rm -Rf *
pip-install:
# Separate, parallel job for testing pip installation
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- run: python setup.py sdist
- run: pip install dist/*.tar.gz
- run: python -c 'import trackhub'
15 changes: 15 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
on: workflow_dispatch
jobs:
publish-testpypi:
name: Upload release to Test PyPI
runs-on: ubuntu-latest
environment:
name: testpypi
url: https://pypi.org/p/trackhub
permissions:
id-token: write
steps:
- name: Publish to Test PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
repository-url: https://test.pypi.org/legacy
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
include ez_setup.py
include *.rst
include *.txt
recursive-include trackhub/test/data *
12 changes: 10 additions & 2 deletions ci/check_hubs.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@

for line in open(args.tsv):
line = line.strip()
if line.startswith('#') or not line:
if line.startswith("#") or not line:
continue

# Dest in trackhub demo is the second column of TSV, without the initial directory
Expand All @@ -59,7 +59,15 @@
# with "can't find database hg19 in hg.conf, should have a default
# named "db"". When checking the hub in the Genome Browser it seems
# fine. So allowing this as a "pass".
if line.startswith("can't find database") and "hic_hub" in str(dest):
#
# Some time later...
# hubCheck for those same tracks is now showing "can't find profile
# central in hg.conf" error. Aain the tracks look fine the browser,
# so we'll consider that OK as well.
if "hic_hub" in str(dest) and (
line.startswith("can't find database")
or line.startswith("can't find profile central in hg.conf")
):
warnings = True
elif not (line.startswith("Found") or line.startswith("warning:")):
error = True
Expand Down
49 changes: 36 additions & 13 deletions doc/source/changelog.rst
Original file line number Diff line number Diff line change
@@ -1,44 +1,67 @@
Changelog
=========

Version 0.3
-----------
Version 1.0 (April 2024)
------------------------

This major release is backwards-compatible but adds many new features and tests.

Testing infrastructure overhaul
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

See :ref:`tests` for details.
Extensive example hubs that demonstrate the various track types and
configuration options. These use data provided by UCSC and are built as part of
the testing framework. Thanks to Eva Jason (@evajason).

Documentation updates
~~~~~~~~~~~~~~~~~~~~~
See :ref:`tests` and :ref:`tracktypes` for details.

- Examples have been added for all of the available track types. See
:ref:`tracktypes` for details.
- New section that details how validation works (see :ref:`validation`).
Build a trackhub configured in Excel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Build a trackhub using only a command-line tool and configuration in Excel.
This new tool, ``trackhub_from_excel``, can write out a template you can use to
fill in details on what tracks should be included, how to organize them, and
configuration options like color, visibility, etc.

Running the tool on your filled-out Excel file will build a trackhub ready to
be uploaded.

Thanks Eva Jason (@evajason) for this new feature.

Semi-automated updating of validated parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dramatically improved semi-automatic updating mechanism for making sure
parameters are up-to-date with the main UCSC instance (see
``trackhub/parse.py`` and ``trackhub/parsed_params.py``). These scrape the UCSC
database document HTML to identify parameters, and writes this out to Python
module that can be diffed against the existing ``parsed_params.py`` module. In
addition to supporting validation, this highlights what has changed in the UCSC
Genome Browser since the last trackhub Python package update.

Other changes
~~~~~~~~~~~~~
- New section that details how validation works (see :ref:`validation`).

- `defaultPos` can now be set when using the `default_hub` helper function

- Support for all track types supported in track hubs (see :ref:`tracktypes`)


- There is now one naming convention, `add_tracks`, rather than multiple
methods names that perform this for different track types. `add_view` and
`add_subtracks` can now simply be replaced with `add_tracks`. These methods
will accept track objects, a list of track object, or a combination of the
two. The older method names are retained for backwards compatibility, but all
documentation has been updated to reflect this change.

- dramatically improved semi-automatic updating mechanism for making sure
parameters are up-to-date with the main UCSC instance (see
``trackhub/parse.py`` and ``trackhub/parsed_params.py``).

- fixed indentation when super tracks, composite tracks, and view tracks are
nested.

- improved handling of rsync and handling symlink modification times

- Dropped support for Python 2.7

- Migrated tests from Travis CI to GitHub Actions and to use pytest fixtures.

Version 0.2.4 (Dec 2017)
------------------------

Expand Down
2 changes: 1 addition & 1 deletion doc/source/excelinstructions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ Super tracks are within the track hub and therefore do not need special fields.


Example ``view_config`` sheet
````````````````````````````
`````````````````````````````

.. list-table::
:header-rows: 1
Expand Down
2 changes: 1 addition & 1 deletion doc/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,6 @@ Contents:
assembly_example
groupAutoScale
tracktypes
autodocs
tests
autodocs
changelog
19 changes: 19 additions & 0 deletions doc/source/tests.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@

.. _tests:

Testing infrastructure
======================
Tests are run on GitHub Actions, configured in ``.github/workflows/main.yml``.

In addition to unit tests, now any code in the documentation that shows how to
build example track hubs is handled like this:

- extract code from documentation (see ``ci/example_hubs.tsv`` for the list of
files from which code is extracted)
- execute code to build track hub (see ``ci//build_examples.py``)
- upload code and built track hub (and data, if relevant) to the `trackhub-demo
<https://github.com/daler/trackhub-demo>`_ repository
- run ``hubCheck`` on the just-uploaded hubs (see ``ci/check_hubs.py``)

The just-built track hubs are then live, and linked to from within the
documentation.
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[build-system]
requires = ["setuptools >= 61.0"]
build-backend = "setuptools.build_meta"
9 changes: 6 additions & 3 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,17 @@
package_dir = {"trackhub": "trackhub"},
scripts=["trackhub/trackhub_from_excel"],
license = 'MIT',
author_email="dalerr@niddk.nih.gov",
author_email="ryan.dale@nih.gov",
classifiers=[
'Intended Audience :: Science/Research',
'License :: OSI Approved :: GNU General Public License (GPL)',
'Topic :: Scientific/Engineering :: Bio-Informatics',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'Programming Language :: Python :: 3.12',
'Programming Language :: Python :: 3',
'Topic :: Software Development :: Libraries :: Python Modules',
],
Expand Down
1 change: 1 addition & 0 deletions test-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ pybedtools
pytest
sphinx>4
pygments
sphinx_rtd_theme>=1.2.2
ucsc-bedgraphtobigwig
ucsc-fatotwobit
ucsc-bedtobigbed
Expand Down
Loading

0 comments on commit 6165bd6

Please sign in to comment.