Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update install to rely on mamba / conda envs only #214

Merged
merged 6 commits into from
Jan 8, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Update to rely on mamba / conda envs only
brynpickering committed Dec 7, 2023
commit 2a913feb76b0938fe2d8876073c56fed925465b0
86 changes: 0 additions & 86 deletions .github/workflows/build_pipeline.yml

This file was deleted.

32 changes: 32 additions & 0 deletions .github/workflows/commit-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Minimal CI

on:
push:
branches:
- "**"
paths-ignore:
- README.md
- CHANGELOG.md
- LICENSE
- CONTRIBUTING.md
- docs/**
- mkdocs.yml

jobs:
test:
uses: arup-group/actions-city-modelling-lab/.github/workflows/python-install-lint-test.yml@main
with:
os: ubuntu-latest
py3version: "11"
notebook_kernel: genet
lint: false
additional_mamba_args: coin-or-cbc "proj>=9.3"

aws-upload:
needs: test
if: needs.test.result == 'success'
uses: arup-group/actions-city-modelling-lab/.github/workflows/aws-upload.yml@main
secrets:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_S3_CODE_BUCKET: ${{ secrets.AWS_S3_CODE_BUCKET }}
91 changes: 29 additions & 62 deletions .github/workflows/daily-scheduled-ci.yml
Original file line number Diff line number Diff line change
@@ -1,69 +1,36 @@
name: Daily GeNet CI Build
name: Daily CI

on:
schedule:
- cron: '37 14 * * 1-5'
- cron: '23 14 * * 1-5'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I INSIST ON RUNNING THIS ON THE 37th MINUTE!!! 🤣

(Actually, thinking about it, 37 actually does carry a very small advantage over 23, in that it makes clear that this is the minute, rather than the hour, but that's admittedly a marginal gain)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You know what, I did think it was hours 😅 maybe I should change it back to 37...


jobs:
build:
get-date:
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v3

- uses: actions/setup-python@v1
with:
python-version: 3.11

- uses: actions/cache@v1
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-

- name: Install dependencies
run: |
export ACCEPT_EULA=Y
sudo apt-get update
python -m pip install --upgrade pip
sudo apt-get install -y python3-pip libgdal-dev locales
sudo apt-get install -y libspatialindex-dev
sudo apt-get install -y coinor-cbc
export CPLUS_INCLUDE_PATH=/usr/include/gdal
export C_INCLUDE_PATH=/usr/include/gdal
sudo apt-get install ca-certificates
export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
pip install GDAL==3.4.1
pip install -e '.[dev]'

- name: Install jupyter kernel
run: python -m ipykernel install --user --name genet

- name: Run tests
run: pytest

- name: Send build success notification
if: success()
uses: rtCamp/[email protected]
env:
SLACK_MESSAGE: ${{ github.repository }} Daily scheduled CI Build ${{ github.run_number }} has succeeded
SLACK_TITLE: Daily Scheduled CI Build Success
SLACK_CHANNEL: city-modelling-feeds
SLACK_USERNAME: GitHub Build Bot
SLACK_ICON: https://slack-files2.s3-us-west-2.amazonaws.com/avatars/2017-12-19/288981919427_f45f04edd92902a96859_512.png
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}

- name: Send build failure notification
if: failure()
uses: rtCamp/[email protected]
env:
SLACK_COLOR: '#FF0000'
SLACK_LINK_NAMES: true
SLACK_MESSAGE: '<!here> ${{ github.repository }} Daily scheduled CI Build ${{ github.run_number }} has failed'
SLACK_TITLE: Daily Scheduled CI Build Failure!
SLACK_CHANNEL: city-modelling-feeds
SLACK_USERNAME: GitHub Build Bot
SLACK_ICON: https://slack-files2.s3-us-west-2.amazonaws.com/avatars/2017-12-19/288981919427_f45f04edd92902a96859_512.png
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
- name: Add date to github output env
run: echo "DATE=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT

test:
needs: get-date
uses: arup-group/actions-city-modelling-lab/.github/workflows/python-install-lint-test.yml@main
with:
os: ubuntu-latest
py3version: "11"
notebook_kernel: genet
pytest_args: '--no-cov' # ignore coverage
cache_mamba_env: false
lint: false
mamba_env_name: daily-ci
additional_mamba_args: coin-or-cbc "proj>=9.3"

slack-notify-ci:
needs: test
if: always()
uses: arup-group/actions-city-modelling-lab/.github/workflows/slack-notify.yml@main
secrets:
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
with:
result: needs.test.result
channel: genet-feed
message: Daily CI action
41 changes: 41 additions & 0 deletions .github/workflows/pr-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Pull Request CI

on:
pull_request:
branches:
- main
paths-ignore:
- README.md
- CHANGELOG.md
- LICENSE
- CONTRIBUTING.md
- docs/**
- mkdocs.yml

jobs:
test:
strategy:
matrix:
os: [windows-latest, ubuntu-latest, macos-latest]
py3version: ["9", "11"]
fail-fast: false
uses: arup-group/actions-city-modelling-lab/.github/workflows/python-install-lint-test.yml@main
with:
os: ${{ matrix.os }}
py3version: ${{ matrix.py3version }}
notebook_kernel: genet
lint: false
pytest_args: '--no-cov' # ignore coverage
upload_to_codecov: false
additional_mamba_args: coin-or-cbc "proj>=9.3"

test-coverage:
uses: arup-group/actions-city-modelling-lab/.github/workflows/python-install-lint-test.yml@main
with:
os: ubuntu-latest
py3version: "11"
notebook_kernel: genet
lint: false
pytest_args: 'tests/' # ignore example notebooks
upload_to_codecov: true
additional_mamba_args: coin-or-cbc "proj>=9.3"
21 changes: 7 additions & 14 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,18 +1,11 @@
FROM python:3.11.4-bullseye
FROM mambaorg/micromamba:1.5.3-bullseye-slim

RUN apt-get update && \
apt-get upgrade -y && \
apt-get -y install gcc git libgdal-dev libgeos-dev libspatialindex-dev curl coinor-cbc && \
rm -rf /var/lib/apt/lists/*
COPY --chown=$MAMBA_USER:$MAMBA_USER . ./src

RUN curl -sL https://deb.nodesource.com/setup_17.x | bash -
RUN apt-get -y install nodejs && node --version && npm --version && \
rm -rf /var/lib/apt/lists/*
RUN micromamba install -y -n base -c conda-forge -c city-modelling-lab python=3.11 "proj>=9.3" pip coin-or-cbc --file src/requirements/base.txt && \
micromamba clean --all --yes
ARG MAMBA_DOCKERFILE_ACTIVATE=1

RUN /usr/local/bin/python -m pip install --no-cache-dir --compile --upgrade pip

COPY . .

RUN pip3 install cmake>=3
RUN pip3 install --no-cache-dir --compile -e . && pip cache purge
RUN pip install --no-deps ./src

ENTRYPOINT ["/usr/local/bin/_entrypoint.sh"]
124 changes: 43 additions & 81 deletions README.md
Original file line number Diff line number Diff line change
@@ -10,18 +10,12 @@
+ [Building the image](#build-the-image)
+ [Running GeNet from the container](#running-a-container-with-a-pre-baked-script)
* [Installation as a Python Package](#installation-as-a-python-package)
+ [Native dependencies](#native-dependencies)
+ [Installing a development environment](#installing-a-development-environment)
+ [A note on the mathematical solver](#a-note-on-the-mathematical-solver)
+ [Installing the native dependencies](#installing-the-native-dependencies)
+ [Install dev prereqs](#install-dev-prereqs)
+ [Install Python dependencies](#install-python-dependencies)
+ [Install GeNet in to the virtual environment](#install-genet-in-to-the-virtual-environment)
+ [Install Kepler dependencies](#install-kepler-dependencies)
- [Developing GeNet](#developing-genet)
* [Unit tests](#unit-tests)
* [Code coverage report](#generate-a-unit-test-code-coverage-report)
* [Linting](#lint-the-python-code)
* [Smoke testing Jupyter notebooks](#smoke-test-the-jupyter-notebooks)


## Overview
@@ -114,103 +108,71 @@ as part of your command:
Note, if you reference data inside your script, or pass them as arguments to the script, they need to reference the
aliased path inside the container, here: `/mnt/`, rather than the path `/local/path/`.

### Installation as a Python Package / CLI
### Installation as a Python Package

You can in your base installation of python or a virtual environment.
You can use GeNet's CLI to run pre-baked modifications or checks on networks.
You can also write your own python scripts, importing genet as a package, use IPython shell or Jupyter Notebook to load
up a network, inspect or change it and save it out to file. Check out the
[wiki pages](https://github.com/arup-group/genet/wiki/Functionality-and-Usage-Guide) and
[example jupyter notebooks](https://github.com/arup-group/genet/tree/master/notebooks)
for usage examples.

**Note:** if you plan only to _use_ GeNet rather than make code changes to it, you can avoid having to perform any
local installation by using [GeNet's Docker image](#using-docker). If you are going to make code changes or use GeNet's
CLI locally, follow the steps below:

#### Native dependencies
GeNet uses some Python libraries that rely on underlying native libraries for things like geospatial calculations and
linear programming solvers. Before you install GeNet's Python dependencies, you must first install these native
libraries.
You can also write your own python scripts, importing genet as a package, use IPython shell or Jupyter Notebook to load up a network, inspect or change it and save it out to file.
Check out the [wiki pages](https://github.com/arup-group/genet/wiki/Functionality-and-Usage-Guide) and [example jupyter notebooks](https://github.com/arup-group/genet/tree/master/notebooks) for usage examples.

**Note:** if you plan only to _use_ GeNet rather than make code changes to it, you can ignore the rest of this section and instead use [GeNet's Docker image](#using-docker).
If you are going to make code changes or use GeNet's CLI locally, follow the steps below to [install a development environment](#installing-a-development-environment).

### Installing a development environment

To create a development environment for genet, with all libraries required for development and quality assurance installed, it is easiest to install genet using the [mamba](https://mamba.readthedocs.io/en/latest/index.html) package manager, as follows:

1. Install mamba with the [Mambaforge](https://github.com/conda-forge/miniforge#mambaforge) executable for your operating system.
2. Open the command line (or the "miniforge prompt" in Windows).
3. Download (a.k.a., clone) the genet repository: `git clone git@github.com:arup-group/genet.git`
4. Change into the `genet` directory: `cd genet`
5. Create the genet mamba environment: `mamba create -n genet -c conda-forge -c city-modelling-lab --file requirements/base.txt --file requirements/dev.txt`
6. Activate the genet mamba environment: `mamba activate genet`
7. Install the genet package into the environment, in editable mode and ignoring dependencies (we have dealt with those when creating the mamba environment): `pip install --no-deps -e .`
8. Create a jupyter kernel linked to the environment to enable example notebook testing: `ipython kernel install --user --name=genet`

All together:
``` shell
git clone git@github.com:arup-group/genet.git
cd genet
mamba create -n genet -c conda-forge -c city-modelling-lab --file requirements/base.txt --file requirements/dev.txt
mamba activate genet
pip install --no-deps -e .
ipython kernel install --user --name=genet
```

#### A note on the mathematical solver

**Note**: The default CBC solver is pre-installed inside [GeNet's Docker image](#using-docker), which can save you some
installation effort

To use methods which snap public transit to the graph, GeNet uses a mathematical solver. If you won't be using such
functionality, you do not need to install this solver.
To use methods which snap public transit to the graph, GeNet uses a mathematical solver.
If you won't be using such functionality, you do not need to install this solver.
Methods default to [CBC](https://projects.coin-or.org/Cbc), an open source solver.
You can install this solver (`coin-or-cbc`) along with your other requirements when creating the environment: `mamba create -n genet -c conda-forge -c city-modelling-lab coin-or-cbc --file requirements/base.txt --file requirements/dev.txt`,
or install it after the fact `mamba install -n genet coin-or-cbc`

Another good open source choice is [GLPK](https://www.gnu.org/software/glpk/).
The solver you use needs to support MILP - mixed integer linear programming.

#### Installing the native dependencies
The commands for installing the necessary native libraries vary according to the operating system you are using, for
example:

| OS | Commands |
|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|Mac OS | `brew install boost` <br/> `brew install spatialindex` <br/> `brew install gdal --HEAD` <br/> `brew install gdal` <br/> `brew tap coin-or-tools/coinor` <br/> `brew install coin-or-tools/coinor/cbc` <br/> `brew install cbc` |
|Ubuntu | `sudo apt install libspatialindex-dev` <br/> `sudo apt install libgdal-dev` <br/> `sudo apt install coinor-cbc` |

#### Install dev prereqs
(Use equivalent linux or Windows package management as appropriate for your environment)

brew install python3.11
brew install virtualenv

#### Install Python dependencies
Create and activate a Python virtual environment

virtualenv -p python3.11 venv
source venv/bin/activate

#### Install GeNet in to the virtual environment
Finally, install `GeNet`'s Python dependencies

pip install -e .

After installation, you should be able to successfully run the help command to see all possible commands:

genet --help

To inspect a specific command, run e.g.:

genet simplify-network --help

#### Install Kepler dependencies

Please follow [kepler's installation instructions](https://docs.kepler.gl/docs/keplergl-jupyter#install) to be able to
use the visualisation methods. To see the maps in a jupyter notebook, make sure you enable widgets.
```
jupyter nbextension enable --py widgetsnbextension
```

## Developing GeNet

We welcome community contributions to GeNet; please see our [guide to contributing](CONTRIBUTING.md) and our
[community code of conduct](CODE_OF_CONDUCT.md). If you are making changes to the codebase, you should use the tools
described below to verify that the code still works. All of the following commands assume you are in the project's root
directory.
We welcome community contributions to GeNet; please see our [guide to contributing](CONTRIBUTING.md) and our [community code of conduct](CODE_OF_CONDUCT.md).
If you are making changes to the codebase, you should use the tools described below to verify that the code still works.
All of the following commands assume you are in the project's root directory.

### Unit tests
To run unit tests within genet python environment:

python -m pytest -vv tests

and within a docker container:

docker run cml-genet pytest -vv tests
To run unit tests within genet python environment, run `pytest`.
To improve test runtime you can focus only on the unit tests and not on the tests of the example jupyter notebooks: `pytest tests/`.
To run tests in the Docker container, you will need to update the Dockerfile to include development dependencies and then you can run `docker run cml-genet pytest`.

### Generate a unit test code coverage report

To generate an HTML coverage report at `reports/coverage/index.html`:

./bash_scripts/code-coverage.sh
pytest --cov-report=html

### Lint the python code

Run `pre-commit install` to install pre-commit, which will lint and format your code whenever you commit staged changes.

### Smoke test the Jupyter notebooks

./bash_scripts/notebooks-smoke-test.sh
4 changes: 4 additions & 0 deletions genet/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import pyproj

__version__ = "4.0.0"

from genet.auxiliary_files import AuxiliaryFile
@@ -19,3 +21,5 @@
from genet.schedule_elements import Route, Schedule, Service, Stop
from genet.use.road_pricing import Toll
from genet.utils import elevation, google_directions, graph_operations

pyproj.network.set_network_enabled(False)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It stops it from trying to connect to the network to download transformation gridfiles on-the-fly (see #213). If it does try, there is a bug with handling self-signed SSL certificates that will cause an error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wasn't needed before because the PROJ environment variable associated with this is set to OFF when downloading from homebrew/apt, but is ON when downloading from conda-forge (because in the former it automatically downloads the proj-data dataset too while it isn't included in the latter)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See OSGeo/PROJ#3977 and pyproj4/pyproj#1233 for conversations I've initiated on it in upstream projects