Route choice (#531)

* Removed `reached_first` from A* The current method of skimming for A* produces out of bound accesses for A*. This occurs when it reconsiders more nodes than are in the network. The `reached_view` array is primarily for skimming, as such I've disabled skimming for A*. Skimming for early exit Dijkstra's should also be reconsidered. * Add preliminary route choice * Return noexcepts * Working RouteChoice generation * Removed unused variables and junk * Notes * fixup! Removed unused variables and junk * fixup! fixup! Removed unused variables and junk * Rename files * Minor fixes * Use comrpessed graph * Avoid unnessacary work when we may fill the route set this iteration * Fix pointer issues, move custom imports * Spelling * Prevent memory leak when destination is unreachable * Remove print out * Prevent infinite loop when depth is unlimited * Prevent oob access and incredible memory consumption (2gb/s) from A* * Scratch testing on Arkansas * Remove working_set, PO1 makes this redundant, reduce memory usage Graphs should also been marked as seen regardless of if the destination is reachable * Remove prints * Add parallelised batched method for running a list of od pairs * Remove dead code and fix A* test now that skimming is disabled * Move `RouteChoice.run` to be wrapper around `RouteChoice.batched` No need for both methods * Fix infinite loop in the case that all possible paths are exhausted * Add comprehensive testing * Warning clean up * Update commentary and fix memory leak * Typos * Rename and remove imports * Disable initialisation and bounds checking * Add docs and support blocking centroid flow with tests * Remove debug method and fix small typos * typo * Allow switching path finding method * Add debug method for display in tests * Linting * Scratch parquet work * Better structuring for the table * Add optional immediately disk saving * Test fixes * Update CI workflows In what I consider a bad move, the PyPi wheel installation of pyarrow requires modifying the installation environment to create symlinks for the linker. * Move pyarrow to a hard dependency for now * Linting * CI again * Warn about duplicate OD pairs, fix memory corruption, disable A* * Fix path order * Add link penalisation * Linting * Add parallelised sparse frequency implementation and test Code is unprofiled, I'm not sure if this is the best approach but it works well. It works by stacking all the paths in a route set into a big vector, then sorting it. By sorting it all the links become grouped and we can just count their occurrences. This has the added benefit of the resulting frequency arrays being sorted so we can bisect then later. Generally this has really simple memory accesses and is easy to read. Another possible implementation might be to sort each path individually, then walk and merge them all (not adding duplicates). This is trickier, requires a lot of book keeping to walk n arrays correctly, upside is lower memory usage provided we sort inplace, if not then it should be the same. * Remove debug test * Add route cost computation and test * Add psl gamma computation, not sure what to test against * Add scratch probability implementation * Fix probability calculations * The great refactor, batch probability computations to dump to disk Moves the computation of the path sized logit inside the main multithreaded loop, this lets us batch them as well and dump them to disk along with the reset of the tables. Enable multithreading by default. Catch a handle of memory leaks. * Revert "Update CI workflows" This reverts commit e3ea0c6. * Move the pyarrow symlink creation to setup.py * CI * . * . * Squashed commit of the following: commit 9337fb611463606b0ca89d1270210fa7fdf46714 Author: Jake-Moss <[email protected]> Date: Tue Feb 27 11:22:48 2024 +1000 . commit 3f2c01b Author: Jake-Moss <[email protected]> Date: Tue Feb 27 10:09:51 2024 +1000 I give up commit c77d5e6 Author: Jake-Moss <[email protected]> Date: Tue Feb 27 10:03:52 2024 +1000 . commit 9dc3650 Author: Jake-Moss <[email protected]> Date: Tue Feb 27 09:58:40 2024 +1000 . commit b4b945c Author: Jake-Moss <[email protected]> Date: Tue Feb 27 09:53:10 2024 +1000 . commit e0c32f2 Author: Jake-Moss <[email protected]> Date: Tue Feb 27 09:38:29 2024 +1000 . commit eedb859 Author: Jake-Moss <[email protected]> Date: Tue Feb 27 09:30:20 2024 +1000 . commit d0def11 Author: Jake-Moss <[email protected]> Date: Tue Feb 27 09:27:33 2024 +1000 . commit a6b2a8e Merge: 0548be6 77edeae Author: Jake Moss <[email protected]> Date: Tue Feb 27 09:19:30 2024 +1000 Merge branch 'develop' into pedro/ci_test commit 0548be6 Author: Jake-Moss <[email protected]> Date: Tue Feb 27 09:13:07 2024 +1000 Macos test commit a10c791 Author: Jake-Moss <[email protected]> Date: Wed Feb 21 20:00:01 2024 +1000 Maybe fix MacOS again commit cad0579 Author: Jake-Moss <[email protected]> Date: Wed Feb 21 19:43:00 2024 +1000 Maybe MacOS fix commit 24de8f7 Author: Jake-Moss <[email protected]> Date: Wed Feb 21 19:31:08 2024 +1000 Hopefully fix CI commit 66ffcc3 Author: pveigadecamargo <[email protected]> Date: Wed Feb 21 17:01:57 2024 +1000 . commit f934c34 Author: pveigadecamargo <[email protected]> Date: Wed Feb 21 16:41:36 2024 +1000 . commit 9f2ca20 Author: pveigadecamargo <[email protected]> Date: Wed Feb 21 16:30:59 2024 +1000 . commit ba0d882 Author: pveigadecamargo <[email protected]> Date: Wed Feb 21 16:27:14 2024 +1000 . commit 315cbce Author: Pedro Camargo <[email protected]> Date: Wed Feb 21 01:01:23 2024 +1000 Update pyproject.toml commit d58e7cc Author: Pedro Camargo <[email protected]> Date: Wed Feb 21 00:58:54 2024 +1000 Update pyproject.toml commit dd6723f Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 23:02:05 2024 +1000 . commit f7ae37e Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 22:58:19 2024 +1000 . commit 0f954e4 Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 22:54:21 2024 +1000 . commit 3dafd88 Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 22:27:05 2024 +1000 . commit ebe3a19 Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 22:12:39 2024 +1000 . commit 9f4413e Merge: daf48a5 2e3c234 Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 22:11:22 2024 +1000 Merge branch 'develop' of github.com:AequilibraE/aequilibrae into pedro/ci_test commit daf48a5 Merge: 3df73da ccb9cfa Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 22:11:13 2024 +1000 . commit 3df73da Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 22:10:13 2024 +1000 . commit 9448e76 Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 22:10:05 2024 +1000 adds emulation commit c9f2aaa Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 21:20:12 2024 +1000 adds emulation commit b2d5d3d Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 21:16:51 2024 +1000 adds emulation commit 2458f9a Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 21:02:26 2024 +1000 adds emulation commit e9e660b Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 20:54:57 2024 +1000 adds emulation commit f112262 Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 20:42:13 2024 +1000 Add ARM architectures for Linux and mac commit bae7d0d Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 20:28:15 2024 +1000 tests cibuildwheels commit 293457a Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 20:09:43 2024 +1000 tests cibuildwheels commit 06b6a44 Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 19:31:21 2024 +1000 tests cibuildwheels commit 0ebce09 Author: pveigadecamargo <[email protected]> Date: Tue Feb 20 19:23:39 2024 +1000 tests cibuildwheels * Re-enable CI * Scratch working for link loading * Rudimentary link loading and path file generation * Fix tests and segfaults * Scratch comments * Separate path file generation and link loading Add method to map compressed link IDs to network IDs * Fix link ID ordering in compressed -> network mapping * We don't need functools for this * Reverse routes during computation, map link IDs during output * Fix tests * Fix windows compilation * Linting * Add ruff to pre-commit hooks * Update black pre-commit hook and drop flake8 * Translate link loads from compressed IDs to graph IDs when link Add decorators as well * Rename Cython file to avoid name clash * Add wrapper object and begin API work * Cannot rely on the ordering of nodes when building the mapping * Rename gamma -> path_overlap * Prevent deadend removal + graph compression introducing simple loops * Move NetworkGraphIndices dataclass, add node mapping, extend API untested * Add link to bfsle paper, add American spelling * Fix lots of small errors in wrapper class * Merges set algorithm and set parameters. Better docs * Add example docs and various bug fixes * Make deadlock case and error, needs a real fix * Enforce single thread for tests * Fix the "deadlock", code wasn't deadlocking but it was running away * Limit pyarrow IO threads, Cython += is funky * Pyarrow IO threads must be > 0, give tests from more freedom * Better type checks and some tests * Fix segfault and infinite loop due to miss count * Spelling, remove clamping, make algorithm positional or keyword arg * Forget import * Skip 3.9 builds * Revert "Skip 3.9 builds" This reverts commit a8c72db. * Drop 3.8 from unit tests * Don't run off the end of the vector * Remove FIXMEs, update docs strings, spelling errors * Add test with known results * Move graph index building to Cython for free 1.5x * Add select link support with sparse matrices * Add select link tests and fix bug * Add sparse matrix writing * Update docs, add small api tests * Add sparse matrix tests and from disk method * Add link loading and select link results saving * WIP: add LP to BFSLE, each depth penalises the next depths base graph * Add optional link penalisation to BFSLE * Add binary logit cut offs for assignment * Update tests * Update example * Some nicer comments * Exclude excluded routes from all calculations, not just probability * Make sure the objects are numpy arrays, not cython memory views * Route choice docs and adjusting (#532) * fixes example (#530) * documentation * docs * Changing API * Docs * Docs * Example for choice set generation * Example for choice set generation * Image thumbnail for notebook * Map for example * Map for example * . * Invert probabilities, cut-off now includes only above it, not below * Clarifies notebook * Clarifies notebook * updates CI * Fix tests for probability cutoff * Support disconnect OD pairs * Fix select link not using filtered graph * Use more copies to avoid link loading issues (hopefully) * Simplifies return of link loading * Makes scheduling of parallel jobs more aggressive (each individual job is very quick, so the overhead is negligible and potential for load balance is huge) * randomizes inputs for load balancing * removes reference to theta as a utility function parameter * removes reference to theta as a utility function parameter * Add missing negation and remove theta parameter from tests * ci test * ci test * Revert "ci test" This reverts commit a34a497. * Revert "ci test" This reverts commit 4eb8cd1. * CI * CI * CI * CI * Documentation icons * Include comments as docs * Add some detail to the modelling with aeq route choice docs * response to comments * . * . * string format * . * . * parameter clarification * move comment one line up for clarity --------- Co-authored-by: Renata Imai <[email protected]> Co-authored-by: pveigadecamargo <[email protected]> Co-authored-by: Jake-Moss <[email protected]> Co-authored-by: Renata Imai <[email protected]> Co-authored-by: Jan Zill <[email protected]> * fixes icon * Test removing version specifier on auditwheel * Revert wheel repair test * Updates pyarrow version * Bump pyarrow in wheel building --------- Co-authored-by: Jake-Moss <[email protected]> Co-authored-by: pveigadecamargo <[email protected]> Co-authored-by: Jan <[email protected]> Co-authored-by: Renata Imai <[email protected]> Co-authored-by: Renata Imai <[email protected]> Co-authored-by: Jan Zill <[email protected]>
AequilibraE · Jun 19, 2024 · 2903ea3 · 2903ea3
1 parent cc57a15
commit 2903ea3
Show file tree

Hide file tree

Showing 58 changed files with 4,020 additions and 274 deletions.
diff --git a/.flake8 b/.flake8
@@ -3,4 +3,4 @@ max-line-length = 120
 ignore = E203, E266, E501, W503, F403, F401, C901, W605
 max-complexity = 20
 select = B,C,E,F,W,T4,B9
-exclude = .idea,.git,__pycache__,sphinx,.venv*,.venv,venv,docs/*,benchmarks/*
+exclude = .idea,.git,__pycache__,sphinx,.venv*,.venv,venv,docs/*,benchmarks/*,*.pyx,*pxd,*.pxi
diff --git a/.github/build_artifacts_qgis.yml b/.github/build_artifacts_qgis.yml
@@ -21,7 +21,7 @@ jobs:
     steps:
     - uses: actions/checkout@v4
     - name: Set Python environment
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
         architecture: ${{ matrix.architecture }}
@@ -40,7 +40,6 @@ jobs:
       run: |
         python -m pip install --upgrade pip setuptools wheel twine
         pip install -r requirements.txt
-        pip install -r requirements_additional.txt
 
     - name: Build MacOS
       if: ${{matrix.os == 'macos-latest'}}
@@ -54,7 +53,6 @@ jobs:
       run: |
         python -m pip install --upgrade pip setuptools wheel twine
         pip install -r requirements.txt
-        pip install -r requirements_additional.txt
 
     - name: Build
       if: ${{matrix.os != 'macos-latest'}}

diff --git a/.github/workflows/build_linux.yml b/.github/workflows/build_linux.yml
diff --git a/.github/workflows/build_mac.yml b/.github/workflows/build_mac.yml
diff --git a/.github/workflows/build_wheels.yml b/.github/workflows/build_wheels.yml
@@ -0,0 +1,58 @@
+name: Build
+
+on: [pull_request]
+
+jobs:
+  build_wheels:
+    name: Build wheels on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up QEMU
+        if: runner.os == 'Linux'
+        uses: docker/setup-qemu-action@v3
+        with:
+          platforms: all
+
+      - name: Build wheels
+        uses: pypa/[email protected]
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
+          path: ./wheelhouse/*.whl
+
+  make_sdist:
+    name: Make SDist
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Build SDist
+        run: pipx run build --sdist
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-sdist
+          path: dist/*.tar.gz
+
+  upload_all:
+    needs: [build_wheels, make_sdist]
+    environment: pypi
+    permissions:
+      id-token: write
+    runs-on: ubuntu-latest
+    if: github.event_name == 'release' && github.event.action == 'published'
+    steps:
+    - uses: actions/download-artifact@v4
+      with:
+        pattern: cibw-*
+        path: dist
+        merge-multiple: true
+
+    - uses: pypa/gh-action-pypi-publish@release/v1
diff --git a/.github/workflows/build_windows.yml b/.github/workflows/build_windows.yml
diff --git a/.github/workflows/debug_tests.yml b/.github/workflows/debug_tests.yml
@@ -13,7 +13,6 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         pip install -r requirements.txt
-        pip install -r requirements_additional.txt
         pip install -r tests/requirements_tests.txt
 
     - name: Download spatialite on Linux
@@ -26,4 +25,4 @@ jobs:
         pip install -e .
 
     - name: Runs test
-      run: python -m pytest
+      run: python -m pytest
diff --git a/.github/workflows/documentation.yml b/.github/workflows/documentation.yml
@@ -1,11 +1,11 @@
 name: Documentation
 
 on:
-  push:
-    branches:
-      - develop
-  pull_request:
-  release:
+ push:
+   branches:
+     - develop
+ pull_request:
+ release:
 
 jobs:
   build:
@@ -17,7 +17,7 @@ jobs:
     steps:
     - uses: actions/checkout@v4
     - name: Set up Python 3.10
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
         python-version: '3.10'
     - name: Install dependencies
@@ -27,7 +27,6 @@ jobs:
         python -m pip install --upgrade pip
         pip install wheel openmatrix pytest
         pip install -r requirements.txt
-        pip install -r requirements_additional.txt
         pip install -r docs/requirements-docs.txt
         
 

diff --git a/.github/workflows/test_linux_with_coverage.yml b/.github/workflows/test_linux_with_coverage.yml
@@ -19,7 +19,6 @@ jobs:
         sudo apt install -y libspatialite-dev
         pip3 install setuptools --upgrade
         pip3 install -r requirements.txt
-        pip3 install -r requirements_additional.txt
         pip3 install spatialite
         pip3 install pytest
         pip3 install pytest-cov
@@ -30,4 +29,4 @@ jobs:
 
     - name: Generate coverage report
       run: |
-        python3 -m pytest --cov=aequilibrae tests/
+        python3 -m pytest --cov=aequilibrae tests/
diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml
@@ -8,7 +8,7 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - name: Set Python environment
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: 3.9
           architecture: x64
@@ -17,7 +17,6 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           pip install -r requirements.txt
-          pip install -r requirements_additional.txt
           pip install -r tests/requirements_tests.txt
 
       - name: Lint with ruff
@@ -31,21 +30,20 @@ jobs:
     runs-on: ${{ matrix.os}}
     strategy:
       matrix:
-        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
+        python-version: ['3.9', '3.10', '3.11', '3.12']
         os: [windows-latest, ubuntu-latest]
 
       max-parallel: 20
     steps:
     - uses: actions/checkout@v4
     - name: Set Python environment
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip setuptools
         pip install -r requirements.txt
-        pip install -r requirements_additional.txt
         pip install -r tests/requirements_tests.txt
 
     - name: Download spatialite on Linux
@@ -66,4 +64,4 @@ jobs:
         python setup.py build_ext --inplace
 
     - name: Runs test
-      run: python -m pytest
+      run: python -m pytest