Lot3 work (#958)

* Feature/lot3 output sql endpoint (#858) * Readded sql endpoint * Split raw files for sql processing. Amend sql view to support both. * Added tests for output api. * Update reader and block endpoints when reader does not support SQL. * added signed url generation to storage for data converter * made it possible to load whole sections from the ini conig as a dict * added task to run data conversion * added missing migrations * made file_storage_link the converted file url if the file has been converted * made the converter raise errors rather than logging can completing successfully * added ability to download converted files * added passing validation configs into the conversion process * added conversion time to related files * added custom logger and logfile gathering for data converter * added group access to mapping files and endpoints for managing access * added tests for file conversion permissions * moved storage objects to lot 3 package * added injecting the model storage config into the OasisLMf bash args * added root dir to model storage * added passing model storage config into tasks * fixed issue when no model storage was set * fixed issues connecting to an azure model storge * fixed tests not being able to override filestora settings * Readded sql endpoint (#848) * Readded sql endpoint * update keycloak * Set version 2.2.0 * Updated Package Requirements: django==3.2.20 * retest * Update changelog * Feautre/1323 reorganize branches plat2 (#849) * Update CI plat2 * Update readme title * Fix piwind branch select * Updated Package Requirements: pyYaml==5.3.1 * fix * test CI workflow without CVE error --------- Co-authored-by: awsbuild <[email protected]> * Fix/migrations plat1 to plat2 (#862) * nuke all current migration files * Add in platform 1 migrations (version 1.28.0) * Apply platform2 migrations ontop of plat1 * Move all of ssl connection string to variable * Update deploy script * test data retention -- testing only * Add helper script to support 2.2.0 and below * Add support to migration between plat2 versions * f * fix * tidy * Revert "test data retention -- testing only" This reverts commit a64e0a8. * Updated Package Requirements: pyyaml==6.0.1 * trigger retest * stricter checking for missing migrations Revert "Revert "test data retention -- testing only"" This reverts commit ecf766d. fix Revert "Revert "Revert "test data retention -- testing only""" This reverts commit 8e4474be5a11869571d10b31ab0ca7b6462e6988. --------- Co-authored-by: awsbuild <[email protected]> * Fix to tag piwind repo on publish (#865) * Fix to tag piwind repo on publish Extract prev versions for latest released worker Disable guards to test release script Set version 2.2.1rc2 Add option to set min CVE errors Fix fix Fix ktools tag and latest publish Fix boolean Update changelog Update changelog test push git tag Update changelog test piwind tag from remote workflow test Revert "test" This reverts commit 82f8ae7. f finish pub script * Revert files edited in release testing * switch piwind tag to main branch * Fix cryptography CVE-2023-38325 - platform 2 (#873) * Updated Package Requirements: cryptography==41.0.2 autobahn pyopenssl * Updated Package Requirements: certifi==2023.7.22 * retest --------- Co-authored-by: awsbuild <[email protected]> --------- Co-authored-by: Sam Gamble <[email protected]> Co-authored-by: awsbuild <[email protected]> Co-authored-by: sambles <[email protected]> Co-authored-by: Dan Bate <[email protected]> * added readme about model storage * upped llvmlite version to fix gul tests * linting * updated lot3 install in ci * added passing df reader configuration to lmf commands * fixed failing tests * added configuration for dask cluster to the docker-compose * made the dask worker and scheduler in docker compose use the host network * fixed failing test * updated lot 3 storage classes * fixed issues reversing sql urls * fixed remaining tests * linting * Hook in oasis-data-manager * Fix clashing Serializer ref_names * Update requriments * Hide lot3 endpoints Skip lot3 tests * update debian worker base image * PEP * Fix image build errors * fix missing system package for dask-sql * WIP - workflow hang when writing files * Add portainer to compose * Strip out dask and redo image packages * f * Fix merge issues * Apply json post fix to v1 files * wip - add oasis-data-manager branch build * fix local runs - post merge * pep * Fix model storage * Update S3 localstack example for testing * f * fix * keep run data for debugging * Disable global env fetch and fallback on local storage * test no global ENV for model storage * pep * test with real bucket * Fix bucket root dir * dump params to oasislmf.json in run dir for debugging * fix pre-analysis-hook * DEBUG * Add missing ENV to fail analysis if RI or IL is requested in settings without the required oasis files * Get V2 worker running with remote piwind data * Updated Package Requirements: oasis-data-manager==0.1.3 * pep * fix tests * WIP * PEP * Fix V1 worker tests * Flake8 * Skip all LOT3 SQL endpoint testing (disabled at the moment) * Fix loading diff between V1 and V2 input tar * Fix settings store in gen losses * Tone down Vun scanning - just run on releases * Update deploy scripts for plat testing --------- Co-authored-by: James Outterside <[email protected]> Co-authored-by: Dan Bate <[email protected]> Co-authored-by: awsbuild <[email protected]>
OasisLMF · May 2, 2024 · 0250a94 · 0250a94
1 parent d155dd6
commit 0250a94
Show file tree

Hide file tree

Showing 87 changed files with 3,718 additions and 1,765 deletions.
diff --git a/.github/workflows/scan-external.yml b/.github/workflows/scan-external.yml
@@ -1,9 +1,6 @@
 name: External Vulnerability Scanning
 
 on:
-  push:
-    branches:
-      - main
   pull_request:
     branches:
       - stable**

diff --git a/.github/workflows/scan.yml b/.github/workflows/scan.yml
@@ -8,8 +8,6 @@ on:
     branches:
       - main
       - stable**  
-  schedule:
-    - cron:  '0 */6 * * *' # Run scan every 6 hours
 
   workflow_dispatch:
     inputs:

diff --git a/.github/workflows/test-python_api.yml b/.github/workflows/test-python_api.yml
@@ -7,7 +7,7 @@ on:
   pull_request:
     branches:
       - main
-      - stable**  
+      - stable**
   workflow_dispatch:
     inputs:
       ods_branch:

diff --git a/.github/workflows/update-python.yml b/.github/workflows/update-python.yml
@@ -38,7 +38,7 @@ jobs:
         python-version: '3.10'
 
     - name: Install piptools
-      run: pip install pip-tools
+      run: pip install pip-tools setuptools-rust
 
     - name: Update Requirements files
       run: |

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -85,6 +85,7 @@ OasisPlatform Changelog
 * [#847](https://github.com/OasisLMF/OasisPlatform/pull/847) - Release 2.2.0
 .. _`2.2.1rc1`:  https://github.com/OasisLMF/OasisPlatform/compare/2.2.0...2.2.1rc1
 
+
 `2.2.0`_
  ---------
 * [#725](https://github.com/OasisLMF/OasisPlatform/pull/843) - Queue priority for same job

diff --git a/Dockerfile.api_server b/Dockerfile.api_server
@@ -14,6 +14,14 @@ RUN if [ ! -z "$ods_tools_branch" ] ; then \
     pip install --user --no-warn-script-location -v git+https://[email protected]/OasisLMF/ODS_Tools.git@${ods_tools_branch}#egg=ods-tools; \
   fi
 
+# Install Oasis-Data-Manager from git branch (Optional) 'docker build --build-arg odm_branch=develop'
+ARG odm_branch
+RUN if [ ! -z "$odm_branch" ] ; then \
+    apt update && apt install -y git; \
+    pip uninstall oasis-data-manager -y; \
+    pip install --user --no-warn-script-location -v git+https://[email protected]/OasisLMF/OasisDataManager.git@${odm_branch}#egg=oasis-data-manager; \
+  fi
+
 USER server
 
 # ---- STAGE 2 ----

diff --git a/Dockerfile.model_worker b/Dockerfile.model_worker
@@ -23,13 +23,22 @@ RUN if [ ! -z "$ods_tools_branch" ] ; then \
     pip install --user --no-warn-script-location -v git+https://[email protected]/OasisLMF/ODS_Tools.git@${ods_tools_branch}#egg=ods-tools; \
   fi
 
+# Install Oasis-Data-Manager from git branch (Optional) 'docker build --build-arg odm_branch=develop'
+ARG odm_branch
+RUN if [ ! -z "$odm_branch" ] ; then \
+    apt update && apt install -y git; \
+    pip uninstall oasis-data-manager -y; \
+    pip install --user --no-warn-script-location -v git+https://[email protected]/OasisLMF/OasisDataManager.git@${odm_branch}#egg=oasis-data-manager; \
+  fi
+
+
 USER worker
 
 # ---- STAGE 2 ----
 FROM ubuntu:22.04
 RUN apt-get update \
  && apt-get upgrade -y \
- && apt-get install -y --no-install-recommends vim git python3 python3-pip libspatialindex-dev curl procps \
+ && apt-get install -y --no-install-recommends vim git  python3 python3-pip libspatialindex-dev curl procps \
  && rm -rf /var/lib/apt/lists/*
 
 # Copy built python packages
@@ -66,5 +75,6 @@ ENV PATH=/home/worker/.local/bin:$PATH
 ENV DEBIAN_FRONTEND=noninteractive
 ENV OASIS_MEDIA_ROOT=/shared-fs
 ENV OASIS_ENV_OVERRIDE=true
+ENV OASIS_CHECK_MISSING_INPUTS=true
 
 ENTRYPOINT ./startup.sh
diff --git a/Dockerfile.model_worker_debian b/Dockerfile.model_worker_debian
@@ -43,6 +43,14 @@ RUN if [ ! -z "$ods_tools_branch" ] ; then \
     pip install -v git+https://[email protected]/OasisLMF/ODS_Tools.git@${ods_tools_branch}#egg=ods-tools; \
   fi
 
+# Install Oasis-Data-Manager from git branch (Optional) 'docker build --build-arg odm_branch=develop'
+ARG odm_branch
+RUN if [ ! -z "$odm_branch" ] ; then \
+    apt update && apt install -y git; \
+    pip uninstall oasis-data-manager -y; \
+    pip install --user --no-warn-script-location -v git+https://[email protected]/OasisLMF/OasisDataManager.git@${odm_branch}#egg=oasis-data-manager; \
+  fi
+
 # Copy startup script + server config
 COPY ./src/startup_worker.sh ./startup.sh
 COPY ./conf.ini ./