Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix to tag piwind repo on publish #865

Merged
merged 3 commits into from
Aug 3, 2023
Merged

Conversation

sambles
Copy link
Contributor

@sambles sambles commented Jul 28, 2023

**IMPORTANT: Please attach or create an issue after submitting a Pull Request.

CI release workflow fixes

  • Add release tag to target piwind branch on release
  • Add new latest tags 2-latest for platform 2 and 1-latest for 1
  • Extract previous component versions from last released worker image
  • added Option to override cve_severity value

@sambles sambles linked an issue Aug 2, 2023 that may be closed by this pull request
3 tasks
@github-advanced-security
Copy link

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script
@sambles sambles merged commit 4a301ec into main Aug 3, 2023
19 of 20 checks passed
@sambles sambles deleted the fix/ci-publish-tag-piwind branch August 3, 2023 12:02
sambles added a commit that referenced this pull request Aug 3, 2023
* Fix to tag piwind repo on publish

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script

* Revert files edited in release testing

* switch piwind tag to main branch
sambles added a commit that referenced this pull request Aug 3, 2023
* Fix to tag piwind repo on publish

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script

* Revert files edited in release testing

* switch piwind tag to main branch
@awsbuild awsbuild added this to the 2.2.1 milestone Aug 3, 2023
sambles added a commit that referenced this pull request Aug 29, 2023
* Fix to tag piwind repo on publish

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script

* Revert files edited in release testing

* switch piwind tag to main branch
sambles added a commit that referenced this pull request Aug 30, 2023
* Fix to tag piwind repo on publish

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script

* Revert files edited in release testing

* switch piwind tag to main branch
@awsbuild awsbuild modified the milestones: 2.2.1, 1.27.6 Aug 30, 2023
OmegaDroid added a commit that referenced this pull request Aug 30, 2023
* Readded sql endpoint

* update keycloak

* Set version 2.2.0

* Updated Package Requirements: django==3.2.20

* retest

* Update changelog

* Feautre/1323 reorganize branches plat2 (#849)

* Update CI plat2

* Update readme title

* Fix piwind branch select

* Updated Package Requirements: pyYaml==5.3.1

* fix

* test CI workflow without CVE error

---------

Co-authored-by: awsbuild <[email protected]>

* Fix/migrations plat1 to plat2 (#862)

* nuke all current migration files

* Add in platform 1 migrations (version 1.28.0)

* Apply platform2 migrations ontop of plat1

* Move all of ssl connection string to variable

* Update deploy script

* test data retention -- testing only

* Add helper script to support 2.2.0 and below

* Add support to migration between plat2 versions

* f

* fix

* tidy

* Revert "test data retention -- testing only"

This reverts commit a64e0a8.

* Updated Package Requirements: pyyaml==6.0.1

* trigger retest

* stricter checking for missing migrations

Revert "Revert "test data retention -- testing only""

This reverts commit ecf766d.

fix

Revert "Revert "Revert "test data retention -- testing only"""

This reverts commit 8e4474be5a11869571d10b31ab0ca7b6462e6988.

---------

Co-authored-by: awsbuild <[email protected]>

* Fix to tag piwind repo on publish (#865)

* Fix to tag piwind repo on publish

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script

* Revert files edited in release testing

* switch piwind tag to main branch

* Fix cryptography CVE-2023-38325 - platform 2 (#873)

* Updated Package Requirements: cryptography==41.0.2 autobahn pyopenssl

* Updated Package Requirements: certifi==2023.7.22

* retest

---------

Co-authored-by: awsbuild <[email protected]>

---------

Co-authored-by: Sam Gamble <[email protected]>
Co-authored-by: awsbuild <[email protected]>
Co-authored-by: sambles <[email protected]>
Co-authored-by: Dan Bate <[email protected]>
sambles added a commit that referenced this pull request Sep 13, 2023
* Update CI 1.28 (#853)

* Feautre/1323 reorganize branches plat1 (#850)

* update readme title

* Update CI plat1

* Fix piwind branch select

* Updated Package Requirements: pyYaml==5.3.1

* Revert "Updated Package Requirements: pyYaml==5.3.1"

This reverts commit 094bcda.

* fix

* bump minor version for testing

---------

Co-authored-by: awsbuild <[email protected]>

* retest

---------

Co-authored-by: awsbuild <[email protected]>

* Fix to tag piwind repo on publish (#865)

* Fix to tag piwind repo on publish

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script

* Revert files edited in release testing

* switch piwind tag to main branch

* Fix cryptography CVE-2023-38325 - platform 1   (#874)

* Updated Package Requirements: cryptography==41.0.2 autobahn pyopenssl

* Updated Package Requirements: certifi==2023.7.22

* retest

---------

Co-authored-by: awsbuild <[email protected]>

* revert settings compatibility to older version (#878)

* revert settings compatibility to older version

* keep both checks

* Update unittest

* Catch and report all execptions from ods-tools val (#884)

* draft actions stable image testing

* Select correct piwind branches

* Fix log storage

* fix

* Use latest piwind meta-data on older workers but skip output compare

* test stable images update

* fix

* update and retest

---------

Co-authored-by: awsbuild <[email protected]>
sambles added a commit that referenced this pull request Sep 13, 2023
* Update CI 1.28 (#853)

* Feautre/1323 reorganize branches plat1 (#850)

* update readme title

* Update CI plat1

* Fix piwind branch select

* Updated Package Requirements: pyYaml==5.3.1

* Revert "Updated Package Requirements: pyYaml==5.3.1"

This reverts commit 094bcda.

* fix

* bump minor version for testing

---------

Co-authored-by: awsbuild <[email protected]>

* retest

---------

Co-authored-by: awsbuild <[email protected]>

* Fix to tag piwind repo on publish (#865)

* Fix to tag piwind repo on publish

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script

* Revert files edited in release testing

* switch piwind tag to main branch

* Fix cryptography CVE-2023-38325 - platform 1   (#874)

* Updated Package Requirements: cryptography==41.0.2 autobahn pyopenssl

* Updated Package Requirements: certifi==2023.7.22

* retest

---------

Co-authored-by: awsbuild <[email protected]>

* revert settings compatibility to older version (#878)

* revert settings compatibility to older version

* keep both checks

* Update unittest

* Catch and report all execptions from ods-tools val (#884)

* draft actions stable image testing

* Select correct piwind branches

* Fix log storage

* fix

* Use latest piwind meta-data on older workers but skip output compare

* test stable images update

* fix

* update and retest

---------

Co-authored-by: awsbuild <[email protected]>
sambles added a commit that referenced this pull request Sep 13, 2023
* Update CI 1.28 (#853)

* Feautre/1323 reorganize branches plat1 (#850)

* update readme title

* Update CI plat1

* Fix piwind branch select

* Updated Package Requirements: pyYaml==5.3.1

* Revert "Updated Package Requirements: pyYaml==5.3.1"

This reverts commit 094bcda.

* fix

* bump minor version for testing

---------

Co-authored-by: awsbuild <[email protected]>

* retest

---------

Co-authored-by: awsbuild <[email protected]>

* Fix to tag piwind repo on publish (#865)

* Fix to tag piwind repo on publish

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script

* Revert files edited in release testing

* switch piwind tag to main branch

* Fix cryptography CVE-2023-38325 - platform 1   (#874)

* Updated Package Requirements: cryptography==41.0.2 autobahn pyopenssl

* Updated Package Requirements: certifi==2023.7.22

* retest

---------

Co-authored-by: awsbuild <[email protected]>

* revert settings compatibility to older version (#878)

* revert settings compatibility to older version

* keep both checks

* Update unittest

* Catch and report all execptions from ods-tools val (#884)

* draft actions stable image testing

* Select correct piwind branches

* Fix log storage

* fix

* Use latest piwind meta-data on older workers but skip output compare

* test stable images update

* fix

* update and retest

---------

Co-authored-by: awsbuild <[email protected]>
@awsbuild awsbuild modified the milestones: 1.27.6, 1.28.2 Sep 18, 2023
sambles added a commit that referenced this pull request May 2, 2024
* Feature/lot3 output sql endpoint (#858)

* Readded sql endpoint

* Split raw files for sql processing. Amend sql view to support both.

* Added tests for output api.

* Update reader and block endpoints when reader does not support SQL.

* added signed url generation to storage for data converter

* made it possible to load whole sections from the ini conig as a dict

* added task to run data conversion

* added missing migrations

* made file_storage_link the converted file url if the file has been converted

* made the converter raise errors rather than logging can completing successfully

* added ability to download converted files

* added passing validation configs into the conversion process

* added conversion time to related files

* added custom logger and logfile gathering for data converter

* added group access to mapping files and endpoints for managing access

* added tests for file conversion permissions

* moved storage objects to lot 3 package

* added injecting the model storage config into the OasisLMf bash args

* added root dir to model storage

* added passing model storage config into tasks

* fixed issue when no model storage was set

* fixed issues connecting to an azure model storge

* fixed tests not being able to override filestora settings

* Readded sql endpoint (#848)

* Readded sql endpoint

* update keycloak

* Set version 2.2.0

* Updated Package Requirements: django==3.2.20

* retest

* Update changelog

* Feautre/1323 reorganize branches plat2 (#849)

* Update CI plat2

* Update readme title

* Fix piwind branch select

* Updated Package Requirements: pyYaml==5.3.1

* fix

* test CI workflow without CVE error

---------

Co-authored-by: awsbuild <[email protected]>

* Fix/migrations plat1 to plat2 (#862)

* nuke all current migration files

* Add in platform 1 migrations (version 1.28.0)

* Apply platform2 migrations ontop of plat1

* Move all of ssl connection string to variable

* Update deploy script

* test data retention -- testing only

* Add helper script to support 2.2.0 and below

* Add support to migration between plat2 versions

* f

* fix

* tidy

* Revert "test data retention -- testing only"

This reverts commit a64e0a8.

* Updated Package Requirements: pyyaml==6.0.1

* trigger retest

* stricter checking for missing migrations

Revert "Revert "test data retention -- testing only""

This reverts commit ecf766d.

fix

Revert "Revert "Revert "test data retention -- testing only"""

This reverts commit 8e4474be5a11869571d10b31ab0ca7b6462e6988.

---------

Co-authored-by: awsbuild <[email protected]>

* Fix to tag piwind repo on publish (#865)

* Fix to tag piwind repo on publish

Extract prev versions for latest released worker

Disable guards to test release script

Set version 2.2.1rc2

Add option to set min CVE errors

Fix

fix

Fix ktools tag and latest publish

Fix boolean

Update changelog

Update changelog

test push git tag

Update changelog

test piwind tag from remote workflow

test

Revert "test"

This reverts commit 82f8ae7.

f

finish pub script

* Revert files edited in release testing

* switch piwind tag to main branch

* Fix cryptography CVE-2023-38325 - platform 2 (#873)

* Updated Package Requirements: cryptography==41.0.2 autobahn pyopenssl

* Updated Package Requirements: certifi==2023.7.22

* retest

---------

Co-authored-by: awsbuild <[email protected]>

---------

Co-authored-by: Sam Gamble <[email protected]>
Co-authored-by: awsbuild <[email protected]>
Co-authored-by: sambles <[email protected]>
Co-authored-by: Dan Bate <[email protected]>

* added readme about model storage

* upped llvmlite version to fix gul tests

* linting

* updated lot3 install in ci

* added passing df reader configuration to lmf commands

* fixed failing tests

* added configuration for dask cluster to the docker-compose

* made the dask worker and scheduler in docker compose use the host network

* fixed failing test

* updated lot 3 storage classes

* fixed issues reversing sql urls

* fixed remaining tests

* linting

* Hook in oasis-data-manager

* Fix clashing Serializer ref_names

* Update requriments

* Hide lot3 endpoints

Skip lot3 tests

* update debian worker base image

* PEP

* Fix image build errors

* fix missing system package for dask-sql

* WIP - workflow hang when writing files

* Add portainer to compose

* Strip out dask and redo image packages

* f

* Fix merge issues

* Apply json post fix to v1 files

* wip - add oasis-data-manager branch build

* fix local runs - post merge

* pep

* Fix model storage

* Update S3 localstack example for testing

* f

* fix

* keep run data for debugging

* Disable global env fetch and fallback on local storage

* test no global ENV for model storage

* pep

* test with real bucket

* Fix bucket root dir

* dump params to oasislmf.json in run dir for debugging

* fix pre-analysis-hook

* DEBUG

* Add missing ENV to fail analysis if RI or IL is requested in settings without the required oasis files

* Get V2 worker running with remote piwind data

* Updated Package Requirements: oasis-data-manager==0.1.3

* pep

* fix tests

* WIP

* PEP

* Fix V1 worker tests

* Flake8

* Skip all LOT3 SQL endpoint testing (disabled at the moment)

* Fix loading diff between V1 and V2 input tar

* Fix settings store in gen losses

* Tone down Vun scanning - just run on releases

* Update deploy scripts for plat testing

---------

Co-authored-by: James Outterside <[email protected]>
Co-authored-by: Dan Bate <[email protected]>
Co-authored-by: awsbuild <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Fixes for OasisPlatform Publish
2 participants