-
Notifications
You must be signed in to change notification settings - Fork 81
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* update to viash 0.9 * update changelog * update github actions * fix ci * ci force * add common resources * Update dataset loaders (#909) * update to viash 0.9 and categorise datasets * group workflows * add api for spatial datasets * add more metadata * update publish dir path * update project config * update namespace * fix id * update example * fix example * update test resources * update helper resources * fix multiple separator --------- Co-authored-by: Robrecht Cannoodt <[email protected]> * Update core components to viash 0.9 (#912) * migrate components * remove resources which were moved to common_resources * remove more resources which were moved to common resources * update components * update scripts * wip migrate reporting components * rename components * update scripts * update components * more fixes * update * remove setup logging helper * fix components * Apply suggestions from code review Co-authored-by: Sai Nirmayi Yasa <[email protected]> * fix references to renamed components --------- Co-authored-by: Sai Nirmayi Yasa <[email protected]>
- Loading branch information
1 parent
559f977
commit 56e4278
Showing
230 changed files
with
4,640 additions
and
10,984 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,88 +1,11 @@ | ||
| name: Test | ||
|
|
||
| on: | ||
| pull_request: | ||
| push: | ||
| branches: [ main ] | ||
| branches: | ||
| - main | ||
| pull_request: | ||
|
|
||
| jobs: | ||
|
|
||
| # phase 1 | ||
| list: | ||
| env: | ||
| s3_bucket: s3://openproblems-data/resources_test/ | ||
| runs-on: ubuntu-latest | ||
|
|
||
| outputs: | ||
| matrix: ${{ steps.set_matrix.outputs.matrix }} | ||
| cache_key: ${{ steps.cache.outputs.cache_key }} | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - uses: viash-io/viash-actions/setup@v6 | ||
|
|
||
| - uses: viash-io/viash-actions/project/sync-and-cache-s3@v6 | ||
| id: cache | ||
| with: | ||
| s3_bucket: $s3_bucket | ||
| dest_path: resources_test | ||
| cache_key_prefix: resources_test__ | ||
|
|
||
| - id: ns_list | ||
| uses: viash-io/viash-actions/ns-list@v6 | ||
| with: | ||
| platform: docker | ||
| format: json | ||
|
|
||
| - id: ns_list_filtered | ||
| uses: viash-io/viash-actions/project/detect-changed-components@v6 | ||
| with: | ||
| input_file: "${{ steps.ns_list.outputs.output_file }}" | ||
|
|
||
| - id: set_matrix | ||
| run: | | ||
| echo "matrix=$(jq -c '[ .[] | | ||
| { | ||
| "name": (.functionality.namespace + "/" + .functionality.name), | ||
| "config": .info.config | ||
| } | ||
| ]' ${{ steps.ns_list_filtered.outputs.output_file }} )" >> $GITHUB_OUTPUT | ||
| # phase 2 | ||
| viash_test: | ||
| needs: list | ||
| if: ${{ needs.list.outputs.matrix != '[]' && needs.list.outputs.matrix != '' }} | ||
| runs-on: ubuntu-latest | ||
|
|
||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| component: ${{ fromJson(needs.list.outputs.matrix) }} | ||
|
|
||
| steps: | ||
| # Remove unnecessary files to free up space. Otherwise, we get 'no space left on device.' | ||
| - uses: data-intuitive/reclaim-the-bytes@v2 | ||
|
|
||
| - uses: actions/checkout@v4 | ||
|
|
||
| - uses: viash-io/viash-actions/setup@v6 | ||
|
|
||
| # use cache | ||
| - name: Cache resources data | ||
| uses: actions/cache@v4 | ||
| timeout-minutes: 10 | ||
| with: | ||
| path: resources_test | ||
| key: ${{ needs.list.outputs.cache_key }} | ||
|
|
||
| - name: Run test | ||
| timeout-minutes: 30 | ||
| run: | | ||
| VIASH_TEMP=$RUNNER_TEMP/viash viash test \ | ||
| "${{ matrix.component.config }}" \ | ||
| --cpus 2 \ | ||
| --memory "5gb" | ||
| test: | ||
| uses: viash-io/viash-actions/.github/workflows/test.yaml@v6 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| [submodule "common"] | ||
| path = common | ||
| url = https://github.com/openproblems-bio/common_resources.git |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,4 @@ | ||
| { | ||
| "yaml.schemas": { | ||
| "src/common/schemas/api_component.yaml": "src/**/api/comp_*.yaml", | ||
| "src/common/schemas/api_file.yaml": "src/**/api/file_*.yaml", | ||
| "src/common/schemas/task_info.yaml": "src/**/api/task_info.yaml", | ||
| "src/common/schemas/task_method.yaml": "src/tasks/**/methods/**/config.vsh.yaml", | ||
| "src/common/schemas/task_control_method.yaml": "src/tasks/**/control_methods/**/config.vsh.yaml", | ||
| "src/common/schemas/task_metric.yaml": "src/tasks/**/metrics/**/config.vsh.yaml" | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,35 @@ | ||
| viash_version: 0.8.6 | ||
| name: openproblems | ||
| version: dev | ||
| organization: openproblems-bio | ||
| viash_version: 0.9.0 | ||
|
|
||
| source: src | ||
| target: target | ||
| description: | | ||
| Open Problems is a living, extensible, community-guided benchmarking platform. | ||
| license: MIT | ||
| keywords: [openproblems, benchmarking, single-cell omics] | ||
|
|
||
| references: | ||
| doi: | ||
| # Malte Luecken, Scott Gigante, Daniel Burkhardt, Robrecht Cannoodt, et al. | ||
| # Defining and benchmarking open problems in single-cell analysis, | ||
| # 03 April 2024, PREPRINT (Version 1) available at Research Square [https://doi.org/10.21203/rs.3.rs-4181617/v1] | ||
| - 10.21203/rs.3.rs-4181617/v1 | ||
|
|
||
| links: | ||
| issue_tracker: https://github.com/openproblems-bio/openproblems/issues | ||
| repository: https://github.com/openproblems-bio/openproblems | ||
| docker_registry: ghcr.io | ||
|
|
||
| config_mods: | | ||
| .functionality.version := 'dev' | ||
| .platforms[.type == 'docker'].target_registry := 'ghcr.io' | ||
| .platforms[.type == 'docker'].target_organization := 'openproblems-bio/openproblems' | ||
| .platforms[.type == 'docker'].target_image_source := 'https://github.com/openproblems-bio/openproblems' | ||
| .platforms[.type == "nextflow"].directives.tag := "$id" | ||
| .platforms[.type == "nextflow"].auto.simplifyOutput := false | ||
| .platforms[.type == "nextflow"].config.labels := { lowmem : "memory = 20.Gb", midmem : "memory = 50.Gb", highmem : "memory = 100.Gb", lowcpu : "cpus = 5", midcpu : "cpus = 15", highcpu : "cpus = 30", lowtime : "time = 1.h", midtime : "time = 4.h", hightime : "time = 8.h", veryhightime : "time = 24.h" } | ||
| .platforms[.type == "nextflow"].config.script := "process.errorStrategy = 'ignore'" | ||
| .runners[.type == "nextflow"].config.labels := { lowmem : "memory = 20.Gb", midmem : "memory = 50.Gb", highmem : "memory = 100.Gb", lowcpu : "cpus = 5", midcpu : "cpus = 15", highcpu : "cpus = 30", lowtime : "time = 1.h", midtime : "time = 4.h", hightime : "time = 8.h", veryhightime : "time = 24.h" } | ||
| .runners[.type == "nextflow"].config.script := "process.errorStrategy = 'ignore'" | ||
| info: | ||
| test_resources: | ||
| - type: s3 | ||
| path: s3://openproblems-data/resources_test/common | ||
| dest: resources_test/common | ||
| - type: s3 | ||
| path: s3://openproblems-data/resources_test/openproblems | ||
| dest: resources_test/openproblems |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| #!/bin/bash | ||
|
|
||
| # get the root of the directory | ||
| REPO_ROOT=$(git rev-parse --show-toplevel) | ||
|
|
||
| # ensure that the command below is run from the root of the repository | ||
| cd "$REPO_ROOT" | ||
|
|
||
| OUT_DIR="resources_test/openproblems/task_results_v1" | ||
|
|
||
| set -e | ||
|
|
||
| echo ">> Removing previously generated results" | ||
| if [ -d "$OUT_DIR" ]; then | ||
| rm -r "$OUT_DIR" | ||
| fi | ||
|
|
||
| echo ">> Fetch results in v1 format" | ||
| mkdir -p "$OUT_DIR/data/" | ||
| TMPDIR=$(mktemp -d) | ||
|
|
||
| wget https://github.com/openproblems-bio/website/archive/refs/tags/v2.3.6.zip -O "$TMPDIR/website-v2.3.6.zip" | ||
| unzip "$TMPDIR/website-v2.3.6.zip" -d "$TMPDIR" | ||
| cp -r "$TMPDIR/website-2.3.6/results/batch_integration_embed/data/" "$OUT_DIR/processed" | ||
|
|
||
| echo ">> Uploading results to S3" | ||
| aws s3 sync --profile op \ | ||
| "resources_test/openproblems/task_results_v1/" \ | ||
| "s3://openproblems-data/resources_test/openproblems/task_results_v1/" \ | ||
| --delete --dryrun |
Oops, something went wrong.