Add support for configurable containers engines (#963)

* Add initial Exacloud page to documentation * Update Exacloud support * Clean up console debug output [no ci] * Add default Worker engine ('docker') [no ci] * Update configurable Container Engine commands * Revert changes to docker implementation and add engine interface [no ci] * Update Exacloud/Container Engine implementation [no ci] * Update ContainerEngine interface [no ci] * Rename ContainerEngine to ContainerType * Add initial Exadocker unit test [no ci] * Update TaskService to pass config to Funnel worker [no ci] * Add small fix in HPC Template config [no ci] * Update worker config path in HPC Template [no ci] * Add debugging from Exacloud testing * Update input/output working example (md5sum) - Test command: ```sh funnel task create examples/md5sum.json ``` [no ci] * Update docker.go to implement ContainerEngine interface - Can be used in HPC environment with unique docker setups (e.g. exadocker) as well as future command templating efforts. [no ci] * Remove exadocker implementation in favor of docker * Add initial HPC documentation and example * Update docker and kin-openapi to compatible versions - github.com/docker/docker versions above v24.0.9 resulted in `panic: Can't connect docker client`: ``` ➜ go test tests/auth/auth_test.go --- FAIL: TestBasicAuthFail (0.01s) panic: Can't connect docker client [recovered] panic: Can't connect docker client FAIL command-line-arguments 0.673s FAIL ``` - This error appears to be originating from [util/dockerutil/docker.go](https://github.com/ohsu-comp-bio/funnel/blob/f5acbabe85d28f45f2f86db959c4d7bc929334a4/util/dockerutil/docker.go#L28). - github.com/getkin/kin-openapi versions above v0.112.0 resulted in type errors: v0.113.0: ``` ➜ go test util/openapi2proto/main.go util/openapi2proto/main.go:54:38: invalid operation: p.Value.AdditionalProperties != nil (mismatched types openapi3.AdditionalProperties and untyped nil) util/openapi2proto/main.go:55:24: cannot use p.Value.AdditionalProperties (variable of type openapi3.AdditionalProperties) as *openapi3.SchemaRef value in argument to getType FAIL command-line-arguments [build failed] FAIL ``` v0.123.0 (latest): ``` ➜ go test util/openapi2proto/main.go util/openapi2proto/main.go:43:7: cannot convert "integer" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:45:7: cannot convert "boolean" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:47:7: cannot convert "number" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:49:7: cannot convert "object" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:54:38: invalid operation: p.Value.AdditionalProperties != nil (mismatched types openapi3.AdditionalProperties and untyped nil) util/openapi2proto/main.go:55:24: cannot use p.Value.AdditionalProperties (variable of type openapi3.AdditionalProperties) as *openapi3.SchemaRef value in argument to getType util/openapi2proto/main.go:60:7: cannot convert "array" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:72:17: cannot use p.Value.Type (variable of type *openapi3.Types) as string value in return statement util/openapi2proto/main.go:84:32: invalid operation: param.Schema.Value.Type != "" (mismatched types *openapi3.Types and untyped string) util/openapi2proto/main.go:175:17: resp.Get undefined (type *openapi3.Responses has no field or method Get) util/openapi2proto/main.go:175:17: too many errors FAIL command-line-arguments [build failed] FAIL ``` * Update util/openapi2proto/main.go with latest kin-openapi version * Update github.com/docker/docker to latest version * Update Docker client creation in tests * Add `WithAPIVersionNegotiation` to docker client creation * Rename 'Driver' to 'DriverCommand' in ContainerConfig * Fix minor linting errors * Update GitHub Actions workflow (linting) * fix: updated config check in hpc_backend.go * Revert tabs/space change in storage/local.go * Update output copying in scratch directory * Update compliance-test.yaml * Debug: add tmate step to Nextflow test * Update Nextflow tests * Update file copying behavior for Nextflow tests * Update local file copying behavior to support directory dests * Update local dest for files with glob patterns * Update Base64Encode test to match Github Actions test - TODO: investigate why this is necessary... * Fix Base64Encoding test value * Revert glob output change based on TesOutput spec - Wildcards in outputs will be uploaded to a directory, with no special handling for the filename. ```yaml tesOutput: properties: url: description: |- URL at which the TES server makes the output accessible after the task is complete. When tesOutput.path contains wildcards, it must be a directory; see `tesOutput.path_prefix` for details on how output URLs are constructed in this case. ``` * Debug: Add tmate step to Nextflow tests * Update wildcard file output destination * Add initial command templating for container engines * Add initial Docker/ContainerEngine tests * Run `go mod tidy` * Update Github Actions * Update container engine IO and command execution * Re-enable all tests for container engines * Update test suite * Nextflow test debug * Nextflow test debug * Nextflow test debug * Nextflow test debug * Update compliance test workflow * Update S3 tests * Update S3 tests * Add comments and clean up debug statements * Clean up code for 0.11.1-rc.1 release * Add container command event to match existing pattern * Update .goreleaser.yaml to replace deprecated fields * Update Nextflow instructions * Update K8s support and documentation - Update docker images to use quay.io/ohsu-comp-bio/funnel * Update README.md * fix: add support for wildcards in AWS S3 paths * Add support for wildcards in AWS S3 output paths * Update actions/upload-artifact to v4 in Github Actions - https://github.blog/changelog/2024-02-13-deprecation-notice-v1-and-v2-of-the-artifact-actions/ * Update version to 0.11.1-rc.4 in Makefile * Fix bug where FileMapper output was being overwritten in every loop * Add initial Exacloud page to documentation * Update Exacloud support * Clean up console debug output [no ci] * Add default Worker engine ('docker') [no ci] * Update configurable Container Engine commands * Revert changes to docker implementation and add engine interface [no ci] * Update Exacloud/Container Engine implementation [no ci] * Update ContainerEngine interface [no ci] * Rename ContainerEngine to ContainerType * Add initial Exadocker unit test [no ci] * Update TaskService to pass config to Funnel worker [no ci] * Add small fix in HPC Template config [no ci] * Update worker config path in HPC Template [no ci] * Add debugging from Exacloud testing * Update input/output working example (md5sum) - Test command: ```sh funnel task create examples/md5sum.json ``` [no ci] * Update docker.go to implement ContainerEngine interface - Can be used in HPC environment with unique docker setups (e.g. exadocker) as well as future command templating efforts. [no ci] * Remove exadocker implementation in favor of docker * Add initial HPC documentation and example * Update docker and kin-openapi to compatible versions - github.com/docker/docker versions above v24.0.9 resulted in `panic: Can't connect docker client`: ``` ➜ go test tests/auth/auth_test.go --- FAIL: TestBasicAuthFail (0.01s) panic: Can't connect docker client [recovered] panic: Can't connect docker client FAIL command-line-arguments 0.673s FAIL ``` - This error appears to be originating from [util/dockerutil/docker.go](https://github.com/ohsu-comp-bio/funnel/blob/f5acbabe85d28f45f2f86db959c4d7bc929334a4/util/dockerutil/docker.go#L28). - github.com/getkin/kin-openapi versions above v0.112.0 resulted in type errors: v0.113.0: ``` ➜ go test util/openapi2proto/main.go util/openapi2proto/main.go:54:38: invalid operation: p.Value.AdditionalProperties != nil (mismatched types openapi3.AdditionalProperties and untyped nil) util/openapi2proto/main.go:55:24: cannot use p.Value.AdditionalProperties (variable of type openapi3.AdditionalProperties) as *openapi3.SchemaRef value in argument to getType FAIL command-line-arguments [build failed] FAIL ``` v0.123.0 (latest): ``` ➜ go test util/openapi2proto/main.go util/openapi2proto/main.go:43:7: cannot convert "integer" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:45:7: cannot convert "boolean" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:47:7: cannot convert "number" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:49:7: cannot convert "object" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:54:38: invalid operation: p.Value.AdditionalProperties != nil (mismatched types openapi3.AdditionalProperties and untyped nil) util/openapi2proto/main.go:55:24: cannot use p.Value.AdditionalProperties (variable of type openapi3.AdditionalProperties) as *openapi3.SchemaRef value in argument to getType util/openapi2proto/main.go:60:7: cannot convert "array" (untyped string constant) to type *openapi3.Types util/openapi2proto/main.go:72:17: cannot use p.Value.Type (variable of type *openapi3.Types) as string value in return statement util/openapi2proto/main.go:84:32: invalid operation: param.Schema.Value.Type != "" (mismatched types *openapi3.Types and untyped string) util/openapi2proto/main.go:175:17: resp.Get undefined (type *openapi3.Responses has no field or method Get) util/openapi2proto/main.go:175:17: too many errors FAIL command-line-arguments [build failed] FAIL ``` * Update util/openapi2proto/main.go with latest kin-openapi version * Update github.com/docker/docker to latest version * Update Docker client creation in tests * Add `WithAPIVersionNegotiation` to docker client creation * Rename 'Driver' to 'DriverCommand' in ContainerConfig * Fix minor linting errors * Update GitHub Actions workflow (linting) * fix: updated config check in hpc_backend.go * Revert tabs/space change in storage/local.go * Update output copying in scratch directory * Update compliance-test.yaml * Debug: add tmate step to Nextflow test * Update Nextflow tests * Update file copying behavior for Nextflow tests * Update local file copying behavior to support directory dests * Update local dest for files with glob patterns * Update Base64Encode test to match Github Actions test - TODO: investigate why this is necessary... * Fix Base64Encoding test value * Revert glob output change based on TesOutput spec - Wildcards in outputs will be uploaded to a directory, with no special handling for the filename. ```yaml tesOutput: properties: url: description: |- URL at which the TES server makes the output accessible after the task is complete. When tesOutput.path contains wildcards, it must be a directory; see `tesOutput.path_prefix` for details on how output URLs are constructed in this case. ``` * Debug: Add tmate step to Nextflow tests * Update wildcard file output destination * Add initial command templating for container engines * Add initial Docker/ContainerEngine tests * Run `go mod tidy` * Update Github Actions * Update container engine IO and command execution * Re-enable all tests for container engines * Update test suite * Nextflow test debug * Nextflow test debug * Nextflow test debug * Nextflow test debug * Update compliance test workflow * Update S3 tests * Update S3 tests * Add comments and clean up debug statements * Clean up code for 0.11.1-rc.1 release * Add container command event to match existing pattern * Update .goreleaser.yaml to replace deprecated fields * Update Nextflow instructions * Update K8s support and documentation - Update docker images to use quay.io/ohsu-comp-bio/funnel * Update README.md * fix: add support for wildcards in AWS S3 paths * Add support for wildcards in AWS S3 output paths * Update actions/upload-artifact to v4 in Github Actions - https://github.blog/changelog/2024-02-13-deprecation-notice-v1-and-v2-of-the-artifact-actions/ * Update version to 0.11.1-rc.4 in Makefile * Fix bug where FileMapper output was being overwritten in every loop * Add initial search functionality to website - Uses Pagefind: https://pagefind.app/docs/ * Update GitHub Actions workflows * Add Pagefind step to hugo.yaml * Run `go mod tidy` * Fix lint errors * debug: Slurm Github Action workflow * debug: Slurm GitHub Actions Workflow * debug: Slurm GitHub Actions Workflow * debug: Slurm GitHub Actions Workflow * debug: Slurm GitHub Actions Workflow * debug: Slurm GitHub Actions Workflow
ohsu-comp-bio · Oct 3, 2024 · 8166db7 · 8166db7
1 parent e157bb8
commit 8166db7
Show file tree

Hide file tree

Showing 67 changed files with 2,015 additions and 681 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -0,0 +1,43 @@
+name: Funnel Build and Cache
+
+on:
+  workflow_call:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Set up Go 1.x
+      uses: actions/setup-go@v5
+      with:
+        go-version: 1.21
+
+    - name: Check out code
+      uses: actions/checkout@v2
+
+    - name: Cache Funnel binary
+      uses: actions/cache@v3
+      with:
+        path: ./funnel
+        key: ${{ runner.os }}-funnel-bin-${{ hashFiles('**/go.sum') }}-${{ github.ref }}
+        restore-keys: |
+          ${{ runner.os }}-funnel-bin-${{ github.ref }}
+
+    - name: Build Funnel (if cache doesn't exist)
+      run: |
+        if [ ! -f ./funnel ]; then
+          make build
+        fi
+    
+    - name: Cache Funnel binary (after build)
+      uses: actions/cache@v3
+      with:
+        path: ./funnel
+        key: ${{ runner.os }}-funnel-bin-${{ hashFiles('**/go.sum') }}-${{ github.ref }}
+
+    - name: Upload Funnel binary as artifact
+      uses: actions/upload-artifact@v4
+      with:
+        name: funnel
+        path: funnel
diff --git a/.github/workflows/compliance-test.yaml b/.github/workflows/compliance-test.yaml
diff --git a/.github/workflows/compliance.yaml b/.github/workflows/compliance.yaml
@@ -0,0 +1,85 @@
+# Workflow for running the TES compliance suite against Funnel
+#
+# This includes the following steps:
+#  1. Build Funnel and store the resulting binary artifact
+#  2. Install tes-compliance-suite and run against every version of TES simultaneously
+#  3. start-report-deployment: Send a dispatch to the funnel-compliance repository to generate and publish
+#     the tes-compliance-suite report to https://ohsu-comp-bio.github.io/funnel-compliance/
+#
+# Optionally debug via SSH
+# Ref: https://fleetdm.com/engineering/tips-for-github-actions-usability
+#
+# To use this step uncomment and place anywhere in the build steps. The build will pause on this step and
+# output a ssh address associated with the Github action worker. Helpful for debugging build steps and
+# and intermediary files/artifacts.
+#
+# - name: "Debug: Package dependancies for tmate (CentOS)"
+#   run: |
+#     yum install -y xz
+#     ln -s /bin/true /bin/apt-get
+#
+# - name: Setup tmate session
+#   uses: mxschmitt/action-tmate@v3
+
+name: Compliance Test
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  build:
+    uses: ./.github/workflows/build.yml
+
+  compliance:
+    strategy:
+      fail-fast: false
+      matrix:
+        version: [1.0.0, 1.1.0]
+        db: ["boltdb", "mongodb"]
+        compute: ["local", "kubernetes"]
+        storage: ["local", "s3"]
+    needs: build
+    runs-on: ubuntu-latest
+    steps:
+      # Required to access the 'tests/mongo.config.yml' file
+      # Perhaps uploading it as an artifact would be more efficient?
+      - name: Check out code
+        uses: actions/checkout@v2
+
+      - uses: actions/download-artifact@v4
+        with:
+          name: funnel
+
+      - name: Start Funnel server
+        run: |
+          touch config.yml
+
+          if [ ${{ matrix.db }} = "mongodb" ]; then
+            make start-mongodb
+            cat `pwd`/tests/mongo.config.yml >> config.yml
+          fi
+
+          if [ ${{ matrix.storage }} = "s3" ]; then
+            docker run -d -p 9000:9000 --name minio \
+                       -e "MINIO_ROOT_USER=minioadmin" \
+                       -e "MINIO_ROOT_PASSWORD=minioadmin" \
+                       -v /tmp/data:/data \
+                       -v /tmp/config:/root/.minio \
+                       minio/minio server /data
+            cat `pwd`/tests/s3.config.yml >> config.yml
+          fi
+
+          chmod +x funnel
+          ./funnel server run --config `pwd`/config.yml &> funnel.logs &
+
+      - name: Run OpenAPI Test Runner
+        run: |
+          git clone https://github.com/elixir-cloud-aai/openapi-test-runner
+          cd openapi-test-runner
+          python3 -m venv venv
+          source venv/bin/activate
+          pip install -r requirements.txt
+          python setup.py install  
+          openapi-test-runner report --version "${{ matrix.version }}" --server "http://localhost:8000/"
+
diff --git a/.github/workflows/hugo.yml b/.github/workflows/hugo.yml
@@ -4,7 +4,9 @@ name: Deploy Hugo site to Pages
 on:
   # Runs on pushes targeting the default branch
   push:
-    branches: ["master"]
+    branches:
+      - main
+      - master
 
   # Allows you to run this workflow manually from the Actions tab
   workflow_dispatch:
@@ -59,6 +61,11 @@ jobs:
             --baseURL "${{ steps.pages.outputs.base_url }}/" \
             --source website \
             --destination public
+
+          # Run pagefind to add search functionality
+          # Reference: https://pagefind.app/docs/
+          npx -y pagefind --site public
+
       - name: Upload artifact
         uses: actions/upload-pages-artifact@v3
         with:

diff --git a/.github/workflows/nextflow.yaml b/.github/workflows/nextflow.yaml
@@ -2,55 +2,34 @@ name: Nextflow Test
 
 on:
   push:
+  pull_request:
 
 jobs:
   build:
-    runs-on: ubuntu-latest
-    steps:
-    - name: Set up Go 1.x
-      uses: actions/setup-go@v2
-      with:
-        go-version: 1.21
-
-    - name: Check out code
-      uses: actions/checkout@v2
-
-    - name: Build Funnel (if cache does not exist)
-      run: make build
+    uses: ./.github/workflows/build.yml
 
-    - name: Store Funnel
-      uses: actions/upload-artifact@v2
-      with:
-        name: funnelBin
-        path: funnel
-
   nextflow:
-    runs-on: ubuntu-latest
     needs: build
+    runs-on: ubuntu-latest
     steps:
     - name: Download Funnel
-      uses: actions/download-artifact@v2
+      uses: actions/download-artifact@v4
       with:
-        name: funnelBin
-        path: funnel
+        name: funnel
 
     - name: Start Funnel
-      working-directory: 
       run: |
-        cd funnel/
         chmod +x ./funnel
         ./funnel server --LocalStorage.AllowedDirs $HOME run &
 
     - name: Install Nextflow
       run: |
-        cd ..
-        git clone https://github.com/nextflow-io/nextflow/ -b tes-update-1.1
+        git clone https://github.com/nextflow-io/nextflow
         cd nextflow
         make compile
 
     - name: Install nf-canary and GA4GH-TES plugin
       run: |
-        cd ..
         git clone https://github.com/seqeralabs/nf-canary
         cd nf-canary
         cat <<EOF >> nextflow.config
@@ -60,8 +39,7 @@ jobs:
           process.executor = 'tes'
           tes.endpoint = 'http://localhost:8000'
         EOF
-  
+
     - name: Run nf-canary tests
       run: |
-        cd ../nf-canary
-        ../nextflow/launch.sh run main.nf
+        ./nextflow/nextflow run nf-canary/main.nf