pipeline health POC #59
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| *.pyc | ||
| venv/ | ||
| __pycache__/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,5 @@ | ||||||
| config: | ||||||
| github:owner: nf-core-tf | ||||||
| # https://start.1password.com/open/i?a=O5GICFDKPNABLLVGMKBL5JWDWA&v=rdfcz6oy6qxxrc4clu467a7dmm&i=4ajrv44kc5lcbboa37fr5oydla&h=nf-core.1password.eu | ||||||
| github:token: | ||||||
| secure: AAABAFMgBNyCNuYsps6YVPV2L7Ji5qBJj0omEQQa9HrdhT2iHo3ex0e9NsDER3Q04itGiY698X/ZQCnTM2zu9op3tcjmzfITdHxGy0FGATuUFamYsSiztHrNAKiIEJ9E0M4Al8/yJeB6X4BXvkLEgik/I+GPvZIXK3tE65Q= | ||||||
|
There was a problem hiding this comment.
Suggested change
should we set this as a github secret? There was a problem hiding this comment. We can, the reason this one isn't is because I was struggling with the 1password Pulumi ESC integration, and I didn't realize you have to copy the plain service key into the environment file, and then it encrypts it in place for that specific environment file. Anyways there's a few options:
This one doesn't really matter, because it's just to the Leaning Pulumi ESC for now as:
We could do all of that with GitHub actions, and pass all of these things, but the secret management is already a complicated web, but it's working currently. TL;DR something to explore, I'll update this one and move it to Pulumi ESC though. |
||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| name: github-repos | ||
| runtime: | ||
| name: python | ||
| options: | ||
| virtualenv: venv | ||
| description: Managing GitHub repos | ||
edmundmiller marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| config: | ||
| pulumi:tags: | ||
| value: | ||
| pulumi:template: https://www.pulumi.com/ai/api/project/3cb51e5f-2548-4d7b-9d9d-1ea680ac96ee.zip | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| # Repos | ||
|
|
||
| Goal is to replace https://oldsite.nf-co.re/pipeline_health | ||
|
|
||
| This repo will be the "Actions" section at the bottom. We can then create a reporting page if we really need to see all the green checks | ||
|
|
||
| ## Initial Roll-out | ||
|
|
||
| The new pipelines that are broken: | ||
|
|
||
| - denovotranscript | ||
| - meerpipe | ||
| - pairgenomealign | ||
| - phaseimpute | ||
| - reportho | ||
|
|
||
| Maybe: | ||
|
|
||
| - scdownstream | ||
| - scnanoseq | ||
|
|
||
| ### Plan | ||
|
|
||
| #### Short-term | ||
|
|
||
| 1. [ ] Import a pipeline that has all the right settings | ||
| 2. [ ] Fix the 5 pipelines above with the correct settings from the "model" repo | ||
| 3. [ ] Keep importing new pipelines until we gain confidence in it. | ||
|
|
||
| #### Long-term | ||
|
|
||
| 1. Wrangle in `core_repos` | ||
| 2. Roll out to all pipelines | ||
| 3. Switch all repos to main | ||
edmundmiller marked this conversation as resolved.
Show resolved
Hide resolved
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| #!/usr/bin/env python | ||
|
|
||
| import yaml | ||
|
|
||
| import pulumi | ||
| import pulumi_github as github | ||
|
|
||
|
|
||
| nf_core_tf = github.Repository( | ||
| "nf-core-tf", | ||
| allow_merge_commit=False, | ||
| allow_rebase_merge=False, | ||
| allow_squash_merge=False, | ||
| default_branch="master", | ||
| description="Repository to host tool-specific module files for the Nextflow DSL2 community!", | ||
| has_downloads=True, | ||
| has_issues=True, | ||
| has_projects=True, | ||
| homepage_url="https://nf-co.re", | ||
| merge_commit_message="", | ||
| merge_commit_title="", | ||
| name="modules", | ||
| security_and_analysis=github.RepositorySecurityAndAnalysisArgs( | ||
| secret_scanning=github.RepositorySecurityAndAnalysisSecretScanningArgs( | ||
| status="disabled", | ||
| ), | ||
| secret_scanning_push_protection=github.RepositorySecurityAndAnalysisSecretScanningPushProtectionArgs( | ||
| status="disabled", | ||
| ), | ||
| ), | ||
| squash_merge_commit_message="", | ||
| squash_merge_commit_title="", | ||
| topics=[ | ||
| "nextflow", | ||
| "pipelines", | ||
| "nf-test", | ||
| "modules", | ||
| "nf-core", | ||
| "dsl2", | ||
| "workflows", | ||
| ], | ||
| visibility="public", | ||
| opts=pulumi.ResourceOptions(protect=True), | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| import yaml | ||
|
|
||
| import pulumi | ||
| import pulumi_github as github | ||
|
|
||
|
|
||
| nf_core_tf = github.Repository( | ||
| "nf-core-tf", | ||
| allow_merge_commit=False, | ||
| allow_rebase_merge=False, | ||
| allow_squash_merge=False, | ||
| default_branch="master", | ||
| description="Repository to host tool-specific module files for the Nextflow DSL2 community!", | ||
| has_downloads=True, | ||
| has_issues=True, | ||
| has_projects=True, | ||
| homepage_url="https://nf-co.re", | ||
| merge_commit_message="", | ||
| merge_commit_title="", | ||
| name="modules", | ||
| security_and_analysis=github.RepositorySecurityAndAnalysisArgs( | ||
| secret_scanning=github.RepositorySecurityAndAnalysisSecretScanningArgs( | ||
| status="disabled", | ||
| ), | ||
| secret_scanning_push_protection=github.RepositorySecurityAndAnalysisSecretScanningPushProtectionArgs( | ||
| status="disabled", | ||
| ), | ||
| ), | ||
| squash_merge_commit_message="", | ||
| squash_merge_commit_title="", | ||
| topics=[ | ||
| "nextflow", | ||
| "pipelines", | ||
| "nf-test", | ||
| "modules", | ||
| "nf-core", | ||
| "dsl2", | ||
| "workflows", | ||
| ], | ||
| visibility="public", | ||
| opts=pulumi.ResourceOptions(protect=True), | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| - .github | ||
edmundmiller marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - basic_training | ||
| - configs | ||
| - logos | ||
| - modules | ||
| - ops | ||
| - prettier-plugin-nextflow | ||
| - references | ||
| - setup-nextflow | ||
| - setup-nf-test | ||
| - sublime | ||
| - test-datasets | ||
| - tools | ||
| - vale | ||
| - vscode-extensionpack | ||
| - website | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| #!/usr/bin/env python | ||
|
|
||
| import yaml | ||
|
|
||
| import pulumi | ||
| import pulumi_github as github | ||
|
|
||
| import pipelines.denovotranscript | ||
| import pipelines.meerpipe | ||
| import pipelines.pairgenomealign | ||
| import pipelines.phaseimpute | ||
| import pipelines.reportho | ||
|
|
||
| # ... | ||
|
|
||
| import core.github | ||
| import core.modules | ||
|
|
||
| # ... | ||
| import core.website |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| #!/usr/bin/env python | ||
|
|
||
| import yaml | ||
|
|
||
| import pulumi | ||
| import pulumi_github as github | ||
|
|
||
| TOPICS = [ | ||
| "nextflow", | ||
| "pipelines", | ||
| "nf-test", | ||
| "modules", | ||
| "nf-core", | ||
| "dsl2", | ||
| "workflows", | ||
| ] | ||
|
|
||
| alpha_test_pipeline_repos = [ | ||
| "denovotranscript", | ||
| "meerpipe", | ||
| "pairgenomealign", | ||
| "phaseimpute", | ||
| "reportho", | ||
| ] | ||
|
|
||
| for pipeline in alpha_test_pipeline_repos: | ||
| github.Repository( | ||
| "nf-core-tf", | ||
| allow_merge_commit=True, | ||
| allow_rebase_merge=True, | ||
| allow_squash_merge=True, | ||
| default_branch="master", | ||
| description="Alpha test repository for nf-core", | ||
| has_downloads=True, | ||
| has_issues=True, | ||
| has_projects=True, | ||
| homepage_url=f"https://nf-co.re/{pipeline}", | ||
| merge_commit_message="", | ||
| merge_commit_title="", | ||
| name=pipeline, | ||
| security_and_analysis=github.RepositorySecurityAndAnalysisArgs( | ||
| secret_scanning=github.RepositorySecurityAndAnalysisSecretScanningArgs( | ||
| status="disabled", | ||
| ), | ||
| secret_scanning_push_protection=github.RepositorySecurityAndAnalysisSecretScanningPushProtectionArgs( | ||
| status="disabled", | ||
| ), | ||
| ), | ||
| squash_merge_commit_message="", | ||
| squash_merge_commit_title="", | ||
| topics=TOPICS, | ||
| visibility="public", | ||
| # NOTE Idk if this will work | ||
| opts=pulumi.ResourceOptions(protect=True), | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| - airrflow | ||
edmundmiller marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - ampliseq | ||
| - atacseq | ||
| - bacass | ||
| - bactmap | ||
| - bamtofastq | ||
| - cageseq | ||
| - callingcards | ||
| - chipseq | ||
| - circdna | ||
| - circrna | ||
| - clipseq | ||
| - coproid | ||
| - createpanelrefs | ||
| - createtaxdb | ||
| - crisprseq | ||
| - cutandrun | ||
| - datasync | ||
| - demo | ||
| - demultiplex | ||
| - denovotranscript | ||
| - detaxizer | ||
| - diaproteomics | ||
| - differentialabundance | ||
| - dualrnaseq | ||
| - eager | ||
| - epitopeprediction | ||
| - fastquorum | ||
| - fetchngs | ||
| - funcscan | ||
| - genomeannotator | ||
| - genomeassembler | ||
| - genomeskim | ||
| - gwas | ||
| - hgtseq | ||
| - hic | ||
| - hicar | ||
| - hlatyping | ||
| - imcyto | ||
| - isoseq | ||
| - lncpipe | ||
| - mag | ||
| - magmap | ||
| - marsseq | ||
| - mcmicro | ||
| - meerpipe | ||
| - metaboigniter | ||
| - metapep | ||
| - metatdenovo | ||
| - methylseq | ||
| - mhcquant | ||
| - mnaseseq | ||
| - molkart | ||
| - multiplesequencealign | ||
| - nanoseq | ||
| - nanostring | ||
| - nascent | ||
| - omicsgenetraitassociation | ||
| - oncoanalyser | ||
| - pairgenomealign | ||
| - pangenome | ||
| - pathogensurveillance | ||
| - pgdb | ||
| - phageannotator | ||
| - phaseimpute | ||
| - phyloplace | ||
| - pixelator | ||
| - proteinfold | ||
| - proteomicslfq | ||
| - radseq | ||
| - rangeland | ||
| - raredisease | ||
| - readsimulator | ||
| - reportho | ||
| - riboseq | ||
| - rnadnavar | ||
| - rnafusion | ||
| - rnaseq | ||
| - rnasplice | ||
| - rnavar | ||
| - sammyseq | ||
| - sarek | ||
| - scdownstream | ||
| - scnanoseq | ||
| - scrnaseq | ||
| - seqinspector | ||
| - setup-nf-test | ||
| - slamseq | ||
| - smrnaseq | ||
| - spatialvi | ||
| - spinningjenny | ||
| - taxprofiler | ||
| - tbanalyzer | ||
| - tfactivity | ||
| - vale | ||
| - variantbenchmarking | ||
| - variantcatalogue | ||
| - viralintegration | ||
| - viralrecon | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| pulumi>=3 | ||
| pulumi_github>=5.20.0 | ||
| ruff>=0.3.7 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,3 @@ | ||
| *.pyc | ||
| venv/ | ||
| __pycache__/ | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this throws an error for me. is that maybe for your personal account?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated the link, you also we're in the
Devvault. Had to make a vault that was specifically accessible to the service accounts, and I didn't want to give them access to everything.