Friday changes: #589

alexiswl · 2024-10-04T04:35:38Z

PierianDx

Added pieriandx pipeline manager (monitors runs and sends events once the reports are generated)
Added nails for stacky to connect cttsov2 to pieriandx pipeline

Workflow Type Renames

wgtsQc to wgts-qc
tumor_normal to tumor-normal

ICAv2 Resilience

Allow a ‘rerun’ of a workflow execution if the analysis id doesn’t exist (but the portal id is in the database).
Reduce concurrency in cttsov2 copy jobs to 5
Reduce concurrency in bsshfastqcopy jobs to 5

Metadata

Refactored metadata tools to match new metadata manager structure in dev
Made metadata mapper lambda construct that allows user to easily collect an orcabus id from an internal id (library id / subject id / project id etc) and vice versa.

Showers

Don’t produce subject based showers.

Stacky

Refactored stacky services to not register subjects / instruments, libraries only, use a scan to find other libraries that have the same subject / on same instrument instead. Reduces number of components within the stack.

## PierianDx * Added pieriandx pipeline manager (monitors runs and sends events once the reports are generated) * Added nails for stacky ## Workflow Type Renames * wgtsQc to wgts-qc * tumor_normal to tumor-normal ## ICAv2 Resilience * Allow a ‘rerun’ of a workflow execution if the analysis id doesn’t exist (but the portal id is in the database). * Reduce concurrency in cttsov2 copy jobs to 5 * Reduce concurrency in bsshfastqcopy jobs to 5 ## Metadata * Refactored metadata tools to match new metadata manager structure in dev * Made metadata mapper lambda construct that allows user to easily collect an orcabus id from an internal id (library id / subject id / project id etc) and vice versa. ## Showers Don’t produce subject based showers. ## Stacky Refactored stacky services to not register subjects / instruments, libraries only, use a scan to find other libraries that have the same subject / on same instrument instead. Reduces number of components within the stack.

victorskl · 2024-10-06T23:28:36Z

Massive changes.! Going through today..

reisingerf · 2024-10-07T02:53:11Z

Uh... way too much for me to go through in detail, so will focus on a few points of interest...

victorskl · 2024-10-07T03:45:14Z

Hey I have 3 coffee ☕ today. All double shots! So why not.

reisingerf

Comments, but no blocker from my side...

reisingerf · 2024-10-07T02:59:06Z

config/constants.ts

-export const wgtsQcIcav2PipelineWorkflowType = 'wgtsQc';
+export const wgtsQcIcav2PipelineWorkflowType = 'wgts-qc';
 export const wgtsQcIcav2PipelineWorkflowTypeVersion = '4.2.4';


Most of those are ...WorkflowType and ...WorkflowTypeVersion combinations,
but there are some that are ...WorkflowName and ...WorkflowTypeVersion or
...WorkflowName, ...WorkflowVersion

The rename/update does not affect all of those?
E.g. bsshFastqCopy

Ah let me fix bsshFastqCopy too, bsshFastqCopy is not used in any output paths so got less attention but I think it would be good to fix for workflow run name consistency

btw: are we OK with cttsov2?

First, we are adding a version where the workflow definition has already a version field. That feels redundant and confusing and may limit our future uses of that workflow (e.g. what if there's a new version 3.0.1?)
Second, we've seen now multiple other places where the analysis and the sample type are separated, e.g. tso-ctdna. I wonder if we are limiting the possible use cases combining those into one term, e.g. could there be a tso-ctrna, a tso600-ctdna or others?

I just want to take this opportunity to rethink our workflow naming based on the experience we have now, not simply following the convenience of what we used so far.
I am not the best person to judge this though, so I'll follow the consensus

Maybe a discussion on Slack where we can reach a consensus.

reisingerf · 2024-10-07T03:38:12Z

...stacks/cttso-v2-pipeline-manager/lambdas/check_num_running_sfns_py/check_num_running_sfns.py

+    Get the Step Functions client
+    :return: SFNClient
+    """
+    return boto3.client('stepfunctions')


Perhaps initialise the client statically?

reisingerf · 2024-10-07T04:23:33Z

lib/workload/stateless/stacks/pieriandx-pipeline-manager/Readme.md

+
+The AWS Launch step function takes in the following inputs
+
+* dag: Object


eh, what's a dag?

This is a great question that not even the PierianDx docs can explain. A dag comprises a name and version (in our case - cromwell_tso500_ctdna_workflow_1.0.4 / tso500_ctdna_workflow), so I think a dag maps to a workflow and as thus we can infer that dag is the acronym 'directed acyclical graph'

reisingerf · 2024-10-07T04:25:48Z

lib/workload/stateless/stacks/pieriandx-pipeline-manager/Readme.md

@@ -0,0 +1,247 @@
+# PierianDx Pipeline Manager
+
+Giant, serverless pipeline attached to a serverless database


This does not create WRSC events, right?
Would it make sense to add those, so the WorkflowManager can track them?

(I might have missed it)

The 'nails' glue creates workflow run state change 'ready' events.

The monitor-pieriandx-runs step functions will generate a workflow run state change event once pieriandx has completed

victorskl

LGTM. Massive effort!

victorskl · 2024-10-07T06:12:53Z

...templates/fastq_list_row_shower_complete_event_to_cttsov2_draft_events_sfn_template.asl.json

@@ -11,25 +11,51 @@
    },
    "Get Libraries from Instrument Run": {
      "Type": "Task",
-      "Resource": "arn:aws:states:::dynamodb:getItem",
+      "Resource": "arn:aws:states:::aws-sdk:dynamodb:scan",


Just a note.

This is ok. Perhaps, create GSI/LSI indexes on those none PK/SK lookup attributes where appropriate.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html

If, however, the use case started to explode such that the need of select * from table use case all the time or, some dynamic with filtering and predicates condition to a table then it is ok to tap into Postgres.

Anyhow. We shall observe this part as-is for now.

This is runs one per instrument run, I don't think this is a burden on our compute resources

Statemachine objects have no object 'current' version.

alexiswl added the feature New feature label Oct 4, 2024

alexiswl self-assigned this Oct 4, 2024

alexiswl requested review from victorskl and reisingerf October 4, 2024 04:36

Validate make-check

9facfdf

alexiswl mentioned this pull request Oct 4, 2024

Add pieriandx pipeline manager #222

Closed

3 tasks

This was linked to issues Oct 4, 2024

Should tumor_normal output dir be tumor-normal? #559

Closed

Set concurrency limits for icav2 copy jobs #580

Closed

Add PierianDx Pipeline Manager as a service #294

Closed

Only give latest versions permissions

00ae37d

Only give latest permissions to sfn on lambda objects

f20da82

Fix cdk-nag permisisons

2104ee6

reisingerf approved these changes Oct 7, 2024

View reviewed changes

CDK Nag fix

be48b09

victorskl approved these changes Oct 7, 2024

View reviewed changes

Fixes cdk-nag error on grant read to statemachine object

6355897

Statemachine objects have no object 'current' version.

alexiswl merged commit 7e68a11 into main Oct 8, 2024
6 checks passed

alexiswl deleted the feature/pieriandx-pipeline-manager branch October 8, 2024 23:51

This was referenced Oct 14, 2024

cttsov2 job checker requires permissions to list executions #603

Merged

Pieriandx needs 'SUCCEEDED' status as positive terminal status not 'COMPLETE' #604

Merged

alexiswl mentioned this pull request Oct 14, 2024

Add fastqc / sequali stats to fastq list row events for first 1 million reads #602

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Friday changes: #589

Friday changes: #589

alexiswl commented Oct 4, 2024 •

edited

Loading

victorskl commented Oct 6, 2024

reisingerf commented Oct 7, 2024

victorskl commented Oct 7, 2024

reisingerf left a comment

reisingerf Oct 7, 2024

alexiswl Oct 7, 2024

reisingerf Oct 7, 2024

pdiakumis Oct 8, 2024

reisingerf Oct 7, 2024

reisingerf Oct 7, 2024

alexiswl Oct 7, 2024

reisingerf Oct 7, 2024

alexiswl Oct 7, 2024

victorskl left a comment

victorskl Oct 7, 2024

alexiswl Oct 8, 2024


		The AWS Launch step function takes in the following inputs

		* dag: Object

		@@ -0,0 +1,247 @@
		# PierianDx Pipeline Manager

		Giant, serverless pipeline attached to a serverless database

Friday changes: #589

Friday changes: #589

Conversation

alexiswl commented Oct 4, 2024 • edited Loading

PierianDx

Workflow Type Renames

ICAv2 Resilience

Metadata

Showers

Stacky

victorskl commented Oct 6, 2024

reisingerf commented Oct 7, 2024

victorskl commented Oct 7, 2024

reisingerf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

victorskl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexiswl commented Oct 4, 2024 •

edited

Loading