Feature/cttso icav2 step function branch

[alexiswl]

Summary

This AWS step functions performs the following actions

• Copies a set of fastq files to a structured location (via the icav2 copy batch utility)
• Uploads a samplesheet to the same structured location
• Invokes the ctTSO v2 pipeline calling the structured location as the run folder

Dependent PRs

• Added feature icav2 copy batch state machine #143
• Feature/bssh icav2 fastq copy manager #144

TODO List

• Implement stateful database to track analysis IDs and runs
• sfn to handle 'state-change' in analysis run (and update Database)
• External event handling on state change
• Raise internal event on terminal state change
• Fix Technical tags, should have it's own statefunctions steps

[victorskl]

LGTM, too.

Yup, same, same. Needa integrate with outer CDK and, perhaps you can break it down pending TODOs as another follow up PR of the same issue implementation story. Leave it with you.!

[reisingerf]

Minor questions, nothing that should stop/delay this.

[alexiswl]

Minor questions, nothing that should stop/delay this.

Yep I'd like to pivot to bclconvert-interop-qc which is a tiny pipeline service to be triggered after the successful completion of the bssh fastq copy manager service

The interop qc workflow is shown here - https://github.com/umccr/cwl-ica/releases/tag/bclconvert-interop-qc%2F1.3.1--1.19__202403122095003

This will mean I can focus on (without accidentally deploying 100 cttso v2 workflows)

1. Event handling
2. Database handling, updating states, entries etc.

We can then move this logic into here (#145) as well.

[alexiswl]

@victorskl @reisingerf Readme should be up to date here - https://github.com/umccr/orcabus/blob/feature/cttso-icav2-step-function-branch/lib/workload/stateless/cttso_v2_pipeline_manager/Readme.md

[victorskl]

Thanks for update, Alexis. I will go through the README, today.

(And, I will come back to this TSOv2 branch once basespace upload service branch #186 review done in tick.)

[victorskl]

Put this branch into draft mode until we finalise

[alexiswl]

WIll prioritise this one over today and tomorrow, shouldn't take more than an hour to get at least the stateless architecture of this PR up to scratch on being part of the Orcabus pipeline

[victorskl]

.. eta; will review tomorrow (30/04/24)

[victorskl]

reviewing

[victorskl]

LGTM with review notes:

• add 1 note
• skip BsshIcav2FastqCopyManagerStack (already reviewed Feature/bssh icav2 fastq copy manager #144)
• skip TypeScript case (already discussed)
• skip Lambda unit test for later (already discussed)
• check to cdk synth success
• heads-up to cdk-nag fail with AwsSolutions-IAM5

[victorskl]

(No action need for this comment. At least for now.)

Just for my note; I said what I need to note here as reviewer. I wanted to say something about this. But. I am torn at the same time. Thinking about argument that invites as side-effect after 3/4 steps. 😭

Perhaps, we just go by documentation that the notion of sample_id is context specific. Such that in which context we are referring this ID to...

This makes me; a quick go and glimpse at our MetadataManager and, feel settle/calm that there are no traces of sample_id notion there, yet.

TL;DR sample_id notion is context specific. It only gets meaningful/complete by its context.

[alexiswl]

In this case, sample_id will match the library id, for v2 samplesheets we strip out everything except the library id, including _topup and _rerun so sample_id and library_id will be synonymous, however, sample_id is the parameter name into the cttso library, can update this to library_id if you want here

[victorskl]

Still, the upstream tso v2 pipeline analysis input would keep it as sample_id at icav2 endpoint, right? This pipeline is not in our control. (sign) If folks happen to look at icav2 endpoint directly, this analysis input would still be confused those users, I think. (Hence, reason I am torn)

[victorskl]

Having said, we can also make a call to harmonise and consistent among ourselves at our system interfaces (especially those system entry point expose to user). i.e. We use library_id throughout our schema and input json formats.

Each services would translate as such mapping need as internal logic when appropriate.

Perhaps, we can establish this dev SOP/guideline, I reckon. What you think.?

[victorskl]

Understood, all good. Thanks for experimenting here. This is valuable explore.

[victorskl]

Would be good to merge this first, before we are following up on WRSC schema alignment, Alexis.

[alexiswl]

Already need to resolve some conflicts again.

Will try align the stack as best I can with respect to the workflowRunStateChange schema and will merge from there.