Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vector CNV pipeline version 1.0.0 #116

Merged
merged 133 commits into from
Jan 10, 2024
Merged
Show file tree
Hide file tree
Changes from 130 commits
Commits
Show all changes
133 commits
Select commit Hold shift + click to select a range
2697dda
Initial commit for the CNV container.
kevinpalis Feb 22, 2023
871033c
Adding utility sript to easily build this image and push to Broad's GCR.
kevinpalis Feb 27, 2023
56a833f
First attempt at CNV WDL.
kevinpalis Mar 1, 2023
4ef3848
Added dockstore entry.
kevinpalis Mar 1, 2023
0011a1b
Test commit
kevinpalis Mar 1, 2023
4c98eb5
Trying to get a hello-world wdl to work so I can import this to Terra…
kevinpalis Mar 3, 2023
1a7872c
Fixing wrong imports
kevinpalis Mar 3, 2023
f8090b8
Trying to get this to work in Dockstore first.
kevinpalis Mar 3, 2023
21eda95
Testing variable assignments in Terra.
kevinpalis Mar 7, 2023
15c06ae
Testing out the new container in terra.
kevinpalis Mar 7, 2023
4ec8bbf
Fixes to dockerbuild for CNV. Adding new draft WDL to CNV.
kevinpalis Mar 20, 2023
483dd78
Fixes to dockerbuild for CNV. Adding new draft WDL to CNV.
kevinpalis Mar 20, 2023
7d98489
Debugging to determine actual dir struct on Terra
kevinpalis Mar 21, 2023
fbc487d
Debugging to determine actual dir struct on Terra
kevinpalis Mar 21, 2023
95e47da
Adding required parameters to the pipeline.
kevinpalis Mar 21, 2023
6d8ceca
New docker built.
kevinpalis Mar 22, 2023
5d1ec66
Merge pull request #113 from malariagen/master
kevinpalis Mar 22, 2023
992e352
Using new container with updated script.
kevinpalis Mar 22, 2023
8f85cc5
Moving away from calling shell scripts, calling python directly form WDL
kevinpalis Mar 23, 2023
2e8b9b9
Removing disk declaration as Terra wasn't happy about it.
kevinpalis Mar 23, 2023
4f1c40f
Changing ram value
kevinpalis Mar 23, 2023
f0ab218
Fixing a mistyped variable
kevinpalis Mar 23, 2023
724b7be
Absolute path for output doesn't seem to work in Terra
kevinpalis Mar 23, 2023
78c74a8
Fixing localization issues.
kevinpalis Mar 24, 2023
313fdd1
Debugging. File handling in terra is a pain.
kevinpalis Mar 24, 2023
e1eca6d
Trying out compressing the output directory
kevinpalis Mar 24, 2023
5ebcc7d
Weird WDL error on mixed whitespaces
kevinpalis Mar 25, 2023
1f81676
Weird WDL error on mixed whitespaces
kevinpalis Mar 25, 2023
a3908be
Fixing directory paths. Adding missing coveragelogs dir.
kevinpalis Mar 27, 2023
be89ec6
Parameterizing window size
kevinpalis Mar 28, 2023
07f6ae4
Setting default values for window size and quality min
kevinpalis Mar 28, 2023
78bb228
Started adding files sent by Eric as parameters to coverage summary task
kevinpalis Apr 4, 2023
6657e46
Adding some documentation on params
kevinpalis Apr 4, 2023
1edcbaf
Added all parameters needed by coverage stats
kevinpalis Apr 4, 2023
588b610
Fixing misnamed param meta
kevinpalis Apr 4, 2023
72ad872
Removing commented out code that apparently was still causing errors …
kevinpalis Apr 4, 2023
6c5c438
Adding coverage script execution to command block
kevinpalis Apr 5, 2023
0061f94
Commenting out CoverageHMM task so I can test the previous steps in T…
kevinpalis Apr 10, 2023
902125f
Forgot a runtime block
kevinpalis Apr 10, 2023
0fe525a
Catching the log file so it doesn't disappear
kevinpalis Apr 11, 2023
396c47e
Adding bai
kevinpalis Apr 11, 2023
b55c5d8
Fixed a typo in calling a sample name variable that caused the covera…
kevinpalis Apr 12, 2023
c45a4a2
Debugging prints because I can't figure out why sample names aren't b…
kevinpalis Apr 12, 2023
9bd88d3
Fixing the fix. Dollar sign vs tilde for bash vs WDL variable got me.
kevinpalis Apr 13, 2023
576a5d3
Additional changes
jessicaway Apr 14, 2023
0f8c038
Merge branch 'kp_create_cnv_container_PD-2152' into jw_add_target_reg…
jessicaway Apr 14, 2023
ddfbdc6
Moving on to HMM task while waiting on new BAM files
kevinpalis Apr 18, 2023
28857c5
Additional changes
jessicaway May 2, 2023
e7ae5be
Merge branch 'kp_create_cnv_container_PD-2152' into jw_add_target_reg…
jessicaway May 2, 2023
0f2ec0c
Setting output of coverage summary
kevinpalis May 12, 2023
d1b5ee8
Setting output of coverage summary
kevinpalis May 12, 2023
5b543f1
Trying out a different approach to dynamic output naming.
kevinpalis May 16, 2023
a1e17fd
Can't have spaces around the equal sign apparently.
kevinpalis May 16, 2023
677bd06
Doubled the double quotes.
kevinpalis May 16, 2023
bded25a
Using WDL string sub
kevinpalis May 16, 2023
8e7485a
Fixing error in substitution with floats
kevinpalis May 16, 2023
4b79228
Replacing with nothing instead of underscore
kevinpalis May 16, 2023
6c579d3
trying to fix missing numbers
kevinpalis May 16, 2023
0415910
trying to fix missing numbers
kevinpalis May 16, 2023
18be7a7
trying to fix missing numbers
kevinpalis May 16, 2023
24a35c0
trying to fix missing numbers
kevinpalis May 16, 2023
bb21a9a
trying to fix missing numbers
kevinpalis May 16, 2023
c3a8e06
trying to fix missing numbers
kevinpalis May 16, 2023
918a5c5
Additional changes
jessicaway May 16, 2023
59c580b
Merge branch 'kp_create_cnv_container_PD-2152' into jw_add_target_reg…
jessicaway May 16, 2023
34a1755
trying to fix missing numbers
kevinpalis May 16, 2023
91f1b06
Additional changes
jessicaway May 17, 2023
5bd60e0
Adding Terra parameters and proper call to the HMM step.
kevinpalis May 22, 2023
e7af857
Fixing path as terra finds output relative to cromwell_root
kevinpalis May 22, 2023
3c0f06d
Forgot them special variable resolution syntax
kevinpalis May 22, 2023
a5cef4b
Fixing missing directory issue.
kevinpalis May 25, 2023
a7eccaa
Fixing missing directory issue.
kevinpalis May 26, 2023
e9340b8
Fixing missing directory issue.
kevinpalis May 26, 2023
b793ef9
Cleaning up the code a bit
kevinpalis May 26, 2023
9b7df10
Enabling parallel runs again
kevinpalis Jun 1, 2023
c2b5684
Additional changes
jessicaway Jun 5, 2023
cde6e31
Merge branch 'kp_create_cnv_container_PD-2152' into jw_add_target_reg…
jessicaway Jun 5, 2023
b034801
Additional changes
jessicaway Jun 5, 2023
3c63454
Additional changes
jessicaway Jun 6, 2023
8e97091
Additional changes
jessicaway Jun 6, 2023
01bd0e2
Additional changes
jessicaway Jun 6, 2023
fa65317
Additional changes
jessicaway Jun 7, 2023
dffca6e
Just a little code cleanup.
kevinpalis Jun 9, 2023
48e6c9f
update docker
jessicaway Jun 12, 2023
657c01f
additional changes
jessicaway Jun 12, 2023
947047e
Additional changes
jessicaway Jun 14, 2023
2c66e85
where is the Rscripts folder?
jessicaway Jun 14, 2023
4bf7c3c
additional changes
jessicaway Jun 14, 2023
d16e176
additional changes
jessicaway Jun 15, 2023
ca73db1
add debugging statements
jessicaway Jun 15, 2023
62a6057
update scripts with new versions
jessicaway Jul 31, 2023
41e4eae
updates
jessicaway Aug 7, 2023
f25b678
Adding consolidation step for all the tarballs
kevinpalis Aug 14, 2023
801b4ab
Weird WDL to bash syntax conflict resolved
kevinpalis Aug 14, 2023
ba6b4ca
Plugging in the call to consolidate the tarballs
kevinpalis Aug 15, 2023
15114bf
Some files are unfortunately lost. Trying if tar can keep them
kevinpalis Aug 16, 2023
90f1bdc
Some files are unfortunately lost. Trying if tar can keep them
kevinpalis Aug 16, 2023
6eddb91
Current method worked well, no need to manually move files.
kevinpalis Aug 16, 2023
6c0779b
updates
jessicaway Aug 18, 2023
731c98a
Merge branch 'kp_create_cnv_container_PD-2152' into jw_add_target_reg…
jessicaway Aug 18, 2023
1628d7c
update docker
jessicaway Aug 18, 2023
267d6d1
remove echos
jessicaway Sep 6, 2023
3c190d5
remove Rdata output
jessicaway Sep 6, 2023
053b44d
remove Rdata output
jessicaway Sep 6, 2023
d0b5b6c
stop piping output to a log that we don't capture
jessicaway Sep 20, 2023
87a8899
explicitly add the coverage dir and diagnostic reads dir
jessicaway Oct 4, 2023
cfe981b
fix things
jessicaway Oct 4, 2023
5e35cdf
fix things again
jessicaway Oct 4, 2023
00c04bb
add outputs
jessicaway Oct 4, 2023
dc6f124
improvements
jessicaway Oct 4, 2023
279935b
Add CNV coverage calls step to CNV pipeline
jessicaway Nov 3, 2023
a5e7b93
Validates now
jessicaway Nov 14, 2023
cdb2682
remove unused input
jessicaway Nov 15, 2023
a6c5397
Update docker
jessicaway Nov 29, 2023
09ce25b
Add debugging statement
jessicaway Nov 29, 2023
4dd046e
fix bash errors
jessicaway Nov 29, 2023
ccbc611
Hopefully this lets it run
jessicaway Nov 30, 2023
ef1e8dd
debugging statements
jessicaway Nov 30, 2023
c8c7b5f
this should help
jessicaway Nov 30, 2023
5effa75
this should help
jessicaway Nov 30, 2023
ad5b553
dynamically set resources for coverage calls step
jessicaway Dec 19, 2023
f5a8c7a
error
jessicaway Dec 19, 2023
761c3ef
fix dependency graph?
jessicaway Jan 9, 2024
e3722e6
fix dependency graph?
jessicaway Jan 9, 2024
c149b59
fix dependency graph?
jessicaway Jan 9, 2024
0bb6d34
Cleanup
jessicaway Jan 9, 2024
bdda415
Update pipelines/copy-number-variation-vector/gcp/CNV.wdl
jessicaway Jan 10, 2024
de4d49e
Update pipelines/copy-number-variation-vector/gcp/CNVCoverageCalls.wdl
jessicaway Jan 10, 2024
2caa765
Update pipelines/copy-number-variation-vector/gcp/CNVCoverageCalls.wdl
jessicaway Jan 10, 2024
d220b4b
Update pipelines/copy-number-variation-vector/gcp/CNVCoverageCalls.wdl
jessicaway Jan 10, 2024
0aa103d
Cleanup
jessicaway Jan 10, 2024
c79ab47
Merge remote-tracking branch 'origin/jw_add_target_regions_subpipelin…
jessicaway Jan 10, 2024
995e919
Update pipelines/copy-number-variation-vector/gcp/TargetRegions.wdl
jessicaway Jan 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .dockstore.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,6 @@ workflows:
- name: MalariaVector_StatisticalPhasing
subclass: WDL
primaryDescriptorPath: /pipelines/phasing-vector/gcp/StatisticalPhasing.wdl
- name: MalariaVector_CNV
subclass: WDL
primaryDescriptorPath: /pipelines/copy-number-variation-vector/gcp/CNV.wdl
34 changes: 34 additions & 0 deletions dockerfiles/CNV/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
FROM --platform=linux/amd64 continuumio/miniconda3

# Update the package lists and install necessary packages
RUN apt-get update && \
apt-get install -y curl bzip2 ca-certificates gnupg && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*

#TODO: Determine if R is really required here
# # Add the R repository and install the latest R
# RUN echo "deb https://cloud.r-project.org/bin/linux/ubuntu hirsute-cran40/" >> /etc/apt/sources.list && \
# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 && \
# apt-get update && \
# apt-get install -y r-base && \
# apt-get clean && \
# rm -rf /var/lib/apt/lists/*

# Copy the environment file to the container
COPY requirements_conda.yml requirements_conda.yml

# Create a conda environment and activate it
RUN mkdir -p /cnv/output && \
mkdir -p /cnv/input && \
conda env create -f requirements_conda.yml && \
echo "conda activate $(head -1 /opt/conda/envs/*/etc/environment.yml | cut -d'/' -f6)" >> ~/.bashrc

# Set the working directory
WORKDIR /cnv

# Copy the source code to the container
COPY . .

# Set the default command to run when the container starts
# CMD ["python", ""]
41 changes: 41 additions & 0 deletions dockerfiles/CNV/R3.6.1/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
## Platform tag ensures that images built on ARM-based machines (ex. M-series macs) are portable
FROM --platform="linux/amd64" ubuntu:18.04

ENV DEBIAN_FRONTEND=noninteractive

# Set parameteres
ARG R_VERSION="3.6.1"

WORKDIR /usr/local/

# Update and install dependencies and wet
RUN sed -Ei 's/^# deb-src /deb-src /' /etc/apt/sources.list && apt-get update \
&& apt-get install -y \
build-essential \
wget

# Install R and addtional dependencies
RUN apt-get build-dep -y r-base \
&& wget https://cloud.r-project.org/src/base/R-3/R-${R_VERSION}.tar.gz \
&& tar xzf R-${R_VERSION}.tar.gz \
&& cd "R-${R_VERSION}" \
&& ./configure --prefix=/opt/R/${R_VERSION}/ --enable-R-shlib --with-blas --with-lapack \
&& make \
&& make install

RUN rm -r R-${R_VERSION} \
&& rm R-${R_VERSION}.tar.gz

# Run an R script to install several libraries
COPY installRDeps.R /usr/local/
RUN /opt/R/${R_VERSION}/bin/Rscript /usr/local/installRDeps.R \
&& rm installRDeps.R

# Add R installation directory to path
ENV PATH="/opt/R/${R_VERSION}/bin/:$PATH"

# Copy source code
COPY . .



Loading
Loading