Skip to content

databricks/genomics-pipelines

Folders and files

NameName
Last commit message
Last commit date
Jan 29, 2021
Jan 29, 2021
Jan 29, 2021
Feb 9, 2022
Jan 30, 2021
Jan 29, 2021
Mar 2, 2022
Mar 2, 2022
Feb 9, 2022
Jan 29, 2021
Feb 1, 2021
Feb 1, 2021
Jan 29, 2021

Repository files navigation

Genomics pipelines. At scale. With Spark and Glow. 🤯

What's inside?

Spark based pipelines for:

  • Variant calling (built on GATK's HaplotypeCaller)
  • Somatic variant calling (built on MuTect2)
  • Joint genotyping (built on GenotypeGVCFs)

Building and testing

  1. Clone the repo
  2. Unpack the big test files archive located in the project root
  • tar -xf big-files.tar.gz
  1. sbt test

Running on a Databricks cluster

  1. Create an init script to download the reference genome from cloud storage (see hls.sh or prepare_reference.py for inspiration.
  2. Build an uber jar (sbt assembly)
  3. Create a cluster with the init script from step 1 and attach the assembly jar.
  4. Run the desired pipeline using one of the attached notebooks.

License

Apache 2.0

Disclaimer

This is not an official Databricks product. This project is released without an expectation of continued development or maintenance.