07_querying_lakehouse.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pyspark\n",
    "from delta import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "builder = pyspark.sql.SparkSession.builder.appName(\"MetadataDeltalakeETL\") \\\n",
    "    .config(\"spark.sql.extensions\", \"io.delta.sql.DeltaSparkSessionExtension\") \\\n",
    "    .config(\"spark.sql.catalog.spark_catalog\", \"org.apache.spark.sql.delta.catalog.DeltaCatalog\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = configure_spark_with_delta_pip(builder).getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cloud Native\n",
    "\n",
    "* At this point, we could simply sync up the whole directory `./lakehouse` into Cloud Object store.\n",
    "* For example for AWS, we could perform these steps:\n",
    "    * Upload our warehouse to S3: `aws s3 sync ./lakehouse s3://org-datalake-prod/v1/lakehouse`\n",
    "    * Setup [AWS Glue to crawl Deltatables](https://aws.amazon.com/blogs/big-data/crawl-delta-lake-tables-using-aws-glue-crawlers/)\n",
    "    * Then query with AWS Athena\n",
    "* Similar cloud services should exist in Azure or GCP.\n",
    "* For other Cloud and private infrastructure, you could leverage PrestoDB, Trino, Hive to front our Genomics data warehouse.\n",
    "* More advanced cases, we could utilise more High throughput dedicated managed data warehouse services such as Redshift, Synapse, etc.\n",
    "\n",
    "> Key point: Our Genomic BigData warehouse is in Cloud-native and can be scale-out by leveraging _state-of-the-art_ Cloud auto-scaling services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load Lakehouse Tables\n",
    "\n",
    "* For now in our local dev, we mimic this Cloud-native effect by loading each Deltatables into separate spark dataframes.\n",
    "* Then, we create in-memory \"Table View\" and, use SparkSQL to mimic SQL like query experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "germline_src = \"./lakehouse/bcbio/germline_table\"\n",
    "somatic_src = \"./lakehouse/bcbio/somatic_table\"\n",
    "metadata_src = \"./lakehouse/bcbio/metadata_table\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "germline_df = spark.read.format(\"delta\").load(germline_src)\n",
    "somatic_df = spark.read.format(\"delta\").load(somatic_src)\n",
    "metadata_df = spark.read.format(\"delta\").load(metadata_src)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "germline_df.createOrReplaceTempView(\"germline_table\")\n",
    "somatic_df.createOrReplaceTempView(\"somatic_table\")\n",
    "metadata_df.createOrReplaceTempView(\"metadata_table\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Describe Table Schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------------------+-------------+-------+\n",
      "|col_name                            |data_type    |comment|\n",
      "+------------------------------------+-------------+-------+\n",
      "|contigName                          |string       |null   |\n",
      "|start                               |bigint       |null   |\n",
      "|end                                 |bigint       |null   |\n",
      "|names                               |array<string>|null   |\n",
      "|referenceAllele                     |string       |null   |\n",
      "|alternateAlleles                    |array<string>|null   |\n",
      "|qual                                |double       |null   |\n",
      "|filters                             |array<string>|null   |\n",
      "|splitFromMultiAllelic               |boolean      |null   |\n",
      "|INFO_platformnames                  |array<string>|null   |\n",
      "|INFO_callsetwithotheruniqgenopassing|array<string>|null   |\n",
      "|INFO_callsetnames                   |array<string>|null   |\n",
      "|INFO_AC                             |array<int>   |null   |\n",
      "|INFO_varType                        |string       |null   |\n",
      "|INFO_DPSum                          |int          |null   |\n",
      "|INFO_datasetsmissingcall            |array<string>|null   |\n",
      "|INFO_AN                             |int          |null   |\n",
      "|INFO_callsets                       |int          |null   |\n",
      "|INFO_callsetwiththisuniqgenopassing |array<string>|null   |\n",
      "|INFO_callable                       |array<string>|null   |\n",
      "|INFO_difficultregion                |array<string>|null   |\n",
      "|INFO_datasets                       |int          |null   |\n",
      "|INFO_platformbias                   |array<string>|null   |\n",
      "|INFO_platforms                      |int          |null   |\n",
      "|INFO_arbitrated                     |string       |null   |\n",
      "|INFO_filt                           |array<string>|null   |\n",
      "|INFO_datasetnames                   |array<string>|null   |\n",
      "|genotypes_sampleId                  |string       |null   |\n",
      "|genotypes_conditionalQuality        |int          |null   |\n",
      "|genotypes_alleleDepths              |array<int>   |null   |\n",
      "|genotypes_phased                    |boolean      |null   |\n",
      "|genotypes_calls                     |array<int>   |null   |\n",
      "|genotypes_depth                     |int          |null   |\n",
      "|genotypes_IGT                       |string       |null   |\n",
      "|genotypes_IPS                       |string       |null   |\n",
      "|genotypes_ADALL                     |array<int>   |null   |\n",
      "|genotypes_PS                        |int          |null   |\n",
      "+------------------------------------+-------------+-------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"describe germline_table\").show(1000, truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------------------+-------------+-------+\n",
      "|col_name                            |data_type    |comment|\n",
      "+------------------------------------+-------------+-------+\n",
      "|contigName                          |string       |null   |\n",
      "|start                               |bigint       |null   |\n",
      "|end                                 |bigint       |null   |\n",
      "|names                               |array<string>|null   |\n",
      "|referenceAllele                     |string       |null   |\n",
      "|alternateAlleles                    |array<string>|null   |\n",
      "|qual                                |double       |null   |\n",
      "|filters                             |array<string>|null   |\n",
      "|splitFromMultiAllelic               |boolean      |null   |\n",
      "|INFO_platformnames                  |array<string>|null   |\n",
      "|INFO_callsetwithotheruniqgenopassing|array<string>|null   |\n",
      "|INFO_callsetnames                   |array<string>|null   |\n",
      "|INFO_AC                             |array<int>   |null   |\n",
      "|INFO_FREQ                           |array<string>|null   |\n",
      "|INFO_varType                        |string       |null   |\n",
      "|INFO_DPSum                          |int          |null   |\n",
      "|INFO_datasetsmissingcall            |array<string>|null   |\n",
      "|INFO_AN                             |int          |null   |\n",
      "|INFO_callsets                       |int          |null   |\n",
      "|INFO_callsetwiththisuniqgenopassing |array<string>|null   |\n",
      "|INFO_callable                       |array<string>|null   |\n",
      "|INFO_difficultregion                |array<string>|null   |\n",
      "|INFO_datasets                       |int          |null   |\n",
      "|INFO_SOMTYPE                        |array<string>|null   |\n",
      "|INFO_platformbias                   |array<string>|null   |\n",
      "|INFO_platforms                      |int          |null   |\n",
      "|INFO_arbitrated                     |string       |null   |\n",
      "|INFO_filt                           |array<string>|null   |\n",
      "|INFO_datasetnames                   |array<string>|null   |\n",
      "|genotypes_sampleId                  |string       |null   |\n",
      "|genotypes_conditionalQuality        |int          |null   |\n",
      "|genotypes_alleleDepths              |array<int>   |null   |\n",
      "|genotypes_phased                    |boolean      |null   |\n",
      "|genotypes_calls                     |array<int>   |null   |\n",
      "|genotypes_depth                     |int          |null   |\n",
      "|genotypes_IGT                       |string       |null   |\n",
      "|genotypes_IPS                       |string       |null   |\n",
      "|genotypes_ADALL                     |array<int>   |null   |\n",
      "|genotypes_PS                        |int          |null   |\n",
      "+------------------------------------+-------------+-------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"describe somatic_table\").show(1000, truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------+---------+-------+\n",
      "|col_name         |data_type|comment|\n",
      "+-----------------+---------+-------+\n",
      "|SequenceRunName  |string   |null   |\n",
      "|SequenceRunID    |string   |null   |\n",
      "|Timestamp        |string   |null   |\n",
      "|SubjectID        |string   |null   |\n",
      "|LibraryID        |string   |null   |\n",
      "|SampleID         |string   |null   |\n",
      "|SampleDescription|string   |null   |\n",
      "|Gender           |string   |null   |\n",
      "|Race             |string   |null   |\n",
      "|Ethnicity        |string   |null   |\n",
      "|Proband          |string   |null   |\n",
      "|ProjectOwner     |string   |null   |\n",
      "|ProjectName      |string   |null   |\n",
      "|StudyID          |string   |null   |\n",
      "|StudyType        |string   |null   |\n",
      "|Pipeline         |string   |null   |\n",
      "|Phenotype        |string   |null   |\n",
      "|Collection       |string   |null   |\n",
      "|DiseaseCode      |string   |null   |\n",
      "|SNOMED           |string   |null   |\n",
      "+-----------------+---------+-------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"describe metadata_table\").show(1000, truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Querying Tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+---------+------+---------+-------+-------------+---------+--------+-----+------------------+-----+-------+\n",
      "|     SequenceRunName|SubjectID|Gender|Phenotype|StudyID|  DiseaseCode|   SNOMED|SampleID|CHROM|               REF|  ALT|ALT_cnt|\n",
      "+--------------------+---------+------+---------+-------+-------------+---------+--------+-----+------------------+-----+-------+\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                AT|  [A]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 C|  [T]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 A|  [G]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 G|  [A]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 G|  [A]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 T|  [C]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 T|  [G]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 G|  [A]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 C|  [G]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 C|  [T]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 T|  [G]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 T|  [C]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 T|  [A]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 G|  [A]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 G|  [C]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 C|  [T]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 A|  [G]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 G|  [A]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|                 G|[GAA]|      1|\n",
      "|221007_A00130_000...| SBJ00001|Female|   normal|NA12878|MONDO:0007254|429740004| NA12878| chr9|AATGTGGGGCATACACAT|  [A]|      1|\n",
      "+--------------------+---------+------+---------+-------+-------------+---------+--------+-----+------------------+-----+-------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.sql(\"select \\\n",
    "    m.SequenceRunName, m.SubjectID, m.Gender, m.Phenotype, m.StudyID, m.DiseaseCode, m.SNOMED, m.SampleID, \\\n",
    "    s.contigName as CHROM, s.referenceAllele as REF, s.alternateAlleles as ALT, array_size(s.alternateAlleles) as ALT_cnt \\\n",
    "from metadata_table as m \\\n",
    "join somatic_table as s on s.genotypes_sampleId = m.SampleID\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Stage 29:==============>                                           (1 + 3) / 4]\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+---------+-------+-------------+---------+--------+-----------+---+---+-----------+\n",
      "|SubjectID|Phenotype|StudyID|  DiseaseCode|   SNOMED|SampleID|GT_SampleID|REF|ALT|num_of_snps|\n",
      "+---------+---------+-------+-------------+---------+--------+-----------+---+---+-----------+\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  G|[A]|     179649|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  C|[T]|     179274|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  T|[C]|     145030|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  A|[G]|     144496|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  G|[T]|      42193|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  C|[A]|      41828|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  C|[G]|      40728|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  G|[C]|      40494|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  A|[C]|      36837|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  T|[G]|      36177|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  T|[A]|      33016|\n",
      "| SBJ00001|   normal|NA12878|MONDO:0007254|429740004| NA12878|    NA12878|  A|[T]|      32659|\n",
      "+---------+---------+-------+-------------+---------+--------+-----------+---+---+-----------+\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "spark.sql(\"select \\\n",
    "    m.SubjectID, m.Phenotype, m.StudyID, m.DiseaseCode, m.SNOMED, m.SampleID, \\\n",
    "    s.genotypes_sampleId as GT_SampleID, s.referenceAllele as REF, s.alternateAlleles as ALT, count(*) as num_of_snps \\\n",
    "from metadata_table as m \\\n",
    "join somatic_table as s on s.genotypes_sampleId = m.SampleID \\\n",
    "where \\\n",
    "    char_length(referenceAllele) = 1 and \\\n",
    "    array_size(alternateAlleles) = 1 and \\\n",
    "    char_length(alternateAlleles[0]) = 1 \\\n",
    "    group by m.SubjectID, m.Phenotype, m.StudyID, m.DiseaseCode, m.SNOMED, m.SampleID, s.genotypes_sampleId, s.referenceAllele, s.alternateAlleles \\\n",
    "    order by num_of_snps desc\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary\n",
    "\n",
    "* In real world scenario, this join table call may or may not make sense. However. It demonstrates the use case possibility.\n",
    "* Often, these warehouse tables may span across multiple sources; AWS Athena, Trino as such distributed query engine enable \"Federated Query\" interface.\n",
    "    * For example, as such Genomic data warehouse could be backend of [GA4GH Beacon](https://github.com/ga4gh-beacon/beacon-v2/) and/or [GA4GH Data Connect](https://github.com/ga4gh-discovery/data-connect) REST endpoint interfaces.\n",
    "* It opens up data into more general tooling for information retrieval e.g. SQL.\n",
    "* As such warehouse table enable OLAP -- Online Analytical Processing and, further bridge into data science such ML/AI.\n",
    "* Depends on data arrangement, we may treat such Genomic Table as central Fact table -- which we could use surrounding Metadata table(s) as dimensional look up.\n",
    "* Or, we may be just focusing on Genomic Tables for some number crunching or aggregation for reporting.\n",
    "* Bring data one step closer to Cloud for computation -- i.e. data & compute closer together. Hence. \"Cloud-native\".\n",
    "\n",
    "To note that;\n",
    "\n",
    "* Certainly, experienced/trained BioInformatician can leverage more efficient tools in a specified ad-hoc setup.\n",
    "* Genomic Table and, as such sourcing data warehouse directly from VCF could easily get out of hand.\n",
    "* Typically, \"best practice\" BioInformatics Pipeline should have finer, distilled, end-of-chain, \"gold\" label VCF product.\n",
    "* This VCF should contain a handful of records per patient that have already analysed & annotated well for given use case.\n",
    "* If sourcing Genomic Table from VCF is not appropriate then one could leverage more summarised tables output from MultiQC or some Cancer Reporter as see fit.\n",
    "\n",
    "Hence, this kind of Genomic BigData warehouse still rely on high quality output of BioInformatics Pipeline. And continuation of overall Data Pipeline as depict below.\n",
    "\n",
    "![GenomicBigData.png](./assets/GenomicBigData.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Future Work\n",
    "\n",
    "* This has yet to evaluate further on real world data workload (case by case) and, setting up some decent size cohort-wide data warehousing.\n",
    "* Explore data partitioning within Deltatable or chosen LakeHouse table framework.\n",
    "* Performance and feasibility study benchmarking on various technologies that underpin the setup.\n",
    "* More specialised Cloud-native BioInfo formats: `BioParquet` _ala_ [GeoParquet](https://github.com/opengeospatial/geoparquet)?\n",
    "* Comparison with other LakeHouse table framework: Iceberg, Hudi\n",
    "    * Similarly, entails more specific to Genomic such that `BioTable` or `GenomicTable`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Page Break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stop Spark Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Continue to next notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}