example-usage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accessing DSS via DOS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data in the HCA DSS is replicated across cloud stores. This allows it to be downloaded from the \"nearest\" location, to avoid egress fees. \n",
    "\n",
    "The data in the DSS has been made available using the Data Object Service schemas, which provide an interoperable way for exposing replicated data and versioned data over a simple HTTP API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the requests module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access services over HTTP we use the requests module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://ekivlnizh1.execute-api.us-west-2.amazonaws.com/api\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import os\n",
    "SERVICE_URL = \"https://ekivlnizh1.execute-api.us-west-2.amazonaws.com/api\"\n",
    "print(SERVICE_URL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ListDataObjects` method has not been implemented yet. However, one can access the DSS' bundle oriented index using `ListDataBundles`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing Data Bundles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'version': '2018-06-07T001704', 'id': 'e9556c3d-53cb-4c24-856e-951085735d45'}, {'version': '2018-06-07T001659', 'id': '15a0ce60-261d-4bc1-8463-f4d87aa483f0'}, {'version': '2018-06-07T001844', 'id': 'b7003567-37a6-4f70-8be3-0e8ee5c1f020'}, {'version': '2018-06-07T001714', 'id': 'd1dca21c-71a3-466d-8690-1212c22491c3'}, {'version': '2018-06-07T001749', 'id': '41680495-06a3-4963-9d2c-9280c6e9979b'}, {'version': '2018-06-07T001808', 'id': 'c3a74a9d-aebb-4fcb-a664-c49f8abbaa8c'}, {'version': '2018-06-07T001848', 'id': '4eb3190b-de14-4248-b143-65084c930741'}, {'version': '2018-06-07T001908', 'id': 'c70c2f7d-5e68-468e-bc66-36e1c4caac80'}, {'version': '2018-06-07T001812', 'id': '29d2be81-e0e0-484b-81ac-efa385f1e9bc'}, {'version': '2018-06-07T001822', 'id': '65b0e718-6b30-4833-8039-526bd9ced487'}]\n"
     ]
    }
   ],
   "source": [
    "BASE_URL = \"ga4gh/dos/v1\"\n",
    "LIST_DATA_BUNDLES_URL = \"{}/{}/{}\".format(SERVICE_URL, BASE_URL, \"databundles\")\n",
    "#r = requests.get(os.path.join(SERVICE_URL, BASE_URL))\n",
    "#print(r.status_code)\n",
    "data_bundles = requests.get(LIST_DATA_BUNDLES_URL).json()['data_bundles']\n",
    "print(data_bundles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Data Bundle details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have some Data Bundle identifiers, we can use `GetDataBundle` to retrieve more information about a bundle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['8ff23235-4435-4929-8fb2-5d55b4564999', '25065a60-6b7d-4c50-abf9-0cf86c5b483a', '44ec7963-5c7a-4974-bb4d-c8b5c002019c']\n"
     ]
    }
   ],
   "source": [
    "DATA_BUNDLE_URL = \"{}/{}/databundles/{}\".format(SERVICE_URL, BASE_URL, data_bundles[0]['id'])\n",
    "data_bundle = requests.get(DATA_BUNDLE_URL).json()['data_bundle']\n",
    "print(data_bundle['data_object_ids'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Data Object Details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now access Data Objects for download using the data object identifiers from the Data Bundle. Both signed URLs and cloud native URLs are available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_object_id = data_bundle['data_object_ids'][0]\n",
    "DATA_OBJECT_URL = \"{}/{}/dataobjects/{}\".format(SERVICE_URL, BASE_URL, data_object_id)\n",
    "data_object = requests.get(DATA_OBJECT_URL).json()['data_object']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Data Object contains a list of URLs and checksums that can be used to download and access the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------URLS------------\n",
      "{'url': 'https://commons-dss.ucsc-cgp-dev.org/v1/files/8ff23235-4435-4929-8fb2-5d55b4564999?replica=aws'}\n",
      "{'url': 'https://commons-dss.ucsc-cgp-dev.org/v1/files/8ff23235-4435-4929-8fb2-5d55b4564999?replica=azure'}\n",
      "{'url': 'https://commons-dss.ucsc-cgp-dev.org/v1/files/8ff23235-4435-4929-8fb2-5d55b4564999?replica=gcp'}\n",
      "-----------checksums-----------\n",
      "{'checksum': 'c873835a74cea9c811cc7799f8897ac480cccf84f631c99b5293900f7a071b53', 'type': 'sha256'}\n",
      "{'checksum': '57db2e71deb4dab5e4b3f251ac9243b0', 'type': 'etag'}\n",
      "{'checksum': '05f818a54510272c17dcda69c948f8d904b5aae3', 'type': 'sha1'}\n",
      "{'checksum': '63439d51', 'type': 'crc32c'}\n"
     ]
    }
   ],
   "source": [
    "print(\"-----------URLS------------\")\n",
    "for url in data_object['urls']:\n",
    "    print(url)\n",
    "print(\"-----------checksums-----------\")\n",
    "for checksum in data_object['checksums']:\n",
    "    print(checksum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, using a HTTP, S3, or GCP downloader, one can access these files."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": ".venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}