Health-Disparities.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NHANES EDA on Health Disparities\n",
    "\n",
    "<p style='text-align: center;'><i><b>Unraveling Health Disparities: A Data-Driven Exploration through Social Determinants of Health (SDOH) using NHANES (Demographic & Questionnaire) data</b></i></p>\n",
    "<center><img src='https://github.com/kkrusere/NHANES-EDA-on-Health-Disparities-and-Inequities/blob/main/assets/nhanes_health_disparities.png?raw=true' width=600/></center>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "1. [Introduction](#introduction)\n",
    "   - [Overview of Health Disparities](#health_disparities)\n",
    "   - [Social Determinants of Health (SDoH)](#social_determinants)\n",
    "   - [NHANES Dataset](#nhanes_dataset)\n",
    "   - [Project Goals and Objectives](#project_goals_objectives)\n",
    "\n",
    "2. [Background and Context](#background_context)\n",
    "   - [Understanding Health Disparities](#understanding_disparities)\n",
    "   - [Role of Social Determinants](#role_of_social_determinants)\n",
    "   - [Significance of NHANES Dataset](#nhanes_significance)\n",
    "\n",
    "3. [Data Collection and Preparation](#data_collection_preparation)\n",
    "   - [Introduction to NHANES-pyTOOL-API](#nhanes_pytool_api)\n",
    "   - [NHANES Demographic Data](#nhanes_demographic_data)\n",
    "   - [NHANES Questionnaire Data](#nhanes_questionnaire_data)\n",
    "   - [Joining Demographic and Questionnaire Data](#joining_data)\n",
    "\n",
    "4. [Exploratory Data Analysis (EDA)](#exploratory_data_analysis)\n",
    "   - [Descriptive Statistics](#descriptive_statistics)\n",
    "   - [Demographic Analysis](#demographic_analysis)\n",
    "   - [Analysis of Health Behaviors](#health_behavior_analysis)\n",
    "   - [Access to Healthcare](#access_to_healthcare)\n",
    "   - [Social Determinants of Health Analysis](#social_determinants_analysis)\n",
    "\n",
    "5. [Statistical Analysis](#statistical_analysis)\n",
    "   - [Inferential Statistics](#inferential_statistics)\n",
    "   - [Multivariate Analysis](#multivariate_analysis)\n",
    "   - [Comparative Analysis](#comparative_analysis)\n",
    "   - [Correlation Analysis](#correlation_analysis)\n",
    "   - [Hypothesis Testing](#hypothesis_testing)\n",
    "\n",
    "6. [Interpretation and Insights](#interpretation_insights)\n",
    "   - [Key Findings from EDA](#key_findings_eda)\n",
    "   - [Implications for Addressing Disparities](#addressing_disparities)\n",
    "\n",
    "7. [Conclusion](#conclusion)\n",
    "   - [Preject Summary](#project_summary)\n",
    "   - [Future Directions](#future_directions)\n",
    "\n",
    "8. [Appendix](#appendix)\n",
    "   - [NHANES-pyTOOL-API Documentation](#api_documentation)\n",
    "   - [Technical Details](#technical_details)\n",
    "\n",
    "9. [References](#references)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Introduction](#introduction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Overview of Health Disparities](#overview-of-health-disparities)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Social Determinants of Health (SDoH)](#social-determinants-of-health-sdoh)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [NHANES Dataset](#nhanes-dataset)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Project Goals and Objectives](#project-goals-and-objectives)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Background and Context](#background-and-context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Understanding Health Disparities](#understanding-health-disparities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Role of Social Determinants](#role-of-social-determinants)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Significance of NHANES Dataset](#significance-of-nhanes-dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Data Collection and Preparation](#data-collection-and-preparation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Introduction to NHANES-pyTOOL-API](#introduction-to-nhanes-pytool-api)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing all the libraries that we are going to be neeing for data collection, processing and cleaning\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "\n",
    "\n",
    "from nhanes_data.nhanes_data_api import NHANESDataAPI\n",
    "\n",
    "# import warnings\n",
    "# warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we are going to create an instance of the NHANES api\n",
    "\n",
    "nhanes_data_api = NHANESDataAPI()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1999-2000',\n",
       " '2001-2002',\n",
       " '2003-2004',\n",
       " '2005-2006',\n",
       " '2007-2008',\n",
       " '2009-2010',\n",
       " '2011-2012',\n",
       " '2013-2014',\n",
       " '2015-2016',\n",
       " '2017-2018']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we are going to list the survey cycles that we have data available for to use this project\n",
    "cycle_list = nhanes_data_api.list_cycle_years()\n",
    "cycle_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['demographics',\n",
       " 'dietary',\n",
       " 'examination',\n",
       " 'laboratory',\n",
       " 'questionnaire',\n",
       " 'limitedaccess']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we are going to list the different data categories that we have access to via the NHANES API\n",
    "data_category_list = nhanes_data_api.list_data_categories()\n",
    "data_category_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that for the project and as explained in the introductory part, we are going to be using the `demographic` and `questionnaire` data. the rest of the data categories we will use in later projects. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [NHANES Demographic Data](#nhanes-demographic-data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Demographic Variables & Sample Weights']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we are going to list the datafile categories within the demographic category, as we know some of these data categories have multiple data files in them\n",
    "demo_files = nhanes_data_api.list_file_names(data_category='demographics', cycle_years='1999-2018')\n",
    "demo_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The demographic data category only have one data file which makes it easier for us in the data retrieval process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SEQN</th>\n",
       "      <th>SDDSRVYR</th>\n",
       "      <th>RIDSTATR</th>\n",
       "      <th>RIDEXMON</th>\n",
       "      <th>RIAGENDR</th>\n",
       "      <th>RIDAGEYR</th>\n",
       "      <th>RIDAGEMN</th>\n",
       "      <th>RIDAGEEX</th>\n",
       "      <th>RIDRETH1</th>\n",
       "      <th>RIDRETH2</th>\n",
       "      <th>...</th>\n",
       "      <th>DMDBORN4</th>\n",
       "      <th>AIALANGA</th>\n",
       "      <th>DMDHHSZA</th>\n",
       "      <th>DMDHHSZB</th>\n",
       "      <th>DMDHHSZE</th>\n",
       "      <th>DMDHRBR4</th>\n",
       "      <th>DMDHRAGZ</th>\n",
       "      <th>DMDHREDZ</th>\n",
       "      <th>DMDHRMAZ</th>\n",
       "      <th>DMDHSEDZ</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>926.0</td>\n",
       "      <td>926.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>125.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>597.0</td>\n",
       "      <td>597.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 175 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   SEQN  SDDSRVYR  RIDSTATR  RIDEXMON  RIAGENDR  RIDAGEYR  RIDAGEMN  RIDAGEEX  \\\n",
       "0   1.0       1.0       2.0       2.0       2.0       2.0      29.0      31.0   \n",
       "1   2.0       1.0       2.0       2.0       1.0      77.0     926.0     926.0   \n",
       "2   3.0       1.0       2.0       1.0       2.0      10.0     125.0     126.0   \n",
       "3   4.0       1.0       2.0       2.0       1.0       1.0      22.0      23.0   \n",
       "4   5.0       1.0       2.0       2.0       1.0      49.0     597.0     597.0   \n",
       "\n",
       "   RIDRETH1  RIDRETH2  ...  DMDBORN4  AIALANGA  DMDHHSZA  DMDHHSZB  DMDHHSZE  \\\n",
       "0       4.0       2.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "1       3.0       1.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "2       3.0       1.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "3       4.0       2.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "4       3.0       1.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "\n",
       "   DMDHRBR4  DMDHRAGZ  DMDHREDZ  DMDHRMAZ  DMDHSEDZ  \n",
       "0       NaN       NaN       NaN       NaN       NaN  \n",
       "1       NaN       NaN       NaN       NaN       NaN  \n",
       "2       NaN       NaN       NaN       NaN       NaN  \n",
       "3       NaN       NaN       NaN       NaN       NaN  \n",
       "4       NaN       NaN       NaN       NaN       NaN  \n",
       "\n",
       "[5 rows x 175 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we are now ging to retrierve all the demographic data from 1999 to 2018\n",
    "demo_dataframe = nhanes_data_api.retrieve_data(data_category='demographics', cycle='1999-2018', filename='Demographic Variables & Sample Weights')\n",
    "demo_dataframe.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#first we change the datatype of the `SEQN` column from float to just an object or a string\n",
    "demo_dataframe['SEQN'] = demo_dataframe['SEQN'].astype(int)\n",
    "demo_dataframe['SEQN'] = demo_dataframe['SEQN'].astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SEQN</th>\n",
       "      <th>SDDSRVYR</th>\n",
       "      <th>RIDSTATR</th>\n",
       "      <th>RIDEXMON</th>\n",
       "      <th>RIAGENDR</th>\n",
       "      <th>RIDAGEYR</th>\n",
       "      <th>RIDAGEMN</th>\n",
       "      <th>RIDAGEEX</th>\n",
       "      <th>RIDRETH1</th>\n",
       "      <th>RIDRETH2</th>\n",
       "      <th>...</th>\n",
       "      <th>DMDBORN4</th>\n",
       "      <th>AIALANGA</th>\n",
       "      <th>DMDHHSZA</th>\n",
       "      <th>DMDHHSZB</th>\n",
       "      <th>DMDHHSZE</th>\n",
       "      <th>DMDHRBR4</th>\n",
       "      <th>DMDHRAGZ</th>\n",
       "      <th>DMDHREDZ</th>\n",
       "      <th>DMDHRMAZ</th>\n",
       "      <th>DMDHSEDZ</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>77.0</td>\n",
       "      <td>926.0</td>\n",
       "      <td>926.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>125.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>597.0</td>\n",
       "      <td>597.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 175 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  SEQN  SDDSRVYR  RIDSTATR  RIDEXMON  RIAGENDR  RIDAGEYR  RIDAGEMN  RIDAGEEX  \\\n",
       "0    1       1.0       2.0       2.0       2.0       2.0      29.0      31.0   \n",
       "1    2       1.0       2.0       2.0       1.0      77.0     926.0     926.0   \n",
       "2    3       1.0       2.0       1.0       2.0      10.0     125.0     126.0   \n",
       "3    4       1.0       2.0       2.0       1.0       1.0      22.0      23.0   \n",
       "4    5       1.0       2.0       2.0       1.0      49.0     597.0     597.0   \n",
       "\n",
       "   RIDRETH1  RIDRETH2  ...  DMDBORN4  AIALANGA  DMDHHSZA  DMDHHSZB  DMDHHSZE  \\\n",
       "0       4.0       2.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "1       3.0       1.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "2       3.0       1.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "3       4.0       2.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "4       3.0       1.0  ...       NaN       NaN       NaN       NaN       NaN   \n",
       "\n",
       "   DMDHRBR4  DMDHRAGZ  DMDHREDZ  DMDHRMAZ  DMDHSEDZ  \n",
       "0       NaN       NaN       NaN       NaN       NaN  \n",
       "1       NaN       NaN       NaN       NaN       NaN  \n",
       "2       NaN       NaN       NaN       NaN       NaN  \n",
       "3       NaN       NaN       NaN       NaN       NaN  \n",
       "4       NaN       NaN       NaN       NaN       NaN  \n",
       "\n",
       "[5 rows x 175 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "demo_dataframe.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['SEQN',\n",
       " 'SDDSRVYR',\n",
       " 'RIDSTATR',\n",
       " 'RIDEXMON',\n",
       " 'RIAGENDR',\n",
       " 'RIDAGEYR',\n",
       " 'RIDAGEMN',\n",
       " 'RIDAGEEX',\n",
       " 'RIDRETH1',\n",
       " 'RIDRETH2',\n",
       " 'DMQMILIT',\n",
       " 'DMDBORN',\n",
       " 'DMDCITZN',\n",
       " 'DMDYRSUS',\n",
       " 'DMDEDUC3',\n",
       " 'DMDEDUC2',\n",
       " 'DMDEDUC',\n",
       " 'DMDSCHOL',\n",
       " 'DMDMARTL',\n",
       " 'DMDHHSIZ',\n",
       " 'INDHHINC',\n",
       " 'INDFMINC',\n",
       " 'INDFMPIR',\n",
       " 'RIDEXPRG',\n",
       " 'RIDPREG',\n",
       " 'DMDHRGND',\n",
       " 'DMDHRAGE',\n",
       " 'DMDHRBRN',\n",
       " 'DMDHREDU',\n",
       " 'DMDHRMAR',\n",
       " 'DMDHSEDU',\n",
       " 'WTINT2YR',\n",
       " 'WTINT4YR',\n",
       " 'WTMEC2YR',\n",
       " 'WTMEC4YR',\n",
       " 'SDMVPSU',\n",
       " 'SDMVSTRA',\n",
       " 'SDJ1REPN',\n",
       " 'DMAETHN',\n",
       " 'DMARACE',\n",
       " 'WTMREP01',\n",
       " 'WTMREP02',\n",
       " 'WTMREP03',\n",
       " 'WTMREP04',\n",
       " 'WTMREP05',\n",
       " 'WTMREP06',\n",
       " 'WTMREP07',\n",
       " 'WTMREP08',\n",
       " 'WTMREP09',\n",
       " 'WTMREP10',\n",
       " 'WTMREP11',\n",
       " 'WTMREP12',\n",
       " 'WTMREP13',\n",
       " 'WTMREP14',\n",
       " 'WTMREP15',\n",
       " 'WTMREP16',\n",
       " 'WTMREP17',\n",
       " 'WTMREP18',\n",
       " 'WTMREP19',\n",
       " 'WTMREP20',\n",
       " 'WTMREP21',\n",
       " 'WTMREP22',\n",
       " 'WTMREP23',\n",
       " 'WTMREP24',\n",
       " 'WTMREP25',\n",
       " 'WTMREP26',\n",
       " 'WTMREP27',\n",
       " 'WTMREP28',\n",
       " 'WTMREP29',\n",
       " 'WTMREP30',\n",
       " 'WTMREP31',\n",
       " 'WTMREP32',\n",
       " 'WTMREP33',\n",
       " 'WTMREP34',\n",
       " 'WTMREP35',\n",
       " 'WTMREP36',\n",
       " 'WTMREP37',\n",
       " 'WTMREP38',\n",
       " 'WTMREP39',\n",
       " 'WTMREP40',\n",
       " 'WTMREP41',\n",
       " 'WTMREP42',\n",
       " 'WTMREP43',\n",
       " 'WTMREP44',\n",
       " 'WTMREP45',\n",
       " 'WTMREP46',\n",
       " 'WTMREP47',\n",
       " 'WTMREP48',\n",
       " 'WTMREP49',\n",
       " 'WTMREP50',\n",
       " 'WTMREP51',\n",
       " 'WTMREP52',\n",
       " 'WTIREP01',\n",
       " 'WTIREP02',\n",
       " 'WTIREP03',\n",
       " 'WTIREP04',\n",
       " 'WTIREP05',\n",
       " 'WTIREP06',\n",
       " 'WTIREP07',\n",
       " 'WTIREP08',\n",
       " 'WTIREP09',\n",
       " 'WTIREP10',\n",
       " 'WTIREP11',\n",
       " 'WTIREP12',\n",
       " 'WTIREP13',\n",
       " 'WTIREP14',\n",
       " 'WTIREP15',\n",
       " 'WTIREP16',\n",
       " 'WTIREP17',\n",
       " 'WTIREP18',\n",
       " 'WTIREP19',\n",
       " 'WTIREP20',\n",
       " 'WTIREP21',\n",
       " 'WTIREP22',\n",
       " 'WTIREP23',\n",
       " 'WTIREP24',\n",
       " 'WTIREP25',\n",
       " 'WTIREP26',\n",
       " 'WTIREP27',\n",
       " 'WTIREP28',\n",
       " 'WTIREP29',\n",
       " 'WTIREP30',\n",
       " 'WTIREP31',\n",
       " 'WTIREP32',\n",
       " 'WTIREP33',\n",
       " 'WTIREP34',\n",
       " 'WTIREP35',\n",
       " 'WTIREP36',\n",
       " 'WTIREP37',\n",
       " 'WTIREP38',\n",
       " 'WTIREP39',\n",
       " 'WTIREP40',\n",
       " 'WTIREP41',\n",
       " 'WTIREP42',\n",
       " 'WTIREP43',\n",
       " 'WTIREP44',\n",
       " 'WTIREP45',\n",
       " 'WTIREP46',\n",
       " 'WTIREP47',\n",
       " 'WTIREP48',\n",
       " 'WTIREP49',\n",
       " 'WTIREP50',\n",
       " 'WTIREP51',\n",
       " 'WTIREP52',\n",
       " 'year',\n",
       " 'SIALANG',\n",
       " 'SIAPROXY',\n",
       " 'SIAINTRP',\n",
       " 'FIALANG',\n",
       " 'FIAPROXY',\n",
       " 'FIAINTRP',\n",
       " 'MIALANG',\n",
       " 'MIAPROXY',\n",
       " 'MIAINTRP',\n",
       " 'AIALANG',\n",
       " 'DMDFMSIZ',\n",
       " 'DMDBORN2',\n",
       " 'INDHHIN2',\n",
       " 'INDFMIN2',\n",
       " 'DMDHRBR2',\n",
       " 'RIDRETH3',\n",
       " 'RIDEXAGY',\n",
       " 'RIDEXAGM',\n",
       " 'DMQMILIZ',\n",
       " 'DMQADFC',\n",
       " 'DMDBORN4',\n",
       " 'AIALANGA',\n",
       " 'DMDHHSZA',\n",
       " 'DMDHHSZB',\n",
       " 'DMDHHSZE',\n",
       " 'DMDHRBR4',\n",
       " 'DMDHRAGZ',\n",
       " 'DMDHREDZ',\n",
       " 'DMDHRMAZ',\n",
       " 'DMDHSEDZ']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we need to take a look at what all theses column mean\n",
    "demo_list_of_variable = list(demo_dataframe.columns)\n",
    "demo_list_of_variable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we are going to create a function thats going to allow us to retrieve the veriable documentation\n",
    "def get_variable_df(url, cycle_list=cycle_list):\n",
    "    \"\"\"\n",
    "    This function inputs the NHANES URL for Variable list.\n",
    "    The Pandas' pandas.read_html() function is used to read tables on the URL.\n",
    "    Resulting data frame is then cleaned (adding a year column and removing some other columns not needed).\n",
    "    Year column matches the survey cycle periods.\n",
    "    Cycle list is used to filter the data to just the cycle of interest.\n",
    "    Returns the data frame of the variable.\n",
    "    \"\"\"\n",
    "    dfs = pd.read_html(url)\n",
    "    df = dfs[0]  # the table of interest is at index 0\n",
    "\n",
    "    Years = [i for i in range(len(df))]\n",
    "    df[\"Years\"] = Years\n",
    "    for i in range(len(df)):\n",
    "        x = df['Begin Year'][i]\n",
    "        y = df['EndYear'][i]\n",
    "        df.loc[i, \"Years\"] = f\"{x}-{y}\"\n",
    "\n",
    "    df.drop([\"Begin Year\", \"EndYear\", \"Component\", \"Use Constraints\"], axis=1, inplace=True)\n",
    "    df = df.loc[df[\"Years\"].isin(cycle_list)]\n",
    "    df.reset_index(drop=True, inplace=True)\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we are going to retrieve the variable table \n",
    "#Demography variable URL\n",
    "demographics_url = \"https://wwwn.cdc.gov/nchs/nhanes/search/variablelist.aspx?Component=demographics\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/19/cc1smq_x3djf4nkk4r1rhl980000gn/T/ipykernel_9465/2256908215.py:19: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '2005-2006' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.\n",
      "  df.loc[i, \"Years\"] = f\"{x}-{y}\"\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Variable Name</th>\n",
       "      <th>Variable Description</th>\n",
       "      <th>Data File Name</th>\n",
       "      <th>Data File Description</th>\n",
       "      <th>Years</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AIALANG</td>\n",
       "      <td>Language of the MEC ACASI Interview Instrument</td>\n",
       "      <td>DEMO_D</td>\n",
       "      <td>Demographic Variables &amp; Sample Weights</td>\n",
       "      <td>2005-2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>DMDBORN</td>\n",
       "      <td>In what country {were you/was SP} born?</td>\n",
       "      <td>DEMO_D</td>\n",
       "      <td>Demographic Variables &amp; Sample Weights</td>\n",
       "      <td>2005-2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>DMDCITZN</td>\n",
       "      <td>{Are you/Is SP} a citizen of the United States...</td>\n",
       "      <td>DEMO_D</td>\n",
       "      <td>Demographic Variables &amp; Sample Weights</td>\n",
       "      <td>2005-2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DMDEDUC2</td>\n",
       "      <td>(SP Interview Version) What is the highest gra...</td>\n",
       "      <td>DEMO_D</td>\n",
       "      <td>Demographic Variables &amp; Sample Weights</td>\n",
       "      <td>2005-2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>DMDEDUC3</td>\n",
       "      <td>(SP Interview Version) What is the highest gra...</td>\n",
       "      <td>DEMO_D</td>\n",
       "      <td>Demographic Variables &amp; Sample Weights</td>\n",
       "      <td>2005-2006</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Variable Name                               Variable Description  \\\n",
       "0       AIALANG     Language of the MEC ACASI Interview Instrument   \n",
       "1       DMDBORN            In what country {were you/was SP} born?   \n",
       "2      DMDCITZN  {Are you/Is SP} a citizen of the United States...   \n",
       "3      DMDEDUC2  (SP Interview Version) What is the highest gra...   \n",
       "4      DMDEDUC3  (SP Interview Version) What is the highest gra...   \n",
       "\n",
       "  Data File Name                   Data File Description      Years  \n",
       "0         DEMO_D  Demographic Variables & Sample Weights  2005-2006  \n",
       "1         DEMO_D  Demographic Variables & Sample Weights  2005-2006  \n",
       "2         DEMO_D  Demographic Variables & Sample Weights  2005-2006  \n",
       "3         DEMO_D  Demographic Variables & Sample Weights  2005-2006  \n",
       "4         DEMO_D  Demographic Variables & Sample Weights  2005-2006  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we call the get_variable_df fuction for the demography URL\n",
    "demographics_var_df = get_variable_df(demographics_url)\n",
    "demographics_var_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AIALANG': 'Language of the MEC ACASI Interview Instrument',\n",
       " 'DMDBORN': 'In what country {were you/was SP} born?',\n",
       " 'DMDCITZN': '{Are you/Is SP} a citizen of the United States? [Information about citizenship is being collected by the U.S. Public Health Service to perform health related research. Providing this information is voluntary and is collected under the authority of the Public Health Service Act. There will be no effect on pending immigration or citizenship petitions.]',\n",
       " 'DMDEDUC2': 'What is the highest grade or level of school {you have/SP has} completed or the highest degree {you have/s/he has} received?',\n",
       " 'DMDEDUC3': 'What is the highest grade or level of school {you have/SP has} completed or the highest degree {you have/s/he has} received?',\n",
       " 'DMDFMSIZ': 'Total number of people in the Family',\n",
       " 'DMDHHSIZ': 'Total number of people in the Household',\n",
       " 'DMDHRAGE': \"HH reference person's age in years\",\n",
       " 'DMDHRBRN': 'In what country {were you/was NON-SP Head} born?',\n",
       " 'DMDHREDU': \"HH reference person's education level\",\n",
       " 'DMDHRGND': \"HH reference person's gender\",\n",
       " 'DMDHRMAR': \"HH reference person's marital status\",\n",
       " 'DMDHSEDU': \"HH reference person's spouse's education level\",\n",
       " 'DMDMARTL': 'Marital status',\n",
       " 'DMDSCHOL': '{Are you/Is SP} now . . .',\n",
       " 'DMDYRSUS': 'Length of time the participant has been in the US.',\n",
       " 'DMQMILIT': 'Did {you/SP} ever serve in the Armed Forces of the United States?',\n",
       " 'FIAINTRP': 'Was an interpreter used to conduct the Family interview?',\n",
       " 'FIALANG': 'Language of the Family Interview Instrument',\n",
       " 'FIAPROXY': 'Was a Proxy respondent used in conducting the Family Interview?',\n",
       " 'INDFMINC': 'Total family income (reported as a range value in dollars)',\n",
       " 'INDFMPIR': 'A ratio of family income to poverty guidelines.',\n",
       " 'INDHHINC': 'Total household income (reported as a range value in dollars)',\n",
       " 'MIAINTRP': 'Was an interpreter used to conduct the MEC CAPI interview?',\n",
       " 'MIALANG': 'Language of the MEC CAPI Interview Instrument',\n",
       " 'MIAPROXY': 'Was a Proxy respondent used in conducting the MEC CAPI Interview?',\n",
       " 'RIAGENDR': 'Gender of the participant.',\n",
       " 'RIDAGEEX': 'Best age in months at date of examination for individuals under 80 years of age at the time of MEC exam.',\n",
       " 'RIDAGEMN': 'Age in months of the participant at the time of screening. Reported for persons aged 24 months or younger at the time of exam (or screening if not examined).',\n",
       " 'RIDAGEYR': 'Age in years of the participant at the time of screening. Individuals 80 and over are topcoded at 80 years of age.',\n",
       " 'RIDEXMON': 'Six month time period when the examination was performed - two categories: November 1 through April 30, May 1 through October 31.',\n",
       " 'RIDEXPRG': 'Pregnancy status for females between 20 and 44 years of age at the time of MEC exam.',\n",
       " 'RIDRETH1': 'Recode of reported race and Hispanic origin information',\n",
       " 'RIDSTATR': 'Interview and examination status of the participant.',\n",
       " 'SDDSRVYR': 'Data release cycle',\n",
       " 'SDMVPSU': 'Masked variance unit pseudo-PSU variable for variance estimation',\n",
       " 'SDMVSTRA': 'Masked variance unit pseudo-stratum variable for variance estimation',\n",
       " 'SEQN': 'Respondent sequence number.',\n",
       " 'SIAINTRP': 'Was an interpreter used to conduct the Sample Person (SP) interview?',\n",
       " 'SIALANG': 'Language of the Sample Person Interview Instrument',\n",
       " 'SIAPROXY': 'Was a Proxy respondent used in conducting the Sample Person (SP) interview?',\n",
       " 'WTINT2YR': 'Full sample 2 year interview weight.',\n",
       " 'WTMEC2YR': 'Full sample 2 year MEC exam weight.',\n",
       " 'DMDBORN2': 'In what country {were you/was SP} born?',\n",
       " 'DMDHRBR2': 'In what country {were you/was NON-SP Head} born?',\n",
       " 'INDFMIN2': 'Total family income (reported as a range value in dollars)',\n",
       " 'INDHHIN2': 'Total household income (reported as a range value in dollars)',\n",
       " 'DMDEDUC': '(SP Interview Version) What is the highest grade or level of school {you have/SP has} completed or the highest degree {you have/s/he has} received?',\n",
       " 'RIDRETH2': 'Linked NH3 Race and Ethnicity Recode.',\n",
       " 'RIDPREG': 'Pregnancy status variable based on all source data. NOTE: This is the previous pregnancy status recode variable. RIDPREG is a more conservative pregnancy status variable. Please review the file documentation.',\n",
       " 'WTINT4YR': 'Interviewed Sample Persons. Used for 1999-2002 data collection years only.',\n",
       " 'WTMEC4YR': 'Both Interviewed and MEC Examined Sample Persons. Used for 1999-2002 data collection years only.',\n",
       " 'DMAETHN': 'Logical Imputation Flag for Ethnicity',\n",
       " 'DMARACE': 'Logical Imputation Flag for Race Recode',\n",
       " 'SDJ1REPN': 'Jack Knife Replicate Number',\n",
       " 'WTIREP01': 'Interview Weight Jack Knife Replicate 01',\n",
       " 'WTIREP02': 'Interview Weight Jack Knife Replicate 02',\n",
       " 'WTIREP03': 'Interview Weight Jack Knife Replicate 03',\n",
       " 'WTIREP04': 'Interview Weight Jack Knife Replicate 04',\n",
       " 'WTIREP05': 'Interview Weight Jack Knife Replicate 05',\n",
       " 'WTIREP06': 'Interview Weight Jack Knife Replicate 06',\n",
       " 'WTIREP07': 'Interview Weight Jack Knife Replicate 07',\n",
       " 'WTIREP08': 'Interview Weight Jack Knife Replicate 08',\n",
       " 'WTIREP09': 'Interview Weight Jack Knife Replicate 09',\n",
       " 'WTIREP10': 'Interview Weight Jack Knife Replicate 10',\n",
       " 'WTIREP11': 'Interview Weight Jack Knife Replicate 11',\n",
       " 'WTIREP12': 'Interview Weight Jack Knife Replicate 12',\n",
       " 'WTIREP13': 'Interview Weight Jack Knife Replicate 13',\n",
       " 'WTIREP14': 'Interview Weight Jack Knife Replicate 14',\n",
       " 'WTIREP15': 'Interview Weight Jack Knife Replicate 15',\n",
       " 'WTIREP16': 'Interview Weight Jack Knife Replicate 16',\n",
       " 'WTIREP17': 'Interview Weight Jack Knife Replicate 17',\n",
       " 'WTIREP18': 'Interview Weight Jack Knife Replicate 18',\n",
       " 'WTIREP19': 'Interview Weight Jack Knife Replicate 19',\n",
       " 'WTIREP20': 'Interview Weight Jack Knife Replicate 20',\n",
       " 'WTIREP21': 'Interview Weight Jack Knife Replicate 21',\n",
       " 'WTIREP22': 'Interview Weight Jack Knife Replicate 22',\n",
       " 'WTIREP23': 'Interview Weight Jack Knife Replicate 23',\n",
       " 'WTIREP24': 'Interview Weight Jack Knife Replicate 24',\n",
       " 'WTIREP25': 'Interview Weight Jack Knife Replicate 25',\n",
       " 'WTIREP26': 'Interview Weight Jack Knife Replicate 26',\n",
       " 'WTIREP27': 'Interview Weight Jack Knife Replicate 27',\n",
       " 'WTIREP28': 'Interview Weight Jack Knife Replicate 28',\n",
       " 'WTIREP29': 'Interview Weight Jack Knife Replicate 29',\n",
       " 'WTIREP30': 'Interview Weight Jack Knife Replicate 30',\n",
       " 'WTIREP31': 'Interview Weight Jack Knife Replicate 31',\n",
       " 'WTIREP32': 'Interview Weight Jack Knife Replicate 32',\n",
       " 'WTIREP33': 'Interview Weight Jack Knife Replicate 33',\n",
       " 'WTIREP34': 'Interview Weight Jack Knife Replicate 34',\n",
       " 'WTIREP35': 'Interview Weight Jack Knife Replicate 35',\n",
       " 'WTIREP36': 'Interview Weight Jack Knife Replicate 36',\n",
       " 'WTIREP37': 'Interview Weight Jack Knife Replicate 37',\n",
       " 'WTIREP38': 'Interview Weight Jack Knife Replicate 38',\n",
       " 'WTIREP39': 'Interview Weight Jack Knife Replicate 39',\n",
       " 'WTIREP40': 'Interview Weight Jack Knife Replicate 40',\n",
       " 'WTIREP41': 'Interview Weight Jack Knife Replicate 41',\n",
       " 'WTIREP42': 'Interview Weight Jack Knife Replicate 42',\n",
       " 'WTIREP43': 'Interview Weight Jack Knife Replicate 43',\n",
       " 'WTIREP44': 'Interview Weight Jack Knife Replicate 44',\n",
       " 'WTIREP45': 'Interview Weight Jack Knife Replicate 45',\n",
       " 'WTIREP46': 'Interview Weight Jack Knife Replicate 46',\n",
       " 'WTIREP47': 'Interview Weight Jack Knife Replicate 47',\n",
       " 'WTIREP48': 'Interview Weight Jack Knife Replicate 48',\n",
       " 'WTIREP49': 'Interview Weight Jack Knife Replicate 49',\n",
       " 'WTIREP50': 'Interview Weight Jack Knife Replicate 50',\n",
       " 'WTIREP51': 'Interview Weight Jack Knife Replicate 51',\n",
       " 'WTIREP52': 'Interview Weight Jack Knife Replicate 52',\n",
       " 'WTMREP01': 'MEC Exam Weight Jack Knife Replicate 01',\n",
       " 'WTMREP02': 'MEC Exam Weight Jack Knife Replicate 02',\n",
       " 'WTMREP03': 'MEC Exam Weight Jack Knife Replicate 03',\n",
       " 'WTMREP04': 'MEC Exam Weight Jack Knife Replicate 04',\n",
       " 'WTMREP05': 'MEC Exam Weight Jack Knife Replicate 05',\n",
       " 'WTMREP06': 'MEC Exam Weight Jack Knife Replicate 06',\n",
       " 'WTMREP07': 'MEC Exam Weight Jack Knife Replicate 07',\n",
       " 'WTMREP08': 'MEC Exam Weight Jack Knife Replicate 08',\n",
       " 'WTMREP09': 'MEC Exam Weight Jack Knife Replicate 09',\n",
       " 'WTMREP10': 'MEC Exam Weight Jack Knife Replicate 10',\n",
       " 'WTMREP11': 'MEC Exam Weight Jack Knife Replicate 11',\n",
       " 'WTMREP12': 'MEC Exam Weight Jack Knife Replicate 12',\n",
       " 'WTMREP13': 'MEC Exam Weight Jack Knife Replicate 13',\n",
       " 'WTMREP14': 'MEC Exam Weight Jack Knife Replicate 14',\n",
       " 'WTMREP15': 'MEC Exam Weight Jack Knife Replicate 15',\n",
       " 'WTMREP16': 'MEC Exam Weight Jack Knife Replicate 16',\n",
       " 'WTMREP17': 'MEC Exam Weight Jack Knife Replicate 17',\n",
       " 'WTMREP18': 'MEC Exam Weight Jack Knife Replicate 18',\n",
       " 'WTMREP19': 'MEC Exam Weight Jack Knife Replicate 19',\n",
       " 'WTMREP20': 'MEC Exam Weight Jack Knife Replicate 20',\n",
       " 'WTMREP21': 'MEC Exam Weight Jack Knife Replicate 21',\n",
       " 'WTMREP22': 'MEC Exam Weight Jack Knife Replicate 22',\n",
       " 'WTMREP23': 'MEC Exam Weight Jack Knife Replicate 23',\n",
       " 'WTMREP24': 'MEC Exam Weight Jack Knife Replicate 24',\n",
       " 'WTMREP25': 'MEC Exam Weight Jack Knife Replicate 25',\n",
       " 'WTMREP26': 'MEC Exam Weight Jack Knife Replicate 26',\n",
       " 'WTMREP27': 'MEC Exam Weight Jack Knife Replicate 27',\n",
       " 'WTMREP28': 'MEC Exam Weight Jack Knife Replicate 28',\n",
       " 'WTMREP29': 'MEC Exam Weight Jack Knife Replicate 29',\n",
       " 'WTMREP30': 'MEC Exam Weight Jack Knife Replicate 30',\n",
       " 'WTMREP31': 'MEC Exam Weight Jack Knife Replicate 31',\n",
       " 'WTMREP32': 'MEC Exam Weight Jack Knife Replicate 32',\n",
       " 'WTMREP33': 'MEC Exam Weight Jack Knife Replicate 33',\n",
       " 'WTMREP34': 'MEC Exam Weight Jack Knife Replicate 34',\n",
       " 'WTMREP35': 'MEC Exam Weight Jack Knife Replicate 35',\n",
       " 'WTMREP36': 'MEC Exam Weight Jack Knife Replicate 36',\n",
       " 'WTMREP37': 'MEC Exam Weight Jack Knife Replicate 37',\n",
       " 'WTMREP38': 'MEC Exam Weight Jack Knife Replicate 38',\n",
       " 'WTMREP39': 'MEC Exam Weight Jack Knife Replicate 39',\n",
       " 'WTMREP40': 'MEC Exam Weight Jack Knife Replicate 40',\n",
       " 'WTMREP41': 'MEC Exam Weight Jack Knife Replicate 41',\n",
       " 'WTMREP42': 'MEC Exam Weight Jack Knife Replicate 42',\n",
       " 'WTMREP43': 'MEC Exam Weight Jack Knife Replicate 43',\n",
       " 'WTMREP44': 'MEC Exam Weight Jack Knife Replicate 44',\n",
       " 'WTMREP45': 'MEC Exam Weight Jack Knife Replicate 45',\n",
       " 'WTMREP46': 'MEC Exam Weight Jack Knife Replicate 46',\n",
       " 'WTMREP47': 'MEC Exam Weight Jack Knife Replicate 47',\n",
       " 'WTMREP48': 'MEC Exam Weight Jack Knife Replicate 48',\n",
       " 'WTMREP49': 'MEC Exam Weight Jack Knife Replicate 49',\n",
       " 'WTMREP50': 'MEC Exam Weight Jack Knife Replicate 50',\n",
       " 'WTMREP51': 'MEC Exam Weight Jack Knife Replicate 51',\n",
       " 'WTMREP52': 'MEC Exam Weight Jack Knife Replicate 52',\n",
       " 'AIALANGA': 'Language of the MEC ACASI Interview Instrument',\n",
       " 'DMDBORN4': 'In what country {were you/was SP} born?',\n",
       " 'DMDHHSZA': 'Number of children aged 5 years or younger in the household',\n",
       " 'DMDHHSZB': 'Number of children aged 6-17 years old in the household',\n",
       " 'DMDHHSZE': 'Number of adults aged 60 years or older in the household',\n",
       " 'DMDHRBR4': \"HH reference person's country of birth\",\n",
       " 'DMQADFC': 'Did {you/SP} ever serve in a foreign country during a time of armed conflict or on a humanitarian or peace-keeping mission? (This would include National Guard or reserve or active duty monitoring or conducting peace keeping operations in Bosnia and Kosovo, in the Sinai between Egypt and Israel, or in response to the 2004 tsunami or Haiti in 2010.)',\n",
       " 'DMQMILIZ': '{Have you/Has SP} ever served on active duty in the U.S. Armed Forces, military Reserves, or National Guard? (Active duty does not include training for the Reserves or National Guard, but does include activation, for service in the U.S. or in a foreign country, in support of military or humanitarian operations.)',\n",
       " 'RIDEXAGM': 'Age in months of the participant at the time of examination. Reported for persons aged 19 years or younger at the time of examination.',\n",
       " 'RIDEXAGY': 'Age in years of the participant at the time of examination. Reported for persons aged 2-19 years at the time of screening.',\n",
       " 'RIDRETH3': 'Recode of reported race and Hispanic origin information, with Non-Hispanic Asian Category',\n",
       " 'DMDHRAGZ': \"HH reference person's age in years\",\n",
       " 'DMDHREDZ': \"HH reference person's education level\",\n",
       " 'DMDHRMAZ': \"HH reference person's marital status\",\n",
       " 'DMDHSEDZ': \"HH reference person's spouse's education level\"}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Variable_Description_my_dict = dict(zip(demographics_var_df['Variable Name'], demographics_var_df['Variable Description']))\n",
    "Variable_Description_my_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we are going to get rid of the below variable as they are of no use to our project:\n",
    "\n",
    "variables_to_be_removed = {'SDJ1REPN': 'Jack Knife Replicate Number',\n",
    " 'WTIREP01': 'Interview Weight Jack Knife Replicate 01',\n",
    " 'WTIREP02': 'Interview Weight Jack Knife Replicate 02',\n",
    " 'WTIREP03': 'Interview Weight Jack Knife Replicate 03',\n",
    " 'WTIREP04': 'Interview Weight Jack Knife Replicate 04',\n",
    " 'WTIREP05': 'Interview Weight Jack Knife Replicate 05',\n",
    " 'WTIREP06': 'Interview Weight Jack Knife Replicate 06',\n",
    " 'WTIREP07': 'Interview Weight Jack Knife Replicate 07',\n",
    " 'WTIREP08': 'Interview Weight Jack Knife Replicate 08',\n",
    " 'WTIREP09': 'Interview Weight Jack Knife Replicate 09',\n",
    " 'WTIREP10': 'Interview Weight Jack Knife Replicate 10',\n",
    " 'WTIREP11': 'Interview Weight Jack Knife Replicate 11',\n",
    " 'WTIREP12': 'Interview Weight Jack Knife Replicate 12',\n",
    " 'WTIREP13': 'Interview Weight Jack Knife Replicate 13',\n",
    " 'WTIREP14': 'Interview Weight Jack Knife Replicate 14',\n",
    " 'WTIREP15': 'Interview Weight Jack Knife Replicate 15',\n",
    " 'WTIREP16': 'Interview Weight Jack Knife Replicate 16',\n",
    " 'WTIREP17': 'Interview Weight Jack Knife Replicate 17',\n",
    " 'WTIREP18': 'Interview Weight Jack Knife Replicate 18',\n",
    " 'WTIREP19': 'Interview Weight Jack Knife Replicate 19',\n",
    " 'WTIREP20': 'Interview Weight Jack Knife Replicate 20',\n",
    " 'WTIREP21': 'Interview Weight Jack Knife Replicate 21',\n",
    " 'WTIREP22': 'Interview Weight Jack Knife Replicate 22',\n",
    " 'WTIREP23': 'Interview Weight Jack Knife Replicate 23',\n",
    " 'WTIREP24': 'Interview Weight Jack Knife Replicate 24',\n",
    " 'WTIREP25': 'Interview Weight Jack Knife Replicate 25',\n",
    " 'WTIREP26': 'Interview Weight Jack Knife Replicate 26',\n",
    " 'WTIREP27': 'Interview Weight Jack Knife Replicate 27',\n",
    " 'WTIREP28': 'Interview Weight Jack Knife Replicate 28',\n",
    " 'WTIREP29': 'Interview Weight Jack Knife Replicate 29',\n",
    " 'WTIREP30': 'Interview Weight Jack Knife Replicate 30',\n",
    " 'WTIREP31': 'Interview Weight Jack Knife Replicate 31',\n",
    " 'WTIREP32': 'Interview Weight Jack Knife Replicate 32',\n",
    " 'WTIREP33': 'Interview Weight Jack Knife Replicate 33',\n",
    " 'WTIREP34': 'Interview Weight Jack Knife Replicate 34',\n",
    " 'WTIREP35': 'Interview Weight Jack Knife Replicate 35',\n",
    " 'WTIREP36': 'Interview Weight Jack Knife Replicate 36',\n",
    " 'WTIREP37': 'Interview Weight Jack Knife Replicate 37',\n",
    " 'WTIREP38': 'Interview Weight Jack Knife Replicate 38',\n",
    " 'WTIREP39': 'Interview Weight Jack Knife Replicate 39',\n",
    " 'WTIREP40': 'Interview Weight Jack Knife Replicate 40',\n",
    " 'WTIREP41': 'Interview Weight Jack Knife Replicate 41',\n",
    " 'WTIREP42': 'Interview Weight Jack Knife Replicate 42',\n",
    " 'WTIREP43': 'Interview Weight Jack Knife Replicate 43',\n",
    " 'WTIREP44': 'Interview Weight Jack Knife Replicate 44',\n",
    " 'WTIREP45': 'Interview Weight Jack Knife Replicate 45',\n",
    " 'WTIREP46': 'Interview Weight Jack Knife Replicate 46',\n",
    " 'WTIREP47': 'Interview Weight Jack Knife Replicate 47',\n",
    " 'WTIREP48': 'Interview Weight Jack Knife Replicate 48',\n",
    " 'WTIREP49': 'Interview Weight Jack Knife Replicate 49',\n",
    " 'WTIREP50': 'Interview Weight Jack Knife Replicate 50',\n",
    " 'WTIREP51': 'Interview Weight Jack Knife Replicate 51',\n",
    " 'WTIREP52': 'Interview Weight Jack Knife Replicate 52',\n",
    " 'WTMREP01': 'MEC Exam Weight Jack Knife Replicate 01',\n",
    " 'WTMREP02': 'MEC Exam Weight Jack Knife Replicate 02',\n",
    " 'WTMREP03': 'MEC Exam Weight Jack Knife Replicate 03',\n",
    " 'WTMREP04': 'MEC Exam Weight Jack Knife Replicate 04',\n",
    " 'WTMREP05': 'MEC Exam Weight Jack Knife Replicate 05',\n",
    " 'WTMREP06': 'MEC Exam Weight Jack Knife Replicate 06',\n",
    " 'WTMREP07': 'MEC Exam Weight Jack Knife Replicate 07',\n",
    " 'WTMREP08': 'MEC Exam Weight Jack Knife Replicate 08',\n",
    " 'WTMREP09': 'MEC Exam Weight Jack Knife Replicate 09',\n",
    " 'WTMREP10': 'MEC Exam Weight Jack Knife Replicate 10',\n",
    " 'WTMREP11': 'MEC Exam Weight Jack Knife Replicate 11',\n",
    " 'WTMREP12': 'MEC Exam Weight Jack Knife Replicate 12',\n",
    " 'WTMREP13': 'MEC Exam Weight Jack Knife Replicate 13',\n",
    " 'WTMREP14': 'MEC Exam Weight Jack Knife Replicate 14',\n",
    " 'WTMREP15': 'MEC Exam Weight Jack Knife Replicate 15',\n",
    " 'WTMREP16': 'MEC Exam Weight Jack Knife Replicate 16',\n",
    " 'WTMREP17': 'MEC Exam Weight Jack Knife Replicate 17',\n",
    " 'WTMREP18': 'MEC Exam Weight Jack Knife Replicate 18',\n",
    " 'WTMREP19': 'MEC Exam Weight Jack Knife Replicate 19',\n",
    " 'WTMREP20': 'MEC Exam Weight Jack Knife Replicate 20',\n",
    " 'WTMREP21': 'MEC Exam Weight Jack Knife Replicate 21',\n",
    " 'WTMREP22': 'MEC Exam Weight Jack Knife Replicate 22',\n",
    " 'WTMREP23': 'MEC Exam Weight Jack Knife Replicate 23',\n",
    " 'WTMREP24': 'MEC Exam Weight Jack Knife Replicate 24',\n",
    " 'WTMREP25': 'MEC Exam Weight Jack Knife Replicate 25',\n",
    " 'WTMREP26': 'MEC Exam Weight Jack Knife Replicate 26',\n",
    " 'WTMREP27': 'MEC Exam Weight Jack Knife Replicate 27',\n",
    " 'WTMREP28': 'MEC Exam Weight Jack Knife Replicate 28',\n",
    " 'WTMREP29': 'MEC Exam Weight Jack Knife Replicate 29',\n",
    " 'WTMREP30': 'MEC Exam Weight Jack Knife Replicate 30',\n",
    " 'WTMREP31': 'MEC Exam Weight Jack Knife Replicate 31',\n",
    " 'WTMREP32': 'MEC Exam Weight Jack Knife Replicate 32',\n",
    " 'WTMREP33': 'MEC Exam Weight Jack Knife Replicate 33',\n",
    " 'WTMREP34': 'MEC Exam Weight Jack Knife Replicate 34',\n",
    " 'WTMREP35': 'MEC Exam Weight Jack Knife Replicate 35',\n",
    " 'WTMREP36': 'MEC Exam Weight Jack Knife Replicate 36',\n",
    " 'WTMREP37': 'MEC Exam Weight Jack Knife Replicate 37',\n",
    " 'WTMREP38': 'MEC Exam Weight Jack Knife Replicate 38',\n",
    " 'WTMREP39': 'MEC Exam Weight Jack Knife Replicate 39',\n",
    " 'WTMREP40': 'MEC Exam Weight Jack Knife Replicate 40',\n",
    " 'WTMREP41': 'MEC Exam Weight Jack Knife Replicate 41',\n",
    " 'WTMREP42': 'MEC Exam Weight Jack Knife Replicate 42',\n",
    " 'WTMREP43': 'MEC Exam Weight Jack Knife Replicate 43',\n",
    " 'WTMREP44': 'MEC Exam Weight Jack Knife Replicate 44',\n",
    " 'WTMREP45': 'MEC Exam Weight Jack Knife Replicate 45',\n",
    " 'WTMREP46': 'MEC Exam Weight Jack Knife Replicate 46',\n",
    " 'WTMREP47': 'MEC Exam Weight Jack Knife Replicate 47',\n",
    " 'WTMREP48': 'MEC Exam Weight Jack Knife Replicate 48',\n",
    " 'WTMREP49': 'MEC Exam Weight Jack Knife Replicate 49',\n",
    " 'WTMREP50': 'MEC Exam Weight Jack Knife Replicate 50',\n",
    " 'WTMREP51': 'MEC Exam Weight Jack Knife Replicate 51',\n",
    " 'WTMREP52': 'MEC Exam Weight Jack Knife Replicate 52',\n",
    " 'AIALANGA': 'Language of the MEC ACASI Interview Instrument',}\n",
    "\n",
    "list_of_variables_to_be_removed = list(variables_to_be_removed.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['SDJ1REPN',\n",
       " 'WTIREP01',\n",
       " 'WTIREP02',\n",
       " 'WTIREP03',\n",
       " 'WTIREP04',\n",
       " 'WTIREP05',\n",
       " 'WTIREP06',\n",
       " 'WTIREP07',\n",
       " 'WTIREP08',\n",
       " 'WTIREP09',\n",
       " 'WTIREP10',\n",
       " 'WTIREP11',\n",
       " 'WTIREP12',\n",
       " 'WTIREP13',\n",
       " 'WTIREP14',\n",
       " 'WTIREP15',\n",
       " 'WTIREP16',\n",
       " 'WTIREP17',\n",
       " 'WTIREP18',\n",
       " 'WTIREP19',\n",
       " 'WTIREP20',\n",
       " 'WTIREP21',\n",
       " 'WTIREP22',\n",
       " 'WTIREP23',\n",
       " 'WTIREP24',\n",
       " 'WTIREP25',\n",
       " 'WTIREP26',\n",
       " 'WTIREP27',\n",
       " 'WTIREP28',\n",
       " 'WTIREP29',\n",
       " 'WTIREP30',\n",
       " 'WTIREP31',\n",
       " 'WTIREP32',\n",
       " 'WTIREP33',\n",
       " 'WTIREP34',\n",
       " 'WTIREP35',\n",
       " 'WTIREP36',\n",
       " 'WTIREP37',\n",
       " 'WTIREP38',\n",
       " 'WTIREP39',\n",
       " 'WTIREP40',\n",
       " 'WTIREP41',\n",
       " 'WTIREP42',\n",
       " 'WTIREP43',\n",
       " 'WTIREP44',\n",
       " 'WTIREP45',\n",
       " 'WTIREP46',\n",
       " 'WTIREP47',\n",
       " 'WTIREP48',\n",
       " 'WTIREP49',\n",
       " 'WTIREP50',\n",
       " 'WTIREP51',\n",
       " 'WTIREP52',\n",
       " 'WTMREP01',\n",
       " 'WTMREP02',\n",
       " 'WTMREP03',\n",
       " 'WTMREP04',\n",
       " 'WTMREP05',\n",
       " 'WTMREP06',\n",
       " 'WTMREP07',\n",
       " 'WTMREP08',\n",
       " 'WTMREP09',\n",
       " 'WTMREP10',\n",
       " 'WTMREP11',\n",
       " 'WTMREP12',\n",
       " 'WTMREP13',\n",
       " 'WTMREP14',\n",
       " 'WTMREP15',\n",
       " 'WTMREP16',\n",
       " 'WTMREP17',\n",
       " 'WTMREP18',\n",
       " 'WTMREP19',\n",
       " 'WTMREP20',\n",
       " 'WTMREP21',\n",
       " 'WTMREP22',\n",
       " 'WTMREP23',\n",
       " 'WTMREP24',\n",
       " 'WTMREP25',\n",
       " 'WTMREP26',\n",
       " 'WTMREP27',\n",
       " 'WTMREP28',\n",
       " 'WTMREP29',\n",
       " 'WTMREP30',\n",
       " 'WTMREP31',\n",
       " 'WTMREP32',\n",
       " 'WTMREP33',\n",
       " 'WTMREP34',\n",
       " 'WTMREP35',\n",
       " 'WTMREP36',\n",
       " 'WTMREP37',\n",
       " 'WTMREP38',\n",
       " 'WTMREP39',\n",
       " 'WTMREP40',\n",
       " 'WTMREP41',\n",
       " 'WTMREP42',\n",
       " 'WTMREP43',\n",
       " 'WTMREP44',\n",
       " 'WTMREP45',\n",
       " 'WTMREP46',\n",
       " 'WTMREP47',\n",
       " 'WTMREP48',\n",
       " 'WTMREP49',\n",
       " 'WTMREP50',\n",
       " 'WTMREP51',\n",
       " 'WTMREP52',\n",
       " 'AIALANGA']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_of_variables_to_be_removed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "175"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(list(demo_dataframe.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we are going to drop the columns from our main demo_dataframe\n",
    "demo_dataframe = demo_dataframe.drop(columns=list_of_variables_to_be_removed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "69"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(list(demo_dataframe.columns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['SEQN',\n",
       " 'SDDSRVYR',\n",
       " 'RIDSTATR',\n",
       " 'RIDEXMON',\n",
       " 'RIAGENDR',\n",
       " 'RIDAGEYR',\n",
       " 'RIDAGEMN',\n",
       " 'RIDAGEEX',\n",
       " 'RIDRETH1',\n",
       " 'RIDRETH2',\n",
       " 'DMQMILIT',\n",
       " 'DMDBORN',\n",
       " 'DMDCITZN',\n",
       " 'DMDYRSUS',\n",
       " 'DMDEDUC3',\n",
       " 'DMDEDUC2',\n",
       " 'DMDEDUC',\n",
       " 'DMDSCHOL',\n",
       " 'DMDMARTL',\n",
       " 'DMDHHSIZ',\n",
       " 'INDHHINC',\n",
       " 'INDFMINC',\n",
       " 'INDFMPIR',\n",
       " 'RIDEXPRG',\n",
       " 'RIDPREG',\n",
       " 'DMDHRGND',\n",
       " 'DMDHRAGE',\n",
       " 'DMDHRBRN',\n",
       " 'DMDHREDU',\n",
       " 'DMDHRMAR',\n",
       " 'DMDHSEDU',\n",
       " 'WTINT2YR',\n",
       " 'WTINT4YR',\n",
       " 'WTMEC2YR',\n",
       " 'WTMEC4YR',\n",
       " 'SDMVPSU',\n",
       " 'SDMVSTRA',\n",
       " 'DMAETHN',\n",
       " 'DMARACE',\n",
       " 'SIALANG',\n",
       " 'SIAPROXY',\n",
       " 'SIAINTRP',\n",
       " 'FIALANG',\n",
       " 'FIAPROXY',\n",
       " 'FIAINTRP',\n",
       " 'MIALANG',\n",
       " 'MIAPROXY',\n",
       " 'MIAINTRP',\n",
       " 'AIALANG',\n",
       " 'DMDFMSIZ',\n",
       " 'DMDBORN2',\n",
       " 'INDHHIN2',\n",
       " 'INDFMIN2',\n",
       " 'DMDHRBR2',\n",
       " 'RIDRETH3',\n",
       " 'RIDEXAGY',\n",
       " 'RIDEXAGM',\n",
       " 'DMQMILIZ',\n",
       " 'DMQADFC',\n",
       " 'DMDBORN4',\n",
       " 'DMDHHSZA',\n",
       " 'DMDHHSZB',\n",
       " 'DMDHHSZE',\n",
       " 'DMDHRBR4',\n",
       " 'DMDHRAGZ',\n",
       " 'DMDHREDZ',\n",
       " 'DMDHRMAZ',\n",
       " 'DMDHSEDZ']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "variable_list = [col for col in demo_dataframe.columns if col != 'year']\n",
    "variable_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#we are going to go retreive the demographic data documentation, that way we can figure out what the variable/columns mean\n",
    "\n",
    "\n",
    "varibale_code_table = dict()\n",
    "variable_sas_label = dict()\n",
    "variable_English_Text = dict()\n",
    "\n",
    "\n",
    "for i, cycle_year in enumerate(cycle_list):\n",
    "   \n",
    "  temp = ''\n",
    "  letter = chr(ord('A') + i)\n",
    "  if letter != 'A':\n",
    "      temp = f\"_{letter}\"\n",
    "      \n",
    "  data_File_Name = f\"DEMO{temp}\"\n",
    "\n",
    "  print(f\"datafile: {data_File_Name} -----> cycle: {cycle_year}\")\n",
    "\n",
    "  url = f\"https://wwwn.cdc.gov/Nchs/Nhanes/{cycle_year}/{data_File_Name}.htm\"\n",
    "  req=requests.get(url)\n",
    "  content=req.text\n",
    "  soup = BeautifulSoup(content)\n",
    "\n",
    "  mydivs = soup.find_all(\"div\", {\"class\": \"pagebreak\"})\n",
    "  for i, div in enumerate(mydivs):\n",
    "    x = div.find_all_next()\n",
    "    variable = x[0][\"id\"]\n",
    "    if variable in variable_list:\n",
    "      #print(variable)\n",
    "      #print(f\"{x[2].text}{x[3].text}\")\n",
    "      variable_sas_label[variable] = x[5].text\n",
    "      #print(f\"{x[4].text}{x[5].text}\")\n",
    "      variable_English_Text[variable] = {x[7].text}\n",
    "      #print(f\"{x[6].text}{x[7].text}\")\n",
    "      if div.find(\"table\") is not None:\n",
    "        table = pd.read_html(str(div.find('table')))[0]\n",
    "        varibale_code_table[variable] = table\n",
    "        #print(data_frame[i-1])\n",
    "      #print(\"#####################\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting html5lib\n",
      "  Downloading html5lib-1.1-py2.py3-none-any.whl (112 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m112.2/112.2 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: six>=1.9 in /opt/homebrew/Caskroom/miniforge/base/envs/nhanes_env/lib/python3.12/site-packages (from html5lib) (1.16.0)\n",
      "Collecting webencodings (from html5lib)\n",
      "  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)\n",
      "Installing collected packages: webencodings, html5lib\n",
      "Successfully installed html5lib-1.1 webencodings-0.5.1\n"
     ]
    }
   ],
   "source": [
    "!pip install html5lib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "ename": "FeatureNotFound",
     "evalue": "Couldn't find a tree builder with the features you requested: html5lib. Do you need to install a parser library?",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFeatureNotFound\u001b[0m                           Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[50], line 4\u001b[0m\n\u001b[1;32m      2\u001b[0m req \u001b[38;5;241m=\u001b[39m requests\u001b[38;5;241m.\u001b[39mget(url)\n\u001b[1;32m      3\u001b[0m content \u001b[38;5;241m=\u001b[39m req\u001b[38;5;241m.\u001b[39mtext\n\u001b[0;32m----> 4\u001b[0m soup \u001b[38;5;241m=\u001b[39m \u001b[43mBeautifulSoup\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcontent\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mhtml5lib\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m \n\u001b[1;32m      6\u001b[0m mydivs \u001b[38;5;241m=\u001b[39m soup\u001b[38;5;241m.\u001b[39mfind_all(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdiv\u001b[39m\u001b[38;5;124m\"\u001b[39m, {\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mclass\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpagebreak\u001b[39m\u001b[38;5;124m\"\u001b[39m})\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m i, div \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(mydivs):\n",
      "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/nhanes_env/lib/python3.12/site-packages/bs4/__init__.py:250\u001b[0m, in \u001b[0;36mBeautifulSoup.__init__\u001b[0;34m(self, markup, features, builder, parse_only, from_encoding, exclude_encodings, element_classes, **kwargs)\u001b[0m\n\u001b[1;32m    248\u001b[0m     builder_class \u001b[38;5;241m=\u001b[39m builder_registry\u001b[38;5;241m.\u001b[39mlookup(\u001b[38;5;241m*\u001b[39mfeatures)\n\u001b[1;32m    249\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m builder_class \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 250\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m FeatureNotFound(\n\u001b[1;32m    251\u001b[0m             \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCouldn\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt find a tree builder with the features you \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    252\u001b[0m             \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mrequested: \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m. Do you need to install a parser library?\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m    253\u001b[0m             \u001b[38;5;241m%\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m,\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39mjoin(features))\n\u001b[1;32m    255\u001b[0m \u001b[38;5;66;03m# At this point either we have a TreeBuilder instance in\u001b[39;00m\n\u001b[1;32m    256\u001b[0m \u001b[38;5;66;03m# builder, or we have a builder_class that we can instantiate\u001b[39;00m\n\u001b[1;32m    257\u001b[0m \u001b[38;5;66;03m# with the remaining **kwargs.\u001b[39;00m\n\u001b[1;32m    258\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m builder \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
      "\u001b[0;31mFeatureNotFound\u001b[0m: Couldn't find a tree builder with the features you requested: html5lib. Do you need to install a parser library?"
     ]
    }
   ],
   "source": [
    "\n",
    "url = f\"https://wwwn.cdc.gov/Nchs/Nhanes/{cycle_year}/{data_File_Name}.htm\"\n",
    "req = requests.get(url)\n",
    "content = req.text\n",
    "soup = BeautifulSoup(content, 'html5lib') \n",
    "\n",
    "mydivs = soup.find_all(\"div\", {\"class\": \"pagebreak\"})\n",
    "for i, div in enumerate(mydivs):\n",
    "    x = div.find_all_next()\n",
    "    variable = x[0][\"id\"]\n",
    "    if variable in variable_list:\n",
    "        variable_sas_label[variable] = x[5].text\n",
    "        variable_English_Text[variable] = {x[7].text}\n",
    "        if div.find(\"table\") is not None:\n",
    "            #pd.read_html with flavor='bs4' to read the HTML table\n",
    "            table = pd.read_html(str(div.find('table')), flavor='bs4')[0]\n",
    "            varibale_code_table[variable] = table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 101316 entries, 0 to 101315\n",
      "Columns: 175 entries, SEQN to DMDHSEDZ\n",
      "dtypes: float64(174), object(1)\n",
      "memory usage: 135.3+ MB\n"
     ]
    }
   ],
   "source": [
    "#we need to do a lot of cleanup on the data, and we are going to do some descriptive statistics on the dataframe\n",
    "demo_dataframe.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SEQN</th>\n",
       "      <th>SDDSRVYR</th>\n",
       "      <th>RIDSTATR</th>\n",
       "      <th>RIDEXMON</th>\n",
       "      <th>RIAGENDR</th>\n",
       "      <th>RIDAGEYR</th>\n",
       "      <th>RIDAGEMN</th>\n",
       "      <th>RIDAGEEX</th>\n",
       "      <th>RIDRETH1</th>\n",
       "      <th>RIDRETH2</th>\n",
       "      <th>...</th>\n",
       "      <th>DMDBORN4</th>\n",
       "      <th>AIALANGA</th>\n",
       "      <th>DMDHHSZA</th>\n",
       "      <th>DMDHHSZB</th>\n",
       "      <th>DMDHHSZE</th>\n",
       "      <th>DMDHRBR4</th>\n",
       "      <th>DMDHRAGZ</th>\n",
       "      <th>DMDHREDZ</th>\n",
       "      <th>DMDHRMAZ</th>\n",
       "      <th>DMDHSEDZ</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>101316.000000</td>\n",
       "      <td>101316.000000</td>\n",
       "      <td>101316.000000</td>\n",
       "      <td>96766.000000</td>\n",
       "      <td>101316.000000</td>\n",
       "      <td>1.013160e+05</td>\n",
       "      <td>6.308500e+04</td>\n",
       "      <td>5.787400e+04</td>\n",
       "      <td>101316.000000</td>\n",
       "      <td>31126.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>39156.000000</td>\n",
       "      <td>23010.000000</td>\n",
       "      <td>3.915600e+04</td>\n",
       "      <td>3.915600e+04</td>\n",
       "      <td>3.915600e+04</td>\n",
       "      <td>28844.000000</td>\n",
       "      <td>9254.000000</td>\n",
       "      <td>8764.000000</td>\n",
       "      <td>9063.000000</td>\n",
       "      <td>4751.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>51134.397193</td>\n",
       "      <td>5.425984</td>\n",
       "      <td>1.955091</td>\n",
       "      <td>1.523407</td>\n",
       "      <td>1.507551</td>\n",
       "      <td>3.112829e+01</td>\n",
       "      <td>3.392977e+02</td>\n",
       "      <td>3.517031e+02</td>\n",
       "      <td>2.895831</td>\n",
       "      <td>2.086584</td>\n",
       "      <td>...</td>\n",
       "      <td>1.241828</td>\n",
       "      <td>1.119991</td>\n",
       "      <td>5.322301e-01</td>\n",
       "      <td>9.717029e-01</td>\n",
       "      <td>4.204975e-01</td>\n",
       "      <td>1.412668</td>\n",
       "      <td>2.860061</td>\n",
       "      <td>2.050776</td>\n",
       "      <td>1.472691</td>\n",
       "      <td>2.110714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>29836.192620</td>\n",
       "      <td>2.850576</td>\n",
       "      <td>0.207105</td>\n",
       "      <td>0.499454</td>\n",
       "      <td>0.499945</td>\n",
       "      <td>2.494308e+01</td>\n",
       "      <td>2.874832e+02</td>\n",
       "      <td>2.831834e+02</td>\n",
       "      <td>1.251255</td>\n",
       "      <td>1.099042</td>\n",
       "      <td>...</td>\n",
       "      <td>1.745682</td>\n",
       "      <td>0.369281</td>\n",
       "      <td>8.111773e-01</td>\n",
       "      <td>1.160525e+00</td>\n",
       "      <td>7.164718e-01</td>\n",
       "      <td>2.808172</td>\n",
       "      <td>0.810059</td>\n",
       "      <td>0.652806</td>\n",
       "      <td>0.721168</td>\n",
       "      <td>0.688517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>25329.750000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000e+01</td>\n",
       "      <td>9.800000e+01</td>\n",
       "      <td>1.182500e+02</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>50658.500000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.400000e+01</td>\n",
       "      <td>2.330000e+02</td>\n",
       "      <td>2.480000e+02</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>5.397605e-79</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>77627.250000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>5.200000e+01</td>\n",
       "      <td>5.600000e+02</td>\n",
       "      <td>5.720000e+02</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>2.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>102956.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>8.500000e+01</td>\n",
       "      <td>1.019000e+03</td>\n",
       "      <td>1.019000e+03</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000e+00</td>\n",
       "      <td>4.000000e+00</td>\n",
       "      <td>3.000000e+00</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 174 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                SEQN       SDDSRVYR       RIDSTATR      RIDEXMON  \\\n",
       "count  101316.000000  101316.000000  101316.000000  96766.000000   \n",
       "mean    51134.397193       5.425984       1.955091      1.523407   \n",
       "std     29836.192620       2.850576       0.207105      0.499454   \n",
       "min         1.000000       1.000000       1.000000      1.000000   \n",
       "25%     25329.750000       3.000000       2.000000      1.000000   \n",
       "50%     50658.500000       5.000000       2.000000      2.000000   \n",
       "75%     77627.250000       8.000000       2.000000      2.000000   \n",
       "max    102956.000000      10.000000       2.000000      2.000000   \n",
       "\n",
       "            RIAGENDR      RIDAGEYR      RIDAGEMN      RIDAGEEX       RIDRETH1  \\\n",
       "count  101316.000000  1.013160e+05  6.308500e+04  5.787400e+04  101316.000000   \n",
       "mean        1.507551  3.112829e+01  3.392977e+02  3.517031e+02       2.895831   \n",
       "std         0.499945  2.494308e+01  2.874832e+02  2.831834e+02       1.251255   \n",
       "min         1.000000  5.397605e-79  5.397605e-79  5.397605e-79       1.000000   \n",
       "25%         1.000000  1.000000e+01  9.800000e+01  1.182500e+02       2.000000   \n",
       "50%         2.000000  2.400000e+01  2.330000e+02  2.480000e+02       3.000000   \n",
       "75%         2.000000  5.200000e+01  5.600000e+02  5.720000e+02       4.000000   \n",
       "max         2.000000  8.500000e+01  1.019000e+03  1.019000e+03       5.000000   \n",
       "\n",
       "           RIDRETH2  ...      DMDBORN4      AIALANGA      DMDHHSZA  \\\n",
       "count  31126.000000  ...  39156.000000  23010.000000  3.915600e+04   \n",
       "mean       2.086584  ...      1.241828      1.119991  5.322301e-01   \n",
       "std        1.099042  ...      1.745682      0.369281  8.111773e-01   \n",
       "min        1.000000  ...      1.000000      1.000000  5.397605e-79   \n",
       "25%        1.000000  ...      1.000000      1.000000  5.397605e-79   \n",
       "50%        2.000000  ...      1.000000      1.000000  5.397605e-79   \n",
       "75%        3.000000  ...      1.000000      1.000000  1.000000e+00   \n",
       "max        5.000000  ...     99.000000      3.000000  3.000000e+00   \n",
       "\n",
       "           DMDHHSZB      DMDHHSZE      DMDHRBR4     DMDHRAGZ     DMDHREDZ  \\\n",
       "count  3.915600e+04  3.915600e+04  28844.000000  9254.000000  8764.000000   \n",
       "mean   9.717029e-01  4.204975e-01      1.412668     2.860061     2.050776   \n",
       "std    1.160525e+00  7.164718e-01      2.808172     0.810059     0.652806   \n",
       "min    5.397605e-79  5.397605e-79      1.000000     1.000000     1.000000   \n",
       "25%    5.397605e-79  5.397605e-79      1.000000     2.000000     2.000000   \n",
       "50%    1.000000e+00  5.397605e-79      1.000000     3.000000     2.000000   \n",
       "75%    2.000000e+00  1.000000e+00      2.000000     4.000000     2.000000   \n",
       "max    4.000000e+00  3.000000e+00     99.000000     4.000000     3.000000   \n",
       "\n",
       "          DMDHRMAZ     DMDHSEDZ  \n",
       "count  9063.000000  4751.000000  \n",
       "mean      1.472691     2.110714  \n",
       "std       0.721168     0.688517  \n",
       "min       1.000000     1.000000  \n",
       "25%       1.000000     2.000000  \n",
       "50%       1.000000     2.000000  \n",
       "75%       2.000000     3.000000  \n",
       "max       3.000000     3.000000  \n",
       "\n",
       "[8 rows x 174 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "demo_dataframe.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [NHANES Questionnaire Data](#nhanes-questionnaire-data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Joining Demographic and Questionnaire Data](#joining-demographic-and-questionnaire-data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Exploratory Data Analysis (EDA)](#exploratory-data-analysis-eda)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Descriptive Statistics](#descriptive-statistics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Demographic Analysis](#demographic-analysis)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Analysis of Health Behaviors and Life-Style](#analysis-of-health-behaviors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Access to Healthcare](#access-to-healthcare)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Social Determinants of Health Analysis](#social-determinants-of-health-analysis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Statistical Analysis](#statistical-analysis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Inferential Statistics](#inferential-statistics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Multivariate Analysis](#multivariate-analysis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Comparative Analysis](#comparative-analysis)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Correlation Analysis](#correlation-analysis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Hypothesis Testing](#hypothesis-testing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Interpretation and Insights](#interpretation-and-insights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Key Findings from EDA](#key-findings-from-eda)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Implications for Addressing Disparities](#implications-for-addressing-disparities)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Conclusion](#conclusion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Preject Summary](#project_summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Future Directions](#future_directions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Appendix](#appendix)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [NHANES-pyTOOL-API Documentation](#api_documentation)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Technical Details](#technical_details)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [References](#references)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nhanes_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}