abstract_grabber.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Need to try to extract training data. \n",
    "- need to create a holdout to use to test the predictions with, can be a small holdout \n",
    "- there are possibly two different patterns - files from Christine \n",
    "- files that have been added since maybe the 4th of March"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "from xlrd import open_workbook\n",
    "import os\n",
    "import time\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "excel_files = glob.glob(\"/Users/ianm/Dropbox/workbench/sage-maker-srm-test/excel-data/*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "97"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(excel_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# get files modified on or after the 4th of March"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def creation_date(path_to_file):\n",
    "    \"\"\"\n",
    "    Try to get the date that a file was created, falling back to when it was\n",
    "    last modified if that isn't possible.\n",
    "    See http://stackoverflow.com/a/39501288/1709587 for explanation.\n",
    "    \"\"\"\n",
    "    stat = os.stat(path_to_file)\n",
    "    try:\n",
    "#         return stat.st_birthtime\n",
    "        return stat.st_mtime\n",
    "    except AttributeError:\n",
    "        # We're probably on Linux. No easy way to get creation dates here,\n",
    "        # so we'll settle for when its content was last modified.\n",
    "        return stat.st_mtime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "selection_epoch_time = 1551653455 # this is unix time for 3rd of March 2019, 23:00 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "post_mar_four_files = []\n",
    "for file in excel_files:\n",
    "    mod_time = creation_date(file)\n",
    "    if mod_time > selection_epoch_time:\n",
    "        post_mar_four_files.append(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'post_mar_four_files' (list)\n"
     ]
    }
   ],
   "source": [
    "%store post_mar_four_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# get files from a team member"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def files_from_team_member(initials, file_list):\n",
    "    team_member_files = []\n",
    "    for file in file_list:\n",
    "        tail_name = file.split(\"/\")[-1]\n",
    "        if tail_name.split()[0] == initials:\n",
    "            team_member_files.append(file)\n",
    "    return team_member_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# get files from Christine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "christine_files = files_from_team_member(\"CS\", excel_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generic xlrd functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_workbook(filename):\n",
    "    wb = open_workbook(filename)\n",
    "    return wb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_worksheet(filename, index):\n",
    "    wb = open_workbook(filename)\n",
    "    sheet = wb.sheets()[index]\n",
    "    return sheet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_sheet_names(wb):\n",
    "    return wb.sheet_names() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_col_names(sheet):\n",
    "    index_row = 0\n",
    "    col_names = row = sheet.row(index_row)\n",
    "    return col_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get abstract and titles of selected and rejected articles from Christine files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sheet_names_with_selections = [\"contenders\", \"to email\", \"names to check for email\", \"double articles- manual email\", \"Contenders\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "abstract_col_names = [\"AB\", \"Abstract\"]\n",
    "title_col_names = [\"TI\", \"Title\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_sheet_names_against_validation(sheet_names, validation_list):\n",
    "    for name in sheet_names:\n",
    "        if name in validation_list: return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "the sheets have to contain both \"original\" and something from a selection list. \n",
    "Then we pull the abstracts from \"original\" and from one of the other columns too. \n",
    "we could just naievely run this twice and have simpler logic, I think that wojld be better\n",
    "so \n",
    "for each file\n",
    "    get the candidate colum\n",
    "    then get the original colum\n",
    "    and if we don't have both skip\n",
    "    else add to the aggregate list! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_col_values_by_col_name(sheet, col_name_candidates):\n",
    "    \"\"\"\n",
    "    the assumption is that we are looking for a colum from one sheet\n",
    "    but the sheet and colum name might have one of a number of differnet values, but only one of them. \n",
    "    \"\"\"\n",
    "    col_names = get_col_names(sheet)\n",
    "    for index, col in enumerate(col_names):\n",
    "        if col.value in col_name_candidates:\n",
    "            col_values = sheet.col_values(index)\n",
    "            return col_values\n",
    "    return None # if we don't find the column return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_col_values_by_col_name_sheet_name(workbook, sheet_name_candidates, col_name_candidates):\n",
    "    \"\"\"\n",
    "    the assumption is that we are looking for a colum from one sheet\n",
    "    but the sheet and colum name might have one of a number of differnet values, but only one of them. \n",
    "    \"\"\"\n",
    "    sheet_names = get_sheet_names(wb)\n",
    "    for index, sheet_name in enumerate(sheet_names):\n",
    "        if sheet_name in sheet_name_candidates:\n",
    "            sheet = wb.sheet_by_name(sheet_name)\n",
    "            col_values = get_col_values_by_col_name(sheet, col_name_candidates)\n",
    "            return col_values\n",
    "    return None # if we don't find the sheet, return None "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# use abstract grabber to get abstracts from Christine's files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "209\n",
      "175\n",
      "\n",
      "427\n",
      "369\n",
      "\n",
      "991\n",
      "707\n",
      "\n",
      "1082\n",
      "720\n",
      "\n"
     ]
    }
   ],
   "source": [
    "all_test_abstracts = []\n",
    "all_selected_abstracts = []\n",
    "for file in christine_files:\n",
    "    wb = get_workbook(file)\n",
    "    original_sheet_candidates = [\"original\"]\n",
    "    original_abstracts = get_col_values_by_col_name_sheet_name(wb, original_sheet_candidates, abstract_col_names)\n",
    "    selected_abstracts = get_col_values_by_col_name_sheet_name(wb, sheet_names_with_selections, abstract_col_names)\n",
    "    if original_abstracts and selected_abstracts:\n",
    "        all_test_abstracts.extend(original_abstracts)\n",
    "        all_selected_abstracts.extend(selected_abstracts)\n",
    "        print(len(all_test_abstracts))\n",
    "        print(len(all_selected_abstracts))\n",
    "        print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "715\n"
     ]
    }
   ],
   "source": [
    "print(len(list(set(all_selected_abstracts))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# let's remove duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_abstracts= list(set(all_test_abstracts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_selected_abstracts = list(set(all_selected_abstracts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "rejected_abstracts = []\n",
    "for abstract in unique_abstracts:\n",
    "    if abstract not in unique_selected_abstracts:\n",
    "        rejected_abstracts.append(abstract)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1071\n",
      "715\n",
      "356\n"
     ]
    }
   ],
   "source": [
    "print(len(unique_abstracts))\n",
    "print(len(unique_selected_abstracts))\n",
    "print(len(rejected_abstracts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'unique_selected_abstracts' (list)\n",
      "Stored 'rejected_abstracts' (list)\n"
     ]
    }
   ],
   "source": [
    "%store unique_selected_abstracts\n",
    "%store rejected_abstracts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Respondent-driven sampling (RDS) is an approach to sampling design and analysis which utilizes the networks of social relationships that connect members of the target population, using chain-referral. RDS sampling will typically oversample participants with many acquaintances. Naive estimators, such as the sample average, will thus be biased towards the state of the most highly connected individuals. Current methodology cannot estimate population size from RDS, and promotes inverse probability weighted estimators for population parameters such as HIV prevalence. We propose to use the timing of recruitment, typically collected and discarded, in order to estimate the population size via a counting process model. Once population size and degree frequencies are made available, prevalence can be debiased in a post-stratified framework. We adapt methods developed for inference in epidemiology and software reliability to estimate the population size, degree counts and frequencies. A fundamental advantage of our approach is that it makes the assumptions of the sampling design explicit. This enables verification of the assumptions, maximum likelihood estimation, extension with covariates, and model selection. We develop large-sample theory, proving consistency and asymptotic normality. We further compare our estimators to other estimators in the RDS literature, through simulation and real-world data. In both cases, we find our estimators to outperform current methods. The likelihood problem in the model we present is separable, and thus efficiently solvable. We implement these estimators in an accompanying R package, chords, available on CRAN. \n",
      "\n",
      "Purpose: Post-traumatic stress disorder (PTSD) and post-traumatic growth (PTG) are two different outcomes that may occur after experiencing traumatic events. Meanwhile, the traumatic exposure level and emotion response played an important role in the process. The present study first evaluated the relationship between PTSD, PTG, and traumatic exposure level and then compared the characteristics of emotional response through response time of the affective priming paradigm. Methods: For the purpose of evaluating the relationship between PTSD, PTG, and trauma exposure level, a sample of 2,395 participants completed measures of posttraumatic stress disorder Checklist-Civilian Version (PCL-C), Post-traumatic Growth Inventory (PTGI) and a trauma exposure-related survey, and Pearson's correlation analysis for the scales were conducted. In order to compare the characteristics of emotional response between PTSD and PTG, we randomly selected 90 participants and divided them into groups of PTSD, PTG, and control according the scores of PCL-C and PTGI, then the 90 participants were asked to do the affective priming task and the response time was recorded, at last analysis of variance was employed to analyze the data. Results: The results indicated that PTSD was not correlated with PTG. It was positively correlated with the traumatic exposure level, but PTG was not observed in this phenomenon. Finally, the data of response time showed that PTSD required more time to do the priming task and PTG demonstrated no difference compared to the control group. Conclusion: Combined with previous research findings, the relationship between PTSD and PTG may depend on the type and severity of the trauma, the exposure level, and other such parameters. In terms of positive outcome of trauma PTG displayed no changes of emotional performance from the perspective of behavior. The preliminary results suggested that PTG was more related to a self-reported or self-experienced state. \n",
      "\n",
      "Background: The impact of intimate partner violence (IPV) on postpartum depression (PPD) has been reported in various countries by many studies. However, the association between IPV and PPD in Japan has been scarce. In addition to the limited number of research on the relationship between IPV and PPD, the number of women seeking help from IPV support centers has been steadily increasing in Japan. Hence, it is of interest to explore the relationship between IPV during pregnancy and PPD in Japan. Methods: Four-page questionnaires assessing sociodemographic characteristics, women's personal situation during pregnancy, and PPD were mailed to participants prior to the checkup and collected at the checkup sites or mailed back to the health center. Of 9,707 eligible mothers, 6,590 responded to a questionnaire at a 3- or 4-month infant health checkup (response rate: 68%). Verbal and physical IPV from partners was assessed with two questions in the questionnaire. Logistic regression analysis was conducted. PPD was evaluated using the Edinburgh Postnatal Depression Scale (EPDS) with a cutoff score of 8/9. Results: Partners' verbal and physical abuse during pregnancy was significantly associated with PPD after adjusting for possible confounders. Specifically, odds ratios (ORs) of PPD for women who had been verbally abused by their partners during pregnancy at a frequency of \"often\" were 4.85 (95% CI, 2.23-10.55). ORs of PPD among women who had been physically abused by their partners during pregnancy at a frequency of \"sometimes or often\" were 7.05 (95% CI, 2.76-17.98). A positive dose-response relationship between both types of IPV and PPD was statistically significant (both p < 0.001). In addition, about 80% of physically abused women reported being verbally abused as well, indicating that these forms of IPV were highly comorbid. Conclusion: Both verbal and physical IPV during pregnancy is associated with PPD in Japan. This is the first study investigating the impact of IPV on PPD using a large number of subjects in the country. Further study using the same participants of the current study would allow us to explore the causality between IPV during pregnancy and PPD. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for x in unique_selected_abstracts[0:3]: print(x, \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pharmaceutical sector of Syrian Arab Republic before the war was characterized by bold and successful development since the late 1980s. With the beginning of war in the country back in March 2011, momentum has changed significantly. Traumatism, communicable diseases related to morbidity and mortality as well as wound infections became particularly hot public health concern. This relates not only to the direct victims of military conflict but also to the displaced civilians, refugees, and ordinary citizens alike. Evolving legislative framework in Syria since 1980s tolerated dispensing of antibiotics without appropriate prescription. Such practice led to spreading of antibiotic resistance among the local bacteria frequently causing both community-acquired and nosocomial infections. Laboratory findings of resistant bacteria strains among the Syrian refugees in some European countries serve as evidence of concern spreading far beyond Middle East. Practice of self-diagnosis and self-medication with antibiotics by patients themselves and restraint to pharmacist advice is widespread. A number of recommendations is presented to stakeholders to compact antibiotic resistance after the peace is established in the country. The successful implementation of such recommendations is the way to preserve shrinking golden reserve of highly potent antibiotics as it is the last defense line against resistant bacterial strains causing severe life-threatening infections. \n",
      "\n",
      "Psychological stressors can contribute to adverse health outcomes and lead to health disparities. To examine associations among psychological stressors, coping, blood pressure, body mass index, and body fat in ethnic minority women, we conducted a secondary analysis using data from 178 African American and Hispanic/Latina women who completed measures of perceived racial discrimination and stress, coping, blood pressure, and body composition. The mean age of participants was 45.3 (+/- 9.3 years), and most were obese (74.2%) and had prehypertensive systolic blood pressure (125.7 +/- 14.6 mmHg). Hierarchical multiple regression models indicated a significant negative relationship between racial discrimination and percent body fat, and positive associations between stress and blood pressure. Coping did not moderate the association between racial discrimination and blood pressure or body composition. Health care providers should consider psychological stressors as underlying causes for hypertension and address tailored stress-reduction coping strategies when treating African American and Hispanic/Latina women with hypertension. \n",
      "\n",
      "BackgroundAfter cryptosporidiosis was reported in three workers caring for preweaned calves at an academic research laboratory, we sought to identify cases, determine risk factors, and implement control measures. MethodsA cryptosporidiosis case was defined as diarrhea duration 72hr, abdominal cramps, or vomiting in an animal research laboratory worker during July 14-July 31. A confirmed case had laboratory evidence of Cryptosporidium infection. Staff were interviewed regarding illness, potential exposures, training, and personal protective equipment (PPE) standard operating procedures (SOPs). ResultsThe cryptosporidiosis attack rate (AR) was 74% (20/27); five were laboratory-confirmed. Median job training was 2hr including respiratory-fit testing. No SOPs existed for doffing PPE. AR for workers who removed their gloves first was 84% (16/19) compared with 20% (1/5) for workers who removed gloves last (risk ratio=4.2; P<0.02). ConclusionsThis outbreak highlights the importance of adequate training, enforced proper PPE procedures, and promoting a culture of safety. Am. J. Ind. Med. 60:208-214, 2017. (c) 2017 Wiley Periodicals, Inc. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for x in rejected_abstracts[0:3]: print(x, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# scramble our lists and get our holdouts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.shuffle(unique_selected_abstracts)\n",
    "random.shuffle(rejected_abstracts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "HOLDOUT_NUMBER = 100\n",
    "holdout_selected_abstracts = unique_selected_abstracts[0:HOLDOUT_NUMBER]\n",
    "training_selected_abstracts = unique_selected_abstracts[HOLDOUT_NUMBER:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "715\n",
      "100\n",
      "615\n"
     ]
    }
   ],
   "source": [
    "print(len(unique_selected_abstracts))\n",
    "print(len(holdout_selected_abstracts))\n",
    "print(len(training_selected_abstracts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'unique_selected_abstracts' (list)\n"
     ]
    }
   ],
   "source": [
    "%store unique_selected_abstracts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Respondent-driven sampling (RDS) is an approach to sampling design and analysis which utilizes the networks of social relationships that connect members of the target population, using chain-referral. RDS sampling will typically oversample participants with many acquaintances. Naive estimators, such as the sample average, will thus be biased towards the state of the most highly connected individuals. Current methodology cannot estimate population size from RDS, and promotes inverse probability weighted estimators for population parameters such as HIV prevalence. We propose to use the timing of recruitment, typically collected and discarded, in order to estimate the population size via a counting process model. Once population size and degree frequencies are made available, prevalence can be debiased in a post-stratified framework. We adapt methods developed for inference in epidemiology and software reliability to estimate the population size, degree counts and frequencies. A fundamental advantage of our approach is that it makes the assumptions of the sampling design explicit. This enables verification of the assumptions, maximum likelihood estimation, extension with covariates, and model selection. We develop large-sample theory, proving consistency and asymptotic normality. We further compare our estimators to other estimators in the RDS literature, through simulation and real-world data. In both cases, we find our estimators to outperform current methods. The likelihood problem in the model we present is separable, and thus efficiently solvable. We implement these estimators in an accompanying R package, chords, available on CRAN.',\n",
       " \"Purpose: Post-traumatic stress disorder (PTSD) and post-traumatic growth (PTG) are two different outcomes that may occur after experiencing traumatic events. Meanwhile, the traumatic exposure level and emotion response played an important role in the process. The present study first evaluated the relationship between PTSD, PTG, and traumatic exposure level and then compared the characteristics of emotional response through response time of the affective priming paradigm. Methods: For the purpose of evaluating the relationship between PTSD, PTG, and trauma exposure level, a sample of 2,395 participants completed measures of posttraumatic stress disorder Checklist-Civilian Version (PCL-C), Post-traumatic Growth Inventory (PTGI) and a trauma exposure-related survey, and Pearson's correlation analysis for the scales were conducted. In order to compare the characteristics of emotional response between PTSD and PTG, we randomly selected 90 participants and divided them into groups of PTSD, PTG, and control according the scores of PCL-C and PTGI, then the 90 participants were asked to do the affective priming task and the response time was recorded, at last analysis of variance was employed to analyze the data. Results: The results indicated that PTSD was not correlated with PTG. It was positively correlated with the traumatic exposure level, but PTG was not observed in this phenomenon. Finally, the data of response time showed that PTSD required more time to do the priming task and PTG demonstrated no difference compared to the control group. Conclusion: Combined with previous research findings, the relationship between PTSD and PTG may depend on the type and severity of the trauma, the exposure level, and other such parameters. In terms of positive outcome of trauma PTG displayed no changes of emotional performance from the perspective of behavior. The preliminary results suggested that PTG was more related to a self-reported or self-experienced state.\",\n",
       " 'Background: The impact of intimate partner violence (IPV) on postpartum depression (PPD) has been reported in various countries by many studies. However, the association between IPV and PPD in Japan has been scarce. In addition to the limited number of research on the relationship between IPV and PPD, the number of women seeking help from IPV support centers has been steadily increasing in Japan. Hence, it is of interest to explore the relationship between IPV during pregnancy and PPD in Japan. Methods: Four-page questionnaires assessing sociodemographic characteristics, women\\'s personal situation during pregnancy, and PPD were mailed to participants prior to the checkup and collected at the checkup sites or mailed back to the health center. Of 9,707 eligible mothers, 6,590 responded to a questionnaire at a 3- or 4-month infant health checkup (response rate: 68%). Verbal and physical IPV from partners was assessed with two questions in the questionnaire. Logistic regression analysis was conducted. PPD was evaluated using the Edinburgh Postnatal Depression Scale (EPDS) with a cutoff score of 8/9. Results: Partners\\' verbal and physical abuse during pregnancy was significantly associated with PPD after adjusting for possible confounders. Specifically, odds ratios (ORs) of PPD for women who had been verbally abused by their partners during pregnancy at a frequency of \"often\" were 4.85 (95% CI, 2.23-10.55). ORs of PPD among women who had been physically abused by their partners during pregnancy at a frequency of \"sometimes or often\" were 7.05 (95% CI, 2.76-17.98). A positive dose-response relationship between both types of IPV and PPD was statistically significant (both p < 0.001). In addition, about 80% of physically abused women reported being verbally abused as well, indicating that these forms of IPV were highly comorbid. Conclusion: Both verbal and physical IPV during pregnancy is associated with PPD in Japan. This is the first study investigating the impact of IPV on PPD using a large number of subjects in the country. Further study using the same participants of the current study would allow us to explore the causality between IPV during pregnancy and PPD.',\n",
       " \"Retrospectively measuring markers on stored baseline samples from participants in a randomized controlled trial (RCT) may provide high quality evidence as to the value of the markers for treatment selection. Originally developed for approximating gene-environment interactions in the odds ratio scale, the case-only method has recently been advocated for assessing gene-treatment interactions on rare disease endpoints in randomized clinical trials. In this article, the case-only approach is shown to provide a consistent and efficient estimator of marker by treatment interactions and marker-specific treatment effects on the relative risk scale. The prohibitive rare-disease assumption is no longer needed, broadening the utility of the case-only approach. The case-only method is resource-efficient as markers only need to be measured in cases only. It eliminates the need to model the marker's main effect, and can be used with any parametric or nonparametric learning method. The utility of this approach is illustrated by an application to genetic data in the Women's Health Initiative (WHI) hormone therapy trial.\"]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_selected_abstracts[0:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we have a couple of duplicate abstracts in our data, boo\n",
    "\n",
    "we have about 1015 uniqe abstracts, of which 715 have been selected.\n",
    "\n",
    "need to pick out a set as a holdout to test the trainig algorithim with \n",
    "\n",
    "I'm going to pick 100 selected, and 100 non selected abstracts as my holdout, I'm also going to randomize the selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps:  \n",
    "\n",
    "- learn about what format SAGE maker needs for training \n",
    "- work though the two lists creating a tagged set of the format that SAGE maker needs  \n",
    "- take out a 100 item set for both the selected and unselcted ones. \n",
    "- shuffle the training set  \n",
    "- send the training set to SAGE maker   \n",
    "- send the holdouts to SAGE maker to see how the model preforms  \n",
    "- repeat the above for titles "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}