Final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One Thousand and One Nights (Arabian nights), is a collection of Middle Eastern folk tales compiled in Arabic during the Islamic Golden Age.\n",
    "\n",
    "It is still not clear where does exactly these stories are originated? Is it Persia, or Arabia?\n",
    "\n",
    "The work was collected over many centuries by various authors, translators, and scholars across West, Central and South Asia, and North Africa. Some tales themselves trace their roots back to ancient and medieval Arabic, Persian, Indian, Greek, Jewish and Turkish folklore and literature.\n",
    "\n",
    "To answer the long-lived question, I implemented two different Natural Language Processing (NLP) methods:\n",
    "\n",
    "1. Topic modelling using LDA (Latent Dirichlet Allocation)\n",
    "2. Word2vec neural language model\n",
    "\n",
    "In both methods, Preprcessing, and cleaning of the data is needed. Data cleaning starts with tokenization, followed by lemmatization.\n",
    "\n",
    "Tokenizing the text:\n",
    "\n",
    "Words are features in text which carries information. Tokenization means to give each word its own identity. \n",
    "\n",
    "Lemmatization:\n",
    "\n",
    "Grouping together the inflected forms of a word so they can be analysed as a single item, identified by the word's lemma, or dictionary form.\n",
    "\n",
    "Removing stop words:\n",
    "\n",
    "A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) which won't add any meaning to the text and should be ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\fsaff\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\fsaff\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "# -*- coding: utf-8 -*-\n",
    "\"\"\"\n",
    "Created on Fri Feb 28 10:20:41 2020\n",
    "\n",
    "@author: fsaff\n",
    "\"\"\"\n",
    "\n",
    "import spacy\n",
    "import gensim\n",
    "spacy.load('en')\n",
    "from spacy.lang.en import English\n",
    "parser = English()\n",
    "\n",
    "# Tokenizing the text meaning to give each word its own identity\n",
    "def tokenize(text):\n",
    "    lda_tokens = []\n",
    "    tokens = parser(text)\n",
    "    for token in tokens:\n",
    "        if token.orth_.isspace():\n",
    "            continue\n",
    "        else:\n",
    "            lda_tokens.append(token.lower_)\n",
    "    return lda_tokens\n",
    "\n",
    "import nltk\n",
    "nltk.download('wordnet')\n",
    "\n",
    "# Present each word based on the root and meaning\n",
    "from nltk.corpus import wordnet as wn\n",
    "def get_lemma(word):\n",
    "    lemma = wn.morphy(word)\n",
    "    if lemma is None:\n",
    "        return word\n",
    "    else:\n",
    "        return lemma\n",
    "    \n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "def get_lemma2(word):\n",
    "    return WordNetLemmatizer().lemmatize(word)\n",
    "# adding stopwords which are thw words that would not add much meaning to overall meaning\n",
    "nltk.download('stopwords')\n",
    "from nltk.corpus import stopwords\n",
    "en_stop = set(nltk.corpus.stopwords.words('english'))\n",
    "HHH=list(en_stop)\n",
    "newStopWords = ['would','could','shall','can','without','nothing','going','allow','ask','saying','hear','thousand','eveything']\n",
    "stopwords=HHH+newStopWords\n",
    "\n",
    "def prepare_text_for_lda(text):\n",
    "    tokens = tokenize(text)\n",
    "    tokens = [token for token in tokens if len(token) > 4]\n",
    "    tokens = [token for token in tokens if token not in stopwords]\n",
    "    tokens = [get_lemma(token) for token in tokens]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LOADING THE TRANSLATION FILES:\n",
    "\n",
    "#I collected two different translation of the Arabian nights in which one is loger with more than 1000000 tokens. Brief version, however, only contains ~200000 tokens.\n",
    "\n",
    "Here, for the sake of time, only the brief version is considered.\n",
    "\n",
    "In this section, all pre-processing functions are applied to the text and words in the sentences are arranged in a list of lits. The list of lists will then be used for a topic modelling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['king', 'gazelle', 'looking', 'telling', 'bring', 'herdsman', 'sitting', 'something', 'please', 'reward', 'agree', 'master', 'daughter', 'young', 'teach', 'magic', 'woman', 'stay', 'yesterday', 'cover', 'tears', 'burst', 'laughter', 'father', 'cheap', 'bring', 'strange', 'strange', 'ask', 'laugh', 'crying', 'master', 'spell', 'mother', 'father', 'laugh', 'reason', 'father', 'kill', 'mother', 'astonish', 'found', 'morning']\n",
      "['noble', 'patience', 'resolute']\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "['hearing', 'voice']\n",
      "['whoever', 'people', 'secret']\n",
      "['birth', 'hero']\n",
      "[]\n",
      "['story', 'marvellous', 'write', 'needle', 'inner', 'corner', 'serve', 'warning', 'study', 'seven', 'reading', 'quran', 'book', 'discuss', 'learning', 'study', 'astronomy', 'poetry', 'branch', 'knowledge', 'surpass', 'people', 'calligraphy', 'unrivalled', 'spread', 'land', 'among', 'king', 'india', 'hear', 'messenger', 'father', 'together', 'gift', 'present', 'suitable', 'royalty', 'father', 'equip', 'ship', 'month', 'voyage']\n",
      "['except', 'pierce', 'heart', 'blood']\n",
      "['patience', 'nature', 'treacherous']\n",
      "['ifrit', 'whore', 'lover', 'look', 'recognize', 'never', 'spite', 'punishment', 'confess', 'ask', 'insist', 'never', 'ifrit', 'sword', 'sword', 'stand', 'gesture', 'eyebrow', 'tears', 'cheek', 'understand', 'gesture', 'reply', 'forgiveness', 'inwardly', 'recite']\n",
      "[]\n",
      "['reason', 'hands', 'lift']\n",
      "['continue', 'serve', 'companion', 'thirty', 'night', 'fortieth', 'youth', 'gladness', 'thanks', 'brother', 'save', 'death', 'blessing', 'blessing', 'bring', 'arrival', 'restore', 'ask', 'water', 'willingly', 'agree', 'warm', 'great', 'quantity', 'water', 'bring', 'using', 'lupin', 'flour', 'help', 'rubbing', 'bringing', 'change', 'clothes', 'couch', 'sleep', 'brother', 'melon', 'dissolve', 'sugar', 'juice', 'store', 'cupboard', 'found', 'melon', 'plate', 'master', 'knife', 'shelf', 'reply', 'quickly', 'knife', 'sheath', 'trip', 'knife', 'youth', 'accordance', 'eternal', 'decree', 'quickly', 'penetrate', 'heart']\n",
      "[]\n",
      "['house', 'turn', 'caliph', 'ja‘far', 'masrur', 'story', 'ja‘far', 'forward', 'story', 'doorkeeper', 'enter', 'hear', 'allow', 'leave', 'outside', 'caliph', 'ask', 'dervish', 'propose', 'break', 'spend', 'night', 'ja‘far', 'bring', 'morning', 'write', 'happen', 'ja‘far', 'caliph', 'palace', 'found', 'unable', 'sleep', 'night']\n",
      "['lightness', 'spirit', 'beauty', 'encompass']\n"
   
     ]
    }
   ],
   "source": [
    "import random\n",
    "text_data = []\n",
    "import re\n",
    "#uploading the file and preprocessing/cleaning sentence by sentence\n",
    "with open('Arabian-Nights1N.txt') as f:\n",
    "    for line in f:\n",
    "        f=str(f)\n",
    "        re.sub(\"\\w*/(?!NOUN)[A-Z]*\",\"NIL\",f)\n",
    "        tokens = prepare_text_for_lda(line)\n",
    "        if random.random() > .99:\n",
    "            print(tokens)\n",
    "            text_data.append(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LDA MODEL:\n",
    "\n",
    "LDA will start with collecting random words \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, '0.010*\"night\" + 0.008*\"bring\" + 0.007*\"slave\" + 0.006*\"father\" + 0.006*\"ask\" + 0.006*\"house\" + 0.006*\"purse\" + 0.006*\"servant\" + 0.005*\"morning\" + 0.005*\"black\" + 0.005*\"master\" + 0.005*\"hundred\" + 0.005*\"break\" + 0.004*\"continue\" + 0.004*\"dawn\"'), (1, '0.007*\"sharkan\" + 0.006*\"dirham\" + 0.006*\"rider\" + 0.005*\"champion\" + 0.005*\"happen\" + 0.005*\"young\" + 0.005*\"night\" + 0.005*\"palace\" + 0.005*\"ja‘far\" + 0.004*\"horse\" + 0.004*\"attack\" + 0.004*\"sword\" + 0.004*\"fighting\" + 0.004*\"kingdom\" + 0.004*\"reply\"'), (2, '0.009*\"father\" + 0.009*\"tears\" + 0.008*\"ask\" + 0.006*\"bring\" + 0.006*\"truth\" + 0.006*\"prince\" + 0.006*\"reply\" + 0.006*\"disobedience\" + 0.006*\"hear\" + 0.005*\"ifrit\" + 0.005*\"brother\" + 0.005*\"return\" + 0.005*\"horse\" + 0.004*\"janshah\" + 0.004*\"palace\"')]\n"
     ]
    }
   ],
   "source": [
    "from gensim import corpora\n",
    "from gensim.corpora import Dictionary\n",
    "words = corpora.Dictionary(text_data)\n",
    "corpus = [words.doc2bow(doc) for doc in text_data]\n",
    "\n",
    "lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,\n",
    "                                           id2word=words,\n",
    "                                           num_topics=3, \n",
    "                                           random_state=2,\n",
    "                                           update_every=1,\n",
    "                                           passes=400,\n",
    "                                           alpha='auto',\n",
    "                                           per_word_topics=True)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results:\n",
    "\n",
    "Disadvantages of the LDA model:\n",
    "\n",
    "1. The meaning of each topic usually spills into other topics which will make topics hard to interpret\n",
    "2. In this case, I could NOT find a topic directly related to \"Persianess\" or \"Arabness\" of each tale\n",
    "\n",
    "Advantages of LDA model:\n",
    "1. It takes advantage of statistical features of a text\n",
    "2. It is usefull to be used for text summarization and topic modelling\n",
    "\n",
    "In the next section we explore the Word2vec neural language model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the era of blooming artificial intelligence (AI) sciences, I introduced an effective Natural language processing (NLP) method to examine the origin of Arabian nights. Furthermore, to explore which cultures have had a more substantial influence on the two major translated English versions of Arabian nights? The highly controversial yet still obscure origins of The Arabian Nights have been rooted in various countries and cultures, including ancient Egypt, India, pre-Islamic Iran, and Arab cultures of the Middle East. The translations used here are Muhsin Mahdi’s Critical Edition (translated by Hussein Haddaway (2011)) and Arabian Nights (2008) translated by Lyons. To find reliable answers to this question we had to go beyond conventional LDA approaches or statistical classifications such as TF/IDF. Earlier statistical methods ignore the effect the neighbors of a word have on its meaning and how those relationships affect the overall meaning of a statement. Using word2vec approach, however,    considers the meaning of neighborhood words around the specific words and showed it to be extremely effective. Latter is done by creating the small bag of words (BOG) of tokens from a “neighborhood” of few words, typically fewer than 10 tokens. The neural language model ensures that these neighborhoods of meaning don’t spill over into adjacent sentences which guarantees the independence and relevance of BOGs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy \n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from spacy.tokenizer import Tokenizer\n",
    "from spacy.lang.en import English\n",
    "sp = spacy.load('en_core_web_sm')\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "file_content = open('Arabian Nights Text A-2020.txt', encoding=\"utf8\").read()\n",
    "file_content=file_content.lower()\n",
    "file_content1=file_content.split()\n",
    "f=[]\n",
    "f=list(file_content1)\n",
    "f2=pd.DataFrame(columns={\"A\"})\n",
    "f2[\"A\"]=f\n",
    "\n",
    "f2['A'] = f2['A'].str.replace('\\d+', '')\n",
    "#Preprocessing of a text and filtering \n",
    "f2=(f2[f2['A'].apply(lambda x: len(x.split('-')) < 2)])\n",
    "f2=(f2[f2['A'].apply(lambda x: len(x.split(',')) < 2)])\n",
    "f2=(f2[f2['A'].apply(lambda x: len(x.split(':')) < 2)])\n",
    "f2=(f2[f2['A'].apply(lambda x: len(x.split('\"')) < 2)])\n",
    "\n",
    "f3=str(f2)\n",
    "\n",
    "parser = English()\n",
    "\n",
    "document = nlp(f3)\n",
    "\n",
    "lemmas = [token.lemma_ for token in document if not token.is_stop]\n",
    "\n",
    "import spacy \n",
    "nlp = spacy.load(\"en\", disable=['parser', 'tagger', 'ner'])\n",
    "from nltk.corpus import stopwords\n",
    "#updating stopwords\n",
    "stops = stopwords.words(\"english\")\n",
    "\n",
    "def normalize(comment, lowercase, remove_stopwords):\n",
    "    if lowercase:\n",
    "        comment = comment.lower()\n",
    "    comment = nlp(comment)\n",
    "    lemmatized = list()\n",
    "    for word in comment:\n",
    "        lemma = word.lemma_.strip()\n",
    "        if lemma:\n",
    "            if not remove_stopwords or (remove_stopwords and lemma not in stops):\n",
    "                lemmatized.append(lemma)\n",
    "    return \" \".join(lemmatized)\n",
    "\n",
    "\n",
    "\n",
    "f=pd.DataFrame()\n",
    "f[\"A\"]=f2[\"A\"]\n",
    "f['A']= f['A'].apply(normalize, lowercase=True, remove_stopwords=True)\n",
    "f['A'].replace('', np.nan, inplace=True)\n",
    "f.dropna(subset=['A'], inplace=True)\n",
    "\n",
    "\n",
    "LLL=[]\n",
    "LLL=list(f[\"A\"])\n",
    "writer = pd.ExcelWriter('TextA.xlsx', engine='xlsxwriter')\n",
    "writer0 = pd.ExcelWriter('errorA.xlsx', engine='xlsxwriter')\n",
    "writer2 = pd.ExcelWriter('BestA13.xlsx', engine='xlsxwriter')\n",
    "\n",
    "NNNN=300\n",
    "new_lst=list(range(0,NNNN))\n",
    "import gensim \n",
    "from gensim.models import Word2Vec \n",
    "from gensim.models import LdaModel\n",
    "import random\n",
    "S1=[]\n",
    "S2=[]\n",
    "S3=[]\n",
    "S4=[]\n",
    "for iii in range (1,NNNN):\n",
    "\n",
    "# Training the neural language network in a loop using Skip-gram method\n",
    "    NN=random.randint(1,101)\n",
    "    model3 = gensim.models.Word2Vec([LLL],min_count = 1, \n",
    "\t\t\t\t\t seed=NN,sg=0, size = 150, window = 5)\n",
    "    model3.build_vocab([LLL], update=True)\n",
    "\n",
    "# Finding the similarities between the-most-common-words and \"Persian\", \"Arab\", \"Persins\", and \"Arabs\".    \n",
    "    S1.append(model3.similarity(\"persian\",\"man\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"man\"))     \n",
    "    S3.append(model3.similarity(\"persians\",\"man\")) \n",
    "    S4.append(model3.similarity(\"arabic\",\"man\"))  \n",
    "   \n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"king\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"king\"))\n",
    "    S3.append(model3.similarity(\"persians\",\"king\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"king\"))\n",
    "   \n",
    " \n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"god\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"god\")) \n",
    "    \n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"girl\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"girl\")) \n",
    "    \n",
    "\n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"morning\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"morning\"))\n",
    "    \n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"night\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"night\"))\n",
    "    \n",
    "\n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"love\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"love\"))  \n",
    "    \n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"old\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"old\"))  \n",
    "    \n",
    "\n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"fortunate\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"fortunate\")) \n",
    "    \n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"slave\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"slave\"))  \n",
    "   \n",
    " \n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"vizier\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"vizier\"))  \n",
    "   \n",
    "\n",
    "   \n",
    "    S1.append(model3.similarity(\"persian\",\"father\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"father\")) \n",
    "    \n",
    "\n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"young\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"young\"))  \n",
    "    \n",
    "\n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"heart\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"heart\"))\n",
    "    \n",
    "\n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"kiss\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"kiss\"))  \n",
    "   \n",
    "    S3.append(model3.similarity(\"persians\",\"god\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"god\"))  \n",
    "\n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"girl\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"girl\"))  \n",
    "\n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"morning\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"morning\"))  \n",
    "\n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"night\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"night\"))  \n",
    " \n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"love\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"love\"))  \n",
    "\n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"old\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"old\"))  \n",
    " \n",
    "\n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"fortunate\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"fortunate\"))  \n",
    "\n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"slave\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"slave\"))  \n",
    "\n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"vizier\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"vizier\"))  \n",
    " \n",
    "   \n",
    "    S3.append(model3.similarity(\"persians\",\"father\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"father\"))  \n",
    " \n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"young\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"young\"))  \n",
    "\n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"heart\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"heart\"))  \n",
    "\n",
    "\n",
    "    S3.append(model3.similarity(\"persians\",\"kiss\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"kiss\")) \n",
    "    \n",
    "    S1.append(model3.similarity(\"persian\",\"tell\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"tell\"))     \n",
    "    S3.append(model3.similarity(\"persians\",\"tell\")) \n",
    "    S4.append(model3.similarity(\"arabic\",\"tell\"))  \n",
    "   \n",
    "\n",
    "    S1.append(model3.similarity(\"persian\",\"shahrazad\")) \n",
    "    S2.append(model3.similarity(\"arab\",\"shahrazad\"))\n",
    "    S3.append(model3.similarity(\"persians\",\"shahrazad\")) \n",
    "    S4.append(model3.similarity(\"arabs\",\"shahrazad\"))\n",
    "\n",
    "\n",
    "# Postprocessing and structuring the data for visualization and statistical analysis\n",
    "NS1=np.array([S1,S3])\n",
    "MNS1=np.array([])\n",
    "MNS1=np.average(NS1, axis=0)\n",
    "LL1n=MNS1.reshape(17,NNNN-1)\n",
    "LL1=np.average(LL1n, axis=1)\n",
    "SLL1=np.std(LL1n, axis=1)\n",
    "DMNS1=pd.DataFrame(LL1)\n",
    "\n",
    "NS2=np.array([S2,S4])\n",
    "MNS2=np.array([])\n",
    "MNS2=np.average(NS2, axis=0)\n",
    "LL2n=MNS2.reshape(17,NNNN-1)\n",
    "LL2=np.average(LL2n, axis=1)\n",
    "SLL2=np.std(LL2n, axis=1)\n",
    "DMNS2=pd.DataFrame(LL2)\n",
    "\n",
    "MNS1=np.average(MNS1, axis=0)\n",
    "SS=pd.DataFrame(S1)\n",
    "SS1=pd.DataFrame(S2)\n",
    "SS2=pd.DataFrame(S3)\n",
    "II=1     \n",
    "pd.DataFrame(LL1).to_excel(writer,index = False,  sheet_name= 'run%d' %(II)) \n",
    "pd.DataFrame(LL2).to_excel(writer,index = False,  sheet_name= 'run%d' %(II+1)) \n",
    "pd.DataFrame(SLL1).to_excel(writer,index = False,  sheet_name= 'run%d' %(II+2)) \n",
    "pd.DataFrame(SLL2).to_excel(writer,index = False,  sheet_name= 'run%d' %(II+3)) \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "writer.save()\n",
    "writer0.save()\n",
    "\n",
    "\n",
    "\n",
    "import spacy\n",
    "from collections import Counter\n",
    "from collections import Counter\n",
    "word_freq = Counter(LLL)\n",
    "NN11=100\n",
    "common_words = word_freq.most_common(NN11)\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "plt.clf()\n",
    "\n",
    "labels = ['man', 'king', 'god', 'girl', 'morning']\n",
    "persian_means = [LL1[0], LL1[1], LL1[2], LL1[3], LL1[4]]\n",
    "Arab_means = [LL2[0], LL2[1], LL2[2], LL2[3], LL2[4]]\n",
    "error1=[SLL1[0],SLL1[1],SLL1[2],SLL1[3],SLL1[4]]\n",
    "        \n",
    "error2=[SLL2[0],SLL2[1],SLL2[2],SLL2[3],SLL2[4]]\n",
    "x = np.arange(len(labels))  \n",
    "width = 0.35  \n",
    "ind = np.arange(5)    \n",
    "width = 0.35      \n",
    "\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "rects1 = ax.bar(x - width/2, persian_means, width, yerr=error1,label='Persianness')\n",
    "rects2 = ax.bar(x + width/2, Arab_means, width,yerr=error2, label='Arabness')\n",
    "\n",
    "# Add some text for labels, title and custom x-axis tick labels, etc.\n",
    "ax.set_ylabel('Similarity')\n",
    "#ax.set_title('Scores by group and gender')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels)\n",
    "ax.legend()\n",
    "\n",
    "\n",
    "#plt.legend((p1[0], p2[0]), ('Men', 'Women'))\n",
    "\n",
    "\n",
    "\n",
    "fig.tight_layout()\n",
    "\n",
    "plt.show()\n",
    "\n",
    "\n",
    "plt.clf()\n",
    "\n",
    "labels = ['night', 'love', 'old', 'fortunate', 'slave']\n",
    "persian_means = [LL1[5], LL1[6], LL1[7], LL1[8], LL1[9]]\n",
    "Arab_means = [LL2[5], LL2[6], LL2[7], LL2[8], LL2[9]]\n",
    "error3=[SLL1[5],SLL1[6],SLL1[7],SLL1[8],SLL1[9]]\n",
    "        \n",
    "error4=[SLL2[5],SLL2[6],SLL2[7],SLL2[8],SLL2[9]]\n",
    "x = np.arange(len(labels))  # the label locations\n",
    "width = 0.35  # the width of the bars\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "rects1 = ax.bar(x - width/2, persian_means, width,yerr=error3, label='Persianness')\n",
    "rects2 = ax.bar(x + width/2, Arab_means, width,yerr=error4, label='Arabness')\n",
    "\n",
    "# Add some text for labels, title and custom x-axis tick labels, etc.\n",
    "ax.set_ylabel('Similarity')\n",
    "#ax.set_title('Scores by group and gender')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels)\n",
    "ax.legend()\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "fig.tight_layout()\n",
    "\n",
    "plt.show()\n",
    "\n",
    "plt.clf()\n",
    "\n",
    "labels = ['vizier', 'father', 'young', 'heart', 'kiss']\n",
    "persian_means = [LL1[10], LL1[11], LL1[12], LL1[13], LL1[14]]\n",
    "Arab_means = [LL2[10], LL2[11], LL2[12], LL2[13], LL2[14]]\n",
    "error5=[SLL1[10], SLL1[11], SLL1[12], SLL1[13], SLL1[14]]\n",
    "        \n",
    "error6=[SLL2[10], SLL2[11], SLL2[12], SLL2[13], SLL2[14]]\n",
    "x = np.arange(len(labels))  # the label locations\n",
    "width = 0.35  # the width of the bars\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "rects1 = ax.bar(x - width/2, persian_means, width, yerr=error5,label='Persianness')\n",
    "rects2 = ax.bar(x + width/2, Arab_means, width,yerr=error6, label='Arabness')\n",
    "\n",
    "# Add some text for labels, title and custom x-axis tick labels, etc.\n",
    "ax.set_ylabel('Similarity')\n",
    "#ax.set_title('Scores by group and gender')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels)\n",
    "ax.legend()\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "fig.tight_layout()\n",
    "\n",
    "plt.show()\n",
    "C=[]\n",
    "for i in range(0,NN11-1):\n",
    "    C.append(common_words[i][0])\n",
    "C1=str(C)\n",
    "from wordcloud import WordCloud\n",
    "import matplotlib.pyplot as plt \n",
    "import pandas as pd \n",
    "  \n",
    "wordcloud = WordCloud(width = 800, height = 800, \n",
    "                background_color ='white', \n",
    "                stopwords = stops, \n",
    "                min_font_size = 5).generate(C1) \n",
    "  \n",
    "# plot the WordCloud image                        \n",
    "plt.figure(figsize = (6, 6), facecolor = None) \n",
    "plt.imshow(wordcloud) \n",
    "plt.axis(\"off\") \n",
    "plt.tight_layout(pad = 0) \n",
    "  \n",
    "plt.show() \n",
    "\n",
    "from spacy.tokenizer import Tokenizer\n",
    "from spacy.lang.en import English\n",
    "sp = spacy.load('en_core_web_sm')\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "#Sentiment analysis of the text\n",
    "\n",
    "import textwrap\n",
    "\n",
    "FFF1=[]\n",
    "content=str(LLL)\n",
    "FFF1= textwrap.wrap(content,7700)\n",
    "\n",
    "S1=[]\n",
    "S2=[]\n",
    "from textblob import TextBlob\n",
    "for i in range(1,10):\n",
    "    blob=TextBlob(FFF1[i])\n",
    "    blob.tags \n",
    "    blob.noun_phrases\n",
    "    S1.append(blob.sentiment.polarity)\n",
    "    S2.append(blob.sentiment.subjectivity)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "for z in range(len(S1)):\n",
    "   x_data=[z]\n",
    "   y_data1=S1[z]\n",
    "   plt.plot(x_data,y_data1, \"r--o\") \n",
    "\n",
    "plt.ylabel('Sentiment-polarity')\n",
    "plt.xlabel('Docs')\n",
    "plt.show() \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "raw",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}