scrapper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Poet Poems Scraping from Ganjoor Website\n",
    "\n",
    "This Python script is designed to scrape poets' poems from the Ganjoor website. It provides a set of functions to extract poems, their excerpts, and related details from the website's pages, and save them into CSV files based on specified criteria.\n",
    "\n",
    "## Functionality\n",
    "\n",
    "The script comprises several functions:\n",
    "\n",
    "1. **`get_poem(url)`**: This function extracts a poem from a provided URL of a webpage containing the poem. It parses the HTML content, retrieves the verses, calculates the number of verses and words, and returns the poem text along with these details.\n",
    "\n",
    "2. **`get_book_urls(url, prefix=\"https://ganjoor.net\")`**: This function retrieves URLs of books from a given URL. It extracts the URLs from the webpage and appends them with a provided prefix, typically the Ganjoor website URL.\n",
    "\n",
    "3. **`get_part_urls(url, prefix=\"https://ganjoor.net\")`**: This function extracts URLs of each poem's full version from a webpage containing a collection of poem excerpts. It appends them with a given prefix to form complete URLs.\n",
    "\n",
    "4. **`get_poet(poet_links, directory=\"./result\", number_of_doc_words=700, print_details=False)`**: This main function takes a list of URLs of poet pages, extracts their poems, and saves them in CSV files according to specified criteria. It iterates over the poet links, retrieves the URLs of books written by each poet, then extracts the URLs of poem excerpts from each book. It further utilizes the `get_poem()` function to extract the poems, considering a maximum number of words per document. Finally, it saves the poems into CSV files and returns a list of dictionaries containing details about the processed documents.\n",
    "\n",
    "## Usage\n",
    "\n",
    "To use this script:\n",
    "- Provide a list of URLs of poet pages.\n",
    "- Specify the directory where the CSV files will be saved.\n",
    "- Optionally set the maximum number of words per document and whether to print details during processing.\n",
    "\n",
    "Example usage:\n",
    "\n",
    "```python\n",
    "poet_links = [\n",
    "        {\n",
    "            \"name\": \"attar\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"asrarname\",\n",
    "                    \"url\": \"https://ganjoor.net/attar/asrarname\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"\",\n",
    "                    \"url\": \"https://ganjoor.net/attar/manteghotteyr/naat\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"saadi\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"golestan\",\n",
    "                    \"url\": \"https://ganjoor.net/saadi/golestan\"\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "details = get_poet(poet_links, directory=\"./poems\", number_of_doc_words=800, print_details=True)\n",
    "print(\"Details:\", details)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "from src.scrape import *"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-05-29T21:52:56.983342Z",
     "end_time": "2024-05-29T21:52:57.060140Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"/home/amir/Documents/university/projects/NLP/IR-Project/persian_authors.csv\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-05-29T21:52:57.062364Z",
     "end_time": "2024-05-29T21:52:57.090801Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [
    {
     "data": {
      "text/plain": "array(['moulavi', 'attar', 'rahi', 'ferdousi', 'saeb', 'iqbal', 'saadi',\n       'nezami', 'jami', 'eraghi'], dtype=object)"
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['author'].unique()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-05-29T21:52:57.093391Z",
     "end_time": "2024-05-29T21:52:57.095299Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "data": {
      "text/plain": "array(['moulavi', 'manteghotteyr', 'rahi', 'ferdousi', 'saeb',\n       'asrar-khodi', 'romooz-bikhodi', 'payam-mashregh', 'saadi',\n       'nezami', 'jami', 'eraghi'], dtype=object)"
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['b_name'].unique()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-05-29T21:52:57.096895Z",
     "end_time": "2024-05-29T21:52:57.139748Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-29T21:52:57.139488Z",
     "end_time": "2024-05-29T21:52:57.139892Z"
    }
   },
   "outputs": [],
   "source": [
    "poets = [\n",
    "        {\n",
    "            \"name\": \"attar\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"asrarname\",\n",
    "                    \"url\": \"https://ganjoor.net/attar/asrarname\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"moulavi\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"shams\",\n",
    "                    \"url\": \"https://ganjoor.net/moulavi/shams\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"rahi\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"all-books\",\n",
    "                    \"url\": \"https://ganjoor.net/rahi\"\n",
    "                },\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"ferdousi\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"shahname\",\n",
    "                    \"url\": \"https://ganjoor.net/ferdousi/shahname\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"saeb\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"divan-saeb\",\n",
    "                    \"url\": \"https://ganjoor.net/saeb/divan-saeb\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"iqbal\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"all-books\",\n",
    "                    \"url\": \"https://ganjoor.net/iqbal\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"saadi\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"golestan\",\n",
    "                    \"url\": \"https://ganjoor.net/saadi/golestan\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"boostan\",\n",
    "                    \"url\": \"https://ganjoor.net/saadi/boostan\"\n",
    "                },\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"nezami\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"5ganj\",\n",
    "                    \"url\": \"https://ganjoor.net/nezami/5ganj\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"jami\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"divanj\",\n",
    "                    \"url\": \"https://ganjoor.net/jami/divanj\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"7ourang\",\n",
    "                    \"url\": \"https://ganjoor.net/jami/7ourang\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"eraghi\",\n",
    "            \"books\": [\n",
    "                {\n",
    "                    \"name\": \"divane\",\n",
    "                    \"url\": \"https://ganjoor.net/eraghi/divane\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"oshaghname\",\n",
    "                    \"url\": \"https://ganjoor.net/eraghi/oshaghname\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2024-05-29T21:52:57.139594Z",
     "end_time": "2024-05-29T22:17:11.357863Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/amir/Documents/university/projects/NLP/IR-Project/src/scrape.py:44: GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"html.parser\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
      "\n",
      "The code that caused this warning is on line 44 of the file /home/amir/Documents/university/projects/NLP/IR-Project/src/scrape.py. To get rid of this warning, pass the additional argument 'features=\"html.parser\"' to the BeautifulSoup constructor.\n",
      "\n",
      "  soup = BeautifulSoup(r.content)\n",
      "/home/amir/Documents/university/projects/NLP/IR-Project/src/scrape.py:67: GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"html.parser\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
      "\n",
      "The code that caused this warning is on line 67 of the file /home/amir/Documents/university/projects/NLP/IR-Project/src/scrape.py. To get rid of this warning, pass the additional argument 'features=\"html.parser\"' to the BeautifulSoup constructor.\n",
      "\n",
      "  soup = BeautifulSoup(r.content)\n",
      "/home/amir/Documents/university/projects/NLP/IR-Project/src/scrape.py:24: GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"html.parser\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
      "\n",
      "The code that caused this warning is on line 24 of the file /home/amir/Documents/university/projects/NLP/IR-Project/src/scrape.py. To get rid of this warning, pass the additional argument 'features=\"html.parser\"' to the BeautifulSoup constructor.\n",
      "\n",
      "  soup = BeautifulSoup(r.content)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://ganjoor.net//attar/asrarname/abkhsh1/sh1 : 1890\n",
      "https://ganjoor.net//attar/asrarname/abkhsh2/sh1 : 1287\n",
      "https://ganjoor.net//attar/asrarname/abkhsh2/sh2 : 1344\n",
      "https://ganjoor.net//attar/asrarname/abkhsh3/sh4 : 842\n",
      "https://ganjoor.net//attar/asrarname/abkhsh4/sh1 : 1199\n",
      "https://ganjoor.net//attar/asrarname/abkhsh5/sh1 : 1339\n",
      "https://ganjoor.net//attar/asrarname/abkhsh6/sh5 : 966\n",
      "https://ganjoor.net//attar/asrarname/abkhsh6/sh7 : 701\n",
      "https://ganjoor.net//attar/asrarname/abkhsh7/sh5 : 839\n",
      "https://ganjoor.net//attar/asrarname/abkhsh7/sh7 : 764\n",
      "https://ganjoor.net//attar/asrarname/abkhsh7/sh11 : 731\n",
      "https://ganjoor.net//attar/asrarname/abkhsh8/sh2 : 1107\n",
      "https://ganjoor.net//attar/asrarname/abkhsh9/sh2 : 811\n",
      "https://ganjoor.net//attar/asrarname/abkhsh9/sh5 : 795\n",
      "https://ganjoor.net//attar/asrarname/abkhsh10/sh3 : 719\n",
      "https://ganjoor.net//attar/asrarname/abkhsh10/sh6 : 859\n",
      "https://ganjoor.net//attar/asrarname/abkhsh11/sh2 : 776\n",
      "https://ganjoor.net//attar/asrarname/abkhsh11/sh5 : 873\n",
      "https://ganjoor.net//attar/asrarname/abkhsh11/sh9 : 737\n",
      "https://ganjoor.net//attar/asrarname/abkhsh12/sh4 : 793\n",
      "https://ganjoor.net//attar/asrarname/abkhsh12/sh6 : 1375\n",
      "https://ganjoor.net//attar/asrarname/abkhsh12/sh10 : 754\n",
      "https://ganjoor.net//attar/asrarname/abkhsh12/sh13 : 737\n",
      "https://ganjoor.net//attar/asrarname/abkhsh13/sh2 : 956\n",
      "https://ganjoor.net//attar/asrarname/abkhsh14/sh2 : 966\n",
      "https://ganjoor.net//attar/asrarname/abkhsh14/sh5 : 766\n",
      "https://ganjoor.net//attar/asrarname/abkhsh15/sh2 : 1269\n",
      "https://ganjoor.net//attar/asrarname/abkhsh15/sh5 : 732\n",
      "https://ganjoor.net//attar/asrarname/abkhsh16/sh4 : 1281\n",
      "https://ganjoor.net//attar/asrarname/abkhsh17/sh3 : 746\n",
      "https://ganjoor.net//attar/asrarname/abkhsh18/sh2 : 845\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh3 : 848\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh7 : 909\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh13 : 951\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh17 : 852\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh21 : 804\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh24 : 983\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh27 : 1051\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh30 : 716\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh33 : 824\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh38 : 814\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh43 : 883\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh46 : 800\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh51 : 702\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh55 : 721\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh59 : 741\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh64 : 818\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh68 : 722\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh73 : 822\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh78 : 792\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh82 : 760\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh88 : 783\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh94 : 819\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh98 : 851\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh102 : 810\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh109 : 740\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh116 : 770\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh120 : 718\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh125 : 731\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh131 : 959\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh136 : 794\n",
      "https://ganjoor.net//moulavi/shams/ghazalsh/sh141 : 751\n",
      "https://ganjoor.net//ferdousi/shahname/aghaz/sh3 : 705\n",
      "https://ganjoor.net//ferdousi/shahname/aghaz/sh7 : 835\n",
      "https://ganjoor.net//ferdousi/shahname/aghaz/sh12 : 1278\n",
      "https://ganjoor.net//ferdousi/shahname/qmars/sh2 : 872\n",
      "https://ganjoor.net//ferdousi/shahname/jamshid/sh1 : 873\n",
      "https://ganjoor.net//ferdousi/shahname/jamshid/sh3 : 1067\n",
      "https://ganjoor.net//ferdousi/shahname/zahak/sh3 : 1225\n",
      "https://ganjoor.net//ferdousi/shahname/zahak/sh6 : 883\n",
      "https://ganjoor.net//ferdousi/shahname/zahak/sh7 : 958\n",
      "https://ganjoor.net//ferdousi/shahname/zahak/sh10 : 900\n",
      "https://ganjoor.net//ferdousi/shahname/zahak/sh12 : 1395\n",
      "https://ganjoor.net//ferdousi/shahname/fereydoon/sh2 : 1335\n",
      "https://ganjoor.net//ferdousi/shahname/fereydoon/sh5 : 820\n",
      "https://ganjoor.net//ferdousi/shahname/fereydoon/sh6 : 1276\n",
      "https://ganjoor.net//ferdousi/shahname/fereydoon/sh9 : 907\n",
      "https://ganjoor.net//ferdousi/shahname/fereydoon/sh11 : 1060\n",
      "https://ganjoor.net//ferdousi/shahname/fereydoon/sh13 : 1917\n",
      "https://ganjoor.net//ferdousi/shahname/fereydoon/sh15 : 827\n",
      "https://ganjoor.net//ferdousi/shahname/fereydoon/sh18 : 731\n",
      "https://ganjoor.net//ferdousi/shahname/fereydoon/sh20 : 1380\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh2 : 1948\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh3 : 1469\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh4 : 750\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh6 : 771\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh7 : 971\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh9 : 1031\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh10 : 1245\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh12 : 1082\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh13 : 863\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh14 : 953\n",
      "https://ganjoor.net//ferdousi/shahname/manoochehr/sh15 : 1526\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh4 : 794\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh8 : 830\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh12 : 709\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh16 : 754\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh21 : 845\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh26 : 742\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh30 : 729\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh35 : 731\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh39 : 895\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh44 : 787\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh49 : 723\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh56 : 900\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh62 : 710\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh67 : 713\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh72 : 738\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh77 : 731\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh83 : 713\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh89 : 862\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh92 : 876\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh96 : 820\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh101 : 729\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh105 : 714\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh109 : 789\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh114 : 745\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh120 : 778\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh126 : 786\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh131 : 858\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh135 : 730\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh139 : 834\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh143 : 777\n",
      "https://ganjoor.net//saeb/divan-saeb/ghazalkasa/sh147 : 729\n",
      "https://ganjoor.net//saadi/golestan/gbab1/sh8 : 713\n",
      "https://ganjoor.net//saadi/golestan/gbab1/sh17 : 791\n",
      "https://ganjoor.net//saadi/golestan/gbab1/sh32 : 713\n",
      "https://ganjoor.net//saadi/golestan/gbab2/sh18 : 724\n",
      "https://ganjoor.net//saadi/golestan/gbab2/sh31 : 766\n",
      "https://ganjoor.net//saadi/golestan/gbab2/sh44 : 736\n",
      "https://ganjoor.net//saadi/golestan/gbab3/sh21 : 711\n",
      "https://ganjoor.net//saadi/golestan/gbab3/sh27 : 814\n",
      "https://ganjoor.net//saadi/golestan/gbab5/sh10 : 724\n",
      "https://ganjoor.net//saadi/golestan/gbab5/sh19 : 708\n",
      "https://ganjoor.net//saadi/golestan/gbab7/sh17 : 757\n",
      "https://ganjoor.net//saadi/golestan/gbab8/sh29 : 703\n",
      "https://ganjoor.net//saadi/golestan/gbab8/sh58 : 709\n",
      "https://ganjoor.net//saadi/golestan/gbab8/sh88 : 724\n",
      "https://ganjoor.net//saadi/boostan/niyayesh/sh1 : 773\n",
      "https://ganjoor.net//saadi/boostan/niyayesh/sh4 : 1224\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh1 : 1094\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh2 : 1636\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh6 : 719\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh10 : 753\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh13 : 711\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh16 : 820\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh22 : 770\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh26 : 761\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh28 : 936\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh31 : 845\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh33 : 744\n",
      "https://ganjoor.net//saadi/boostan/bab1/sh39 : 758\n",
      "https://ganjoor.net//saadi/boostan/bab2/sh5 : 931\n",
      "https://ganjoor.net//saadi/boostan/bab2/sh10 : 889\n",
      "https://ganjoor.net//saadi/boostan/bab2/sh15 : 779\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh3 : 1124\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh5 : 927\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh7 : 946\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh9 : 1246\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh10 : 809\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh12 : 974\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh13 : 723\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh15 : 1021\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh18 : 1675\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh20 : 970\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh22 : 702\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh23 : 1593\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh24 : 1006\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh25 : 1792\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh28 : 1115\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh29 : 986\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh30 : 840\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh33 : 867\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh35 : 832\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh36 : 1011\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh38 : 814\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh40 : 1484\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh41 : 2233\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh43 : 1277\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh44 : 785\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh45 : 737\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh46 : 747\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh48 : 1554\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh51 : 1048\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh52 : 2550\n",
      "https://ganjoor.net//nezami/5ganj/khosro-shirin/sh53 : 1080\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh3 : 728\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh7 : 819\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh11 : 814\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh17 : 922\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh22 : 706\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh27 : 834\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh29 : 734\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh33 : 788\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh39 : 810\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh43 : 705\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh48 : 853\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh56 : 932\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh59 : 727\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh64 : 757\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh68 : 777\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh71 : 714\n",
      "https://ganjoor.net//jami/7ourang/7-2/sh73 : 717\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh3 : 1348\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh6 : 886\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh9 : 1102\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh12 : 905\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh15 : 908\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh18 : 965\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh20 : 996\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh22 : 831\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh25 : 749\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh28 : 1074\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh31 : 768\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh34 : 1129\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh37 : 793\n",
      "https://ganjoor.net//jami/7ourang/7-3/sh40 : 1035\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh7 : 770\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh13 : 745\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh18 : 784\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh23 : 716\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh28 : 737\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh34 : 792\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh40 : 760\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh46 : 720\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh52 : 716\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh58 : 770\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh63 : 718\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh70 : 715\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh76 : 770\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh81 : 770\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh87 : 882\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh93 : 764\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh99 : 733\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh105 : 775\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh110 : 739\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh116 : 706\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh122 : 776\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh126 : 724\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh131 : 721\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh137 : 702\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh143 : 775\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh150 : 836\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh155 : 875\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh162 : 874\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh167 : 836\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh174 : 749\n",
      "https://ganjoor.net//eraghi/divane/ghazale/sh179 : 721\n"
     ]
    }
   ],
   "source": [
    "for poet in poets:\n",
    "    get_poet(poet, directory=\"datasets/raw\", print_details=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "path = '/home/amir/Documents/university/projects/NLP/IR-Project/datasets/raw'\n",
    "all_files = [os.path.join(path, file) for file in os.listdir(path) if file.endswith('.csv')]\n",
    "\n",
    "li = []\n",
    "\n",
    "for filename in all_files:\n",
    "    df = pd.read_csv(filename, index_col=None, header=0)\n",
    "    li.append(df)\n",
    "\n",
    "# Concatenate all dataframes into one\n",
    "frame = pd.concat(li, axis=0, ignore_index=True)\n",
    "\n",
    "# Save the merged DataFrame to a new CSV file in the desired location\n",
    "frame.to_csv('datasets/raw/persian_authors.csv', index=False)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2024-05-30T00:34:27.286902Z",
     "end_time": "2024-05-30T00:34:27.379456Z"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}