Web_Scraping_with_Corey_Schafer.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "import lxml\n",
    "import csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "source = requests.get('https://coreyms.com/').text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(source, 'lxml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "29"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_file = open('corey_schafer.csv', 'w', encoding='utf-8')\n",
    "\n",
    "csv_writer = csv.writer(csv_file)\n",
    "csv_writer.writerow(['headline', 'summary', 'video_link'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python Tutorial: Zip Files – Creating and Extracting Zip Archives\n",
      "In this video, we will be learning how to create and extract zip archives. We will start by using the zipfile module, and then we will see how to do this using the shutil module. We will learn how to do this with single files and directories, as well as learning how to use gzip as well. Let’s get started…\n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n",
      "Python Data Science Tutorial: Analyzing the 2019 Stack Overflow Developer Survey\n",
      "In this Python Programming video, we will be learning how to download and analyze real-world data from the 2019 Stack Overflow Developer Survey. This is terrific practice for anyone getting into the data science field. We will learn different ways to analyze this data and also some best practices. Let’s get started…\n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n",
      "Python Multiprocessing Tutorial: Run Code in Parallel Using the Multiprocessing Module\n",
      "In this Python Programming video, we will be learning how to run code in parallel using the multiprocessing module. We will also look at how to process multiple high-resolution images at the same time using a ProcessPoolExecutor from the concurrent.futures module. Let’s get started…\n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n",
      "Python Threading Tutorial: Run Code Concurrently Using the Threading Module\n",
      "In this Python Programming video, we will be learning how to run threads concurrently using the threading module. We will also look at how to download multiple high-resolution images online using a ThreadPoolExecutor from the concurrent.futures module. Let’s get started…\n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n",
      "Update (2019-09-03)\n",
      "Hey everyone. I wanted to give you an update on my videos. I will be releasing videos on threading and multiprocessing within the next week. Thanks so much for your patience. I currently have a temporary recording studio setup at my Airbnb that will allow me to record and edit the threading/multiprocessing videos. I am going to be moving into my new house in 10 days and once I have my recording studio setup then you can expect much faster video releases. I really appreciate how patient everyone has been while I go through this move, especially those of you who are contributing monthly through YouTube \n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n",
      "Python Quick Tip: The Difference Between “==” and “is” (Equality vs Identity)\n",
      "In this Python Programming Tutorial, we will be learning the difference between using “==” and the “is” keyword when doing comparisons. The difference between these is that “==” checks to see if values are equal, and the “is” keyword checks their identity, which means it’s going to check if the values are identical in terms of being the same object in memory. We’ll learn more in the video. Let’s get started…\n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n",
      "Python Tutorial: Calling External Commands Using the Subprocess Module\n",
      "In this Python Programming Tutorial, we will be learning how to run external commands using the subprocess module from the standard library. We will learn how to run commands, capture the output, handle errors, and also how to pipe output into other commands. Let’s get started…\n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n",
      "Visual Studio Code (Windows) – Setting up a Python Development Environment and Complete Overview\n",
      "In this Python Programming Tutorial, we will be learning how to set up a Python development environment in VSCode on Windows. VSCode is a very nice free editor for writing Python applications and many developers are now switching over to this editor. In this video, we will learn how to install VSCode, get the Python extension installed, how to change Python interpreters, create virtual environments, format/lint our code, how to use Git within VSCode, how to debug our programs, how unit testing works, and more. We have a lot to cover, so let’s go ahead and get started…\n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n",
      "Visual Studio Code (Mac) – Setting up a Python Development Environment and Complete Overview\n",
      "In this Python Programming Tutorial, we will be learning how to set up a Python development environment in VSCode on MacOS. VSCode is a very nice free editor for writing Python applications and many developers are now switching over to this editor. In this video, we will learn how to install VSCode, get the Python extension installed, how to change Python interpreters, create virtual environments, format/lint our code, how to use Git within VSCode, how to debug our programs, how unit testing works, and more. We have a lot to cover, so let’s go ahead and get started…\n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n",
      "Clarifying the Issues with Mutable Default Arguments\n",
      "In this Python Programming Tutorial, we will be clarifying the issues with mutable default arguments. We discussed this in my last video titled “5 Common Python Mistakes and How to Fix Them”, but I received many comments from people who were still confused. So we will be doing a deeper dive to explain exactly what is going on here. Let’s get started…\n",
      "https://youtube.com/watch?v=IEEhzQoKtQU\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for article in soup.find_all('article'):\n",
    "    headline = article.h2.a.text\n",
    "    print(headline)\n",
    "    \n",
    "    summary = article.find('div', class_='entry-content').p.text\n",
    "    print(summary)\n",
    "    \n",
    "    #vid_src = article.find_all('iframe', class_= 'youtube-player')['src']\n",
    "    \n",
    "    vid_id = vid_src.split('/')[4]\n",
    "    vid_id = vid_id.split('?')[0]\n",
    "    #print(vid_id)\n",
    "    \n",
    "    yt_link = f'https://youtube.com/watch?v={vid_id}'\n",
    "    print(yt_link)\n",
    "    \n",
    "    print()\n",
    "    \n",
    "    csv_writer.writerow([headline, summary, yt_link])\n",
    "    \n",
    "csv_file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'C:\\\\Users\\\\Administrator\\\\Web Scraping'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}