notebook/3-1-requests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# requests: an alternative to urllib\n",
    "**requests has more functions to replace urlopen. Use request.get() to replace urlopen() and pass some parameters to the webpage.  The webbrowser is to open the new url and give you an visualization of this result.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://www.baidu.com/s?wd=%E8%8E%AB%E7%83%A6Python\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "import webbrowser\n",
    "param = {\"wd\": \"莫烦Python\"}\n",
    "r = requests.get('http://www.baidu.com/s', params=param)\n",
    "print(r.url)\n",
    "webbrowser.open(r.url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## post\n",
    "\n",
    "**We test the post function in this [page](http://pythonscraping.com/pages/files/form.html). To pass some data to the server to analyse and send some response to you accordingly.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello there, 莫烦 周!\n"
     ]
    }
   ],
   "source": [
    "data = {'firstname': '莫烦', 'lastname': '周'}\n",
    "r = requests.post('http://pythonscraping.com/files/processing.php', data=data)\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## upload image\n",
    "\n",
    "**We still use post function to update image in this [page](http://pythonscraping.com/files/form2.html).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The file image.png has been uploaded.\n"
     ]
    }
   ],
   "source": [
    "file = {'uploadFile': open('./image.png', 'rb')}\n",
    "r = requests.post('http://pythonscraping.com/files/processing2.php', files=file)\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## login\n",
    "\n",
    "**Use post method to login to a [website](http://pythonscraping.com/pages/cookies/login.html).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'username': 'Morvan', 'loggedin': '1'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hey Morvan! Looks like you're still logged into the site!\n"
     ]
    }
   ],
   "source": [
    "payload = {'username': 'Morvan', 'password': 'password'}\n",
    "r = requests.post('http://pythonscraping.com/pages/cookies/welcome.php', data=payload)\n",
    "print(r.cookies.get_dict())\n",
    "r = requests.get('http://pythonscraping.com/pages/cookies/profile.php', cookies=r.cookies)\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## another general way to login\n",
    "\n",
    "**Use session instead requests. Keep you in a session and keep track the cookies.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'username': 'Morvan', 'loggedin': '1'}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hey Morvan! Looks like you're still logged into the site!\n"
     ]
    }
   ],
   "source": [
    "session = requests.Session()\n",
    "payload = {'username': 'Morvan', 'password': 'password'}\n",
    "r = session.post('http://pythonscraping.com/pages/cookies/welcome.php', data=payload)\n",
    "print(r.cookies.get_dict())\n",
    "r = session.get(\"http://pythonscraping.com/pages/cookies/profile.php\")\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}