To lift the hood on a website and see how it works, you'll need to use a browser's developer tools:
Here are a few of the most popular workhorses in the Python community for web scraping tasks:
- requests to fetch web pages/files
- BeautifulSoup for parsing HTML
- lxml for more serious HTML/XML parsing firepower (it's a "heavy" install but offers more speed and features than BeautifulSoup)
- Selenium for stateful web scraping and working with dynamic page content (among other thorny issues)
- Getting Started with HTML (Mozilla)
- BeautifulSoup - HTML and XML
- Automate the Boring Stuff - HTML section of Chapter 11
- Inspecting the Web with Chrome's Dev Tools - work through all three sections for a primer on HTML, CSS and Dev Tools