Skip to content

web scraper for Carleton College's Knowledge Base

Notifications You must be signed in to change notification settings

carleton/KB-Scraper

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

39 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Installation

Executables

Windows:

Mac:

  1. Clone this repository (git clone https://github.com/samgjl/KB-Scraper.git).
  2. Locate the executable file at mac/dist/KB Deletion Scraper.exe.
    You can either move this file to your desired location, or create a shortcut.

Full code:

  • Clone this repository (git clone https://github.com/samgjl/KB-Scraper.git)
  • You can launch from the python files n /src/.

Usage

Launching Without Executable

If you aren't using the executable, make sure to install all the requirements!
(Install Python, then run python -m pip install -r requirements.txt)

  • Launch the base program by running python kb_scraper.py.
  • Launch the GUI by running python scraper_gui.py.

Fields

  • Username: your username for Carleton College SSO
  • Password: your password for Carleton College SSO
  • CSV Path: the path to the CSV file (note: if the file is not in the same folder, you must provide the absolute filepath)
  • Headless: If checked, this will allow the program to open the browser without visuals. This is best for multitasking

Building from source

This software requires PyInstaller (pip install pyinstaller),
along with all packages in requirements.txt (python -m pip install -r requirements.txt)

Windows

  1. Open a terminal at this directory
  2. Enter the windows directory (cd ./windows/)
  3. Run PyInstaller's initialization script: python -m PyInstaller --onefile --name "KB Deletion Scraper" ../src/scraper_gui.py
    (This will take several minutes to complete)
  4. This will create the build and dist folders, along with the KB Deletion Scraper.spec file. Replace the text in the KB Deletion Scraper.spec file in this folder with the text from windows.spec in the base directory. The new file will have all the correct compilation settings.
  5. Recompile the app with python -m PyInstaller "KB Deletion Scraper.spec".

For debugging, see Kivy's Documentation.

MacOS

  1. Open a terminal at this directory
  2. Enter the mac directory (cd ./mac/)
  3. Run PyInstaller's initialization script:
pyinstaller -y --clean --onefile --windowed --name "KB Deletion Scraper" \
  --exclude-module _tkinter \
  --exclude-module Tkinter \
  --exclude-module enchant \
  --exclude-module twisted \
  ../src/scraper_gui.py
  1. This will create the build and dist folders, along with the KB Deletion Scraper.spec file. Replace the text in the KB Deletion Scraper.spec file in this folder with the text from mac.spec in the base directory. The new file will have all the correct compilation settings.
  2. Recompile the app with pyinstaller -y --clean 'KB Deletion Scraper.spec'.

For debugging, see Kivy's Documentation.

About

web scraper for Carleton College's Knowledge Base

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • HTML 72.4%
  • TeX 27.4%
  • Other 0.2%