ScheduleStorm.com

You Can Find the Front-End Repo Here

Welcome to the back-end that powers Schedule Storm. The back-end is written entirely in Python using Falcon, Beautiful Soup, ldap3, requests, MongoDB, and pymongo.

As you might expect, Schedule Storm is reliant upon class data for numerous universities. Since many universities don't have APIs to query, the vast majority of scraping is done on HTML using Beautiful Soup and Requests. Here you'll be able to find documentation on how to add your University to Schedule Storm.

Feel free to send pull requests that add universities/features

How to Run

Python 3 Dependencies

• Requests
• Beautiful Soup 4
• ldap 3
• pymongo
• falcon
• html5lib
• lxml

You can automatically install the dependencies using pip install -r requirements.txt

Note: Requires MongoDB to be running

Rename settings.example.json to settings.json and set the enabled and scrape settings to true for every university you'd like to enable.

The default port is 3000, you can change this in the settings file

Simply execute (tested on Python 3.4+): python index.py

You can browse the API by going to http//localhost:{port}/v1/unis or http://localhost:{port}/v1/unis/{uni}/{term}/all

If you'd like to use the front-end with your local API, clone it and change the URLs at the top of ClassList.js and Welcome.js

How to Add Your University

1. Settings File
2. Creating Your University Python File
3. Terms
4. Subjects
5. Course Descriptions
6. Classes

Settings File

Using the settings file, you can tell Schedule Storm your University's rmpid, api key, username, password, etc...

Every University Must Have An Entry in settings.json

{
    "Universities": {
        ...
        "<uniID>": {
            "fullname": "<University Name>"
            "enabled": true,
            "scrape": true,
            "scrapeinterval": 7200,
            "rmpid": <rmpid>,
        ...
        },
    ...

key	Type	Optional	Notes
uniID	string	No	ID/Abbreviation of the University (ex. UCalgary, MTRoyal)
fullname	string	No	Full name of the university shown to the user (ex. University of Calgary)
enabled	bool	No	Boolean as to whether this university is enabled or not
scrape	bool	No	If true, starts the university thread to fetch updated course info
scrapeinterval	int	No	Amount of seconds to sleep between subsequent scrapes
rmpid	int	Yes	RMP ID of the University to fetch professor data from
lastUpdated	int	Yes	(Auto-generated) UNIX timestamp of the last successful scrape in seconds

Within the university's JSON block, you can have as many more attributes as you'd like. Here, you can specify usernames, passwords, api keys, and they'll all be passed to your University thread upon creation.

How do I get the rmpid?

Simply go to Rate My Professors and search for your university in the search bar. Make sure you click on your university in the bottom "Schools" section.

Afterwards, you will be forwarded to a URL such as: http://www.ratemyprofessors.com/campusRatings.jsp?sid=1416

The sid parameter is the rmpid, so this rmpid is 1416.

Creating Your University Python File

File Name

All of the universities are located in the "uni" folder with their names being <uniID>.py

For example: University of Calgary has a uniID of "UCalgary" in the settings file, so its file is UCalgary.py.

Creating the Class

Each university inherits the University class, which inherits the threading.Thread class.

Within your Uni file, you must import the University superclass at the top: from .University import University. The University superclass contains the API handlers and DB interaction methods.

Next, you'll want to create a class that inherits University and is named your uniID along with instantiating the University superclass in your __init__ method.

ex. If your uniID is "UCalgary"

from .University import University

class UCalgary(University):
    def __init__(self, settings):
        super().__init__(settings)

    def scrape(self):
        """
        Scraping function that obtains updated course info
        :return:
        """
        self.log.info("Obtain course info here!")

Within the scrape method, you should fetch updated course data

Terms

Since each university inherits the University superclass, all interaction with the MongoDB database is done through it. You can use the following methods to interface with the terms.

Term Object

Term Objects are dictionaries with the specified keys

key	Type	Optional	Unique	Notes
id	string	No	Yes	ID of the Term (ex. "20235")
name	string	No	No	Name of the term (ex. "Fall 2016")

Within the DB, it also contains an enabled flag that specifies whether it is shown to users or not. The Term methods abstract this from you.

Term Methods

updateTerms(list terms)

Arguments:

name	Type	Optional	Notes
terms	list	No	List of term objects to update in the DB

NOTE: This sets the only enabled terms to be the specified terms in the list

Updates the terms specified into the DB. If a term doesn't exist in the DB yet, it is inserted.

updateTerm(dict term)

Arguments:

name	Type	Optional	Notes
term	dict	No	Term object to update in the DB

Updates the term specified into the DB and sets its enabled flag to True (shows it to users). If the term doesn't exist in the DB yet, it is inserted.

getTerm(str/int termid)

Arguments:

name	Type	Optional	Notes
termid	string	No	ID of the term for fetch the term obj for

Returns: Term object for the specified termid if succesful, False if not

resetEnabledTerms()

Sets every term within the DB to have a False enabled flag (isn't shown to users).

Subjects

Subject Object

Subject Objects are dictionaries with the specified keys

key	Type	Optional	Unique	Notes
subject	string	No	Yes	Subject abbreviation (ex. CPSC)
name	string	Yes	No	Subject name (ex. Computer Science)
faculty	string	Yes	No	Faculty that this subject belongs to (ex. Faculty of Science)

Subject Methods

updateSubjects(list subjects)

Arguments:

name	Type	Optional	Notes
subjects	list	No	List of subject objects to update in the DB

Updates the subjects specified into the DB. If a subject doesn't exist in the DB yet, it is inserted.

updateSubject(dict subject)

Arguments:

name	Type	Optional	Notes
subject	dict	No	Subject object to update in the DB

Updates the subject specified into the DB. If the subject doesn't exist in the DB yet, it is inserted.

Course Descriptions

Course Description Object

Course Description Objects are dictionaries with the specified keys

key	Type	Optional	Unique	Notes
coursenum	string	No	No	Course number (ex. "300")
subject	string	No	No	Subject abbreviation (ex. "CPSC")
name	string	Yes	No	Name/title of the course (ex. "Introduction to Computer Science")
desc	string	Yes	No	Description of the course (ex. "You'll learn about computers in this course")
units	int	Yes	No	How many units this course is worth (ex. 3)
hours	string	Yes	No	Distribution of hours between types of classes (ex. "H(3-3)")
prereq	string	Yes	No	Human-readable course prerequisites (ex. "Must take CPSC 299 or CPSC 256")
coreq	string	Yes	No	Human-readable course corequisites (ex. "Must take CPSC 301 with this course")
antireq	string	Yes	No	Human-readable course antirequisites (ex. "Student must not have taken CPSC 302")
notes	string	Yes	No	Any further human-readable notes for this class (ex. "You might learn too much!")
aka	string	Yes	No	Equivalent/old names for this class (ex. "Formally CPSC 234")

The coursenum and subject fields together form a unique constraint.

Course Description Methods

updateCourseDesc(dict coursedesc)

Arguments:

name	Type	Optional	Notes
coursedesc	dict	No	Course description object to update in the DB

Updates the course description specified into the DB. If the course description doesn't exist in the DB yet, it is inserted.

getCourseDescription(string coursenum, string subject)

Arguments:

name	Type	Optional	Notes
coursenum	string	No	Course number of the description to obtain (ex. "300")
subject	string	No	Subject abbreviation of the description to obtain (ex. "CPSC")

Returns: Course Description object of the specified coursenum and subject if successful, False if not

Classes

Class Object

Class Objects are dictionaries with the specified keys

key	Type	Optional	Unique	Notes
id	int	No	Yes	ID of the class (ex. "34534")
subject	string	No	No	Subject abbreviation (ex. "CPSC")
term	string	No	No	Term ID that this class belongs to (ex. "2342")
coursenum	string	No	No	Course number of the description to obtain (ex. "300")
rooms	list	No	No	List of strings that contain the rooms that this class is situated in (ex. ["MFH 164", "HEH 101"]). If there are no rooms, set it to ["TBA"] or ["N/A"]
teachers	list	No	No	List of strings that contain the teachers teaching this class (ex. ["Jack Shepard", "Hugo 'Hurley' Reyes"]). If there are no teachers, set it to ["TBA"] or ["N/A"]
type	string	No	No	Type of the class (ex. "LEC")
times	list	No	No	List of strings that contain the times in which this class is (ex. ["MWF 12:00PM - 2:00PM"]). If there are no times, set it to ["TBA"] or ["N/A"]. See below for time formatting.
group	list	No	No	Group(s) of this class (ex. ["1", "2"]). See below for further details.
status	string	No	No	If the class is open, set to "Open", otherwise, set the enrollment status to "Closed" or "Wait List"
location	string	No	No	Location of the class (ex. "Main Campus")
section	string	Yes	No	Shows this value instead of group to the user when applicable
restriction	bool	Yes	No	True if this class has a restriction to some students
curEnroll	int	Yes	No	Amount of students currently enrolled in the class
capEnroll	int	Yes	No	Maximum amount of students that are able to be enrolled in the class
waitEnroll	int	Yes	No	Amount of students waitlisted to take the course
capwaitEnroll	int	Yes	No	Total amount of students that can waitlist the course

group Attribute

Most universities have a system where you must take some specific classes together (and they are incompatible with others). In order to resolve this, the group attribute defines the classes that can be taken together. Two classes that have at least one match in their group lists can be taken together.

The generator ensures that it only matches compatible classes with each other and if a user tries to manually specify two incompatible classes, it doesn't generate any possibilities.

If a specific course doesn't care whether two classes are compatible, just set each class' group to ["1"].

Ex.

• CPSC 301 LEC group: ["1", "2"]
• CPSC 301 LEC2 group: ["2", "3"]
• CPSC 301 TUT group: ["2"]
• CPSC 301 TUT2 group: ["3"]

LEC and TUT are compatible, LEC2 is compatible with TUT and TUT2

Time Formatting

Each time for a given class must be in the specified format:

<DaysOfTheWeek> <StartTime><AM/PM> - <EndTime><AM/PM>

• Days of the Week

  • Concatenated series of days in which this time is applicable
  • Possible Days
    • M/Mo - Monday
    • T/Tu - Tuesday
    • W/We - Wednesday
    • R/Th - Thursday
    • F/Fr - Friday
    • S/Sa - Saturday
    • U/Su - Sunday
  • Examples
    • "M"
    • "MTR"
    • "FWM"
  • Order of the days does not matter

• StartTime/EndTime

  • 12-hour Start/End time
  • Format: <Hour>:<Minutes>
  • Examples
    • "12:00"
    • "1:23"

• Examples

  • "TR 12:00PM - 1:20PM"
  • "MWF 9:50AM - 10:30AM"
  • "MoWeFr 9:00AM - 11:00AM"
  • "MoTWe 2:00PM - 3:00PM"

Class Methods

updateClasses(list classes)

Arguments:

name	Type	Optional	Notes
classes	list	No	List of class objects to update

Updates the classes specified into the DB. If a class doesn't exist in the DB yet, it is inserted.

updateClass(dict class)

Arguments:

name	Type	Optional	Notes
class	dict	No	Class object to update

Updates the class specified into the DB. If the class doesn't exist in the DB yet, it is inserted.

General Tips When Adding a University

• Don't use print statements, the University superclass contains a logger, you can use it with self.log
  • Examples
    • self.log.error("You can't do this")
    • self.log.info("Scraping courses")
• You can access your uni settings object with self.settings within the uni class
• Look at Example.py for a starting point, make sure you edit the settings file though!
• If you want to add more attributes to a class/subject/term/coursedesc object, go ahead! They won't be used by the front-end, but we can add support for it later on!
• If you have any questions/concerns, feel free to file an issue or talk to us!

Authors

• Stepan Fedorko-Bartos - Email, Github, LinkedIn
• Ceegan Hale - Email, Github, LinkedIn
• Thanks to any contributors!

Inspiration from:

• Hey Winston for University of Alberta