OATutor is an Open-source Adaptive Tutoring System (OAT) based on Intelligent Tutoring System principles. It uses Bayesian Knowledge Tracing for skill mastery estimation and is implemented entirely in React JS with optional logging using Firebase. The system can be deployed to git-pages without the use of any backend. For LMS integration, a middleware backend is required by Learning Tools Interoperability (LTI). Our hosted backend server can be used or the middleware can be launched independently. OATutor is Section 508 accessibility compliant.
Quick clone and deploy notebook example
To credit this system, please cite our CHI'23 paper:
Zachary A. Pardos, Matthew Tang, Ioannis Anastasopoulos, Shreya K. Sheel, and Ethan Zhang. 2023. OATutor: An Open-source Adaptive Tutoring System and Curated Content Library for Learning Sciences Research. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 416, 1–17. https://doi.org/10.1145/3544548.3581574
@inproceedings{pardos2023oat,
title={OATutor: An Open-source Adaptive Tutoring System and Curated Content Library for Learning Sciences Research},
author={Pardos, Z.A., Tang, M., Anastasopoulos, I., Sheel, S.K., Zhang, E},
booktitle={Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems},
pages={1--17},
organization={Association for Computing Machinery},
doi={https://doi.org/10.1145/3544548.3581574},
year={2023}
}
Our new pre-print, reporting preliminary finding on learning gains and ChatGPT-based hint evaluation: https://arxiv.org/abs/2302.06871
Content Repository: Link
This content repository contains problems curated from OpenStax and classroom syllabus with hints and scaffolds authored by the OATutor Project. As of Fall 2024, these materials have been piloted in classrooms. The hints and scaffolds for the problems have been curated by the OATutor team. If you identify any errors please issue a pull request or bug report.
Jump to our website: OATutor.io
The textbooks contained in this repo are:
- Elementary Algebra 2e
- Intermediate Algebra 2e
- College Algebra 2e with Corequisite Support
- Introductory Statistics 2e
- Calculus Volume 1
- Pre-Calculus Essentials (UC Berkeley Math 1B)
- Matematik 4 (OpenStax Calculus Content in Swedish)
All content is available in JSON format within the repository. To view the raw spreadsheets for specific textbooks, use the links below:
- Elementary Algebra 2e: Link to Spreadsheet
- Intermediate Algebra 2e: Link to Spreadsheet
- College Algebra 2e with Corequisite Support: Link to Spreadsheet
- Introductory Statistics 2e: Link to Spreadsheet
- Calculus Volume 1: Link to Spreadsheet
- Pre-Calculus Essentials (UC Berkeley Math 1B): Link to Spreadsheet
All content in this repository is made available under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. Attribution is given within each json file, indicating the authoring organization and license for each hint, scaffold, and problem.
The installation assumes that you already have Git, Node.js, and npm installed.
git clone --recurse-submodules https://github.com/CAHLR/OATutor.git
cd OATutornpm installYou may use an alternative package manager such as yarn or pnpm.
npm run startnpm run build
npx serve -s buildThe build folder now contains all of the static assets necessary to make a complete deployment on a static site hosting provider.
OATutor can use Firebase to persistently store log data.
- Navigate to the Firebase website
- Add new project. Configure it as you wish (the options are not important for setup)
- Click on Database and then create database. Start in test mode, leave the cloud location as is
- Click on Project settings --> general. Copy SDK Setup & Configuration --> Config
- Put configuration in
src/config/firebaseConfig.js
- Scaffolding/hint system - modularize the type of help
- Adaptive item selection - Pick items to master weakest skills (isolate skills to master individually)
- Centralized skill model -
src/content-sources/*/skillModel.json - Data logging/collection - Based off of the Cognitive Tutor KDD dataset.
- User login/registration - JSON Web Tokens
- Frontend: ReactJS
- Theme: Material UI
- Database: localForage (localStorage, WebSQL, IndexedDB)
- Deployment: Github Actions to Github Pages
- [Optional] Logging: Firebase (Cloud Firestore)
- Middleware: ExpressJS
- Database: Level-DB
- Offline Computation/Iteration:
- Python (dAFM Machine Learning algorithm)
Code for this project is located in the src directory.
App.css: Top level style sheet, contains colors for headers/logoApp.js: Top level script, creates firebase object. Sets up the application context.index.js: RendersApp.js
BKT-brain.js: Containsupdatefunction that implements the standard BKT update algorithm.problem-select-heuristics/*.js: These files contain a configurable heuristic for adaptive problem selection. The default heuristic iterates across the problems and chooses the one with the lowest average probability of mastery across all of its knowledge components, but this can be changed to any heuristic depending on the problem information and the previously completed problems.
Firebase.js: Class with methods to read/write to Firebase (Cloud Firestore).
-
HintSystem.js: Expandable panel component to display all the hints. -
HintTextbox.js: Textbox for scaffold types of hints with answers. -
Problem.js: The "problem" component created in App.js. The component is initailized with a "problem" object as one of its props. It then creates a series of "ProblemCards" inconst parts(currently keys for ProblemCard components are random values, should be UUIDs in the future).The
answerMadefunction is passed to each ProblemCard component and is called whenever an answer is submitted to a ProblemCard. This enables the Problem component to update the knowledge component variables after each answer and transition to the next problem after all answers are correct. -
ProblemCard.js: This component displays an individual card, updates the input text field to display the result of an answer attempt, and calls theProblem.jsanswerMadefunction when the submit button is pressed.
These two files heavily rely on Material-UI syntax (eg. all the useStyles and classes references). Check their
website for more info on this syntax.
problemCardStyles.js: This file contains all the styles forProblemCard.js
-
checkAnswer.js: Function to check answers. 3 different types of answers are supported: Algebraic, String, Numeric. Algebraic will simplify numeric expressions, numeric checks numeric equivalence, string requires answers to exactly match. -
Platform.js: Creates top "AppBar" and presents the first "problem" (everything under the app bar is part of the problem component). Also imports all of the problem files and stores them inconst problemIndex. The functionnextProblemis used to determine the next problem to be displayed. -
Firebase.js: Class with methods to read/write to Firebase (Cloud Firestore). -
renderText.js: Method called to render text. Fills in dynamic text generation.
- Each sub-folder can be considered its own isolated content source.
oatutorcontains OATutor-curated content but can be removed if the content is not being used.
- See the Content Source section for more details.
- All
\must be escaped as\\because values are strings - Wrap Latex in
$for inline LaTeX - Newlines can be created with
\n, escaped as\\n
-
config.js: Central place where options can be configured. Also includes function to get the treatment id given a userID, imports all appropriate treatments (Ex. BKTParam, HintPathway, Adaptive Problem selection heuristic) -
firebaseConfig.js: File containing firebase set up configuration.
Data parsing and spreadsheet populating tools are stored in src/tools. If you would
like to use any of the tools in this directory, the following steps must be taken to
ensure Firebase access.
- Navigate to the Firebase website
- Click on the project made in Firebase Setup
- Click on Project settings --> service accounts. Generate a new
Node.jsprivate key. - Put that private key in
src/tools/service-account-credentials.json cd src/toolsnpm install
This tool allows you to sync the feedback received from your website to a Google Spreadsheet of your choosing.
- Navigate to the Firebase website
- Click on the project made in Firebase Setup
- Click on Project settings --> service accounts
- Create another service account and save its private key to
src/tools/sheets-service-account.json - Copy the email address for the service account
- Share the target spreadsheet with that email address and give them editor access
- Create
.env.localinsrc/toolsand add this lineSPREADSHEET_ID=YOUR_SPREADSHEET_ID_HERE - If your school runs on a quarter system, you may change the first line in
common/global-config.jstoQUARTER
From the src/tools directory, node populateGoogleSheets.js
This tool requires no additional set up, and allows you to download a CSV of the Firebase collections.
Contains common helper methods for the frontend React app.
- Size of screen is the size of the scrollable browser canvas
- Wrap
<Platform />with the following inApp.js
<ReactCursorPosition onPositionChanged={(data) => {
if (DO_LOG_MOUSE_DATA) {
this.firebase.mouseLog(data);
}
}}>
<Platform props_here/>
<ReactCursorPosition/>
}}>- Records the times at which the user enters/leaves the tab
- Uses efficient Firestore storage by partitioning data into collections
- Turn off by editing config.js setting
DO_FOCUS_TRACKING = false;- Install the React component for the listener.
- Wrap the Platform component with the listener. The listener must take a prop that is a function to log data that it receives.
- In
/ProblemLogic/Firebase.jsadd a new function to log the new type of data. Create a new collection for this listener logs. - Configure buffer size and granularity of logging
- OATutor can support multiple content sources simultaneously, compartmentalizing courses, lessons, and problems of different topics
- Currently, the
oatutorcontent source is included in this repository as a git submodule to enable separate versioning - However, content sources can be copied in as entire folders as well and committed to this repository
- Each problem is contained in its own folder.
- Problems can contain steps which are contained in their own sub-folder.
- Steps can contain hints which are stored as pathways in the
tutoringsub-folder. - All problems are pre-processed before being ingested by the frontend platform.
All problems are accumulated in the
generated/processed-content-pool/[source_name].jsonfile prior to each run or build.
bktParams.js: Contains the mastery, transit, slip, and guess probabilities for each skill. Used by the BKT model.
skillModel.json: This file contains all the problem to skill mappings. The format is as follows:
{
problemID1a: ['skill1', 'skill2'],
problemID1b: ['skill2', 'skill3'],
// ...
}coursePlans.json: This file contains all of the courses relating to this content source. Each course specified in this file is associated with multiple lessons. See the creating lesson plans section for more details.
- Create a folder in
./content-poolfor that problem (Ex.circle1) - Create a metadata json file for that problem id (Ex.
circle1.json) - Create a folder called
figuresif the problem has image figures - Create a sub-folder for each problem step (Ex.
circle1a,circle1b) - In each sub-folder, create a json file for that problem step (Ex.
circle1a.json)- Ensure that the step name matches the folder name
- Create a sub-folder within the step's sub-folder called
tutoring - Place each hint pathway within the folder (Ex.
circle1aDefaultPathway.json) - In
./skillModel.json, tag each problem with the appropriate skills - If the skill does not already exist in
bktParamsand you are using the BKT model, add its BKT parameters in the appropriatebkt-params/bktParams.jsonfiles
TextBox: Box for student to enter answer. 3 different types of answers are supported: Algebraic, String, Numeric. Algebraic will simplify numeric expressions, numeric checks numeric equivalence, string requires answers to exactly match.MultipleChoice: List choices aschoices: ["Choice A", "Choice B"], must haveanswerType: "string"
content-sources/
└── oatutor [submodule]/
├── bkt-params/
│ ├── bktParams1.json
│ └── bktParams2.json
├── content-pool/
│ ├── circle1/
│ │ ├── circle1.json
│ │ └── steps/
│ │ ├── circle1a/
│ │ │ ├── circle1a.json
│ │ │ └── tutoring/
│ │ │ └── circle1aDefaultPathway.json
│ │ └── circle1b/
│ │ ├── circble1b.json
│ │ └── tutoring/
│ │ └── circle1bDefaultPathway.json
│ └── slope1/
│ ├── slope1.json
│ └── ...
├── coursePlans.json
└── skillModel.json
{
"id": "circle1",
"title": "Buying a Big Rug",
"body": "Bob wants to surprise Alice by buying a new rug for their living room. Their living room is 28 feet wide and 20 feet long. To further surprise Alice, Bob wants to buy the biggest circular rug that will fit.",
"variabilization": {},
"oer": "https://example.com",
"lesson": "1.1 Circle Radius",
"courseName": "Geometry"
}{
"id": "circle1a",
"stepAnswer": [
"10"
],
"problemType": "TextBox",
"stepTitle": "1. Maximum Radius",
"stepBody": "What is the maximum radius of a circular rug that will fit in the room?",
"answerType": "numeric",
"variabilization": {}
}[
{
"id": "circle1a-h1",
"title": "Size of the room",
"text": "Consider the shape of the room and the limitations this has on the radius of the rug.",
"type": "hint",
"dependencies": [],
"variabilization": {}
},
{
"id": "circle1a-h2",
"title": "Constricting dimension",
"text": "The length (20ft) creates limitations on the size of the circle. What is the maximum diameter that the circle can be?",
"hintAnswer": ["20"],
"problemType": "TextBox",
"answerType": "numeric",
"type": "scaffold",
"dependencies": [0],
"variabilization": {}
},
{
"id": "circle1a-h3",
"title": "Solution",
"text": "Recall that the radius is half the diameter, so $r = \\frac{d}{2} = 10$",
"type": "solution",
"dependencies": [1],
"variabilization": {}
}
]{
"id": "pythag1", //Substeps will be in the form problem.id + 'a' and so on
"title": "Car Forces",
"body": "A %CAR% experiences three horizontal forces of -3.10N, 1.70N and -4.00N. It also experiences three vertical forces of -4.30N, 0.20N and 4.20N. \\n Round all answers to the hundredths place. \\n##triangle.png## ",
"variabilization": {}
}- Create a lesson plan by making a new item in
[content_source]/coursePlans.json - Each lesson plan has learning objectives which you can also list the target mastery level
- Lesson plans can have multiple learning objectives (for cumulative review)
- Users select a lesson upon visiting the site
{
id: "lesson1",
name: "Lesson 1",
topics: "Pythagorean Theorem",
allowRecycle: true,
learningObjectives:
{
pythagorean: 0.95
}
}OATutor was designed with the research case in mind and thus supports AB testing for many features. The benefit of the open source nature of the platform allows researchers to insert AB testing logic into any part of the platform they would like. To show that this is possible, we have included several examples of how one could use AB testing.
AB testing is conducted by randomly assigning users into one of two groups. The treatment split is 50/50 by default, but it can be easily changed to a different split percentage or more than two splits. The userID is recorded in all data logs to infer the treatment.
// src/App.js
this.userID = generateRandomInt().toString();
getTreatment = () => {
return this.userID % 2;
}
getTreatmentObject = (targetObject) => {
return targetObject[this.getTreatment()]
}One example is to include different heuristics for problem selection. One heuristic is to choose problems with a knowledge component that is lowest (meaning the student is weakest in this subject) to round out the student's knowledge. Another heuristic is to choose problems with a knowledge component that is highest (meaning the student is strongest in this subject) to fully master a skill before moving on to another. New heuristic files can be added to the appropriate folder (see below code snippet) and imported accordingly to be used.
// src/App.js
import { heuristic as defaultHeuristic } from "./models/BKT/problem-select-heuristics/defaultHeuristic.js";
import { heuristic as experimentalHeuristic } from "./models/BKT/problem-select-heuristics/experimentalHeuristic.js";
...
const treatmentMapping = {
heuristic: {
0: defaultHeuristic,
1: experimentalHeuristic
},
}Different BKT parameters can also be used in AB testing. New bktParam files can be added to the appropriate folder (see below code snippet) and imported accordingly to be used.
// src/App.js
import defaultBKTParams from "./content-sources/oatutor/bkt-params/defaultBKTParams.json";
import experimentalBKTParams from "./content-sources/oatutor/bkt-params/experimentalBKTParams.json";
...
const treatmentMapping = {
bktParams: {
0: cleanObjectKeys(defaultBKTParams),
1: cleanObjectKeys(experimentalBKTParams)
},
}Most content in the OATutor-Content repository currently only contains one hint pathway (the defaultPathway), but
additional hint pathways can easily be added. AB testing can be done with multiple hint pathways for efficacy tests.
New hint pathway files can be added to the tutoring folder of within a step.
// src/App.js
const treatmentMapping = {
hintPathway: {
0: "DefaultPathway",
1: "YourNewPathwayHere"
}
}Example content directory with multiple hint pathways:
content-sources/
└── oatutor [submodule]/
├── content-pool/
│ ├── circle1/
│ │ ├── circle1.json
│ │ └── steps/
│ │ ├── circle1a/
│ │ │ ├── circle1a.json
│ │ │ └── tutoring/
│ │ │ ├── circle1aDefaultPathway.json
│ │ │ └── circle1aYourNewPathwayHere.json <--- Add your new pathway here
│ │ └── circle1b/
│ │ ├── circble1b.json
│ │ └── tutoring/
│ │ ├── circle1bDefaultPathway.json
│ │ └── circle1bYourNewPathwayHere.json <--- Add your new pathway here
│ └── slope1/
│ ├── slope1.json
│ └── ...
├── ...
Knowledge components (KCs) are assigned at the step level in the file skillModel.json. A KC is defined as a string that contains corresponding BKT parameters (existing in bktParams.js file) including probMastery, probTransit, probSlip, and probGuess. Each step can be assigned any number of KCs in an array format (['kc1', 'kc2', ... 'kcN']). skillModel.json stores a mapping between step IDs and the KCs array as a JSON object.
bktParams.js contains a JSON object that maps KCs to their corresponding BKT parameters (probMastery, probTransit, probSlip, and probGuess). These values are typically empirically determined and can be AB tested (see above).
What the format of a section looks like Problems are decomposed into steps. Problems do not contain an answer field ( problems without real steps are formatted as a problem with only one step). Steps can be one of 2 answer types: textbox or multiple choice. Steps can contain help items which can be toggled to be shown by clicking a raised hand icon on each step. There are two possible help items: hints which are purely textual and have no user input, or scaffolds which contain user input (again, either textbox or multiple choice). Scaffolds can contain help items themselves, except this is the deepest level of content (the scaffolds's scaffolds cannot contain any help items).
OATutor uses Bayesian Knowledge Tracing to determine model mastery based on an input. (If you need to describe how BKT works, just copy the descriptions of the 4 model parameters used in BKT from wikipedia along with equations a thru d. The implementation is exactly identical to wikpedia, nothing special here)
Problem selection is determined using a heuristic which is fully configurable and can be AB tested (see above). For the purposes of this paper, let us assume we are using a heuristic that selects problems prioritizing the lowest mastery first. Upon receiving user input, the standard BKT update equations will update the predicted user's mastery. Upon completion of a problem, OATutor will iterate through all problems and compute each problem's mastery level(note: mastery level is computed at the problem granularity not the step) for the user. This is done by multiplying all the mastery priors for all KCs of that step (as labelled by the researcher in the KC model) and then multiplying all step masteries together to get the problem mastery. The heuristic will be applied, which in this case is lowest mastery first, so the problem with the lowest mastery is selected to give to the user. In the case that the first problem is being chosen in the session, equation a from the BKT model is used and the default probMastery is considered the user's mastery. Ties (of equal mastery) in the heuristic selection algorithm are broken by randomly choosing a problem.
- giveStuFeedback: controls correctness feedback (i.e. whether a user inputted the correct answer or not)
- giveStuHints: controls whether hints should be displayed or not (i.e. controls the hint icon as well)
- doMasteryUpdate: controls whether OATutor should track student mastery
- allowRecycle: controls whether problems/steps can be repeated or not
- showStuMastery: controls whether matters should be displayed to the user in the upper right corner
- unlockFirstHint: controls whether the first hint should be auto-expanded when the user clicks the hint icon
- allowDynamicHint: controls whether a dynamically generated hint should be given to the user
- giveStuBottomHint: controls whether the suer should receive a bottom-out hint (last hint in the hint pathway that contains the answer)
- giveHintOnIncorrect: controls whether an incorrect response should automatically force the user into the hint pathway
- keepMCOrder: controls whether to preserve the order of MCQ choices in the spreadsheet