-
Notifications
You must be signed in to change notification settings - Fork 97
Usability study 2
Similar to usability study 1, Colorado used the think aloud protocol, with two Berkeley graduate students (one focused in NLP and the other focused in computer graphics). Colorado provided a brief verbal description of metacademy (described as an apt-get for knowledge) and then observed the students interacting with metacademy for roughly 15 minutes, starting from metacademy.org's landing page. Both users used chrome on their high-performance mac laptops. Here's the summary points:
Here's some higher-level directions gleaned from the sessions (see below)
[Colorado] I think we should focus on:
-
focus on improving the experience for non-logged in users, i.e. saving learned/starred concepts. Most metacademy user's aren't going to create an account -
figure out how to explain/justify the validity of the various concepts/resources
-
inform the user of resource type (video|text|course|etc)
-
improve check/star clickability -
figure out how users can navigate from a concept to subsequent concepts
-
figure out search results for broad concepts
-
fix "all text selected" problem
-
perhaps we should figure out how to make the resources more central to the learning display (emphasize that this isn't wikipedia)
-
Noticed GP reference in main search box and said "ah, I can see who you're orienting this towards," and then read the footer on main page and said, "ah, right"
-
search for "hierarchical dircihlet process" -- search return HDP page, clicked on HDP
-
scanned summary and asked "where did this text come from, why should I trust this author?" (I believe Roger brought up this point previously)
-
clicked disabled hide/show learned concepts button, nothing happened
-
clicked the graph view button -- big scary graph
-
said that the graph was essentially too big/complicated to be helpful, especially on more complex topics (he mentioned later that it might provide a nice initial visualization, but probably wouldn't have much of a purpose when actually trying to learn the concepts)
-
started at the top of the list and started systematically checking off the concepts he knew
-
found checks easily on the list, but had a few misclicks, and at first, he thought he couldn't unclick the checks once they were clicked (NOTE: creating clickable margins around the check/star might help)
-
This user checkout the resources and liked the exact location references
-
this user did not explore the star buttons
-
tons of clicking to mark the concepts he'd learned (at the end of the session, I asked if a "linear algebra" course would be helpful to mark a bunch of concepts at once, he was skeptical of the idea since linear algebra courses vary so much, but he was receptive to the idea of clicking a "linear algebra" course and then seeing a full checked list of concepts that he could curate
-
navigated to a different page then back to the original page and was a bit perturbed to see his checked concepts had disappeared
-
checkmark doesn't seem to disappear when clicking an on checkmark to an off checkmark (the color change wasn't very noticeable on his laptop)
-
question: does checking a shortcut concept also check the main concept?
-
thought it would be cool to group the concepts by topic in the learning view (e.g. work though probability theory and then linear algebra)
-
thought color brackets or tags to separate the different concept categories might be interesting
-
When examining resources, thought it would be nice to know if a resource was a video or pdf or textbook in advance
-
question when scanning the HDP: is this everything on metacademy or is this just the HDP?
-
user used the back button to navigate between views
-
searched for broad concept, "measure theory," and was disappointed by the results
-
confused by dashed lines in explore view around the shortcut nodes
-
encountered the "all text selected problem" where all of the titles become selected in the explore view and its hard to unselect the text
-
initial question -- only for machine learning?
-
searched for "logistic regression" (I mentioned this concept while explaining metacademy)
-
was a little disoriented by the learning view
-
starting from the HDP he climbed the dependency structure by clicking on the links he didn't know in the prereqs section
-
found the graph view easily
-
likes the quick summary in the explore view
-
"oh that's nice" when noticing the resources
-
confused by star vs check buttons, finally decided "oh those must have something to do with a logged in account"
-
tried clicking the greyed out clear/show learned buttons, not sure what they do
-
reading about kernels, he asks "what's a kernel" and wasn't able to answer this questions from the summary text of "kernel trick" and nearby concepts -- seemed frustrated -- confused if a kernel is an inner product or a kernel is a specific linear subspace. Clicked on Coursera link and quickly clicked away -- "oh I have to sign up for an entire course" (didn't notice notes mentioning that he could click the preview button)
-
User mentioned that he's largely viewing this resource like wikipedia and wanted to be able to learn the concepts without going to external resources
-
the used liked the graph/list layout and display; he complemented the color scheme, presentation, and ample use of whitespace
-
searched for "neural network" and was confused as to why their wasn't an entry for "neural network"
-
scrolled through the full concept list and clicked on QR decomposition -- wanted to know what he could learn given that we knew QR decomposition; want to see what depends on QR decompostion (would be nice to be able to do this from the graph view)
-
thinks that a border around the list icon in the explore view (on the nodes) would make it more obvious that it's a list
-
thought a light gradient in the learning view list could indicate a progression of the concepts
-
clicked the root concept and then clicked hide and was confused as to why everything disappeared
Notes: the user mentioned that dynamic graph generation/manipulation is an open problem in the field and removing/adding nodes and keeping fluidity in the display is a really hard problem (he was skeptical that we could improve much on our current graph generation technique)