Skip to content
This repository has been archived by the owner on Jul 6, 2023. It is now read-only.

Usability study 2

Colorado Reed edited this page Sep 23, 2013 · 9 revisions

Similar to usability study 1, Colorado used the think aloud protocol, with two Berkeley graduate students (one focused in NLP and the other focused in computer graphics). Colorado provided a brief verbal description of metacademy (described as an apt-get for knowledge) and then observed the students interacting with metacademy for roughly 15 minutes, starting from metacademy.org's landing page. Both users used chrome on their high-performance mac laptops. Here's the summary points:

Here's some higher-level directions gleaned from the sessions (see below)

Directions from feedback

[Colorado] I think we should focus on:

  • focus on improving the experience for non-logged in users, i.e. saving learned/starred concepts. Most metacademy user's aren't going to create an account

  • figure out how to explain/justify the validity of the various concepts/resources

  • inform the user of resource type (video|text|course|etc)

  • improve check/star clickability

  • figure out how users can navigate from a concept to subsequent concepts

  • figure out search results for broad concepts

  • fix "all text selected" problem

  • perhaps we should figure out how to make the resources more central to the learning display (emphasize that this isn't wikipedia)

User 1 (NLP focused phd student)

  1. Noticed GP reference in main search box and said "ah, I can see who you're orienting this towards," and then read the footer on main page and said, "ah, right"

  2. search for "hierarchical dircihlet process" -- search return HDP page, clicked on HDP

  3. scanned summary and asked "where did this text come from, why should I trust this author?" (I believe Roger brought up this point previously)

  4. clicked disabled hide/show learned concepts button, nothing happened

  5. clicked the graph view button -- big scary graph

  6. said that the graph was essentially too big/complicated to be helpful, especially on more complex topics (he mentioned later that it might provide a nice initial visualization, but probably wouldn't have much of a purpose when actually trying to learn the concepts)

  7. started at the top of the list and started systematically checking off the concepts he knew

  8. found checks easily on the list, but had a few misclicks, and at first, he thought he couldn't unclick the checks once they were clicked (NOTE: creating clickable margins around the check/star might help)

  9. This user checkout the resources and liked the exact location references

  10. this user did not explore the star buttons

  11. tons of clicking to mark the concepts he'd learned (at the end of the session, I asked if a "linear algebra" course would be helpful to mark a bunch of concepts at once, he was skeptical of the idea since linear algebra courses vary so much, but he was receptive to the idea of clicking a "linear algebra" course and then seeing a full checked list of concepts that he could curate

  12. navigated to a different page then back to the original page and was a bit perturbed to see his checked concepts had disappeared

  13. checkmark doesn't seem to disappear when clicking an on checkmark to an off checkmark (the color change wasn't very noticeable on his laptop)

  14. question: does checking a shortcut concept also check the main concept?

  15. thought it would be cool to group the concepts by topic in the learning view (e.g. work though probability theory and then linear algebra)

  16. thought color brackets or tags to separate the different concept categories might be interesting

  17. When examining resources, thought it would be nice to know if a resource was a video or pdf or textbook in advance

  18. question when scanning the HDP: is this everything on metacademy or is this just the HDP?

  19. user used the back button to navigate between views

  20. searched for broad concept, "measure theory," and was disappointed by the results

  21. confused by dashed lines in explore view around the shortcut nodes

  22. encountered the "all text selected problem" where all of the titles become selected in the explore view and its hard to unselect the text

User 2 (computational graphics focused phd student)

  1. initial question -- only for machine learning?

  2. searched for "logistic regression" (I mentioned this concept while explaining metacademy)

  3. was a little disoriented by the learning view

  4. starting from the HDP he climbed the dependency structure by clicking on the links he didn't know in the prereqs section

  5. found the graph view easily

  6. likes the quick summary in the explore view

  7. "oh that's nice" when noticing the resources

  8. confused by star vs check buttons, finally decided "oh those must have something to do with a logged in account"

  9. tried clicking the greyed out clear/show learned buttons, not sure what they do

  10. reading about kernels, he asks "what's a kernel" and wasn't able to answer this questions from the summary text of "kernel trick" and nearby concepts -- seemed frustrated -- confused if a kernel is an inner product or a kernel is a specific linear subspace. Clicked on Coursera link and quickly clicked away -- "oh I have to sign up for an entire course" (didn't notice notes mentioning that he could click the preview button)

  11. User mentioned that he's largely viewing this resource like wikipedia and wanted to be able to learn the concepts without going to external resources

  12. the used liked the graph/list layout and display; he complemented the color scheme, presentation, and ample use of whitespace

  13. searched for "neural network" and was confused as to why their wasn't an entry for "neural network"

  14. scrolled through the full concept list and clicked on QR decomposition -- wanted to know what he could learn given that we knew QR decomposition; want to see what depends on QR decompostion (would be nice to be able to do this from the graph view)

  15. thinks that a border around the list icon in the explore view (on the nodes) would make it more obvious that it's a list

  16. thought a light gradient in the learning view list could indicate a progression of the concepts

  17. clicked the root concept and then clicked hide and was confused as to why everything disappeared

Notes: the user mentioned that dynamic graph generation/manipulation is an open problem in the field and removing/adding nodes and keeping fluidity in the display is a really hard problem (he was skeptical that we could improve much on our current graph generation technique)