A4. Preparation for User Testing

Home / Learning about and preparing for a user testing study

Interview criteria and principles

• Listen more than you speak
• Ask questions in a straightforward, clear, and nonthreatening manner
• Use 'I' instead of 'we' to keep the dialogue more personal
• Eliminate cues that lead participants to respond in a particular way
• Be curious, interested, and engaged
• Enjoy the interview
• Refer to Interview Guidelines for more information

User Research Schedule

• Pilot Date and Time
• Research Date and Time
• School
• Subject(s) Names
• Subject(s) Roles
• Summary Document Location
• Recording Location
• Review Material Document Location

Introductions and Initial Discussion Questions

• Let us introduce ourselves.
• Please provide a quick overview of your role for the benefit of the recording.
• Do you use Canvas and how often?
• Do you consider yourself an expert, fluent or beginner Canvas user?

Introduction

• I will be leading the discussion, but they may jump in from time to time with questions of their own.
• Quick overview of project (example): This is a redesign of the Canvas interface. Nothing you will see has been released yet; we are still very much in the design and development phases, and your feedback will directly impact the design direction that we go in. Please be mindful that, although very useful and impactful, we cannot guarantee that the outcome this meeting will be implemented at a specific time and design.
• Don't hesitate to ask questions during the session. I may not answer them right away, because I want to see what conclusions you come to on your own. But it's still important that you ask, so we can see what questions are arising for you as you explore the prototype.
• You’ll notice that I will often ask “why?” to your answers. This is not an indication of lack of clarity, rather a technique for prompting additional details and feedback from you.
• Keep in mind that we're testing the prototype, not you. There are no right or wrong answers. Please think out loud as much as possible as you view the prototype and say whatever comes to mind, without any concern that your feedback or criticism may be taken other than constructive - I just want to hear your honest reactions and thoughts on what we're showing you.
• For two participants: For the sake of simplicity, I will only direct questions toward primary participant. However, if the other person has a difference of opinion on any portion of the prototype, then please speak up so we can make note of your opinion as well.
• In order for us to capture your experience to the fullest, we would like your permission to record you as you work through the tasks mentioned above. The recording will help us to collect your feedback with more accuracy. This recording will be kept private and only shared with the team that is working on the development of the tool. Do we have your permission?
• Any questions at this time?

##Instructions for user testing using a task list read by the user I’m going to give you a printed copy of the tasks. I’d like you to read each task aloud, and then use the site to perform the task. As you work through the tasks, please talk aloud and describe what you’re thinking as you answer. If you aren’t sure what to do, feel free to say that. This isn’t a test of your abilities - if it’s confusing to you, it may be for others and it’s our job to fix that. After each task, we may need to pause as I write a few notes down. [Add any special instructions here.]

Instructions for initial questions

• How often do you anticipate that you would use this tool?
• How often do you use Canvas?
• Which Canvas tool do you enjoy using the most?
• I’m going to show the [tool name], could you answer some questions:
• Do you feel comfortable to say that; you know where you are? If not, what would you change?
• Could you describe the page as you see it?
• Is there anything missing within the context of the app?

Instruction for pilot session

Take advantage of this exercise to test the meeting room and equipment (if possible) and other preparation activities by using the very same setup as for the actual user testing session. This often helps to learn early enough about problems (display cable, internet availability and strength, privacy, distracting noises and activities from adjacent spaces) that could be resolved by the time the actual testing is staged.

Instruction for testing session

Before each test: Use an incognito browser window and have website open. [insert link]

User research questions

• How was your experience with the prototype?
• Is there a feature that worked particularly well or bad for you?
• How do you see this tool enhancing your effectiveness at work?
• How often will you use this tool?
• Have you used any other A/B testing tool?
• What did you like/dislike?
  • What would you like to see implemented in ours?
  • What is the No. 1 thing you want to be able to do with this tool?
• What does success looks like to you?
• Tell me about your job.
• Walk me through a typical day in your life.
• How often are you online?
• What is your biggest pain points?
• What computers or devices do you use?
  • When do you use each of them?
  • Do you share any of them?
• What do you typically do online?
• How do you use Canvas?
  • Where does it fall short?
  • What doesn't it do or do well that you need it to do?