Usability Research Plan – Manage User Access to the Admin Feature

Story #855 / Sprint: TBD

Lisa McMichael / 9.25.2019

Key link: https://github.com/USDAForestService/fs-open-forest-platform/issues/855

Research overview / goal / approach

The purpose of the study is to evaluate the design of the Open Forest User Management feature with participants from Region 6. The goal of the study is three-fold: to understand if the design of the Open Forest User Management feature needs improvement to satisfy users, to identify areas for improvement, and to understand what is satisfying about the design.

Research questions

• Can users complete critical tasks (i.e., success rates of 70% or higher)?
• Will users rate tasks as “Easy” or “Very Easy” with the Single Ease Question (SEQ)?
• What are the most common problems users make when completing tasks?
• Will terminology, navigational structure, and page components be easy to understand?
• Does the User Management feature meet user’s expectations (i.e., “satisfaction”)?

Participant considerations

For the “Super Admin” role, Aaron” can participate in a “cognitive walkthrough” with the researcher. And for the “Write” and “Read” roles, those tasks will be covered in testing with the Admin role “Judy” by default.

Location	Personas	#users	Access type	Forests
R6	Admin “Judy”	6	Grant + manage all features	1

Research method

The study will be a 60-minute remote-moderated test with warm-up questions, tasks for completion and closing questions. Task completion ratings (i.e., Single Ease Questions) will be captured, along with the researchers’ observations and participants’ comments.

Research findings

Each test session will be analyzed against the goals of the study. The Product Owner will receive a study summary after all tests are completed. The summary will be located on Pinyon in the Research folder. The researcher will meet with the PO and the design/development team to discuss the findings and outline next steps such as feature or content enhancements, and any follow-on research.

Research limitations

Running a Pilot test will tease out any known issues with the methodology of this research plan. If an unforeseen problem pops up while testing, adjustments can be made to the study. All of these details will be included in the study summary.

Moderator Guide/Script

The guide is not a literal script to follow, rather a framework to facilitate research activities.

Hi, there. My name is ___________, and I’m going to be walking you through this session today. Before we begin, let me remind you why we asked you here. You’ll be trying out a new feature on the Open Forest website where you can go in and manage your employees’ access to Open Forest. Please keep in mind that we’re testing the site and how it works, not you. You can’t do anything wrong here. We’d really like your honest feedback. If you have any questions as we go along, just ask them. I may not be able to answer them right away, but I’ll try to answer them at the end. And if you need to take a break at any point, just let me know. Also, your participation here is voluntary, so at any point we can stop the session if you need to.

With your permission, we’re going to record what happens on the screen and during our conversation. The recording will only be used to help us figure out how to improve the site, and it won’t be seen by anyone except the people working on this project. And it helps me, because I don’t have to take as many notes. You should have already received an informed consent form and filled it out ahead of time, so if you haven’t already, please sign that and return it after the session.

Thanks so much for helping us today. Any questions before we begin?

Introduction/Warm-up (5 mins)

Before we start and before you see the design, I’d like to hear a little bit about your role when it comes to Christmas tree permitting.

• Do you have any expectations about selling Christmas tree permits online? Or, who has access to your Open Forest to view or edit information?

Thanks. Now I’m going to ask you to respond to a few scenarios. I’m going to read each scenario out loud, and I’ll also share the text of each scenario in the chat window.

Task overview and screenshare (2 mins)

The first thing I’d like you to do is share your screen. You can do that by clicking [include details to where to click]. Before you share, make sure to close anything you don’t want recorded.

(Start the recording.) Use the observation worksheet and confirm h/she reached Staging site.

Task scenarios and questions (30 mins)

• Prototype link: https://invis.io/TZU7WFD83JS

Ask participant to Log into eAuth_

Scenario 1: Interface tour “landing page”

Take a moment and scan this page. (Prompt: Remind the participant to think out loud)

• What are you most curious about or interested in exploring? [Probe on answer].
• What do you think you can do here? [Probe on answer with “Why?”]
• Based on what you can do on this page, what would be the most important feature to you and why?

Scenario 2: Task, Select Manage Users link

You have been granted access (permission) to see your forest’s employees who are added to the administration pages. Now you want to see who they are. Show me how you would get started.

[insert image] Success target: click Manage users link

Explain the Scale and ask the Single Ease Question:

SEQ: Overall, this task was?

1 Very difficult ……………………………………. 7 Very Easy

What prompted you to give that rating?

Scenario 3: Interface tour of the page “Add user”

Take a moment and scan this page. [insert image]

• What do you think you can do here? [Probe on answer]
• Based on what you can do here, what would be the most important feature to you and why?
• We’ve included Search, what do you think you would use it for? [Probe on answer]

Scenario 4: Task, Add User

You have 2 new employees who will be “front liners” during the Christmas tree cutting season. They will need access to run reports. Show me how you would add them? Success target: click Add user link Explain the Scale and ask the Single Ease Question: SEQ: Overall, this task was? 1 Very difficult ……………………………………. 7 Very Easy What prompted you to give that rating?

Scenario 4a: Add User modal window

Tell me your impression of this information. What do you think of how it’s laid out?  

Scenario 5: Interface tour of the page “Add access”

Tell me your impression of this information. What do you think of how it’s laid out? [Encourage talking out loud, to understand if the participant knows how to get started with the design without hesitation.]

Scenario 6: Task, Add access

You want to give your employee Mona, access to make changes to season cutting dates and to run reports. Show me how you would do that. [Note: this will help us learn if terminology is understandable.]

What does “Manage access” and “Write access” suggest to you?

Success target: select Manage-access radio button and participant understands the selection made.

Explain the Scale and ask the Single Ease Question:

SEQ: Overall, this task was?

1 Very difficult ……………………………………. 7 Very Easy

What prompted you to give that rating?

Scenario 7: Confirm new admin

Tell me your impression of this information. What do you think of how it’s laid out?

• What will happen if you click “Confirm”?
• What will happen if you click “Go Back”?

Wrap-up questions

• Now that you’ve seen the design, how would you describe it to a colleague in one sentence?
• If we only changed one thing about the design to make it better for you, what would it be?
• What is your biggest concern about using this design?

Thanks for your time today. If you think of anything you want to share after this session, contact me or Amber.

I’ll end the recording now.

Enjoy your day!