Usability Testing — Special Use permits evaluation content (June 2019)

This study involves testing different content flow strategies for the Special Use permit website to see that the right amount of content surfaces at the appropriate time, resulting in faster, more efficient, and better decision-making by the prospective special use permit applicant.

Test session details

• 5 outfitter participants. No non-commercial testers were available which may have skewed the data slightly; conclusions take this into consideration.

• 5 total A/B testing sessions were performed where two designs were tested. Outfitter testers were professional outfitters and/or instructors who were found through product owner Amber Sprinkle’s contact list. Each testing session was scheduled for 45 minutes and were on time.

Session overview

Each tester was given two scenarios per session as noted below. Scenario’s were given alternative for each of the five sessions:

• Scenario One: Outfitter persona —Site A, then Non-commercial persona —Site B
• Scenario Two: Non-commercial persona —Site B, then Outfitter persona —Site A

General Outcome

POSITIVE FEEDBACK

• Participants appreciated the digitized, online permitting effort.
• Expectations are high and a lot of anticipation by the users for an online permit for USFS.
• The site experience was very appreciated, participants noted it’s much better than other government sites.
• Participants liked the use of images and USFS branding colors.

OPPORTUNITIES

• Participant noted uncertainty about the use of “Open Forest” on the site without context.
• Participants preferred for the permit landing page to have two buttons in the boxes “Learn More” and “Apply” to accommodate the different user groups.
• Participants preferred less text on the home page, but agreed it was important that it be enough to make the right choice.
• It was agreed that the content in the two permit boxes needed to be refined:
• • Enough information to make a decision, but not too wordy.
• • Bullets might be easier to read
• • No italics
• • Use plain language
• The site in general needs a navigation menu so it's easier to access different pages quickly and easily.
• Observed that users rarely used the breadcrumbs (which was added above the left anchor nav list on the detail page) and instead relied on the back button to navigate to pages.
• Temporary Outfitter participants did not find the wizard helpful.
• Participants noted the details page language too much like a legal doc, should be more like the homepage using “you”. Not so technical, more friendly.
• Participants noted the detail pages weren’t appealing, too much text, not easy to scan.
• Participants noted that the flow to the Login.gov page was a bit jarring (very different look and feel, not expecting to login).
• Participants asked if we could provide templates or examples of the information needed like liability insurance or the application - this would be helpful and reduce the time to complete applications
• Left navigation menu not used - new solution for long detail page.

Results and insights

General Landing Page

• Participants saw a clear choice between two permits
• Participant noted if there was a need for a non-standard permit that did not fit into this scenario —what they would do in that case
• Most participants were able to make a decision by just reading the title of the two boxes and most did not read the content until asked by facilitator
• Participant mentioned seeing “permits” instead of “Special Use permit” was confusing
• Participant mentioned that other uses cases —not non-commercial group nor temporary use— were not clearly accounted for on the landing page
• Participants preferred option of LEARN NOW and APPLY NOW buttons on Site A landing page
• Participants preferred simple shorter text of Site B landing page (not verbiage but length)
• Participant mentioned the lack of mapping tools to understand what use area is applicable
• Participant mentioned “Open Forest” and not having enough context in branding
• Wizard was not used by any user tested - testers were only temp-outfitters - “First time and non-commercial might find the wizard helpful”

Site A

Landing Page

Once encouraged to read content the following comments were made

• Both the time the application takes and the things you need to apply were important and appreciated.
• Other details were lost in the paragraph and hard to read in italic (image very pixelated because it’s an image twice reproduced so made this difficult to assess).

Site B

Landing Page

• Participants were able to quickly decided on which permit they needed based on the scenario and the title in each box—most didn’t take the time to read the content until asked later
• Most liked and preferred the shorter content of this prototype. Participants knew to click button when asked “What would you do to apply for this permit?”
• Some participants felt having to go to the second page to find the ‘apply’ button wasn’t useful in the case that they’ve applied for this type of permit before and knew what to do

General Detail Page

• Participants found the content very useful and informative
• Participants thought the language needed to be less legalese in some areas; ex. remove "revocable, suspendable, and not appealable"
• Participants didn’t notice or use the left navigation
• When asked to return to the home page, bread crumbs were not seen - all testers used back button after searching the page with the exception of one
• One participant saw the bread crumbs but wasn’t sure what it was - wasn’t sure what the name “Special Use Permits’ was so hesitant to click on it. (Participant suggested calling it ‘home’)
• Participants mentioned the cost structure table and service days were confusing
• Participant mentioned the Evaluation Period and What to Expect sections on were conflicted and confusing.
• Participants thought it wasn’t clear how long it would take overall; concerned it might interfere with scheduling.
• Participant mentioned that templates of expected documents for the application might help to speed up the submission process
• Participant noted that on What to Expect that defined due dates might be helpful

General Login.gov page

• Some felt it was jarring to go to a page so different, others recognized it as part of another site that’s secure.
• Participants wanted forewarning about being led to a differently branded site to authenticate
• One participant thought the difference in the look and feel of the logon.gov page was fine, screenflow matched expectations
• Some noted there was no way to get back to home page if you change your mind.
• No link (in our prototype) to create a new account.

Recommendations

LANDING

• Pain point 1: Can’t access Apply button from landing page—makes it hard for a returning applicant who doesn’t need to read details to apply quickly without clicking to another page

• • Possible solution: Give the option of two buttons on the landing page (Learn More and Apply) like site A

• Pain point 2: Participants noted a preference for less-is-more content and preferred bulleted text and non-italics on the landing

• • Possible solution: Use the copy and styling of site B for home page content but make it bulleted for each item to make it easy to scan, and add length of time it takes, and what they need

• Pain point 3: The use of Open Forest and Special Use Permits still unclear and could be clarified

• • Possible solution: Add Open Forest to top banner, and write a short two sentences about Open Forest and the plan on the page someplace.
• • Possible solution: Add “Special use permits” as the title of the page

DETAILS PAGES

• Pain point 4: Plain language needed for details page—use 'you' instead of third party

• • Possible solution: Rewrite content to be more user friendly

• Pain point 5: Formula for number of service and permit days was very confusing to everyone who tried to understand the information presented

• • Possible solution: Possible use a small formula, or maybe a wizard to help users understand the fee

• Pain point 6: Length of time the permit takes was not clear. Currently the screen says, “'...after approval within 2 weeks" but not how long the approval will take

• • Possible solution: Suggest there is a general statement such as "Expect at least three weeks for approval and the delivery of your permit after your request is submitted to the Open Forest system" so they can plan

LOGIN

• Paint point 7: Some users were surprised when the page appeared (not all), so better context needed for user
• • Possible solution: Include some clue on the permit site in the graphical user interface that prepares them for the difference by bringing more of login.gov branding to the link

OPEN FOREST TEMPLATES

• Pain point 8: Detail pages are more like a legal document. Should be more engaging.

• • Possible solution: Alternate text with illustrations like the Christmas tree site (to maintain look and feel). Add some design elements like styled divider bars

• Pain point 10: Left anchor nav not utilized or identified—participants scrolled instead

• • Possible solution: Remove left nav so content can spread full page, break the content up into sections with dividers, find a solution to anchor links that aren't in a left nav - possible text links at the top that are 'sticky, or at the top of each section

• Pain point 11: Bread crumb not used when user encouraged to go back to home page from details page (used back button)

• • Possible solution: Bread crumbs stand out more away from other elements. Or, adding a site navigation might help solve the issue of how to return to the landing page.

Select quotes

• “Good to see the Forest Service take this project on, as we all have had frustrations with the permits”
• "The more text [there is], the less people read them"
• “It might be useful to add some text that explains that if these aren’t the permits you’re looking for, here is a link to a page that will help”
• “The preference is A (prototype) - because I can skip the ‘Learn more’, I just want to get to it as someone who applies or renews every year.”
• “If I’ve done the application many times it helps to have the (apply) button on the home page, but not a deal breaker.”
• "Really like the decision tree, one decision at each point" for B-prototype
• “I like the B landing descriptions because they are simpler—I would merge the two, adding the learn more button to B landing would be helpful”
• "Would remove the passive voice on learn more page"
• "Defined due dates [on What to Expect] would be helpful"
• "I find it confusing as far as permitting rules"
• “You need to know what to do before hitting apply first” (about landing A/B)
• “Information to get approved is important”
• “Would be good to have a map so we can see the area so we can contact the land manager. We are using a printed map we sell in our store, so that would be very helpful online.”
• “National parks and forests, I get confused. Both have different processes. I usually get a pdf, I’ll open it and fill out in acrobat, or throw it right into docusign and fill it out and digital sign. That way I don’t have to fill it out with a pen.”
• “It was good (information on the site), got all the info I needed for this permit. It’s very simple.”
• “A lot nicer than I’ve seen previously for the forest service. Laid out in a clear way.”
• “Will there be a landing page for us 10 year holder where we can upload our info up and just go and it’s done?”