Usability Testing at OpenEd (1 of 2): Testing Methods and Procedure

We conducted our first round of usability testing at this years Open Education Conference. It took us nearly four months to create a mockup that we think is a good starting point for something that educators might want to use and that is comprehensive enough to be tested. The mockup we tested is shown in the photo below.

A photo of the mockup we tested:

For those interested in seeing more than just a photo, below are links to the actual mockups we used for testing (beware the mockups are not fully functional and have some bugs — also, the mockups work only in Firefox):

Links to the Mockups we tested:

The Usability Test

The purpose of this test was to get answers to the following questions:

  • What editing programs do educators currently use to create educational material?
  • Do participants have positive reactions to the editor?
  • Is it clear that the editor is to be used to create educational material?
  • What parts of the editor do participants discover on their own?
  • Will users be able to discover and properly use key parts of the editor including:
    • inserting pedagogical supports
    • inserting tables, adding some content, adding a row
    • inserting an image, caption and descriptive text
    • creating section headers
    • creating links
    • inserting math
  • How usable do participants perceive the editor to be?

Our usability test consisted of the following tasks:

  1. Explore the editor
  2. Insert an exercise (here is a short video showing what that looks like in the editor)
  3. Insert a table, add some content, and add a new row
  4. Insert an image, title, caption and descriptive text for the visually impaired
  5. Make a section header
  6. Create a link to a web page
  7. Create a link to another part of the document

If the participant was a math instructor, math author, or volunteered to test the math portion of the editor, they were also given these tasks to execute using the math editing mockup:

  1. Edit an existing equation (another short video showing this operation)
  2. Add a new equation
  3. Convert LaTeX mark-up, or plain text, into an equation

For those interested, here is a link to our actual testing script–try taking the test yourself!

The Testing Procedure

Upon arrival, all participants were given this pre-test questionnaire. The questionnaire assessed: 1) whether the participant authored educational content (and what tools they used if they did); 2) whether the participant felt comfortable editing math, and 3) whether the participant saw a lecture and demonstration about the editor beforehand. Next, participants were read this orientation script informing them of what to expect during the test. Then participants were successively given each testing task to complete. Lastly, participants were asked to complete the Subjective Usability Scale (SUS) via SurveyMonkey. After the testing session, participants were allowed to ask any technical questions they had.

 Read the next post for a summary of the general findings from each task.

Next blog post: Usability testing at OpenED 2 of 2