Workshop | UX Study
This workshop is designed to offer you experience with executing a scenario-driven, think-aloud protocol in preparation for designing your own user-centered usability study. During the workshop, you will follow a study prompt that I created to help you understand what it means to run a research trial, review and interpret your data, and organize your thoughts in a Summary Report.
Learning Objectives
- Examine materials relevant to designing and executing a usability study.
- Execute one trial of a usability study.
- Interpret data from that study.
- Present your results in a summary report.
Workshop Deliverables
Your work on this assignment will result in the crafting of a summary report. Link the video of your study trial from your Screencast.com account to your report.
For the complete details about how to submit your work, see Submission Requirements.
Workshop Pathways & Knowledge Building
The primary focus of this workshop experience is learning what it means to participate in a scenario-driven think-aloud protocol. This is a specific kind of UX study that connects prospective users of a design with the design itself with the goal of completing one or more meaningful tasks appropriate to the function of the object of study. This is another research method that comes from UX Design. In the "UX Methods Toolkit," I provide the following description of these linked research tools.
Problem-Solving Scenarios (PSSs) are a hybrid method that spans the experience-centered and narrative-dialogic categories. PSSs represent one of the most commonly applied methods for studying usability. The concept is simple. To study the way users interact with a particular design, you construct a task (or series of tasks) typical to using that design, and observe as participants attempt to complete the scenario.
Some designs (e.g., apps, web sites, video games) embed instructions for using the thing itself in the interface. In the case of something like that, or in the case of a technology that we generally conceive of as intuitive (e.g., web sites), then the scenario ought to be designed based on the content of the site, or based on outcomes the thing is designed to create. PSSs are
the standard method for assessing instructional documents.
Think-aloud Protocols (TAP) are mixed methods (i.e., experience-centered, narrative-dialogic) that invite participants to share their thoughts (e.g., decision making, choices, evaluations, reflections) in the moment about a design encounter as they are engaged in a study. By asking
participants to think out loud, we can understand why they choose the pathway they do through the research trial in which they are participating.
Think-aloud protocol is a complementary method that we
will employ in combination with problem-solving scenarios.
Workshop Details
This workshop requires you to conduct a single trial for a usability study that I have designed. It implements the problem-solving scenario method in combination with a think-aloud protocol. This is one of the methods available to you for your work on the Design Assessment project.
Ordinarily, this study would be completed in a team with 3 or 4 participants, each taking on a different role for the study. If you are able to work with one or more classmates to complete the study (or even someone who is not enrolled in the course, but who is willing to assist you), do so. If not, follow the alternative instructions for solo work (designated from now on by the term Solo).
Review each stage of the description for more details. The workshop has 4 stages.
- Prepare for the study.
- Conduct the study.
- Interpret the results of the study.
- Complete and submit a summary report.
Prepare for the Study
When you prepare to complete this workshop, attend to the following details. Note that the first decision is who will work with you (required during in-person instruction, optional for remote sections).
- Option 1. Team up with two classmates. Studies are most efficiently managed if you have three people: one to run the study, one to serve as the trial participant, and one to film the process. If the number of people present for the workshop does not allow for an even distribution of three-person groups, some groups can have four members. Decide on your roles before you go any further (coordinator, participant, film assistant).
- Option 2. Work alone (Solo). If you are unable to partner with another person to complete the study, work alone. Look for the ways that the instructions vary to allow you to do so. In those instances where there is no Solo variation offered, follow the standard instruction for how to proceed.
Once your team organization is decided, continue with your preparations.
- Review Krug's think-aloud protocol. Review Stephen Krug's demonstration video for scenario-driven think-aloud protocols: YouTube.com: Usability Test Demo by Steve Krug. Listen as he explains the basic procedure for this kind of research trial. Note the way he prompts the participant to explain to him what she is doing. You'll want to emulate his actions. (Note that if the link fails for some reason, search YouTube.com for the phrase displayed for the link above. It will get you to the content you need.)
Do not skip this step. Krug's demonstration is essential to your knowledge for how to conduct this kind of usability study.
- Retrieve the workshop scenario. Retrieve the scenario document from SVSU Canvas Files in the Workshop Support folder. If you are working with a team, only the study Coordinator should view the scenario. The Coordinator should take a moment to review the document to make sure s/he/they understand the steps involved.
- Retrieve the scenario (Solo). Retrieve the workshop scenario document from SVSU Canvas Files in the Workshop Support folder. Review the scenario to make sure you understand the steps involved.
- Identify your recording device. Recording video and audio of this trial is essential to your success with the workshop.
- If you work in a team, I recommend that your designated film person record on his/her/their smart phone.
- If you work alone, it will be less easy to record yourself. If you have access to an app or service that allows you to record your screen and yourself at the same time (e.g., Techsmith Capture from your Screencast.com account, OBS Studio - visit OBSProject.com), that is definitely the easy option. If not, do the best you can to record yourself and your screen using a smartphone or digital video camera, whichever you have access to. The goal is to capture you and your screen in the video.
-
- Test your recording device. Make sure your video recording device/system is ready to go. If you use a device such as a smartphone, make sure you have plenty of memory free for recording. Make sure the camera and microphone are both functioning properly. Conduct a brief test recording to make sure your video and audio are clear.
Conduct the Study
The study itself is the core part of this research process. Attention to detail during preparation and the study itself make it more likely that your work will result in meaningful data for later examination. Note that a video recording with clear audio is essential to your ability to gather quality data.
Prior to and during the study, keep in mind the following details.
- Compose the shot. Once the participant and coordinator have settled into their spots, frame the shot you will record. If possible, get both people in frame. If that is not possible, make sure you have a clear shot of the participant and the work space, including the computer screen.
- Designate the work screen as your recording window (Solo). Make sure that the browser window is designated as your record window if you are able to record a screencast with you on screen as well. Otherwise, do your best to capture you and your work screen on your video.
- Monitor the recording as you go. Make sure there are no problems during the recording itself. If any problems arise, ask the participant to pause for a moment while you sort things out.
- Direct the study. The study coordinator introduces the study. Use the script from the task description. Remind the participant that however she or he performs, the study gathers data. In other words, the study does not evaluate the participant, but rather the web site. Keep things moving, if there is a need to do so.
- Keep the protocol on task. Remind the participant to talk about what she or he is doing during the study. If this is necessary, phrase your prompt as if you are encouraging dialog between the participant and yourself: "Explain to me what you are doing." "Tell me what you are thinking."
- Intervene, if necessary. If the participant gets stuck, you are free to assist. Before you intervene, however, ask the participant to think her or his way through the process: "What is giving you trouble? Do you see any way of working out the problem?"
- Work through the scenario to the best of your ability (Solo). You will not be able to monitor or direct the study or intervene if you are working alone. Do your best to explain to the camera what you are doing and why as you work. If you get stuck, talk your way through it.
- Save your video. Export your video from the device you recorded with in MP4 format. Note that this is a standard video format. You will upload this video to your Screencast.com account later.
Interpret the Results of the Study
Review your video after the trial is complete. What do you notice about the participant's/your thinking and doing strategies? Does the process go easily? If not, what problems arise? Were there steps in the process that the study participant struggled with? What caused those issues? Do you notice any problems you have in managing the study? What could have gone better?
Note. Do not skip this step. This is essential to the study.
Complete the Summary Report
The final stage of the workshop requires you to craft and submit a summary report. Whether you worked independently or in a team, prepare and submit your own report. Locate the forum dedicated to this workshop on Canvas Discussions, and follow the guidelines offered there.
Your report should include the following discussion elements.
- Overview. Describe the overall sense of how your study went (e.g., it went well, there were problems), and how easily you/your participant managed the scenario (50 words).
- Observations. What were the key moments from the study? Where did you/the participant seem to have an easy time with the trial? Where did you/the participant struggle? Share direct quotes/statements from the think-aloud protocol (150 to 200 words).
- Reflections. What did you learn about the Object of Study from the trial? What conclusions might you draw from the participant's experience? What did you learn about conducting this kind of study? What are the biggest challenges, and how might you overcome them? (150 words).
- Link to video. Include the SHARE link to your video on Screencast.com. Do not attach your video file to your post. Do not store you video file in your Dropbox account.
Craft your report directly in the forum or in a simple text editor (and then paste it into a response field). Signal sections of your discussion with appropriate headings.
Submission Requirements
Craft your summary report in the forum dedicated to this workshop on Canvas Discussions. Do not create a separate document for this report. Compose it in the forum.
Upload the MP4 of your session recording to Screencast.com, and include the SHARE link to that file in your report.
Note: Do not attach the recording of your work session. Do not post it to your Dropbox space either. Doing so will consume storage space you may need later this semester.
Evaluation Standards
This section describes the standards by which your work will be evaluated for this workshop. Attend carefully to these details. If you do so, you will earn full credit for the workshop.
Evaluation Summary
There are 50 possible points for this workshop. You will earn points according to this standard.
- 50 points (full credit) for a complete summary report posted to the Discussion forum on SVSU Canvas, including sharing the video from your study by the means described above. The report must meet the specifications defined in this description.
- 40 points (partial credit) for work submitted that does not fulfill all of the posted workshop requirements, or that is submitted late, but within 72 hours of the posted deadline.
- 0 points (no credit) for work that is submitted more than 72 hours late.
A Note to Teaching Colleagues and Other Professionals
This material has evolved over many years of teaching & research, and is protected by U.S. copyright laws.
If you are here because of random chance, or because this content came up in a search, then please feel free to explore the site. If you are a teacher or other professional in any context who would like to use any of my course content in your work, I grant you permission to do so with the following limitations.
- Give me credit for my work through proper citation practices.
- Contact me at ProfBillWilliamson @ gmail.com to let me know what you are using and how you are implementing it into your work. If you adapt or alter the content, please share with me how you did so. (I appreciate seeing the ways that others apply the ideas I have developed.)
- You do not have permission to reuse, repackage, or otherwise alter my content to sell in any context.