Moderated usability testing is crucial during the discovery phase of a project. The research stage helps gather feedback from real users to uncover new customer problems and opportunities that aid in the creation of a roadmap for product and experimentation teams.
To ensure relevant and useful feedback, research participants should reflect the main characteristics and behaviours of your target users, making effective recruitment fundamental.
The recruitment process, therefore, significantly influences the success of usability testing efforts. Recent projects at Creative CX have highlighted several critical do’s and don’ts in moderated usability testing recruitment.
1. The Incentive Dilemma
Some might think that providing incentives to participants could introduce bias and alter their natural behaviour. This is why careful moderation and observation are crucial.
Right at the start of a session, it’s important to tell the participant that they should behave as if they were at home. If they would abandon a task, or leave a website while on their own, they should do so too during the test.
Also, observing is often more insightful than relying on what participants say. For example, a user might express that a task was easy when asked, but close observation may reveal struggles.
Incentives, even if not extravagant, play a significant role in attracting participants. People value their time, and compensating them for their input shows respect and motivates them.
Whether in the form of a gift card, discount, or small token of appreciation, incentives can substantially increase interest and commitment.
In our experience, we’ve run projects where the recruitment process had to be managed internally, and no incentive was provided to potential participants, although we did not recommend this approach. This decision caused considerable disruption, affecting recruitment success, project timeline, and budget.
Without incentives, we faced challenges in generating enough interest for the moderated usability sessions. We had to send multiple rounds of recruiting emails to reach the required number of participants for testing. Even when we believed we had all participants, there were last-minute cancellations that forced us o recruit again, delaying the project.
A lack of motivation not only slows down the project, but also diminishes the quality of insights gained. For instance, one participant who had to use their phone for the moderated usability session wasn’t happy about it, and without the motivation of an inventive payment, decided to cut the session short.
2. Recruiting Agency vs Recruitment Platform
At times, we have been restricted to recruiting participants via the user panels which come with usability testing platforms. While this approach might appear efficient, it can bring along its own set of challenges that impact the authenticity of the feedback.
Striving for a diverse and representative group of users is essential. However, a consistent challenge with using panels of users, and the main reason why we try not to use panels, is the ‘professional’ testers that we encounter.
These factors affect the quality of the feedback, as some participants’ expertise in moderated usability testing influences their reactions in a way that isn’t entirely genuine. Furthermore, the profiles of other participants might not always be as representative as required.
When avoiding the use of user panels isn’t feasible, a robust screener becomes indispensable. This screener is essential for filtering out profiles, particularly those of highly experiences users, whose involvement could potentially skew the testing outcomes. For example, a question that informs users that “we’re looking for participants with a range of experience in attending research” can be a particularly effective way of filtering out these experienced candidates.
However, these challenges will only be completely avoided when using a professional recruitment company. These recruiters specialise in sourcing participants which meet your target audience requirements, based on a wide variety of criteria. They also assess participant quality by looking at factors like communication skills, reliability, and technological proficiency. Crucially, they can also find people who are actively looking for the product/service that you need to test, helping to keep your tests as realistic as possible.
3. Ensuring Participants Attendance
Adding to the challenges mentioned above, due to time constraints, we have sometimes had to schedule moderated usability sessions on both Mondays and Fridays. These days are typically avoided for such sessions because, even with incentives, people are more likely to cancel or not show up due to work-related commitments or weekend plans.
Scheduling sessions on these days further validated our prior experiences and reiterated the significant of selecting the right timing for moderated usability testing. This significantly impacts participants rates. So, focusing on usability testing sessions from Tuesday to Thursday is also a key consideration to minimise no-shows.
4. Recruitment for Smaller Budgets
We’ve also encountered clients whose budget couldn’t accommodate a recruitment agency or platform. In such instances, we had to resort to more creative solutions, such as guerrilla testing in strategic areas where we knew we’d find potential target users for the brand. For example, we suggested a gold equipment brand visit golf clubs to recruit participants on-site, offering them a beer for a fifteen-minute session. Though not ideal, this approach allowed us to reach our target customers and gather relevant feedback.
In Conclusion
The moderated usability testing recruitment process is a critical phase that requires careful consideration and strategic planning. To make sure that this happens smoothly and the results are fruitful, begin by carefully selecting and offering an attractive incentive that resonates with the target user group. This incentive can range from monetary rewards to gift cards or even exclusive access to your product’s new features, depending on the budget and audience.
When recruiting participants, the most effective approach, the one that we prefer to recommend to our clients, is to work with a professional recruitment agency. This ensures that your reach your actual target audience, whether they are looking to book a flight, buy sports equipment, or switch insurance providers. While this option comes with an additional cost, it is a worthwhile investment that enhances the quality of your testing.
Alternatively, if using an agency isn’t feasible, create a thorough screener to filter out users who may not be a good fit for your project.
By implementing these strategies, you can streamline the recruitment process and set the stage for productive and insightful moderated usability testing sessions.
