When it comes to usability testing, one of the most important decisions you’ll make is whether someone should moderate the session.
So which one should you choose? Read on to learn the pros and cons of each.
If natural user behavior generates the most accurate usability data, then of course testing should be unmoderated, right? But if that data is unfocused and misdirected, then won’t you want to moderate the tests? In that case, aren’t you then biasing the results with someone else in the room?
You’ve probably heard of the observer effect, even if you don’t know it by name. It’s the idea that people being observed act differently than when they’re alone. While this is a highly documented phenomenon, this should not be the only influence on whether or not to use a moderator.
Budgets, schedules, manpower, the types of tests, the necessary data — these are the more important considerations we discussed in the Guide to Usability Testing.
Let’s dive deeper so you can make the most informed decision.
Moderated Usability Tests
It’s true, moderated tests give you more control. Moderators can answer any immediate questions posed by the user, and guide the test to meet the goals of the study.
Of course, moderated tests require a moderator — someone with people skills and knowledge of the product to keeps things running smoothly. This also means more difficulty in scheduling with the user and restricting at what they take the test. If you’re conducting the test on-site, this will require even more resources, and make it harder to recruit participants willing to go out of their way.
On the plus side, many of these difficulties are eliminated through remote testing. This tactic allows users to take the test where they want, and gives them some leeway with scheduling, while still giving you some of the control.
Because moderated testing requires more effort, it is highly recommended in the earlier stages of design. When you’re working with concepts and prototypes, you definitely want to be able to ask the follow-up questions to your users.
We agree with this advice from UserTesting:
1. Incomplete or buggy interfaces — In other words, early-stage prototypes. If you’re testing a product that risks confusing the user, you need a moderator to help answer questions and guide them through. While this may seem uncommon, starting a usability test as early as possible is always a good idea.
2. Complex interfaces — If your product has a high learning curve, a moderator can salvage a test that might otherwise yield no useful results. For example, a moderated test makes sense for products that don’t always have a linear sequence of steps to accomplish tasks.
3. Products with security concerns — While rare occurrences, if you’re testing a product with access to data that you don’t want public, a moderator will keep the user where they belong — although there are other ways to restrict access in this scenario.
Remember that moderated and unmoderated testing are complementary, not competing. For example, you can run a moderated test on your prototype, then iron out any major usability issues. As you finalize your product, you can then run an unmoderated test to gauge how it performs “in the wild”.
When you’re ready to get started, make sure you follow Jeff Sauro’s 20 tips for moderated usability testing.
Unmoderated Usability Tests
Unmoderated tests are cheaper and easier to quickly recruit more users. Let’s take a practical look at the pros and cons.
Source: UserZoom
Benefits
- Less time — Free from scheduling concerns and limited manpower, unmoderated tests allow you to test users simultaneously and at all hours. This means more data in a quicker time. As we described in the Guide to Usability
- Less chance of bias — While the observer effect shouldn’t be the only factor, it — and several other types of social and technical biases — are still relevant factors. The absence of a moderator will alleviate many of these, giving you more natural and reliable results.
- Cheaper — If you need to hire an outside user research company, then the cost will certainly exceed unmoderated tests. Unmoderated tests don’t require a reserved location, and all the equipment can be provided by any number of usability tools (see below). In fact, the cost of these tools is scalable, so you can pay whatever amount you deem appropriate.
- Easier recruitment — The relaxed environment for unmoderated tests creates a larger pool of willing candidates. Unmoderated tests can be done virtually anywhere at any time, by anyone who meets your criteria.
Of course, unmoderated testing, with its lesser degree of control, has its drawbacks as well.
Downsides
- No follow up questions — While you can include forms for open-ended questions to users, nothing beats the conversation between moderator and user. The back-and-forth of moderated testing gives you a rare peek inside why users behave the way they do. If time and scheduling permits, we recommend following up with users either in email or Skype to for a post-test discussion.
- Less forgiving tasks — Because it’s up to the user to decide when a task is complete (and not a moderator), users may move onto the next task prematurely. That’s why you must write clear tasks with an easily defined state of success or failure.
- More time spent filtering for quality — While moderated testing may introduce some bias, unmoderated testing runs the risk of users motivated purely by the compensation. While you’ll spend less time recruiting people, you’ll need to spend more time setting up filters to qualify responses.
If you’re interested in unmoderated tests, we can recommend these tools to help you along.
Lookback — Free beta. Compatible with Mac, Android, and iOS, this is one of the most promising tools we’ve seen to date. It simultaneously records facial reactions, mobile screens, and gestures at 60fps. The video is then played in a sleek storytelling fashion, which you can see for in this actual recording.
UserTesting — $39 per user, per test. Known for its quick feedback, UserTesting allows you to test anything from websites (yours or your competitors), apps, prototypes, Facebook games, or Google ads.
Userlytics — $49 per user, per test. After setting the parameters, Userlytics will recruit participants and conduct the tests on their own, with a similar spectrum to UserTesting.
Solidify — $19 per month (basic), $49 per month (plus). Solidify allows you to test interfaces for desktop or mobile devices at any stage of development, even sketches and wireframes. The plus membership gives you demographics, logo customization, and report customization.
UserZoom — $1,000 per month. UserZoom operates a little differently, offering its suite of utilities for a monthly fee. Members can use a variety of tests, including click tests, card sorts, tree tests, and surveys.
Conclusion
Unmoderated tests provide statistically significant usability results in a short period of time. However, there is far less room for error since you can’t revise the tasks or questions once the test has started (unless you don’t mind wasting all the money you’ve spent).
Remember that you’ll also need to sift back through the data for a full analysis since you’ll lack the insights possible from the dialogue of a moderated session. For polished products, however, unmoderated testing is a serious force multiplier when it comes to scaling up your usability research.
The answer as to which is better is what you’d expect for most design questions: it depends. Are you testing early in the process, or later? Is there a shortage of time and money?
Each type of test has its place and purpose, and the best advice we can offer is to treat usability testing as an iterative exercise. Different stages of design benefit from moderated vs. unmoderated testing, so don’t be afraid to mix it up.
If you’d like to get additional practical advice on usability testing, feel free to download our free Guide to Usability Testing. We explore over 30 different types of usability tests, providing tips and examples for each.
