Tips for moderating remote usability tests

One of the things that we’ve been asked about a lot in recent months is to provide some amazing tips for moderating remote usability tests.

It’s a service we’ve always offered but this year we’ve conducted hundreds of remote moderated usability testing sessions. Obviously Covid19 social distancing restrictions have been the main driver for the shift but as the months wore on and the whole world became accustomed to using different technology to facilitate their work remotely, we’ve seen clients and participants become increasingly comfortable with remote testing.

There will always be a place for in-person studies but now that more people have been able to experience the advantages of remote studies it’s hard to imagine we’ll be downing our remote testing tools anytime soon.

Arguably the biggest drawback to remote moderated usability testing is the additional preparation time required to set up a successful test and if you’ve never moderated a remote session before there are a few watch outs you might not be ready for. So we’ve put together some of our top tips for moderating remote usability testing

Our tips for moderating remote usability tests

Recruitment

You may be recruiting your own participants or using a recruiting service like our Research Network services in APAC. It’s important to make sure you understand which tools you’ll be using to communicate with the participant and which tools you’ll be using to communicate with observers before you start recruiting. Pay particular attention to installation requirements for any software participants may need to use and their compatibility with operating systems or browsers so that you can include all of your software and hardware requirements in your initial screener.

Ideally, you’ll also want to have determined your scenarios and tasks for the session so that you can understand things like whether you’ll need the participants to complete any pre work, have anything with them like photos or examples, have access to a second screen or printed stimulus during the session.

Planning scenarios and tasks

Administering tasks in a remote setting is a little more complicated and requires more planning than in-person testing. It will depend on the tool you’re using. You want to make sure the participant is able to process the scenario and /or task and depending on how your study is structured there will be more than one way to manage it. Tasks should Ideally be delivered one at a time; in-person you can do this by handing the participant a written task and asking them to read it aloud to ensure that they have read the complete task. Remotely you don’t have this option, here’s what we do:

• If you’re using a screen sharing tool with a chat feature, you can simply deliver task instructions one at a time through the chat and ask the participant to read aloud. It’s best to have them visible on your own second screen along with any links so you can quickly copy and paste.

• If your screen sharing tool doesn’t have a chat function, provide participants with a document with a numbered list of tasks (the document should begin with a blank page and each task should appear on a separate page) – the benefit is that you can still ask the participant to read aloud to ensure they have read the entire task and that they will be able to easily refer back to it in case they forget. The downside is that you won’t be delivering the tasks one at a time and if you want to avoid priming for subsequent tasks it’s unlikely you’ll be able to prevent participants from reading the full document prior to testing even if you ask nicely. It’s also best to make sure participants have a second screen or are able to print the document and have it ready for the session so that they’re not distracted or flustered by having to switch screens during the session.

• If you’re restricted by the study or tool limitations, you can read each task to the participant. Make sure you remind them that it is not a memory test and they can ask you to repeat it at any time if they forget or are unsure of the details. Make sure you keep all of the tasks and scenarios on hand so that you’re able to repeat them when necessary without losing time.

Test environment

As a moderator the test environment can be the most intimidating aspect of remote usability testing because unlike in-person testing, you have very little control of where the participant will be when they connect to the session.

A good recruiter will be able to screen and prepare your participants to make sure they have access to the right software, hardware and a strong and stable internet connection.

They will also clarify that the participant has access to and understands that they need to be in a quiet and private space with good lighting and minimal background noise or distractions during the session.

Performing a ‘tech check’ 1-2 days prior to the session can help to verify that participants have everything they need to participate and they are able to use the tools. It’s also beneficial to help them feel comfortable using the tools prior to the session.

It’s worth noting that whilst remote test environments can be a little risky simply due to a lack of control, it’s our experience that in the overwhelming majority of cases they actually have more upside than down.

Participants are generally more relaxed and open in their own surroundings, using their own devices and arguably more likely to behave more naturally than they might do in a lab setting.

It can generally be cheaper to run overall because incentives for participants can be less than in-person rates, there’s no room rental costs or requirements to provide refreshments.

On top of that, the convenience of a remote connection reduces the dropout rate and it allows us to recruit participants from a much broader geographical pool without incurring travel time and expense.

Dry run and observers

You’re probably used to doing dry runs for in-person usability testing sessions and realise that it’s a no-brainer to do them for remote sessions as well.

Aside from having all of the same benefits like helping you to refine your process, facilitation, tasks, scenarios, technology, time management etc. it’s a great opportunity to give your observers a chance to get comfortable with the observation software.

You can use the dry run to develop a brief for observers which includes how to install and use the software, observer guidelines e.g. they must remain on mute, how to communicate with the moderator during the session.

And a list of scenarios and tasks as well as what to watch out for during the testing.

Moderating a remote session

Understanding that there are some nuances to moderating a remote session can help you be prepared to keep it running smoothly.

For instance, during an in-person session when you need to interrupt a participant there are a few ways to do it that can feel natural, in a remote setting a sound or a gesture won’t do it, often the best way can be to simply say their name, so make sure you know how to pronounce it correctly.

You may do this already for in-person testing because it’s polite and it helps build rapport, the same is true for remote testing as well but you’ll quickly realise how much more often you’ll need to use the participants’ name and it’s better for everyone if you get it right!

Most importantly, accept that things will go wrong and have plans in place to manage it so that you won’t jeopardise the research.

Especially when using moderated usability testing tools for mobile be aware that you will always be dealing with stability issues because it’s no small ask of a software provider to thoroughly support every type of mobile device.

Operating systems can automatically update overnight, prototypes and apps have bugs, firewalls can be aggressive, wi-fi can fail and servers can crash.

Build in a little extra session time for troubleshooting tech issues and make sure you recruit at least one backup.

Having a plan B for both your observers and your participants will help you keep your cool when it does go wrong so you can get it back on track and feel confident about the insights you’ll uncover.

Those are some of our tips, we hope you’ve found something handy to help you feel confident running your own remote usability testing sessions.

Let us know about your remote testing experiences and any great advice you can share.