TriCountry Remote Bilingual UX Research – A Case Study

UX Research

At Objective Experience, we have always used remote testing methods to augment our face-to-face testing. Pre-covid, most of our participants were locals and the farthest anyone might travel from was the central coast, which is an hour away. The move to remote has certainly opened up new possibilities. This project example really highlights the fact that we can now recruit and test with a much broader audience, across multiple countries and even in different languages.

The project was managed out of our Singapore office. The 18 participants and our clients were in Hong Kong. One of the Moderators was in Australia, another moderator/observer was in Singapore and the Cantonese/English translator was in Hong Kong, with the participants in a physical lab.

The challenge that we surmounted was coordinating everyone across different time zones and synchronising the technology so that we were all capable of operating the software.

Research Objectives

  • To identify the usability issues and pain points current and new investors encounter when using the Mobile app to make investments

  • To offer design recommendations to improve the investment experience on the Mobile app

Basic Info

Client: A Global Bank

Goals: To evaluate the usability of their investment/wealth products: Opening an investment account with

  1. Mutual Fund Purchase in app journey

  2. FX trade in app journey

  3. Equities trade in app journey

Platform: iOS & Android Mobile App

Fieldwork dates: June 2020 – Between 2-5 sessions per day over 2 weeks in English & Cantonese

Fieldwork sessions: 60 mins with 60 mins break in between

Participants: 18 Participants: 9 x English speaking participants, 9 x Cantonese speaking participants

Elements tested

Journeys:

  • Opening of Investment Account

  • Mutual Fund Purchase

  • FX trade

  • Equities trade

Specific app elements:

  • Investment Dashboard

  • Assessment of Investment Portfolio

Methodology

We used 3 research techniques in tandem:

Recruitment

A mix of current customers for whom the bank is their main financial institution with varying levels of investment in wealth products, and current customers who use a competitor as their main financial institution but actively invest with the bank. Age requirements were generationally defined from Gen Z through to Boomer and each generation was represented in the research across the 4 wealth journeys.

Software

  • Google Drive: Drive as the repository for all the files. Sheets for note taking during the sessions and project planning. Docs for discussion guides and recruitment

  • Zoom: Communication between the translator and the Scribe

  • Lookback: Mobile usability testing, observation and video recording the sessions (including the participant’s facial expressions during the interview)

  • Quicktime: Editing the videos to make VOC snippets for the client

  • YouTube: To host the videos

  • Keynote: As the software for the analysis document to be delivered to the client

Tech set up

When I was notetaking for the project and the client was speaking Cantonese I watched the testing through Lookback but was listening to the translator through Zoom. To avoid getting two audio feeds I muted Lookback through the Chrome browser.

Analysis

We analyzed on the go. This was made easier by having a scribe for each session and having an hour in between sessions. At the end of each session the moderator and the scribe would get together and brainstorm the six major findings and then we would rate the severity of these findings. Rather than have to go back through the scribe notes and the videos it’s much fresher in the memory to recall details directly after each session.

Finally at the completion of all the sessions we went back and looked at the objectives for the testing and attempted to answer those objectives with the insights from testing.

Findings

We produced a report that was filled with Voice of the Customer (VOC) videos as requested by the client, and we outlined the major findings from every participant. Voice of the customer was used to demonstrate in detail to the senior exec of the client what the experience was really like. We delivered these VOC videos in a timely fashion by having one person concentrate on editing the videos and using the note taking time stamps to locate these snippets quickly.

The videos were hosted in a private channel on Youtube negating the need for large file transfers and the requirement to host the videos on the client’s servers. We categorised them by participant number and then link number so they were easy to identify.

The client was unable to watch all sessions live so they were very keen for us to highlight the major findings in video snippets. It’s also a much more consumable and convincing finding when it comes directly from the participants mouth. The client’s team was also more likely to digest the findings if they were served up in an organised video format rather than in a large document.

As illustrated below we also summarised the finding from every journey/task into usability highlights and recommendations.

Remote Bilingual User Research Checklist

Pre Testing

  • Read the agreed proposal to understand the brief, the timescale and the reporting requirements

  • Create a Gantt chart for project planning

  • Communicate with the participant recruiter so that you are aligned

  • Arrange a Kick Off meeting with the client

  • Clearly establish the project objectives or hypotheses that you are going to test

  • Discuss recruitment demographics with the client

  • Clear up any questions that you might have with the client

  • Establish communication channel for the project (Whatsapp, Slack etc)

  • Establish repository for the project (Drive or Dropbox)

While Planning the Remote Usability Test Sessions

  • Familiarise yourself with the prototype

  • Create the discussion guide with reference to the project objectives

  • Check if you have all the equipment and applications that you need (e.g., laptop, camera, mobile device, online conferencing software)

  • Arrange a participant for the dry run

  • Establish viewing links for the client

  • Check on recruitment progress

  • Run a dry run about one day before the first session

  • Clarify how and if the client can ask questions during the sessions

  • Think about a backup plan in case of technical issues during a session

Shortly before the Session (Approx. 30 Minutes before the Session Begins)

  • Set up and check the technical devices and the applications you’re going to use

  • Open the online conferencing tool

  • Open the prototype

  • Familiarise yourself with the details of the participant

  • Turn off any messaging applications that might disturb you during the session (if you’re using only one monitor)

  • Close any unnecessary applications (e.g Email)

  • Turn off the screen saver

Once the Participant Has Joined the Session

  • Start the software recording session

  • Welcome the participant and describe the procedure

  • Explain to the participant the backup plan for technical issues.

  • Share the prototype

  • Ask the participant if the prototype is displayed well (e.g., if the texts are legible)

  • Give the participant presenter rights if he or she is to interact with the prototype

  • The translator will need to have a Zoom link open for the notetaker to access the translation

  • The note taker will have to watch the interaction on Lookback but listen through Zoom. This will necessitate muting Lookback through the Chrome browser.

  • Keep a close eye on the time so as to not exceed the allotted time

  • Near the end of the session check if there are any questions from the client or scribe

After the Session

  • Debrief the project team and discuss the most important findings

  • If you’ve recorded the session, save the recording and share it with your project team