WA Notify

Working with the Washington State Department of Health, myself and three other master students worked together to conduct a usability test for the Washington Notify COVID-19 Exposure Notification System for iOS and Android devices.

MY ROLE

UX Researcher


PROJECT TYPE

Usability Testing


TIMEFRAME

January 2021 - March 2021

PROJECT SUMMARY

My team and I completed an evaluative usability study to investigate the overall user experience of the WA Notify system, perceived differences between the iOS and Android systems, and the usefulness of informational resources provided within the system and in accompanying web-based resources. The findings of this study are intended to inform interaction changes to WA Notify and accompanying web-based informational resources.

THE CHALLENGE

The primary client for this project was the Washington State Department of Health, with

secondary stakeholders including DOH partners from Apple and Google.


Client goals stated at the onset of the project were to gather first reactions to the app, the

accompanying web-based information resources, exposure notifications, and overall user

experience. Also, to gather evidence based suggestions on improving user interaction and

positive test results (ie. scenario-based testing). In relation to the “Apple/Google Exposure

Notifications Express” framework, the goal is more about interaction than user interface design

(parts of the app cannot change, but app content and information provided can be changed).

THE SOLUTION


What methods were used?

Participant Screener

We posted the screener (via Google Forms) to UW slack channels, and within our own professional and personal networks. We received 13 responses, and 11 of them moved forward into the next step, which was a 60 minute usability test. We captured data around demographics, their experience with WA Notify, which device they have (Android or iPhone), and their interest in an interview.


Interviews

We conducted interviews remotely, via Zoom Video Conference. Participants and members of the research team connected to the session from the location of their choosing, using personal devices. The study was conducted in English, although 2/11 participants listed English as a secondary language. This is noted as the WA Notify system is available in 36 languages, and future studies may be focused on varying language systems.


For the pre-task interview, we started the usability test via Zoom by asking general questions around what the participants typical day looks like, how often they go outside, the impact Covid has had on their activities, and their knowledge and expectations around WA Notify.


For the post-task interview questions, we dove into the participants overall impressions, what they would change, which device flow they liked better, how likely they would recommend WA Notify, and also gave them the opportunity to give closing thoughts or ask questions.


Usability Testing

To help us align what type of tasks to create for the participants, we created user flows highlighting three main flows for both Android and iOS:

● Onboarding

● Exposure Notifications

● Covid Positive

During the usability testing sessions participants shared their screens via Zoom, while they went through a series of Figma prototypes. We paid attention to non-verbal cues, where they clicked on the prototype, and if there were spots they were confused about. We chose to test both systems (Android and iOS) on each participant to gather comparison data and to be able to gain insight on which system participants preferred.

For each system (Android and iOS), the user went through three tasks:


  • Task A

    • Onboarding: Using the DOH resource link, enable WA Notify on your device

  • Task B

    • Exposure Notifications: You received a text from the DOH that you may have been exposed. What do you do now?

  • Task C

    • COVID-19 Positive Result: You tested positive for COVID-19 and received a text from the DOH. What do you do now?


We gathered task impressions, areas of confusion, recommendations, and how iPhone tasks compared to Android tasks. It is important to call out that we alternated which flow was viewed first by each participant to eliminate the possibility of bias.

Findings

To analyze findings, we took the data from our notes and carried out affinity mapping to look for patterns and common themes. We color coded each participant to stay organized.

Major Common Findings & Recommendations

THE RESULTS


Limitations

  • Due to the sample size, we weren’t able to draw statistical significance from the quantitative data we collected (ie. NPS, task-based metrics)

  • For recruiting, we relied on convenience and snowball sampling

        • This recruitment method biases the participant sample, as opposed to recruiting randomly from the general population. Many participants were either affiliated or had past affiliation with the University of Washington (6/11 participants). Due to this, findings may not be fully representative of the general public of Washington

  • Our research was limited to remotely observing participants interaction within the prototypes, rather than using the system on an actual mobile phone

  • This is limits our participants experiences due to the constraints of the prototype features in comparison to the fully functioning WA Notify product.

  • For testing the DOH website, we were limited to only testing users’ interaction within the “What to do Next” page

        • We were unable to gather data on other portions of the website that may have been useful to the user



Future Work

  • Diversify participant demographics

        • Include participants from a larger range of ages, technological literacy, languages spoken, etc.

        • For non-english speaking participants, unmoderated usability tests with post-task likert scales and other quantitative measures can be used to determine the usability of the system in other languages covered by the ENX system.

        • P6 mentioned as an international student with her primary language as Chinese, she had trouble digesting and understand the content, and we realize this brings some bias into the data since the usability test was conducted only in English.

  • Testing prototypes mockups on phones in-person rather than remotely

        • Additionally, using diary studies to capture participants’ genuine reactions to using the app over a longer period of time

  • Increase the sample size to make a more meaningful results from the usability metrics, aiming for statistically significant data results

  • Focus on a broader range of user behavior, such as how users find vaccine information, and how this can help improve the information provided by the system