Vrbo Project | Yung-Sheng Chang

Information Architecture Testing - Vrbo

I was a User Research Intern at Vrbo - Expedia Group on the Customer Insights, Partner Research team in the summer of 2019. The aim of my research project was to evaluate the information architecture (IA) of the partner production website and three proposed IA prototypes. A two-phase study, involving first-click study and in-lab testing, was conducted. The results of the project contributed to understanding users' navigation pain points. Design recommendations were given to help improve users' experience in navigating the partner production website.

Supervisor: Karl Steiner
Mentors: Anikó Sándor, Jessica White-Sustaita

Overview

Problem:

Partners were having a hard time navigating and finding information on the website, which made it time-consuming and difficult for them to manage their properties.

Objective:

Identify information on the website that was hard to find.
Evaluate the production site and proposed IA prototypes.
Understand users' navigation and thought process of locating hard-to-find information on the production site and prototype.
Provide design recommendations to help improve the partner website's IA.

Method:

First click study was first performed to test the global navigation menu among the current IA and three IA prototypes by sending out surveys. Survey was selected to rapidly collect large-scale responses from partners.
In-lab testing was then conducted to have in-depth knowledge of users' navigation pain points in finding information on the production site and prototype IA.

Results:

4 tasks were hard to find on the production website.
Criteria were developed to classify each task into three levels of difficulty (production site) or levels of user agreement (prototype).
IA and content related recommendations were given to the UI design teams and product managers.
Some tasks need further research in determining where to best locate the information in the IA.

Phase 1 - First Click Study

Goal:

The purpose of the first click study was to gather data on the global navigation of the Vrbo partner's website.

Process:

20 tasks that were perceived as difficult to find on the website were selected by the UX researchers and designers.
The inclusion/exclusion criteria for selecting respondents were discussed and determined.
Tasks scenarios and 4 surveys were designed: One for testing the production site and the other three for testing prototypes' global navigation menu (between-subject design).
Around 100 partners responded to each survey within 2 days.
Results were analyzed based on respondents' agreement in selecting the same menu item to find information across the production site and prototypes.

Outcome:

Only 3 tasks had high agreement across all IAs, the other 17 tasks had moderate-to-low agreement.
In-lab testing was required to have an in-depth understanding of users' experience navigating the website.
A revised IA prototype was proposed and needed to be tested in the in-lab study.

Phase 2 - In-Lab Testing

Goal:

The purpose of this study was to test the IA of the current partner production site and a new partner website prototype.

Process:

14 tasks were tested, with 6 tasks tested in both the production site and the prototype. The 14 tasks were selected based on low agreement around where they belonged in the first-click survey.
12 participants were recruited, including 6 experienced and 6 novice partners.
Participants were given task scenarios and asked to find information on the website. Concurrent think-aloud was performed while they were searching information. Participants' eye-movement was captured to assist the researcher in understanding participants' cognitive attention.

For the production site, tasks were categorized into difficult, moderate, and easy to find, based on two criteria:
- Average number of wrong navigation path
- Average participants needing guidance from researcher
For the prototype, tasks were categorized into high, moderate, and low agreement, based on two criteria:
- Number of selected locations
- Percentage of the most freqent response

Screen Shot 2020-09-06 at 9.30.23 AM.png

Outcome:

4 tasks were labeled as difficult on the production site and only 1 task had low user agreement on the prototype.
Finding info was challenging, and many were unable to retrace their steps or orient themselves in the navigation. The website needed to increase consistency across navigation and page labeling
Some of the labels' content confused participants. Recommendations were suggested to rename the labels or provide additional explanations.
The results from the study was inclusive of one task's location in the IA. Future research is required.
IA and content related recommendations were given for designing future prototype.

Lesson Learned

How to utilize newly learned skills and methods in performing fast-paced yet rigorous research.
How to collaboratively work and communicate with cross-functional teams (product managers and UI designers) on the same project.
How research is conducted in Expedia Group's subsidiaries and learning the company's structure, working culture, and environment.

Information Architecture Testing - Vrbo

Overview

​

Problem:

Partners were having a hard time navigating and finding information on the website, which made it time-consuming and difficult for them to manage their properties.

​

Objective:

Identify information on the website that was hard to find.

Evaluate the production site and proposed IA prototypes.

Understand users' navigation and thought process of locating hard-to-find information on the production site and prototype.

Provide design recommendations to help improve the partner website's IA.

​

​

Method:

First click study was first performed to test the global navigation menu among the current IA and three IA prototypes by sending out surveys. Survey was selected to rapidly collect large-scale responses from partners.

In-lab testing was then conducted to have in-depth knowledge of users' navigation pain points in finding information on the production site and prototype IA.

​

Results:

4 tasks were hard to find on the production website.

Criteria were developed to classify each task into three levels of difficulty (production site) or levels of user agreement (prototype).

IA and content related recommendations were given to the UI design teams and product managers.

Some tasks need further research in determining where to best locate the information in the IA.

Phase 1 - First Click Study

Goal:

The purpose of the first click study was to gather data on the global navigation of the Vrbo partner's website.

​

Process:

20 tasks that were perceived as difficult to find on the website were selected by the UX researchers and designers.

The inclusion/exclusion criteria for selecting respondents were discussed and determined.

Tasks scenarios and 4 surveys were designed: One for testing the production site and the other three for testing prototypes' global navigation menu (between-subject design).

Around 100 partners responded to each survey within 2 days.

Results were analyzed based on respondents' agreement in selecting the same menu item to find information across the production site and prototypes.

​

​​

Outcome:

​Only 3 tasks had high agreement across all IAs, the other 17 tasks had moderate-to-low agreement.

In-lab testing was required to have an in-depth understanding of users' experience navigating the website.

A revised IA prototype was proposed and needed to be tested in the in-lab study.

Phase 2 - In-Lab Testing

Goal:

The purpose of this study was to test the IA of the current partner production site and a new partner website prototype.

​

Process:

14 tasks were tested, with 6 tasks tested in both the production site and the prototype. The 14 tasks were selected based on low agreement around where they belonged in the first-click survey.

12 participants were recruited, including 6 experienced and 6 novice partners.

Participants were given task scenarios and asked to find information on the website. Concurrent think-aloud was performed while they were searching information. Participants' eye-movement was captured to assist the researcher in understanding participants' cognitive attention.

​

​

​

​

​

For the production site, tasks were categorized into difficult, moderate, and easy to find, based on two criteria:

Average number of wrong navigation path

Average participants needing guidance from researcher

For the prototype, tasks were categorized into high, moderate, and low agreement, based on two criteria:

Number of selected locations

Percentage of the most freqent response

​

Outcome:

4 tasks were labeled as difficult on the production site and only 1 task had low user agreement on the prototype.

Finding info was challenging, and many were unable to retrace their steps or orient themselves in the navigation. The website needed to increase consistency across navigation and page labeling

Some of the labels' content confused participants. Recommendations were suggested to rename the labels or provide additional explanations.

The results from the study was inclusive of one task's location in the IA. Future research is required.

IA and content related recommendations were given for designing future prototype.

Lesson Learned

How to utilize newly learned skills and methods in performing fast-paced yet rigorous research.

How to collaboratively work and communicate with cross-functional teams (product managers and UI designers) on the same project.

How research is conducted in Expedia Group's subsidiaries and learning the company's structure, working culture, and environment.

​

​

​

​

Attended Expedia Group's Research Summit

Only 3 tasks had high agreement across all IAs, the other 17 tasks had moderate-to-low agreement.