Chun-Ling Hsu (Vanya), Claremont Graduate University, December 2022
Project Overview
This report details the usability testing of the Carfax website prototype by Chun-Ling Hsu (Vanya). Carfax is a company providing vehicle data to individuals and businesses.
The website includes a used-car product list, market price, vehicle history report and other information.
This study aims to estimate the usability problems of the Carfax website and see whether the target customer base can easily navigate the workflow of searching for a used car.
Executive Summary:
The result shows that 6 of the 10 participants needed help finding key information on the Carfax website. We consistently received negative feedback about the product from users.
The recommendations included redesigning the interface, especially the market price and the information display, to improve the site's usability and provide a satisfying experience for customers to enjoy.
Research Goals & Hypotheses
The research goal is to estimate the usability problems of the Carfax website. Especially testing product concepts with the target audience reveals friction points, unclear information, and confusing experiences. It helps understands whether Carfax's key features are communicated to the user and see whether the target customer base can easily navigate the workflow of searching for a used car.
Research question: How problematic would users find information on the website and use the workflow?
Hypotheses I: Less than 70% of users complete the task (Variable 1).
Using the summative benchmarking method. Binary data
Variable 1: Success rate - Success (participants who complete all the tasks)/ all the participants
Hypotheses II: Users have less than 80 scores on usability satisfaction (Variable 2).
Using the summative benchmarking method. Continuous data
Variable 2: Usability satisfaction- the usability satisfaction of this workflow in the prototype, using UMUX scale.
User Test Participants
Screening criteria:
1. The user is personally in the process of finding a used car.
2. The user is searching for a used car to a related other.
3. The user has used the Carfax website before.
While the screening questionnaire primarily intend to source participants that would be potential Carfax users, it also annotated participant knowledge of the used car. The questionnaire also went into greater depth by asking users about searching behavior and information necessity.
Sample size: The study also calculates it that we need 10.824 participants to have a medium effect size in 95% confidence. Therefore, we have 10 participants with one sample in the study.
Design & Procedures
Ten participants completed the moderated in-person usability test.
Step 1: Introduction
Step 2: Open-ended pre-task questions about used car purchase behavior
Step 3: Carfax website task
Step 4. Post-task satisfaction scale
Analysis
Hypotheses I: Less than 70% of the users complete the task. (Variable 1) Binary data, Success (participants who complete four subtasks), and Fail (participants who fail either one or more subtasks).
Using a summative benchmarking method with a confidence interval to understand at 95% confidence whether 70% or fewer users would successfully complete the task.
Hypotheses II: Users have less than 80 scores on usability satisfaction. (Variable 2)
Continuous data, UMUX score.
Using a summative benchmarking method with one sample t-test to see whether the average usability satisfaction would be less than 80.
(The one-sample t-test is a statistical hypothesis test used to determine whether an unknown population mean is different from a specific value.)
Participant Data Insights
From the data gathered, participants emphasized used car information, including accident history, car condition, mileage, overall cost, model, and market value.
We also observed that participants spent 3 minutes 31 seconds on average to finish the
usability test. 40% of the participants completed the task, 60% of the participants
failed the task in either one of the subtasks. When participants were asked about their
satisfaction with the Carfax prototype, there was an average 68.33 UMUX score.
Findings & Recommendations
The following section details our major findings, hypothesis testing and recommendations for improving the Carfax website usability.
Hypotheses I: Less than 70% of users complete the task.
Variable: Percentage of successful rate through 10 participants.
The result shows that 40% of users completed the task. If 40% of users complete a task, we can be 95% confident that actual completion rate is between 17% and 69%. Because the higher boundary of the confidence interval is 69%, anything over 69% becomes an improbable result. This supports hypothesis I, less than 70% of users complete the task.
Hypotheses II: Users have less than 80 score on usability satisfaction.
Variable: Average UMUX score. One sample t-test.
The result shows that there is an average 68.33 score of satisfaction of the users, probability of 0.0698, meaning we can be around 93% confident that this satisfaction score has an average score lower than the expected average of 80, t (9) = -1.62, one-tailed, which support hypotheses II, users’ usability satisfaction is less than 80.
Recommendation and Action Items
Several recommendations for the Carfax Website:
The low task completion rate shows that website needs to be improved, users had a hard time searching for necessary information.
40% of the users could not find the market value, and most were confused by the price. Carfax needs to address the cost and value more carefully.
Overall usability satisfaction needs to be improved.
Next steps:
Research what information is considered a priority and most necessary using unmoderated card sorting.
Redesign the Carfax website and compare it with the present study.
Unexpected functions should be tested again clearly.
Targeting users with different levels of car knowledge since different ages, education levels, and familiarity with cars would highly impact the priority information from the user.
Key takeaways:
Unexpected features should be tested again clearly. For example, the interactive design, information clustering, position, etc.
Targeting users with different levels of car knowledge since different ages, education levels, and familiarity with cars would highly impact the priority information from the user.
When doing a summative benchmarking method, a larger sample size is necessary.
Disclaimer: I am not an employee of Carfax and this does not represent the official strategic direction of Carfax website.
Appendix.
Prototype of the Carfax Website: Link
