Competitive evaluations let you assess if your design is better or worse than your competitors and discover the relative strengths and weaknesses of competing designs. They allow you to take an in-depth look at how others solve the same design problems.
What Is a Competitive Evaluation
Competitive usability evaluation: Comparing your product against several competing designs.
The comparison can be holistic, ranking the designs by some overall site-usability metrics, or more focused, comparing features, content, or design elements across sites.
Evaluations can take the form of expert reviews, where an experienced usability practitioner reviews the designs based on their UX expertise, or competitive usability testing, where users complete a set of tasks using two or more competing designs.
Goals of Competitive Usability Evaluations
The goal of any competitive evaluation is to see what competitors are doing, how they’re doing it, what’s working, and what’s not.
Competitive evaluations are often a good initial research activity for a project. They can inform the design direction for that project and uncover the need for a particular feature. The goals of the competitive evaluation should be clear before any work is started. Ask yourself:
- What design challenges are you trying to solve?
- What features of your competitors seem interesting or appealing?
- What features on your site do you want to compare to others?
Organizations with a high UX maturity and a substantial investment in strategically managed user experience can also conduct longitudinal competitive evaluations to track relative status over time. However, these evaluations tend to be expensive and might not be feasible for organizations with small UX budgets or low UX maturity.
Benefits of Competitive Usability Evaluations
Competitive usability evaluations have multiple benefits, including:
- Objective decision making. The insights from competitive usability evaluations enable UX teams to base their decisions on data rather than on opinions or popular trends.
- Risk reduction. Basing decisions on insights rather than opinions decreases the risk of costly failures.
- Value-driven improvements. Identifying features or design approaches that significantly benefit users of competing products can enable you to integrate them seamlessly into your own product.
- Learn from others. Your team can gain insights based on the usability work of your competitors. Evaluating others’ designs can show you the strengths and weaknesses of a design that may have already been through one or several rounds of usability testing or reviews. Your team benefits from the work that the competitors’ teams have already done.
- Test multiple approaches. Competitive evaluations allow you to evaluate several approaches to the same design and can be done before doing any design work.
- Empathize with customers. Competitive evaluations reflect what your customers do as they look for information on the web. They compare your site's content, functionality, and overall experience to your competitors. People typically browse in parallel, moving between several sites before deciding which company to choose. It is beneficial for your team to know what your users see and do as they look around at the competition.
Defining the Competition
A typical competitive evaluation focuses on 2 to 4 competitors’ sites. Any more than that can be too expensive and too overwhelming to analyze. If you have many competitors, do an initial review of several of them to determine which sites:
- Offer similar content and functionality to your site or to what your site strives to provide
- Provide the best overall user experience
- Use innovative designs that set them apart
- Are your strongest or most important competitors
- Are the competitors that your customers are most likely to compare you against
Conducting a competitive analysis goes beyond merely comparing your product to major competitors. The goal is to extract valuable insights that can elevate your user experiences. While larger competitors may seem like natural benchmarks, valuable insights can also be gleaned from examining smaller or more innovative companies.
Additionally, exploring businesses that are only tangentially related to yours can offer unique perspectives, as they may employ similar practices but with variations influenced by traditions from other industries or market segments.
Competitive evaluations go beyond just finding badly designed products and claiming that yours is better. While you can learn what doesn't work from poorly designed sites, it's equally important to study sites with better usability than yours.
Methods for Competitive Usability Evaluations
There are 2 methods for conducting competitive usability evaluations:
Competitive Reviews
In a competitive review, a usability expert takes an in-depth look at a series of related products. The expert is looking for relative strengths and weaknesses, trends, patterns, and differences. Looking at similar content across several designs can reveal holes in your content or functionality; holes on others’ sites may inspire an addition to your site.
Competitive reviews can be as broad or narrow as desired. They can dive into a particular feature or area, or encompass an entire site. You may look only at ordering processes, or you may review the overall site experience. In any broad evaluation, it is best if some key areas are identified to focus the review and its results.
Competitive Testing
In a competitive test, an expert runs a usability test on your design and on the designs of your competitors. For best results, follow these guidelines:
- Realistic scope. Each study participant typically completes a series of tasks on 2 or 3 different sites. Testing more than 3 sites with any given participant can overwhelm the study participants.
- Randomization. Alternate which sites are paired together and which is tested first for each user. This helps prevent learnability effects biasing your study results.
- Key tasks. Focus on representative and key tasks that can be completed across sites, as well as on tasks where sites offer different approaches to information, functionality, or design.
- Final comparison. While you want to observe what users do, rather than solely asking for their opinion, it will be beneficial to ask participants at the end of the session to compare the sites they used. Comparing the two designs can help users verbalize what was clear or confusing in each design and can help you gain further insight into strengths and weaknesses.
Competitive Evaluation of Design Variations
A competitive evaluation can also involve multiple variations of a single design instead of competitor products. Usually, these variations are developed through a parallel design process; a competitive test or review will help you identify the design that will work best for your users.
Competitive evaluations of design variations are applicable in the early stages of design. Designs don't necessarily need to be fully developed or interactive to be evaluated. Depending on the design, paper-prototype testing can yield valuable results before committing resources to development.
Analyzing Results
By collecting metrics, a “winner” can emerge from competitive testing or competitive evaluations. The winner can be based on success scores, time on task, users’ subjective ratings, or a scorecard developed to analyze findings. However, the main goal of a competitive evaluation is not to declare a winner. The goal is to improve your design.
While metrics can emphasize the point that your site performed, say, 30% better — or worse -— than your competitors’ sites, the important information lies in the details of what worked and what didn’t across designs. Focus on what’s relevant to your design questions or challenges.
- What are the biggest strengths of competing designs?
- What trends arise across products – and are they good or bad?
- Did any product offer unique solutions to a common problem?
- How did they perform?
- Is there any opportunity to stand out from the competition?
- Were there any gaps in the information, content, or features that the designs provided?
Don’t forget to look for opportunities beyond what your competitors offer. You want to beat the competition, not copy them.
When writing – or reading – a competitive report, focus on information that can help you improve your site. Knowing what worked well can inform design decisions. Knowing what failed and why can help teams avoid others’ mistakes or fix their own.
A good competitive evaluation assesses the competition’s designs and translates that assessment into recommendations for your product. The results must help you design your own solution.
Summary
Competitive evaluations can benefit your team by learning from your competition, avoiding costly mistakes, and focusing on data-driven strategic improvements of your website.
