Formative vs Summative Evaluations in User Research: What’s the Difference?

A fundamental aspect of user research is evaluation, an essential process used to measure and understand a product's usability, effectiveness, and overall impact on users. When we talk about evaluation, two terms often come to the forefront: formative and summative evaluations.

But what are they, and how do they differ from each other? More importantly, how can they be effectively leveraged to shape a product's development and ensure it successfully meets user needs?

In this post, we will explore these two types of evaluations, explaining their purpose, the methods used in each, their timing in the development process, and the specific questions they help answer. 

What is user research evaluation? 

User research evaluation refers to the systematic examination of a product or service from the perspective of its end users. It involves various methodologies that assess how real users interact with, understand, and experience the product or service.

The main goal of user research evaluation is to inform the design process from the users' point of view rather than your organization's internal goals or assumptions. 

Evaluative user research can include qualitative and quantitative methods to assess the usability of your product and determine if the project is on the right track. 

Surveys are one of the most popular methods to collect user feedback in both formative and summative methods, including the standardized UMUX template below: 

What is formative evaluation?

Formative evaluation is about identifying issues in the design of your product or service while it’s being designed. Although it’s usually conducted in the early stages of the development process, it can be deployed later on, too. 

This method is great when you want to determine which parts of your design are working well from the perspective of your end users, which ones are not, and the reasons why. 

The techniques used in formative evaluation are often qualitative and include user interviews, usability testing, field studies, and observational studies. However, quantitative methods can also be used as part of the approach. 

The results you obtain at the end of the process are generally used internally to inform design decisions and iterate on the product or service.

You should conduct a formative evaluation at least twice. 

The first one should be done in the early design concept phase to validate your prototype.

The second one should be deployed on your MVP when some of the features are ready, or you’ve just entered the development phase. 

Here are some of the questions formative evaluation aims to answer:

• What are the needs, preferences, and motivations of our target users? What problems are they trying to solve?
• What challenges or issues are users encountering while using our product or service? 
• In what context are users likely to use our product or service? How does this context affect their experience?
• Are the proposed features useful and meaningful to our target users? 

Benefits of formative evaluation

Since it’s often used at the beginning of the development process, formative evaluation can help you identify any bugs or usability issues with your product or service while it’s still relatively easy and inexpensive to fix them.

It can also identify unnecessary features that might seem like a good idea to the product designers, but they’re not really contributing to the usability of your product.  

This discovery-based approach is useful for incrementally improving the quality of your design. The data you collect in the process will help you shape the overall product strategy and roadmap to align business objectives with user needs.

What is summative evaluation?

Summative user research evaluation assesses the quality, effectiveness, and overall impact of a product, service, or system after it has been fully developed or at the end of a development cycle. Due to this fact, it’s conducted less often than formative evaluations. 

Summative evaluation is often used to compare the performance of a new version of a product or service against an older one. It aims to verify whether the implemented changes have been successful and the objectives set at the beginning of the development process have been met. 

The data collected as part of summative evaluation is usually, although not always, quantitative in nature, and the methods used include user surveys, benchmark studies, and A/B testing.

Some of the questions this approach can answer include:

• Does the product solve the problems it was intended to address?
• What is the overall impact of the product or service on the users' tasks or life? 
• Are users adopting and using the product or service as intended?
• How does the product or service compare to its competitors or to previous versions?

Summative studies can serve as reference points to benchmark your design over time and compare it to your competitors.  

Benefits of summative evaluation

Summative evaluation provides a way to assess how well a product or service meets its intended goals and user needs and validate the effectiveness of the design and functionality.

It helps determine the level of user satisfaction with the product or a specific feature. It often makes use of surveys, such as the Net Promoter Score, to assess how likely your users are to recommend your product to others:  

Summative evaluations often provide quantitative data, which can be used to track performance over time, identify trends, and provide objective evidence of the product’s success. Having access to historical data also allows for benchmarking against competitors or previous versions of the product or service, providing a competitive advantage and driving continuous improvement.

Collect more user insights with surveys

Regardless of whether your research is formative or summative, the goal is always to release a product that delights its end users. 

Throughout the process, surveys are a common tool used in both formative and summative evaluations. They offer a straightforward method of gathering user feedback and can be conducted at various stages of product development to answer a range of essential questions.

As a powerful and effortless survey platform, Survicate offers a robust set of features designed to facilitate high-quality, user-centric research. It supports various types of surveys and can integrate natively with numerous third-party tools and platforms, making it an excellent choice for both formative and summative evaluations.

Sign up for your free account today and kick-start your journey towards superior user experiences and product success.