.

 

 

While the old English saying is true about how asking too many questions can turn ugly in certain areas of life, being inquisitive about your audience is key to an effective customer experience strategy. Still, it’s important to be equally attentive to trends and changes in audience behavior, as it is to determine a cohesive survey design approach.

So, how do you, as a marketer or user researcher, find the golden mean between structure and constant adjustment?

I’m going to show you how understanding the fundamentals of survey methodology can boost the quality and relevance of your feedback collection strategy.

You’ll learn how to set your survey efforts up for success with the right tools, types of insights you’re looking to derive, and most fitting customer satisfaction metrics.

Regardless of whether you’re new to survey design – or already have some experience, but never got around to catching up on the theoretical basics – you’ve come to the right place.

All set? Let’s begin!

 

What you need to know about survey methodology

 

If you search for the term “survey methodology” online, you’ll see it’s used interchangeably to describe all sorts of survey tools, questionnaire construction methods, or feedback collection hacks.

In fact, they all somewhat fall into the definition.

According to UCLA Labor Center, “survey methodology is the study of survey methods and the sources of error in surveys”. Errors are all the factors that deviate your survey efforts from the desired outcome. Survey methodology studies aim at minimizing their occurrence in the future.

Now, depending on your previous experience with feedback collection, the word “survey” may bring various pictures to mind.

For some, it’s that evening call from the national opinion center prior to presidential elections. Others are more likely to think of the occasional website pop-up on their favourite online store, or live polls carried out in the street.

These are all examples of various survey instruments. But before we dive in, let’s think about the very core of all feedback collection:

Cause.

Why do you want to survey your audience?

Are you planning to collect feedback as a reactive measure to a single event, or do you want to analyze results for the same question across various points in time?

Are you looking for structured responses that can be quickly analyzed with a data analysis tool, or do you want descriptive answers to open-ended questions?

All these questions, among others, need to be taken into account if you want to optimize your survey design efforts.

So, let’s take a look at the pros and cons of various survey tools, and consider how some of the most popular customer satisfaction metrics can serve your goals.

 

Survey instrumentation

At the highest level, surveys can be divided into two groups: questionnaires and interviews.

User researcher Tahmina Tanny accurately points out that the main difference between both approaches lies in who collects the answers.

In questionnaires, it’s the participant, whereas during interviews, it’s the researcher who notes down not necessarily answers, per se, but key takeaways.

Now, at first glance, you might contemplate drawing a line between the two – especially if you’ve come to this post with a plan to collect feedback through questionnaires only.

Potentially, interviews may seem as too much of an elaborate research method for your current needs. And, chances are, your gut feeling is just right.

Still, the thing is, this might change quickly.

As you derive more and more information about your respondents, there may come a day longer conversations will make a lot of sense – and a world of a difference for your product development plan. And, when the time is right, you’ll know exactly what tool to employ to expand your feedback collection efforts.

Another great thing?

Interviews and questionnaires are pretty much a marriage made in heaven. While they’re doing quite well solo, combined, they’re a whole other dimension of powerful feedback. One method can accelerate the efforts of the other – interviews can complement what you’ve already learned from questionnaires, and questionnaires can pre-qualify the right people for interviews. Pretty great, right?

So, let’s take a look at the ways you can ace the feedback collection game.

 

Questionnaires

Website questionnaire

In a digitalized world like ours, no one needs to be convinced of the incredible potential of website feedback.

These surveys take on various forms – from large pop-ups in the middle of the screen, to discrete widgets on the edge of the browser window.

Here’s a general breakdown of what you can expect from website questionnaires:

Pros:

  • With the right tool, you can target an audience as broad or narrow as you like
  • The fastest way to collect crucial feedback from your website crowd
  • A highly effective way to measures customer satisfaction metrics such as NPS®.


Potential cons:

  • If overused, you can exhaust your respondents’ attention or patience and cause survey fatigue 
  • Without prior knowledge on your audience, you can target users inaccurately and receive unactionable feedback
  • It may be difficult to receive answers to open-ended questions if you haven’t previously proven your respondents that their insights and time are valued.

 

Now, here’s the good news:

With the right tool, you can make sure you don’t experience the downsides described above.

 

With Survicate, you can target relevant respondents right from the beginning, as you can easily import data you already have on your audience from external marketing and CRM tools.

Depending on your preference, you can also choose to run website questionnaires through your favourite customer communication tools. This approach, loved by many marketers, allows you to use the same tool for all audience communication, as well as cross-analyze and export survey responses within seconds.


Click here to read more.

 

Let’s proceed to another type of digital survey…

 

Mobile app questionnaire

These surveys, run inside mobile apps, are both relatively new and specific in terms of the beneficiary in mind.

Here’s an example – one you’re likely well accustomed with if you use ride-hailing or hotel apps:

Similarly to website questionnaires, mobile app surveys are displayed to users who interact with the product/service.

Pros:

  • Ideal for companies that have mobile apps and want to know how users find their way around/what they think about the service
  • You can reach users “on the go” – ideal for single-question surveys that are quick to answer
  • You can target users contextually and ask highly relevant questions.

 

Potential cons:

  • Threat of being quickly discarded, if too many steps or open-ended answers are required (i.e. inconvenience of typing on small screen)
  • If questionnaire covers entire screen or appears too often, there’s the risk of coming across as obtrusive.

Let’s carry on to the last online questionnaire on the list…

 

Email questionnaire

Similarly to website questionnaires, emails are one of the fastest ways to reach your audience. Assuming you’re using emails provided by your audience (which really is the only way to go), you can count on incredibly valuable feedback from users who have expressed interest in your service/product.

That is, of course, if you approach email questionnaires strategically.

Pros:

  • Similarly to website questionnaires, you can target specific segments of your email list.
  • Quick way to reach an engaged audience – assuming emails have been provided to you willingly (for ex. newsletter list), your respondents are likely to be more engaged than those who only sporadically visit your page
  • Effective for sending targeted customer satisfaction surveys

 

Potential cons:

  • Weak email subject lines can set you up for low response rates. Make sure to stand out!
  • If you send out all your questionnaires to the same audience (i.e. no segmentation), you might disturb respondents who are sensitive about spam (which, of course, your emails are not).

 

Want to hear a pro tip?

It has been proven that embedding only the first question in an email, and displaying remaining questions in a new window (as opposed to emailing a full questionnaire) boosts survey completion rates.

Sounds like something you’d like to try? You can do so on a free Survicate account!

Example of a first question in an email questionnaire

Now, let’s put digital questionnaires aside and take a look at some other options.

 

Mail questionnaire

While it may come as a surprise, many survey methodology resources online report that mail surveys are the most popular feedback collection method. Whether they’ve been dethroned by digital questionnaires remains unclear, though it certainly seems inevitable, given all the convenience online questionnaires bring.

Still, as of today, paper questionnaires are still appreciated in certain lines of business.

Pros:

  • You can collect feedback from clients who did not provide an address or phone number
  • Respondents can fill the survey in when it’s most convenient
  • Certain groups of respondents may trust a traditionally mailed survey more than an email.

 

Potential Cons:

  • Feedback is collected slowly, and response collection can take months
  • A lot of paperwork – responses need to be handled and scanned individually, if any modern-day analysis is to be performed.

 

Telephone questionnaire

Not as common as they were back in the day, but still a favourite of many companies that rely hugely on sales and customer service teams.

Pros:

 

Potential cons:

  • Response rates can be very low, if the call is automated
  • May come across as obtrusive, if calls are initiated too often or at the wrong time
  • Since the introduction of GDPR, and after the famous Cambridge Analytica scandal, respondents have become reluctant to reveal information on the phone.
Customer success team member on phone with client

In-person questionnaire

These used to perform much better decades back, before online surveying methods emerged. Nowadays, face-to-face questionnaires have mostly taken the form of meetings via video conferencing software.

As they’re quite time-consuming, today the approach is mostly carried out among random respondents (i.e. on the street), at an incentive to the respondent, or among very specific, carefully selected members of the audience (which, in its own right, might require previous pre-qualifying questionnaires).

 

Pros:

  • Good for modestly-sized feedback collection goals
  • If carried out on the street, it provides a good overview among random respondents
  • People tend to be more invested in questionnaires in person, and might be more prone to respond to open-ended questions.

 

Potential cons:

  • Time-consuming, both for the respondent and researcher
  • In-person questionnaires with carefully targeted groups often require running pre-qualification surveys weeks ahead.

 

Which leads us to…

 

Interviews

According to Geneva Foundation for Medical Education and Research, there are two main criteria to think of when we distinguish interviews. These are:

Interview structure

  • Structured (question order is predetermined, researchers only go off script to clarify question or ask for more details)
  • Semi-structured (questions are predetermined, but researcher is at liberty to be use informal language and tweak wording)
  • In-depth (questions aren’t predetermined, conversation relies mostly on one’s subjective point of view/opinion).

Number of participants

  • Individual interviews

In-person interviews: these will be invaluable if you’re looking to read between the lines – especially, if you adapt an in-depth or semi-structured approach. The same way as in the case of in-person questionnaires, respondents are more prone to answer more precisely when addressed face-to-face.

 

Phone interviews: Similarly to phone questionnaires, the response rates are high – even more so, as interviews can’t (at least, as of yet) be effectively carried out without a fellow human researcher on board. There is a significant difference though – while phone questionnaires remain on script, interviews, with their more open nature, encourage sharing opinions and, as an effect, often collect insights of highly subjective, emotional nature.

 

  • Focus groups

While these aren’t necessarily interchangeable with other surveys that made the list, it’s worth noting nonetheless. Focus groups are a way to trigger discussion on a given subject among several participants. It is an amazing way to uncover your audience’s attitudes, as it not only promotes openness, but also shows how diversified opinions evolve throughout a live discussion.

Now that you know all the survey instrumentation options, the question is:

What are some tell tale signs you’d benefit from a mixed approach?

Here are a couple examples:

  • You’ve been uncovering recurring behavior within a group, that can’t be explained through correlating other behavioral or attitudinal data.
  • Your respondents are less prone to answering open-ended questions (you’ll read more about closed/open-ended questions here) than they have before, and leave more and more empty fields.
  • Your respondents answer open-ended questions, but the responses are less insightful than you’ve imagined. Also, you don’t know how to formulate a questionnaire that triggers detailed response anymore.
  • There’s a very specific user group you’re increasingly interested in hearing more from – ideally, in a less structured, individual way.
  • You’ve been measuring specific customer metrics via surveys (for example, NPS® survey or CSAT survey, as explained below) but the results have been introducing confusion. You don’t really know what your respondents think anymore – or, more importantly, what they’re actually evaluating.


Now, the latter example is something I’d like us to look into a bit further.

 

Customer metrics – a blessing, or a downward spiral?

 

The quick answer? Potentially, both.


A common mistake made by survey design beginners is to start measuring specific metrics only because they’d heard their competitors or company they look up to do so.

An even more gruesome scenario? Collecting such data with zero-to-none knowledge on the basics of survey methodology or survey data analysis.

To put things simple, if you start measuring your audience’s satisfaction with the wrong metric, you’ll likely create more confusion than derive valuable, actionable information.

So how do you determine which metric – if any – will work?

Let’s take a quick look at how the way you formulate your questions defines the way to go.

 

Qualitative v. quantitative questions

Earlier on in the article, we mentioned how open-ended and closed-ended questions often serve different purposes. An equally important way of thinking about questions is through feedback priorities:

Are you looking to collect structured, quantifiable (so, alphanumerical) responses that can be easily visualized in survey data reports?

Or would you rather see what’s between the lines and find insights in answers that aren’t entirely measurable, but speak about quality?

All this determines how well customer metrics serve and drive your customer experience strategy.

 

So, let’s take a look at the three most popular customer satisfaction metrics and when they bring most benefits.

For starters, the metric that took the user research world by storm:

 

NPS® survey

It’s quite possible that the term NPS® rings a bell or two, as it’s easily one of the most popular (if not the most popular) customer metric among marketers.

Customers are asked a simple question about the likelihood of recommending your product or service. They rate it on a 0-10 scale.

Here’s an example of an NPS® survey created with Survicate:

Intercom NPS survey

Now, when it comes to analyzing NPS®, it all comes down to the assumption that your entire audience can be divided into three groups:

Detractors, who provide an answers between 0 to 6,

Passives, who answer either 7 or 8, and

Promoters, who answer either 9 to 10.

The assumption is that anyone who answered between 0 and 6 is a source of bad PR. Passives (7-8 scores) don’t have a particular stand regarding your service, whereas those who provided a score of 9 or 10 are active evangelists and would recommend it in a heartbeat.

NPS® can range from -100 to 100, and is measured simply by subtracting the percentage of detractors from promoters.

 

Is it worth measuring and tracking NPS®?

NPS® can do wonders if you’re looking for a metric that reveals a general opinion on your product/service, as an entirety. It’s just a single question, which often makes response rates go through the roof in comparison to other surveys.

Be prepared, though, that NPS® numbers may reflect all sorts of factors that constitute your brand, and setting follow-up questions is key. With a tool like Survicate, you can create automatic follow-up questions to nail down the areas that are most lauded for or disliked, and incorporate these insights into product development plans.

Interested in a real-life example?

Here’s how Droplr, content communication app for Slack, revolutionized their customer experience approach with Survicate NPS® surveys.

 

Let’s carry on to…

 

CSAT survey

Contrary to NPS®, CSAT (short for Customer Satisfaction Survey) enables you to ask your audience to rate a specific feature/event on a 5-point scale, from “Very unsatisfied” to “Very satisfied”. CSAT questions can be asked according to the following pattern:

“How would you rate your experience with …” – like in the example below:

 

Example of a Survicate CSAT survey

 

Calculating CSAT score could not be easier. You simply sum up the number of “Satisfied” and “Very satisfied” answers, and aim at the highest possible number.

What makes CSAT so popular among user researchers and marketers is that you can ask about virtually anything. Specific in nature, it provides quick, actionable insights to guide your customer experience.

Still, the results you receive relate only to the specific area you inquired about. They do not extend to your entire brand. This, in turn, may require you to reach for an additional metric that measures overall customer loyalty, such as NPS®. Or CES, which is often referred to as a hybrid of both:

 

CES survey

Depending on feedback collection goals and preferences, some researchers decide to run CES surveys as an alternative to carrying out NPS® surveys and CSAT surveys simultaneously.

But does it really kill two birds with one stone?

The answer is – somewhat.

 

CES gained momentum in the last decade, with Harvard Business Review praising “its ability to capture customer impressions at the transactional level (as opposed to NPS®, which captures more-holistic impressions of a company) and its ability to capture negative experiences as well as positive ones”.

CES measures how much effort your audience needs to put into performing a specific interaction with your company/product. So, he/she is asked a contextually accurate question (or statement, like the one below):

 

Survicate CES survey

The respondent answers on a scale from 1 to 7, where “Strongly disagree” and “Strongly agree” are on opposite ends. To measure the score, you simply calculate the average of all collected answers. The lower the results, the higher effort (and potential frustration) of your audience.

A definite upside of CES is that you not only measure customer satisfaction with your brand and/or specific department. You also check the entire usefulness and efficiency, which can be invaluable insight for any UX enhancements.

Want a pro hint?

If you do decide to run with CES surveys, make sure to add follow-up questions to hear the “why” behind all the answers that indicate dissatisfaction.

 

Other customer satisfaction metrics to consider


In this post, you’ve learned how various tools, goals, and customer satisfaction metrics can all influence the quality of your feedback collection strategy.

Naturally, the three metrics we just discussed aren’t all you can reach for.

With Survicate, you can choose from over 100 survey templates that have proven effective and note high response rates among respondents.


Here’s just a sneak peak of what you can expect:

 

Survey template library at Survicate

 

Like what you just learned?

For more guides and actionable tips, be sure to visit our Feedback Academy and see how you can make your response rates go through the roof!