C11: Using Questionnaires and Surveys in UX Research

Abstract

Surveys are a widely used method to collect user feedback concerning the user experience of products or services, as they allow for collecting data from larger groups of users with low effort. A survey allows the collection of data concerning demographics and usage behavior (for example, frequency of use or experience with a product), quantitative measurements of UX quality over standardized UX questionnaires and user comments concerning various aspects of the user interaction.

The course describes the typical research questions that can be answered through a UX survey. It will be discussed in which cases alternative methods (for example, inspection methods, usability tests or usage tracking data) may be more appropriate and how a survey can be combined with such methods.

Some common standardized UX questionnaires are described in detail to provide an overview of the various existing measurement concepts, item formats and response scales.

We will discuss in detail the practical setup of a UX survey. How to find the most suitable UX questionnaire for the research questions? How to formulate effective survey questions? Which channels can be used to distribute the survey and what are their advantages and disadvantages? Which legal and ethical requirements must be considered, for example, data privacy regulations? How to analyze the structured data with suitable statistical methods? What are typical pitfalls in data analysis and how to avoid them? How to analyze larger volumes of user comments efficiently using GenAI? How can the results be interpreted? How are the findings presented to managers and software developers in industry or in research? How can UX insights collected over surveys be transferred successfully into UX improvements?

Course Content

Participants in this course will gain the ability to:

• Understand common UX research questions that can be addressed using surveys.
• Understand the advantages and disadvantages of UX surveys compared to other popular UX methods, such as interviews, usability tests or inspection methods.
• Understand the types of information that can be collected through surveys, including demographic data, usage behaviour data, user comments and UX metrics from standardized questionnaires.
• Understand the structure and use cases of popular standardized UX questionnaires that enable quantitative measurement of UX.
• Understand the key quality criteria (objectivity, reliability, validity) for standardized UX questionnaires.
• Learn how to set up a UX survey, including writing clear instructions, selecting appropriate and easy to understand questions, etc.
• Learn about various channels for collecting user feedback through surveys and the biases associated with each channel.
• Learn how to analyse structured data and avoid common pitfalls when drawing conclusions from such data.
• Discover the potential and associated risks of leveraging GenAI tools for analyzing unstructured data, specifically user comments collected from surveys. How to classify and summarize collected user comments with GenAI?
• Receive tips on delivering a successful presentation of survey results.
• Get some insights on how to establish a UX quality control process based on survey data in a larger enterprise, especially what is important to turn UX insights collected over surveys into concrete product improvements.
• Several exercises will help participants to get a deeper understanding of the covered topics.

Table of Contents

1. Basic concepts
2. Example of a typical UX survey
3. Typical application scenarios of UX surveys
4. Some popular standardized UX questionnaires
5. Comparison of surveys with other popular UX evaluation methods
6. How to define effective items?
7. Which UX aspects should be measured?
8. Setting up the survey and collecting data
9. Analyzing UX scores, demographic data and comments
10. How to use GenAI tools to classify and summarize user comments
11. Presenting survey results and turning insights into action
12. Course summary and concluding remarks