Published: 10/31/2024
Steven Millman
Global Head of Research & Data Science, Dynata
It’s well understood that the quality and reliability of survey data are critical for effective decision-making. But how do you define the elements of survey data quality so that you can evaluate it? At Dynata we talk about the four pillars of data quality which serve as foundational elements for evaluating the integrity and usefulness of the insights we gather. Let’s explore each pillar in detail and see how they contribute to high-quality survey data.
- Open-End Quality
Open-ended questions allow respondents to express their thoughts and opinions in their own words, offering rich insights that structured questions might not be able to capture. The quality of these responses can vary substantially with high quality open-ends characterized by clear, relevant, and insightful answers. When there’s fraud in the survey, open-ended questions can be unrelated to the question posed, short, and sometimes incomprehensible. Disengaged but real respondents may have very surface-level response or responses with marginal value. We utilize both manual and automated text analysis tools to identify poor quality in open-ended responses to clear out respondents that will provide no value to our clients.
- Response Quality
Response quality goes beyond just an evaluation of the open-ends and looks at the overall quality of the survey response in their totality. It involves the accuracy, honesty, attention, and consistency of the answers that were provided. To promote high response quality, we look at the consistency of question responses, straight-lining behavior, speeding, and nearly 200 other behaviors and feed them into our proprietary AI tool, QualityScore™. QualityScore allows us to look at all of these behaviors simultaneously and detect not just fraud but also inattention so that we can filter the data we collect so that they genuinely reflect real opinions. - Demographic Representativeness
For survey results to be actionable, they must reflect the demographic composition of the target population. Demographic representativeness ensures that key characteristics—such as age, gender, ethnicity, socio-economic class, and geographic location—are accurately represented in the survey sample. We employ stratified sampling techniques, adjusting our panels to align with demographic benchmarks. We also recruit panelists in a wide variety of different ways and offer different kinds of incentives in order to create the most diverse and representative panel possible. This commitment to diversity not only enhances the validity of our findings but also builds trust with stakeholders who rely on our insights for decision-making.
- Behavioral Representativeness
Demographics alone don’t capture the full picture, understanding respondents’ behaviors is equally essential to ensure representative sample. Behavioral representativeness considers how well our survey sample reflects the actual behaviors, attitudes, and preferences of the broader population. This includes factors like purchase behavior, health and lifestyle choices and activities. Our comprehensive recruitment approach described above allows us to create a highly representative sample with respect to behaviors as well, providing insights that resonate with real-world behaviors, making our data even more actionable for clients.
Conclusion
The net result of our relentless pursuit of excellence in these pillars of quality is trusted, reliable data that have the highest sample acceptance rate in the industry. Understanding and innovating in the survey data quality space is a multifaceted challenge that relies on careful attention to these pillars. Each play a crucial role in ensuring that our findings are not only accurate but also meaningful for data-driven decision making. For more information on how we create the industry’s highest-quality data, please visit our Data Quality page.