Assessing Data Sources for a Question
Learn how to judge whether a data source can answer a particular question by checking its scope, collection and update practices, definitions, transformations, and record-level grain.
Is This Data Fit for the Question?
Start with the question, not the spreadsheet
A is the system, process, or publication from which data is obtained. Examples include transaction and operational systems, surveys, spreadsheets, application logs, and external datasets.
A dataset can be reliable and still be the wrong evidence for a particular question. To assess whether it is fit for purpose, first identify what it represents—and what it leaves out. Then check who collected it, how and when it was collected, how often it is updated, and what definitions, units, filters, or transformations shaped it.
Consider a question about people who bought a product. Transaction data may reliably record completed purchases, but it does not include people who never bought. It can support conclusions about recorded purchases; by itself, it cannot describe all potential customers. Similarly, a survey may capture opinions, but its results reflect the answers of people who responded. The key issue is not simply whether the records are accurate: it is whether the source covers the people or events your question concerns.
Find out what one record means
The is what one record represents. Knowing it prevents a common counting error: treating every row as if it represented the same kind of thing.
For example, an order-line dataset has one record per line in an order. An order with several products may therefore appear in several rows. Counting rows counts order lines, not orders. The dataset may be well suited to questions about products sold, but a count of its rows does not answer how many orders there were.
Before using a count or total, state the in plain language: “one row represents one ___.” Then check whether that unit matches the question. If it does not, the analysis needs a suitable way to count the intended entities rather than assuming that rows and entities are interchangeable.
Make a source-fit check
Work through these questions before interpreting results:
Scope: Which people, events, or objects are included? Which are excluded?
Collection and timing: Who gathered the data, how and when was it gathered, and how frequently is it updated? Does its date range match the question?
Meaning: What do the fields, categories, and units mean? Check definitions and metadata rather than relying on column names alone.
Processing: Were filters or other transformations applied? Could they affect which records or values appear?
: What does one record represent, and does that match the unit you need to analyze?
For instance, before using purchase records to describe a customer group, check whether the records cover the intended dates and population, whether the source excludes non-buyers, and whether each row is a purchase, an order, or an order line. These checks clarify what conclusions the data can support—and where its limits matter.