6 Exploratory Data Analysis

Purpose and questions

Exploratory data analysis (EDA) examines a dataset to understand its structure, describe important features, discover patterns, and identify questions for further investigation. It combines numerical summaries with graphs. EDA can reveal anomalies and suggest explanations, but a pattern found during exploration is not, by itself, proof of a hypothesis or a causal relationship.

A useful investigation starts with a question that the available data can answer. Questions may be descriptive, such as asking what a typical score is; comparative, such as asking whether scores differ by course section; or relational, such as asking whether study hours are associated with scores.

Define the investigation

A clear question identifies the outcome being investigated and the possible explanatory variables that might be related to it. It also specifies the population and time period represented by the observations and the groups, conditions, or time periods to compare. These details help determine which variables to inspect and which summaries or plots will be useful.

For example, “Do students who study more earn higher exam scores?” identifies exam score as the outcome and study hours as a possible explanatory variable. It asks about a relationship, but does not establish that study time causes higher scores; other factors, such as prior preparation, could also matter.

Inspect the data

Before looking for patterns, determine what each row represents and what each column means. Check the dataset’s size, variable names, measurement units, and data types. Identify categorical variables, numerical measurements, dates, and any outcome or group labels relevant to the question.

Check for missing values, repeated records, impossible values, inconsistent category labels, and unusual ranges. These issues can distort summaries and graphics. Do not automatically delete an unusual observation: first consider whether it is an error or a genuine, informative case.

In Python, pandas can provide a quick overview. Treat the summaries as an initial check, not a substitute for understanding the data. For example, a numeric code representing a category may be incorrectly summarized as a measured quantity.

Summarize individual variables

Start by examining each variable on its own. For numerical data, the mean and median describe location. The range, interquartile range (IQR), and standard deviation describe spread; minimum and maximum values can help check extremes. An unusually large or small value can pull the mean, while the median is often more representative when the distribution is skewed.

For categorical data, count observations in each category and calculate proportions when comparing groups is useful. Note how many observations contribute to each summary, especially when values are missing.

Graphs can reveal features that a single summary hides. Histograms and dot plots show the shape of numerical distributions, while box plots compactly show the median, quartiles, and potential extreme observations. Bar charts show category counts or proportions. Look for skew, multiple clusters, gaps, and unusual values.

Examine relationships

Choose a display that matches the variables and the question:

• Two numerical variables: Use a scatterplot. Ask whether the association is positive, negative, curved, or weak, and whether there are clusters or unusual points.

• Numerical outcome and categorical group: Use side-by-side box plots, dot plots, or distributions by group. Compare typical values and spreads, and check whether the distributions overlap.

• Two categorical variables: Use a contingency table or grouped bar chart to compare category proportions across groups.

• Measurement over time: Use a time plot to look for a trend, seasonality, an abrupt change, or a gap.

A scatterplot of study hours against exam score, for example, may show that scores tend to rise with study time. Add group labels or compare sections if the question calls for it. A visual pattern may be driven by a few observations, a subgroup, or a third variable, so examine the data more closely before interpreting it.

Develop and assess hypotheses

EDA supports an iterative process for developing and assessing a hypothesis:

1. State a testable expectation, such as: “Students reporting more study hours tend to have higher exam scores.”

2. Choose evidence relevant to the claim. Inspect a scatterplot and summaries of both variables, and consider whether the pattern is consistent across groups.

3. Look for alternative explanations. Check for outliers, missingness, measurement problems, and variables that could account for the apparent relationship.

4. Refine the question. A pattern may suggest a more specific comparison or a new variable to investigate.

5. Separate discovery from confirmation. If the same data are repeatedly inspected and used to choose a hypothesis, the apparent pattern may be exaggerated. Treat exploratory findings as leads; assess a pre-specified claim with appropriate inferential methods or new data.

A hypothesis is consistent with the observed data when the patterns support its expected direction or differences. This is more careful than saying EDA proves the hypothesis. Formal statistical tests address uncertainty under stated assumptions. Causal claims generally require a suitable study design, not just an association in an exploratory plot.

Communicate findings

A clear EDA report states the question, describes the observations and important data limitations, and presents only summaries and plots that help answer the question. Label axes and units, identify groups, and explain the visible pattern in plain language. Distinguish what the data show from what you infer, and mention uncertainty or alternative explanations where relevant.

The overall process is to inspect the data and variable types, check data quality, summarize individual variables, and visualize relationships that match the question. Use the patterns to guide further analysis, without treating exploratory evidence as proof of causation or a replacement for confirmatory methods.