8 End-to-End Data Analysis Project
Learn how to plan, clean, explore, model, evaluate, and document a reproducible data analysis project, using a synthetic customer dataset as a practical example.
Plan for a reproducible analysis
A data analysis project turns a question and a dataset into a result that other people can check and reproduce. The work includes understanding and cleaning the data, exploring patterns, communicating findings, and evaluating any model on data it has not learned from.
Before analyzing data, specify the question, what one row represents, the target to predict (if there is one), which features will be available at prediction time, and which summaries, plots, or metrics will help answer the question. In the example, each row represents a customer; the goal is to explore spending and predict the continuous target spend using age, income, and region.
Organize inputs, code, and outputs so the work can be rerun. Keep original data separate from cleaned data, save figures rather than relying only on notebook display, and record Python and package versions. A fixed random seed makes generated data and a split repeatable; recording software versions helps make the computing environment repeatable.
Inspect and clean the data
A describes what values a variable can hold and which operations make sense for it. A number stored as text remains text until it is parsed. Treating a category code as a measurement, or a date as an ordinary string, can lead to invalid analysis.
Rows in a table are observations, and columns are variables. Start by checking the table's shape, sample rows, column types, unique values, duplicate records, and missing values. These checks can reveal problems before they affect a plot or model. Tabular tools such as pandas provide operations for reading data, checking types and missingness, and calculating summaries.
Cleaning should follow explicit, documented rules rather than aim for tidiness alone. Check whether values are valid for their meaning and units; standardize inconsistent labels; parse numeric and date fields; identify duplicates; and examine missing values. A missing value might mean “not recorded,” “not applicable,” or something else, so do not automatically replace it with zero. Record the chosen treatment because it depends on what the missingness means.
Explore patterns and visualize results
uses summaries and visualizations to understand distributions, unusual observations, and possible relationships. Useful descriptive statistics include counts, mean, median, standard deviation, minimum, maximum, and quartiles. Extreme values can pull the mean, while the median is often more representative of skewed data. For categorical variables, inspect counts and proportions.
Choose plots to match the question: histograms show distributions, scatter plots show relationships between two numeric variables, and bar charts compare category counts. Label axes and units, choose readable scales, and show missing data or groups when relevant. Save figures so results can be reviewed and shared.
A pattern in a plot or correlation table is evidence of association, not proof that one variable causes another.
Fit models without leaking test information
A learns a pattern from examples to make predictions. In supervised learning, each example has a target: regression predicts a numeric target, while classification predicts a category. Linear regression is a possible starting point for a numeric target, but its predictions do not demonstrate a causal relationship.
Set aside a test set before fitting preprocessing or a model. Fit preprocessing only on training data, then apply the learned transformations to the test data. Otherwise, can make evaluation overly optimistic. A combines preprocessing with an estimator; a column transformer applies different steps to different types of columns.
In the example, numeric features are imputed with their training-set medians and standardized. The categorical feature is imputed with its most frequent training-set value and one-hot encoded. These steps are learned on the training split and applied through the model . Standardization and encoding are preprocessing choices for this example, not reasons to use the test set when fitting transformations.
Follow the synthetic customer example
The example generates a synthetic dataset of 500 customers using a fixed random seed. Age is generated as an integer from 18 through 70, and income is generated around 60,000, then clipped to the range from 15,000 to 150,000. Region is selected from north, south, east, and west. Constructed region effects are 120, 60, 0, and , respectively. Spending is generated as
where the noise is generated with standard deviation 250. The constructed values illustrate a workflow, not a claim about real customers.
To make the inputs deliberately messy, some ages and incomes are missing, some ages contain the text unknown, and some region labels have inconsistent capitalization or whitespace. Income values include currency symbols and thousands separators; some region values are missing too. The cleaning steps parse numeric text, turn unparseable age values into missing values, remove the currency symbols and separators before parsing income, and trim and lowercase region labels. Empty region labels are treated as missing.
The example inspects shape, types, duplicates, missing values, numeric summaries, region counts, and numeric correlations. It plots the spending distribution and income against spending, with points grouped by region. It then splits the data into training and test sets, reserving one fifth for testing, and uses random seed 42 for that split. The script requires Python with numpy, pandas, matplotlib, and scikit-learn; it generates data locally and does not require an external dataset download.
Evaluate, interpret, and document
measures average absolute prediction error in the target's units; lower is better. compares the model with predicting the mean of the test-set targets. It can be negative when predictions are worse than that comparison. Compare a model's performance with a simple instead of interpreting a metric in isolation.
In the example, the predicts the average training-set spending for every test example. If the model does not meaningfully improve on it, reconsider the features, data quality, target definition, and whether the task is predictable. Do not repeatedly tune decisions against the test set; use cross-validation on training data for model selection, then reserve the test set for final evaluation.
A useful result includes more than a score. Check whether errors are concentrated in particular ranges or groups, and compare predictions with observed values. Consider whether the data represents the intended population, whether each feature would be available at real prediction time, and whether the evaluation split matches how predictions will be used. For example, a random split may not represent future prediction when data is time-ordered.
Document the question, data origin, cleaning decisions, exclusions, software versions, random seeds, plots, metrics, and limitations. Keep the original input unchanged so the analysis can be rerun from its starting point. Conclusions from this synthetic demonstration apply only to its constructed data.