8. From Data to Insight

8. From Data to Insight

The central idea is that computation alone does not create knowledge. A defensible insight connects a clear question to suitable data, accurate representation, careful preparation, appropriate analysis, honest interpretation, and responsible communication.

A useful workflow is:

1. Define the question and the decision it may inform.

2. Identify relevant observations and variables.

3. Collect data using an appropriate method and record its context.

4. Represent the information in a form a computer can store and process.

5. Clean, validate, and document the data.

6. Explore patterns with filters, summaries, and comparisons.

7. Visualize results in a way that matches the question.

8. Interpret evidence while distinguishing association from causation.

9. Communicate findings, uncertainty, limitations, and justified actions.

10. Consider privacy, fairness, security, and accountability throughout the process.

Takeaway: Reliable insight is a complete process, not merely the output of a calculation or chart.

1. Define Questions and Collect Data

Begin with a purpose. Data are recorded representations of objects, events, measurements, or claims. A table commonly uses one row for each observation and one column for each variable. For example, a cafeteria-waste dataset might record the date, food prepared, food sold, and food discarded.

Before collecting information, ask:

• What question are we trying to answer?

• Which observations and variables are relevant?

• How will each variable be measured or categorized?

• What sampling method will be used?

• What privacy, consent, safety, or legal constraints apply?

Metadata describes how, when, where, and by whom the data was collected. It helps determine whether the data fits the question and how its limitations should affect interpretation.

Collection methods influence results. A voluntary online poll may overrepresent people who are especially interested in the topic. A measurement that excludes a group or time period may not represent the wider population. Unrelated personal information should not be collected simply because it is available.

Takeaway: Define the question, collect only relevant information when possible, and preserve enough context to judge the data’s suitability.

2. Represent Information Digitally

Computers store information as patterns of bits. A Bit has one of two values, $0$ or $1$. A Byte contains eight bits, so it can represent $2^8 = 256$ different patterns. More generally, $n$ bits can represent $2^n$ patterns.

The binary system, also called base 2, uses powers of two. For example:

A pattern has meaning only because an encoding rule gives it meaning. Text encodings map numbers to characters; image encodings map values to pixels and color components; sound encodings store numerical samples of air-pressure changes; video commonly combines a sequence of images with audio and metadata.

The same bits can mean different things under different formats. File formats and metadata tell software how to decode the stored patterns. Binary storage units also differ from decimal units: one kibibyte is $1{,}024$ bytes, whereas a kilobyte conventionally uses $1{,}000$ bytes.

Takeaway: Digital meaning comes from both the stored bit pattern and the encoding rule used to interpret it.

3. Store Data Efficiently

Compression reduces the number of bits needed to store or transmit data. The appropriate method depends on whether exact reconstruction is required.

Lossless compression preserves every original detail. It can replace repeated patterns with shorter references, assign shorter codes to frequent symbols, or represent repeated sequences efficiently. It is appropriate for source code, spreadsheets, databases, legal records, and many scientific datasets.

Lossy compression creates smaller files by discarding information considered less important for the intended use. It is common for photographs, streaming audio, and video. Repeatedly editing and resaving a lossy file can make discarded detail and visual or audible artifacts more noticeable.

Choose a method by considering:

• Required quality and accuracy.

• File size and transmission limits.

• Whether the original must be preserved exactly.

• Whether the data will be edited repeatedly.

• Accessibility and long-term archival needs.

Takeaway: Use lossless methods when exact values matter; use lossy methods only when the reduction in detail is acceptable for the purpose.

4. Prepare and Clean Data

Raw data may contain missing values, inconsistent labels, duplicate records, impossible measurements, or incorrect formats. Data cleaning begins by inspecting what each row and column represents, identifying data types and units, and checking missing and duplicated values.

A practical process includes:

• Standardizing formats: Represent equivalent labels, dates, currencies, capitalization, decimal separators, and units consistently.

• Handling missing data: Determine whether a blank means not collected, declined, or not applicable. Do not silently convert a blank to zero, because zero usually means that a quantity was measured as zero.

• Checking validity: Apply type checks, range checks, cross-field rules, and comparisons with trusted reference data.

• Investigating duplicates: Prevent the same observation from being counted more than once.

• Investigating an Outlier: Determine whether an unusual value is an error, a faulty measurement, a rare event, or evidence of a process change.

• Documenting changes: Record which fields changed, why they changed, which rules were used, and how many records were removed or altered.

Keep the original data unchanged and store transformations in a separate, reproducible procedure. Data quality is relative to intended use and includes accuracy, completeness, currency, relevance, consistency, reliability, appropriate presentation, and accessibility.

Takeaway: Cleaning is not the same as deleting inconvenient values; it is a documented process of making data fit for a justified use.

5. Explore Patterns and Comparisons

After cleaning, explore the dataset before making claims. Filtering selects records that meet conditions. Sorting orders records, while grouping combines observations by categories such as month, product type, location, or route.

Useful exploratory operations include:

• Counting records and categories.

• Calculating totals, means, medians, minimums, and maximums.

• Comparing groups with rates or percentages.

• Examining changes over time.

• Looking for relationships between variables.

• Checking whether results differ across demographic or geographic groups.

Use appropriate denominators. If a school serves different numbers of students in different months, the percentage of meals sold may be more informative than the total number sold. Likewise, incidents per $1{,}000$ people may be more comparable than raw incident counts when populations differ.

Tools should match scale and complexity. A spreadsheet may be suitable for a small, well-structured table, while a database, query language, statistical program, or data-processing system may be better for a large dataset. Whatever the tool, retain a record of filters, formulas, queries, and assumptions.

Takeaway: Exploration reveals patterns, but summaries are meaningful only when the comparison, denominator, and context are appropriate.

6. Summarize and Visualize Results

Spreadsheets combine tables, formulas, filters, summaries, and charts. A PivotTable can summarize a large dataset by category, apply filters, calculate subtotals, and rearrange fields. A PivotChart displays the summary so comparisons and trends are easier to inspect.

Choose Data visualization according to the question:

• Use a line chart to show how a value changes over time.

• Use a bar chart to compare categories.

• Use a histogram or box plot to show a distribution.

• Use a scatter plot to examine the relationship between two numeric variables.

• Use a stacked bar chart, or a carefully justified pie chart, to show parts of a whole.

• Use a map when geographic location is meaningful.

An honest chart should:

• State what is being measured.

• Include a clear title, labels, units, and time periods.

• Use a scale that does not exaggerate or hide differences.

• Avoid unnecessary decoration and confusing three-dimensional effects.

• Use color consistently and provide alternatives for readers who cannot distinguish some colors.

• Identify relevant uncertainty and distinguish counts from percentages and averages from totals.

A truncated axis is not automatically wrong, but it must be clear and appropriate. A change from $98$ to $100$ can look dramatic if the vertical axis begins at $97$, even though the absolute difference is small.

Takeaway: A chart is an argument about what matters, so its design must make the evidence easier to understand rather than more dramatic than it is.

7. Interpret Evidence Carefully

Exploration produces observations; interpretation connects them to the original question while accounting for uncertainty and context. Correlation means that two variables change together, but it does not by itself prove Causation.

For example, ice-cream sales and sunburn cases may both increase during hot weather. Temperature is a possible confounding variable. The observed association does not show that ice cream causes sunburn.

Strong interpretation distinguishes among:

• What the data directly shows.

• What is a reasonable inference.

• What remains unknown.

Also consider:

• Sample size and representativeness.

• Measurement error.

• Missing or excluded records.

• Changes in definitions over time.

• Selection bias and survivorship bias.

• Whether the analysis was exploratory or based on a pre-specified test.

• Whether totals should be replaced or supplemented by rates, proportions, or other denominators.

A large total may result from a large population rather than a high rate. Averages may also hide important differences in distributions. Conclusions should state how confident we can be and what evidence could change them.

Takeaway: Interpret patterns cautiously, compare like with like, and avoid claiming more than the data and design can establish.

8. Communicate Findings Responsibly

A clear report or presentation should answer:

1. What question was investigated?

2. What data was collected, and from where?

3. How was it cleaned and analyzed?

4. What are the most important findings?

5. How certain are those findings?

6. What limitations or alternative explanations exist?

7. What action, if any, is justified?

A useful communication structure is:

• Context: Explain the problem and why it matters.

• Method: Describe the population, variables, and key processing steps.

• Evidence: Present the most relevant tables or visualizations.

• Interpretation: Explain the pattern without overstating it.

• Limitations: State missing data, uncertainty, possible bias, and scope.

• Recommendation: Suggest an action or a next investigation when appropriate.

Reproducibility strengthens communication. Another person should be able to understand the data used, repeat the main calculations, and determine how the conclusions were reached.

Takeaway: A persuasive data communication links a question to documented evidence and makes uncertainty visible.

9. Apply Data Ethics

Data systems can improve public health, transportation, education, research, and access to services, but they can also create harm. Ethical practice must consider the full data lifecycle, from deciding what to measure through storing, analyzing, sharing, and deleting information.

Ask:

• Privacy: What personal information is collected, and who can access it?

• Purpose limitation: Is the data used only for the purpose originally explained?

• Consent and choice: Do people understand the collection and have meaningful choices?

• Security: How is data protected from unauthorized access or misuse?

• Fairness: Could the dataset underrepresent or disadvantage particular groups?

• Transparency: Can affected people understand how data influences decisions?

• Accountability: Who can correct errors or challenge harmful outcomes?

• Retention: When should data be deleted or anonymized?

• Access and inequality: Do communities have the technology, skills, and resources needed to benefit?

Data-driven systems are not automatically neutral. Choices about what to measure, whose information to collect, how categories are defined, which records are excluded, and which outcomes are optimized can all influence results. Anonymization can reduce risk but does not guarantee that reidentification is impossible when datasets are combined. Access controls, encryption, limited retention, privacy impact assessments, and clear governance reduce risk but do not replace ethical judgment or public accountability.

Takeaway: Responsible data work requires technical quality and respect for privacy, fairness, human oversight, inclusion, and the people affected by decisions.

10. Apply the End-to-End Workflow

Consider a city investigating delays on a bus route. The workflow connects every stage:

1. Define the question: Which stops and times are associated with the longest delays?

2. Collect data: Record scheduled arrival time, actual arrival time, route, stop, date, weather conditions, and service disruptions.

3. Represent the data: Store each arrival as one row with consistent timestamps and units.

4. Clean the data: Remove duplicates, correct invalid timestamps, identify missing arrivals, and investigate extreme delays.

5. Filter and group: Group arrivals by stop and hour; calculate average and median delay and the percentage exceeding five minutes.

6. Explore: Use a PivotTable to compare stops and time periods.

7. Visualize: Use a line chart for delay by hour and a bar chart for delay by stop.

8. Interpret: Consider whether delays are concentrated at particular locations or times, while accounting for weather and route changes.

9. Communicate: Report the methods, findings, uncertainty, and limitations.

10. Act responsibly: Avoid publishing personally identifying information about riders or drivers, and explain how the analysis will affect service decisions.

This example shows why insight depends not only on computation, but also on appropriate questions, valid representations, careful preparation, sound reasoning, and responsible communication.

Final takeaway: A defensible conclusion is one that can be traced from a clearly defined question through documented evidence to a proportionate and ethically considered action.