1 Introduction to Data Science
Learn how to plan and carry out an iterative data science project, choose formats for different data needs, and handle data responsibly.
Start with a clear purpose
Begin with the purpose of the project, not with a particular tool. Clarify the question to answer, who will use the result, and what evidence would count as a useful answer. These choices guide the data and methods that are appropriate.
The
The connects the project purpose to its results through six activities:
Acquire data from appropriate sources, recording where it came from and how it was obtained.
Understand the data by checking its structure, meaning, coverage, and limitations.
Prepare the data for analysis, for example by correcting errors, handling missing values, and combining relevant sources.
Analyze or model the prepared data with methods suited to the question.
Evaluate whether the results are reliable and address the original purpose.
Communicate or deploy the findings, explain their limits, and monitor results when they are used in an ongoing process.
This workflow is iterative rather than a one-way checklist. Analysis may reveal data problems, while evaluation may show that the question or approach needs revision. Documenting transformations and decisions helps others understand and reproduce the work.
For example, to answer whether bus delays are getting worse, a city team defines “delay,” obtains schedule and arrival records, checks which routes and dates are represented, prepares consistent timestamps, analyzes delay patterns, checks whether the results answer the question, and presents both findings and data gaps. If records omit certain routes, that limitation should be reported rather than hidden.
Choose a format for the data
A data format affects how information is stored, exchanged, and interpreted. Choose a format suited to the data and the tools that need to read it.
(comma-separated values) stores rows and fields as text. It is convenient for simple tables and widely supported. Quoting, delimiters, character encodings, and missing-value conventions can cause ambiguity, so document assumptions and verify how software reads the file.
(JavaScript Object Notation) represents structured data using objects of name–value pairs and ordered arrays. It is useful for nested records and data exchanged between software systems.
is a column-oriented format designed for efficient storage and retrieval, often useful for larger analytical datasets.
For example, a small table of daily temperatures may fit naturally in ; a record containing a person's address and a list of observations may be represented in ; and a large table queried by selected columns may be stored in .
Preserve and
Keep a or data dictionary that explains field names, types, units, and meanings. provides context that raw values alone may not convey. Preserve this information alongside the data so that its structure and meaning are available when the data are used.
Handle data responsibly
Responsible data work considers risks throughout collection, analysis, sharing, and disposal—not only when a dataset is published. A voluntary approach such as the NIST Privacy Framework can help identify and manage privacy risks.
Collect only what is needed. Apply for a defined purpose, and keep personal information only as long as necessary.
Protect access and storage. Limit access to people who need it, use appropriate safeguards, and follow applicable organizational requirements.
Preserve meaning and context. Record sources, definitions, changes, and known gaps. Check whether missing or inaccurate records could distort conclusions.
Consider who is represented. Ask whether data omit groups or circumstances relevant to the question, and avoid presenting uncertain results as universal.
Share carefully. Before sharing, consider whether people could be identified from the data or combinations of details. Share only what is appropriate for the stated purpose.
Explain limitations. Communicate how data were collected, what they cannot establish, and any important uncertainty.
Set retention and disposal practices. Keep data only as needed and dispose of it securely according to applicable policy.