2 Working with Data and Data Types
Learn how to structure, import, inspect, and prepare tabular data in pandas while checking data types, missing values, and inconsistent entries.
Rows, columns, and pandas tables
A tabular dataset organizes observations in rows and variables in columns. For example, a table of measurements might have one row per sample and columns for sample ID, collection date, temperature, and location. Each column should generally describe one kind of information, and rows should follow the same structure.
In pandas, a is a two-dimensional table whose columns can have different data types. Each column behaves like a labeled one-dimensional , making this structure suitable for spreadsheet-like and database-style data.
Importing and inspecting data
(comma-separated values) is a common format for tabular files. In pandas, read_csv() loads a file into a ; other formats have corresponding import functions, such as read_excel() for Excel files.
The na_values argument can identify dataset-specific markers as missing. Choose these markers using the dataset’s conventions: for example, unknown may indicate a in one dataset but be a meaningful category in another.
Choosing and checking data types
A column’s tells software how to interpret its values and which operations make sense. Common types include:
Integer and floating-point number: counts and measurements, such as number of visits or temperature.
String (text): names, descriptions, and other character data.
Boolean: true/false values, such as whether a record is active.
Date or time: calendar dates and timestamps that support chronological operations.
Categorical: values from a limited set of groups, such as a product type or rating level.
The stored type is not always the type that best represents a variable. A postal code or customer ID may contain digits but is usually an , not a quantity, so averaging its values has no useful meaning. A rating such as low, medium, and high has an order that ordinary text does not express. Choose types according to what the values mean, not only how they look.
Check imported types before analysis. A numeric-looking column may be read as text if it contains symbols or inconsistent entries, and a date may initially be text rather than a date-time value. Convert values only after checking their format and deciding how to handle entries that do not convert; otherwise, valid information can be lost or misrepresented.
Missing and
A means an observation is unavailable or unrecorded. It is not the same as zero, an empty string, or a category such as “none,” unless the dataset defines it that way. Missing values may appear as blank cells, special codes, or values such as ?. pandas uses markers that vary by to represent missing values; use isna() or notna() to detect them rather than comparing values directly with NaN.
can be harder to detect because entries may look complete. Examples include North, north, and NORTH for the same category; extra spaces around text; dates written in different formats; or height values recorded in both centimeters and inches. Check distinct values in each column and compare them with the data dictionary, expected units, and plausible ranges. A missing-value count alone will not reveal these issues.
To review the categories in a column while keeping missing values visible, use value_counts(dropna=False):
Standardizing text, such as removing surrounding spaces or unifying capitalization, can resolve formatting differences, but only when those differences are not meaningful. Investigate unusual numbers before changing them: a value may be an error, a valid extreme observation, or the result of a different unit. Keep the original data available and document cleaning decisions.
Preparing data for analysis
Import a tabular dataset and inspect its dimensions, sample rows, column types, and missing-value counts before analysis. Check that each stored type matches the variable’s meaning, and look for both explicit missing markers and inconsistent conventions. Careful inspection helps prevent data-quality problems from producing misleading results.