Python Data Analysis with Pandas: Complete Tutorial (2026)

Pandas is the foundation of data analysis in Python. Whether you are cleaning messy CSV exports, building reports from database queries, or preparing features for a machine learning model, Pandas is almost certainly part of the workflow. It handles tabular data the way NumPy handles numerical arrays: fast, flexible, and with an API designed for real-world messiness.

This tutorial walks through every core Pandas operation with working code. Each example uses realistic data so you can paste it directly into a Jupyter notebook or Python script and see results immediately.

Setup and Installation

# Install Pandas (and optional dependencies)
pip install pandas matplotlib openpyxl

# Import convention
import pandas as pd
import numpy as np

The standard import alias is pd. Every tutorial, documentation page, and Stack Overflow answer uses it. Do not deviate from this convention.

Creating DataFrames

A DataFrame is a two-dimensional labeled data structure. Think of it as a spreadsheet with named columns and numbered (or labeled) rows.

From a Dictionary

import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
    'department': ['Engineering', 'Marketing', 'Engineering', 'Sales', 'Marketing'],
    'salary': [95000, 72000, 88000, 78000, 71000],
    'years': [5, 3, 7, 2, 4]
})

print(df)
#       name   department  salary  years
# 0    Alice  Engineering   95000      5
# 1      Bob    Marketing   72000      3
# 2  Charlie  Engineering   88000      7
# 3    Diana        Sales   78000      2
# 4      Eve    Marketing   71000      4

From a List of Dictionaries

# Each dict is a row - common when parsing API responses
records = [
    {'product': 'Widget A', 'price': 29.99, 'qty': 150},
    {'product': 'Widget B', 'price': 49.99, 'qty': 85},
    {'product': 'Widget C', 'price': 19.99, 'qty': 300},
]
df = pd.DataFrame(records)

Quick Inspection

df.shape        # (5, 4) - rows, columns
df.dtypes       # data type of each column
df.info()       # column names, types, non-null counts
df.describe()   # statistical summary (count, mean, std, min, max)
df.head(3)      # first 3 rows
df.tail(2)      # last 2 rows
df.columns      # column names as an Index
df.index        # row labels

Reading Data: CSV, Excel, JSON

CSV Files

# Basic read
df = pd.read_csv('sales_data.csv')

# Specify column types (reduces memory, prevents type guessing errors)
df = pd.read_csv('sales_data.csv', dtype={
    'order_id': 'int32',
    'customer_id': 'int32',
    'product': 'category',
    'amount': 'float32'
})

# Parse dates automatically
df = pd.read_csv('sales_data.csv', parse_dates=['order_date'])

# Read specific columns
df = pd.read_csv('large_file.csv', usecols=['name', 'email', 'total'])

# Handle different delimiters
df = pd.read_csv('european_data.csv', sep=';', decimal=',')

# Read in chunks (for files too large for memory)
chunks = pd.read_csv('huge_file.csv', chunksize=10000)
for chunk in chunks:
    process(chunk)

Excel Files

# Requires: pip install openpyxl
df = pd.read_excel('report.xlsx')

# Read a specific sheet
df = pd.read_excel('report.xlsx', sheet_name='Q4 Data')

# Read specific columns and skip header rows
df = pd.read_excel('report.xlsx', usecols='A:D', skiprows=2)

# Read all sheets into a dict of DataFrames
all_sheets = pd.read_excel('report.xlsx', sheet_name=None)
# all_sheets['Sheet1'], all_sheets['Q4 Data'], etc.

JSON Files

# Standard JSON array of objects
df = pd.read_json('data.json')

# Nested JSON (needs normalization)
import json
with open('nested.json') as f:
    raw = json.load(f)

df = pd.json_normalize(raw['results'], record_path='orders',
                        meta=['customer_id', 'customer_name'])

If you need to convert between CSV and JSON formats for data pipeline work, the JSON to CSV Converter and CSV to JSON Converter handle the transformation in your browser without uploading data to any server.

Writing Data Back Out

# CSV
df.to_csv('output.csv', index=False)

# Excel
df.to_excel('output.xlsx', index=False, sheet_name='Results')

# JSON
df.to_json('output.json', orient='records', indent=2)

# Clipboard (paste into spreadsheet)
df.to_clipboard(index=False)

Selecting and Filtering Data

Column Selection

# Single column (returns Series)
df['name']

# Multiple columns (returns DataFrame)
df[['name', 'salary']]

# Dot notation (only for simple column names without spaces)
df.name

Row Selection with loc and iloc

# loc: label-based selection
df.loc[0]                    # row with index label 0
df.loc[0:2, 'name':'salary'] # rows 0-2, columns name through salary
df.loc[df['salary'] > 80000] # boolean filter

# iloc: integer position-based selection
df.iloc[0]        # first row
df.iloc[0:3]      # first 3 rows
df.iloc[:, 0:2]   # all rows, first 2 columns
df.iloc[-1]       # last row

Filtering (Boolean Indexing)

# Single condition
high_earners = df[df['salary'] > 80000]

# Multiple conditions (use & for AND, | for OR, ~ for NOT)
eng_senior = df[(df['department'] == 'Engineering') & (df['years'] >= 5)]

# Filter with isin()
target_depts = df[df['department'].isin(['Engineering', 'Sales'])]

# Filter with string methods
df[df['name'].str.startswith('A')]
df[df['name'].str.contains('li', case=False)]

# query() method (cleaner for complex conditions)
df.query('salary > 80000 and department == "Engineering"')
df.query('years >= @min_years')  # use @ for Python variables

Adding and Modifying Columns

# New column from calculation
df['annual_bonus'] = df['salary'] * 0.10

# Conditional column with np.where
df['seniority'] = np.where(df['years'] >= 5, 'Senior', 'Junior')

# Multiple conditions with np.select
conditions = [
    df['years'] >= 7,
    df['years'] >= 3,
    df['years'] >= 0
]
labels = ['Senior', 'Mid', 'Junior']
df['level'] = np.select(conditions, labels)

# Apply a function to a column
df['name_upper'] = df['name'].apply(str.upper)

# Drop columns
df = df.drop(columns=['name_upper'])

GroupBy: Split-Apply-Combine

GroupBy splits data into groups based on column values, applies a function to each group, and combines the results. It is the Pandas equivalent of SQL's GROUP BY.

# Average salary per department
df.groupby('department')['salary'].mean()
# department
# Engineering    91500.0
# Marketing      71500.0
# Sales          78000.0

# Multiple aggregations
summary = df.groupby('department').agg(
    avg_salary=('salary', 'mean'),
    max_salary=('salary', 'max'),
    headcount=('name', 'count'),
    avg_years=('years', 'mean')
)

# Group by multiple columns
df.groupby(['department', 'seniority'])['salary'].mean()

# Custom aggregation with a lambda
df.groupby('department')['salary'].agg(
    lambda x: x.max() - x.min()
).rename('salary_range')

Transform and Filter

# transform: broadcast group result to original DataFrame shape
df['dept_avg'] = df.groupby('department')['salary'].transform('mean')
df['salary_vs_avg'] = df['salary'] - df['dept_avg']

# filter: keep only groups meeting a condition
large_depts = df.groupby('department').filter(lambda x: len(x) >= 2)

Merging, Joining, and Concatenating

pd.merge() -- SQL-Style Joins

employees = pd.DataFrame({
    'emp_id': [1, 2, 3, 4],
    'name': ['Alice', 'Bob', 'Charlie', 'Diana'],
    'dept_id': [10, 20, 10, 30]
})

departments = pd.DataFrame({
    'dept_id': [10, 20, 40],
    'dept_name': ['Engineering', 'Marketing', 'Legal']
})

# Inner join (only matching rows)
pd.merge(employees, departments, on='dept_id', how='inner')

# Left join (keep all employees, NaN for unmatched departments)
pd.merge(employees, departments, on='dept_id', how='left')

# Right join (keep all departments)
pd.merge(employees, departments, on='dept_id', how='right')

# Outer join (keep everything)
pd.merge(employees, departments, on='dept_id', how='outer')

# Join on different column names
pd.merge(employees, departments,
         left_on='dept_id', right_on='dept_id')

pd.concat() -- Stacking DataFrames

# Vertical stack (add rows)
q1 = pd.DataFrame({'month': ['Jan', 'Feb', 'Mar'], 'revenue': [100, 120, 110]})
q2 = pd.DataFrame({'month': ['Apr', 'May', 'Jun'], 'revenue': [130, 140, 125]})

full_year = pd.concat([q1, q2], ignore_index=True)

# Horizontal stack (add columns)
pd.concat([df1, df2], axis=1)

For a quick visual comparison of your data before and after merging, the JSON Diff tool highlights structural differences between two datasets.

Pivot Tables and Reshaping

sales = pd.DataFrame({
    'date': ['2026-01-01', '2026-01-01', '2026-01-02', '2026-01-02'],
    'product': ['Widget', 'Gadget', 'Widget', 'Gadget'],
    'region': ['North', 'South', 'North', 'South'],
    'revenue': [500, 300, 450, 320]
})

# Pivot table: average revenue by product and region
pd.pivot_table(sales, values='revenue', index='product',
               columns='region', aggfunc='mean')

# With totals
pd.pivot_table(sales, values='revenue', index='product',
               columns='region', aggfunc='sum', margins=True)

# Melt: wide to long format
wide = pd.DataFrame({
    'name': ['Alice', 'Bob'],
    'math': [90, 85],
    'science': [88, 92],
    'english': [95, 78]
})

long = wide.melt(id_vars='name', var_name='subject', value_name='score')
# name    subject  score
# Alice   math     90
# Alice   science  88
# Alice   english  95
# Bob     math     85
# ...

Handling Missing Data

df = pd.DataFrame({
    'name': ['Alice', 'Bob', None, 'Diana'],
    'salary': [95000, np.nan, 88000, 78000],
    'bonus': [5000, 3000, np.nan, np.nan]
})

# Detect missing values
df.isna()          # Boolean DataFrame
df.isna().sum()    # Count NaN per column

# Drop rows with any NaN
df.dropna()

# Drop rows where specific columns are NaN
df.dropna(subset=['salary'])

# Drop columns with any NaN
df.dropna(axis=1)

# Fill with a constant
df['bonus'].fillna(0)

# Fill with column mean
df['salary'].fillna(df['salary'].mean())

# Forward-fill (use previous value)
df['salary'].fillna(method='ffill')

# Back-fill (use next value)
df['salary'].fillna(method='bfill')

# Interpolate (linear by default)
df['salary'].interpolate()

# Replace specific values
df.replace({'salary': {0: np.nan}})

Time Series Analysis

# Create a DatetimeIndex
dates = pd.date_range('2026-01-01', periods=365, freq='D')
ts = pd.DataFrame({
    'date': dates,
    'sales': np.random.randint(100, 500, size=365)
})
ts.set_index('date', inplace=True)

# Slice by date range
jan = ts['2026-01']
q1 = ts['2026-01':'2026-03']

# Resample: daily to monthly
monthly = ts.resample('M').sum()

# Resample: daily to weekly with multiple aggregations
weekly = ts.resample('W').agg({'sales': ['sum', 'mean', 'max']})

# Rolling average (7-day window)
ts['rolling_7d'] = ts['sales'].rolling(window=7).mean()

# Expanding (cumulative) sum
ts['cumulative'] = ts['sales'].expanding().sum()

# Shift (lag/lead)
ts['prev_day'] = ts['sales'].shift(1)    # yesterday's sales
ts['next_day'] = ts['sales'].shift(-1)   # tomorrow's sales

# Percentage change
ts['pct_change'] = ts['sales'].pct_change()

# Day of week, month extraction
ts['dow'] = ts.index.day_name()
ts['month'] = ts.index.month

Visualization with Matplotlib

Pandas has built-in plotting that wraps Matplotlib. For quick exploration, it is faster than writing raw Matplotlib code.

import matplotlib.pyplot as plt

# Line plot (great for time series)
ts['sales'].plot(figsize=(12, 4), title='Daily Sales')
plt.ylabel('Revenue ($)')
plt.tight_layout()
plt.savefig('daily_sales.png', dpi=150)
plt.show()

# Bar chart
df.groupby('department')['salary'].mean().plot.bar(
    color='#00d4ff', title='Average Salary by Department'
)
plt.ylabel('Salary ($)')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

# Histogram
df['salary'].plot.hist(bins=20, edgecolor='black',
                       title='Salary Distribution')
plt.xlabel('Salary ($)')
plt.show()

# Scatter plot
df.plot.scatter(x='years', y='salary', alpha=0.6,
                title='Experience vs Salary')
plt.show()

# Multiple subplots
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
df.groupby('department')['salary'].mean().plot.bar(ax=axes[0], title='Avg Salary')
df['years'].plot.hist(ax=axes[1], bins=10, title='Years Distribution')
plt.tight_layout()
plt.show()

# Box plot (detect outliers)
df.boxplot(column='salary', by='department', figsize=(8, 5))
plt.suptitle('')  # Remove automatic title
plt.title('Salary Distribution by Department')
plt.show()

When you need to fine-tune the colors in your charts, the Color Picker generates hex, RGB, and HSL values that you can drop directly into your Matplotlib style parameters.

Performance Tips

1. Use Vectorized Operations, Not Loops

# Slow: iterating with iterrows
for idx, row in df.iterrows():
    df.at[idx, 'tax'] = row['salary'] * 0.3

# Fast: vectorized operation (100x+ faster on large datasets)
df['tax'] = df['salary'] * 0.3

2. Optimize Data Types

# Before: 304 bytes for a category column stored as object
df['department'].memory_usage(deep=True)

# After: convert to category (fewer unique values = big savings)
df['department'] = df['department'].astype('category')
df['department'].memory_usage(deep=True)  # significantly less

# Downcast numeric columns
df['salary'] = pd.to_numeric(df['salary'], downcast='integer')
df['years'] = pd.to_numeric(df['years'], downcast='integer')

3. Filter Early

# Slow: load everything, then filter
df = pd.read_csv('huge.csv')
filtered = df[df['region'] == 'North']

# Faster: only read what you need
df = pd.read_csv('huge.csv', usecols=['region', 'sales', 'date'])
filtered = df[df['region'] == 'North']

4. Use query() for Complex Filters

# Boolean indexing (creates temporary boolean arrays)
result = df[(df['salary'] > 80000) & (df['department'] == 'Engineering')]

# query() can be faster on large DataFrames
result = df.query('salary > 80000 and department == "Engineering"')

5. Consider Alternatives for Big Data

• Polars -- a Rust-based DataFrame library that is 5-10x faster than Pandas for many operations, with a Pandas-like API.
• DuckDB -- an in-process SQL database that can query Pandas DataFrames, Parquet files, and CSV files with SQL syntax, often faster than Pandas for aggregations.
• Dask -- parallel Pandas that works on datasets larger than memory by splitting work across multiple cores.

For formatting the JSON output from your Pandas data pipelines, the JSON Formatter validates structure and formats output with consistent indentation.

Related Developer Tools

Free browser-based tools for working with the data formats you encounter in Pandas workflows.

---