Why Pandas?
While NumPy provides the computational horsepower for numerical arrays, it lacks the high-level features needed for real-world data analysis, such as:
- Handling missing data (NaN).
- Working with labeled axes (instead of just integer indices).
- Merging and joining datasets (SQL-like operations).
- Time-series functionality.
Pandas builds on top of NumPy to provide these features. It is the primary tool for data “munging” or “wrangling”—the process of cleaning and transforming raw data into a format suitable for analysis.
Core Data Structures: Series and DataFrame
There are two primary data structures in Pandas:
- Series: A 1D array-like object with an associated index.
- DataFrame: A 2D table-like structure with labeled rows and columns.
The Series Object
Think of a Series as a cross between a NumPy array and a Python dictionary.
Interactive Lab
import pandas as pd
import numpy as np
# Creating a Series from a list
data = pd.Series([0.25, 0.5, 0.75, 1.0], index=['a', 'b', 'c', 'd'])
print("Series:\n", data)
# Accessing by label
print("\nValue at index 'b':", data['b'])
# Series behaves like a numpy array
print("\nMean of Series:", data.mean())
Expected output
Series:
a 0.25
b 0.50
c 0.75
d 1.00
dtype: float64
Value at index 'b': 0.5
Mean of Series: 0.625
1
2import pandas as pd
3import numpy as np
4
5
6data = pd.Series([0.25, 0.5, 0.75, 1.0], index=['a', 'b', 'c', 'd'])
7print("Series:\n", data)
8
9
10print("\nValue at index 'b':", data['b'])
11
12
13print("\nMean of Series:", data.mean())
14
The DataFrame Object
The DataFrame is the most important structure in Pandas. It represents a table of data, similar to a spreadsheet or a SQL table. Each column in a DataFrame is itself a Series.
Interactive Lab
import pandas as pd
data = {
'Name': ['Alice', 'Bob', 'Charlie', 'David'],
'Age': [25, 30, 35, 40],
'City': ['New York', 'London', 'Paris', 'Tokyo']
}
df = pd.DataFrame(data)
print("DataFrame:\n", df)
# Check basic info
print("\nColumns:", df.columns)
print("Index:", df.index)
Expected output
DataFrame:
Name Age City
0 Alice 25 New York
1 Bob 30 London
2 Charlie 35 Paris
3 David 40 Tokyo
Columns: Index(['Name', 'Age', 'City'], dtype='object')
Index: RangeIndex(start=0, stop=4, step=1)
1
2import pandas as pd
3
4data = {
5 'Name': ['Alice', 'Bob', 'Charlie', 'David'],
6 'Age': [25, 30, 35, 40],
7 'City': ['New York', 'London', 'Paris', 'Tokyo']
8}
9
10df = pd.DataFrame(data)
11print("DataFrame:\n", df)
12
13
14print("\nColumns:", df.columns)
15print("Index:", df.index)
16
Data Selection and Indexing
Selection in Pandas can be confusing because there are multiple ways to do it.
1. loc: Label-based selection
loc is used to select data using the labels of rows and columns.
2. iloc: Integer-based selection
iloc is used to select data using the 0-based integer position.
Interactive Lab
import pandas as pd
df = pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 6]
}, index=['row1', 'row2', 'row3'])
print("Selection with loc (labels):\n", df.loc['row2', 'A'])
print("\nSelection with iloc (positions):\n", df.iloc[1, 0])
Expected output
Selection with loc (labels):
2
Selection with iloc (positions):
2
1
2import pandas as pd
3
4df = pd.DataFrame({
5 'A': [1, 2, 3],
6 'B': [4, 5, 6]
7}, index=['row1', 'row2', 'row3'])
8
9print("Selection with loc (labels):\n", df.loc['row2', 'A'])
10print("\nSelection with iloc (positions):\n", df.iloc[1, 0])
11
Importing Data
In practice, you rarely create DataFrames by hand. You load them from files (CSV, Excel, JSON, SQL).
# Reading a CSV file
df = pd.read_csv('data.csv')
# Writing to an Excel file
df.to_excel('output.xlsx')
Handling Missing Data
Real-world data is messy. Pandas uses NaN (Not a Number) to represent missing values and provides robust tools to handle them.
Interactive Lab
import pandas as pd
import numpy as np
df = pd.DataFrame({
'A': [1, 2, np.nan, 4],
'B': [5, np.nan, np.nan, 8],
'C': [1, 2, 3, 4]
})
print("Original DF with NaNs:\n", df)
# Fill NaNs with a value
print("\nFilled NaNs:\n", df.fillna(value=0))
# Drop rows with any NaN
print("\nDropped NaNs:\n", df.dropna())
Expected output
Original DF with NaNs:
A B C
0 1.0 5.0 1
1 2.0 NaN 2
2 NaN NaN 3
3 4.0 8.0 4
Filled NaNs:
A B C
0 1.0 5.0 1
1 2.0 0.0 2
2 0.0 0.0 3
3 4.0 8.0 4
Dropped NaNs:
A B C
0 1.0 5.0 1
3 4.0 8.0 4
1
2import pandas as pd
3import numpy as np
4
5df = pd.DataFrame({
6 'A': [1, 2, np.nan, 4],
7 'B': [5, np.nan, np.nan, 8],
8 'C': [1, 2, 3, 4]
9})
10
11print("Original DF with NaNs:\n", df)
12
13
14print("\nFilled NaNs:\n", df.fillna(value=0))
15
16
17print("\nDropped NaNs:\n", df.dropna())
18
GroupBy and Aggregation
Similar to GROUP BY in SQL, Pandas allows you to split data into groups and apply functions to each group separately.
Interactive Lab
import pandas as pd
df = pd.DataFrame({
'Company': ['GOOG', 'GOOG', 'MSFT', 'MSFT', 'FB', 'FB'],
'Person': ['Sam', 'Charlie', 'Amy', 'Vanessa', 'Carl', 'Sarah'],
'Sales': [200, 120, 340, 124, 243, 350]
})
by_comp = df.groupby("Company")
print("Mean sales per company:\n", by_comp['Sales'].mean())
Expected output
Mean sales per company:
Company
FB 296.5
GOOG 160.0
MSFT 232.0
Name: Sales, dtype: float64
1
2import pandas as pd
3
4df = pd.DataFrame({
5 'Company': ['GOOG', 'GOOG', 'MSFT', 'MSFT', 'FB', 'FB'],
6 'Person': ['Sam', 'Charlie', 'Amy', 'Vanessa', 'Carl', 'Sarah'],
7 'Sales': [200, 120, 340, 124, 243, 350]
8})
9
10by_comp = df.groupby("Company")
11print("Mean sales per company:\n", by_comp['Sales'].mean())
12
In the following modules, we will explore advanced joining techniques and time-series analysis.