Why Time Series in Pandas?
Pandas was originally developed in a financial context (at AQR Capital Management), which explains why it has world-class support for time series. Temporal data often requires specialized operations like resampling (converting frequency), time zone conversion, and rolling windows.
The DatetimeIndex
The key to time series in Pandas is having a DatetimeIndex.
Interactive Lab
import pandas as pd
import numpy as np
# Create a range of dates
dates = pd.date_range('2023-01-01', periods=6, freq='D')
print("Dates Index:\n", dates)
# Create a Series with the date index
ts = pd.Series(np.random.randn(6), index=dates)
print("\nTime Series:\n", ts)
Expected output
Dates Index:
DatetimeIndex(['2023-01-01', '2023-01-02', '2023-01-03', '2023-01-04',
'2023-01-05', '2023-01-06'],
dtype='datetime64[ns]', freq='D')
Time Series:
2023-01-01 -1.11
2023-01-02 0.45
2023-01-03 1.23
2023-01-04 -0.56
2023-01-05 0.89
2023-01-06 -0.12
Freq: D, dtype: float64
1
2import pandas as pd
3import numpy as np
4
5
6dates = pd.date_range('2023-01-01', periods=6, freq='D')
7print("Dates Index:\n", dates)
8
9
10ts = pd.Series(np.random.randn(6), index=dates)
11print("\nTime Series:\n", ts)
12
Resampling: Upsampling and Downsampling
Resampling is the process of changing the frequency of your time series observations.
- Downsampling: Aggregating data (e.g., daily to monthly).
- Upsampling: Increasing frequency (e.g., monthly to daily), often requiring interpolation.
Interactive Lab
import pandas as pd
import numpy as np
# Hourly data for 3 days
rng = pd.date_range('1/1/2023', periods=72, freq='H')
ts = pd.Series(np.random.randn(len(rng)), index=rng)
# Downsample to Daily frequency and get the mean
daily_summary = ts.resample('D').mean()
print("Daily Mean:\n", daily_summary)
Expected output
Daily Mean:
2023-01-01 -0.01
2023-01-02 0.04
2023-01-03 -0.12
Freq: D, dtype: float64
1
2import pandas as pd
3import numpy as np
4
5
6rng = pd.date_range('1/1/2023', periods=72, freq='H')
7ts = pd.Series(np.random.randn(len(rng)), index=rng)
8
9
10daily_summary = ts.resample('D').mean()
11print("Daily Mean:\n", daily_summary)
12
Moving Windows (Rolling Operations)
Rolling operations allow you to calculate statistics over a sliding window of time. This is common for smoothing noisy data or calculating moving averages in finance.
Interactive Lab
import pandas as pd
import numpy as np
# Create 100 days of data
ts = pd.Series(np.random.randn(100), index=pd.date_range('1/1/2023', periods=100))
# Calculate 7-day rolling mean
rolling_mean = ts.rolling(window=7).mean()
print("Original Data (first 10):\n", ts.head(10))
print("\n7-day Moving Average (first 10):\n", rolling_mean.head(10))
Expected output
Original Data (first 10):
2023-01-01 0.12
2023-01-02 -0.45
2023-01-03 0.89
2023-01-04 -0.34
2023-01-05 1.56
2023-01-06 0.11
2023-01-07 -0.78
2023-01-08 0.23
2023-01-09 1.11
2023-01-10 -0.56
Freq: D, dtype: float64
7-day Moving Average (first 10):
2023-01-01 NaN
2023-01-02 NaN
2023-01-03 NaN
2023-01-04 NaN
2023-01-05 NaN
2023-01-06 NaN
2023-01-07 0.3157
2023-01-08 0.3342
2023-01-09 0.4568
2023-01-10 0.3456
Freq: D, dtype: float64
1
2import pandas as pd
3import numpy as np
4
5
6ts = pd.Series(np.random.randn(100), index=pd.date_range('1/1/2023', periods=100))
7
8
9rolling_mean = ts.rolling(window=7).mean()
10
11print("Original Data (first 10):\n", ts.head(10))
12print("\n7-day Moving Average (first 10):\n", rolling_mean.head(10))
13
Handling Time Zones
Global data often requires reconciling different time zones.
Interactive Lab
import pandas as pd
ts = pd.Series([1, 2, 3], index=pd.date_range('2023-01-01', periods=3, freq='D'))
# Localize to UTC
ts_utc = ts.tz_localize('UTC')
print("UTC Series:\n", ts_utc)
# Convert to US Eastern time
ts_eastern = ts_utc.tz_convert('US/Eastern')
print("\nEastern Time Series:\n", ts_eastern)
Expected output
UTC Series:
2023-01-01 00:00:00+00:00 1
2023-01-02 00:00:00+00:00 2
2023-01-03 00:00:00+00:00 3
Freq: D, dtype: int64
Eastern Time Series:
2022-12-31 19:00:00-05:00 1
2023-01-01 19:00:00-05:00 2
2023-01-02 19:00:00-05:00 3
Freq: D, dtype: int64
1
2import pandas as pd
3
4ts = pd.Series([1, 2, 3], index=pd.date_range('2023-01-01', periods=3, freq='D'))
5
6
7ts_utc = ts.tz_localize('UTC')
8print("UTC Series:\n", ts_utc)
9
10
11ts_eastern = ts_utc.tz_convert('US/Eastern')
12print("\nEastern Time Series:\n", ts_eastern)
13
Shifting and Lagging
In time series modeling, you often want to shift data forward or backward in time (e.g., to calculate percentage changes).
Interactive Lab
import pandas as pd
ts = pd.Series([10, 20, 30, 40], index=pd.date_range('2023-01-01', periods=4))
# Shift forward by 1
shifted = ts.shift(1)
print("Shifted (Lag 1):\n", shifted)
# Calculate percentage change
pct_change = (ts - ts.shift(1)) / ts.shift(1)
print("\nPercent Change:\n", pct_change)
Expected output
Shifted (Lag 1):
2023-01-01 NaN
2023-01-02 10.0
2023-01-03 20.0
2023-01-04 30.0
Freq: D, dtype: float64
Percent Change:
2023-01-01 NaN
2023-01-02 1.00
2023-01-03 0.50
2023-01-04 0.33
Freq: D, dtype: float64
1
2import pandas as pd
3
4ts = pd.Series([10, 20, 30, 40], index=pd.date_range('2023-01-01', periods=4))
5
6
7shifted = ts.shift(1)
8print("Shifted (Lag 1):\n", shifted)
9
10
11pct_change = (ts - ts.shift(1)) / ts.shift(1)
12print("\nPercent Change:\n", pct_change)
13
Period Containers
While Timestamp represents a point in time, Period represents a duration (a day, a month, a year).
Interactive Lab
import pandas as pd
p = pd.Period('2023-01', freq='M')
print(f"Period: {p}")
print(f"Next month: {p + 1}")
# Converting from Timestamp to Period
ts = pd.Timestamp('2023-01-15')
period = ts.to_period('M')
print(f"Timestamp to Monthly Period: {period}")
Expected output
Period: 2023-01
Next month: 2023-02
Timestamp to Monthly Period: 2023-01
1
2import pandas as pd
3
4p = pd.Period('2023-01', freq='M')
5print(f"Period: {p}")
6print(f"Next month: {p + 1}")
7
8
9ts = pd.Timestamp('2023-01-15')
10period = ts.to_period('M')
11print(f"Timestamp to Monthly Period: {period}")
12
In the next section, we will integrate these data manipulation skills with Scikit-Learn to build predictive models.