Machine Learning Operations (MLOps)

MLOps standardizes the automation, deployment, and monitoring of production ML systems.

Model Decay and Drift

Unlike traditional software, ML models decay in production as real-world data distributions drift:

• Data Drift: statistical properties of inputs change ($P(X_{\text{prod}}) \neq P(X_{\text{train}})$) due to demographic or environment shifts.
• Concept Drift: relationships between inputs and targets change ($P(Y|X_{\text{prod}}) \neq P(Y|X_{\text{train}})$) due to structural changes (e.g., macroeconomics).

MLOps Core Architecture

Production MLOps pipelines introduce Continuous Training (CT) alongside CI/CD:

skinparam backgroundColor transparent
rectangle "Feature Store" as Data
rectangle "CT Pipeline" as CT
rectangle "Model Registry" as Registry
rectangle "Production API" as Prod
rectangle "Drift Monitor" as Mon
Data --> CT : training data
CT --> Registry : registers binary
Registry --> Prod : rolls out model
Prod --> Mon : logs inference
Mon --> Data : triggers CT on drift

Core Components

• Feature Store: stores standardized features for training/inference, preventing serve skew.
• Model Registry: catalogs model binaries, metadata, and version tags.
• Metadata Store: logs hyperparameters and evaluation histories.
• Data Versioning (DVC): tracks datasets using git-like hashes.

MLOps Maturity Levels

• Level 0: manual training and deployment. No automated tracking.
• Level 1: automated model retraining (CT) triggered by drift.
• Level 2: automated CI/CD pipelines deploy both code and retraining steps.

Example: Simulating Drift Detection

The following example demonstrates a basic drift detection agent comparing production feature means with the training baseline:

import numpy as np

train_mean, train_std = 50.0, 5.0
np.random.seed(42)
prod_data = np.random.normal(loc=52.5, scale=5.0, size=100)
prod_mean = np.mean(prod_data)
deviation = abs(prod_mean - train_mean) / train_std

print(f"Production Mean: {prod_mean:.2f}")
print(f"Deviation (in STDs): {deviation:.2f}")

if deviation > 0.4:
  print("ALERT: Statistical data drift detected! Trigger retraining.")
else:
  print("Status: Stable")

Production Mean: 52.34
Deviation (in STDs): 0.47
ALERT: Statistical data drift detected! Trigger retraining.

Exercise

Test your understanding of concept and data drift mechanisms:

References & Further Reading

• Géron, A. (2022). Hands-On Machine Learning (3rd ed.). O’Reilly Media.