Derivatives: Rates of Change

The derivative is the mathematical tool for measuring change. If limits are about “approaching,” derivatives are about “moving.”

1. The Instantaneous Rate of Change

We define the derivative $f'(x)$ as the limit of the average rate of change as the interval goes to zero: $f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}$

Instead of jumping to rules, let’s calculate the slope of $f(x) = x^2$ at $x=1$ by making $h$ smaller and smaller. This is exactly what your computer does when it performs Numerical Differentiation.

def f(x): return x**2

x_target = 1.0
h_values = [0.1, 0.01, 0.001, 0.0001]

print(f"{'h':<10} | {'Estimated Slope':<15}")
print("-" * 30)

for h in h_values:
  slope = (f(x_target + h) - f(x_target)) / h
  print(f"{h:<10} | {slope:<15.6f}")

print("-" * 30)
print("The limit seems to be 2.0, which matches the power rule (2x).")

2. Linear Approximation

Around a point $a$, we can approximate a complex function $f(x)$ with a simple line defined by the derivative: $f(x) \approx f(a) + f'(a)(x-a)$

This is why derivatives are so powerful: they turn complex, curvy problems into simple, linear ones.

import numpy as np
import matplotlib.pyplot as plt

def f(x): return np.sin(x)
def df(x): return np.cos(x) # Derivative of sin is cos

a = 1.0 # Point of approximation
x_range = np.linspace(0, 2, 100)

# Linear approx: L(x) = f(a) + f'(a)(x-a)
linear_approx = f(a) + df(a) * (x_range - a)

plt.plot(x_range, f(x_range), label='True sin(x)', linewidth=2)
plt.plot(x_range, linear_approx, '--', label='Linear Approx', color='r')
plt.scatter([a], [f(a)], color='black', zorder=5)
plt.title("Linear Approximation (Tangent Line) at x=1")
plt.legend()
plt.show()

3. The Rules of Calculus

To avoid numerical limits, we use rules. The Chain Rule is particularly vital for modern AI, as it powers backpropagation. $\frac{d}{dx} f(g(x)) = f'(g(x)) \cdot g'(x)$

4. Optimization: Finding Extremes

If a smooth function reaches a peak or valley, its slope must be zero. These are Critical Points.

import numpy as np
import matplotlib.pyplot as plt

# f(x) = x^4 - 4x^2
def f(x): return x**4 - 4*x**2
def df(x): return 4*x**3 - 8*x # 4x(x^2 - 2) = 0 => x=0, x=±sqrt(2)

x = np.linspace(-2.5, 2.5, 500)
plt.plot(x, f(x))
plt.axhline(0, color='k', alpha=0.3)
plt.title("Find the valleys: f'(x) = 0")
plt.show()

5. Summary Check