10 · Capstone — Derive & Verify a Gradient¶
This capstone combines every skill from Level 1: reading vector/matrix notation, computing partial derivatives, assembling a gradient, and cross-checking hand work against NumPy — both symbolically and numerically.
The task¶
Consider the two-variable cost function
This isn't tied to a specific dataset — it's a hand-crafted "toy" cost surface, deliberately built so we can derive its gradient exactly and then verify every step. (Real cost functions, like Module 9's linear regression \(J\), have the same structure: a sum of terms, each differentiable by the rules from Modules 6–8.)
Step 1: Derive \(\frac{\partial J}{\partial w}\) by hand¶
Differentiate term by term (sum rule), treating \(b\) as constant:
- \(\frac{\partial}{\partial w}(w-3)^2\): chain rule, outer \(u^2\) with \(u=w-3\), inner derivative \(\frac{\partial u}{\partial w}=1\), so this term gives \(2(w-3)\cdot 1 = 2(w-3)\).
- \(\frac{\partial}{\partial w}(b+2)^2 = 0\) (no \(w\) in this term, so it's a constant with respect to \(w\)).
- \(\frac{\partial}{\partial w}(wb) = b\) (treating \(b\) as a constant multiplier of \(w\), power rule on \(w^1\)).
Step 2: Derive \(\frac{\partial J}{\partial b}\) by hand¶
Symmetric process, treating \(w\) as constant:
- \(\frac{\partial}{\partial w \to b}(w-3)^2 = 0\) (no \(b\) in this term).
- \(\frac{\partial}{\partial b}(b+2)^2 = 2(b+2)\) (chain rule, same as above).
- \(\frac{\partial}{\partial b}(wb) = w\) (treating \(w\) as a constant multiplier of \(b\)).
Step 3: Assemble the gradient¶
Step 4: Evaluate by hand at a specific point¶
Let's evaluate at \((w,b) = (1, 1)\):
Also compute \(J(1,1)\) itself for the cross-check:
Step 5: Verify in NumPy — symbolic gradient formula¶
import numpy as np
def J(w, b):
return (w - 3) ** 2 + (b + 2) ** 2 + w * b
def grad_exact(w, b):
dJ_dw = 2 * (w - 3) + b
dJ_db = 2 * (b + 2) + w
return np.array([dJ_dw, dJ_db])
w0, b0 = 1.0, 1.0
print("J(1,1) :", J(w0, b0))
print("exact gradient :", grad_exact(w0, b0))
Expected output (matches Step 4's hand computation):
Step 6: Verify with numerical (finite-difference) gradient¶
As a completely independent check — using no symbolic derivative rules at all, just the raw definition of a partial derivative from Module 8 — confirm the same result:
def grad_numeric(w, b, h=1e-5):
dJ_dw = (J(w + h, b) - J(w, b)) / h
dJ_db = (J(w, b + h) - J(w, b)) / h
return np.array([dJ_dw, dJ_db])
print("numeric gradient :", grad_numeric(w0, b0))
Expected output (matches the exact gradient to within finite-difference error):
The symbolic derivation (Steps 1–4), the direct formula in code (Step 5), and the independent finite-difference check (Step 6) all agree on \(\nabla J(1,1) \approx [-3, 7]\) — this triple-agreement is the standard way to trust a hand-derived gradient before using it in any real training loop.
Step 7: Use the gradient to take one gradient-descent step¶
As a preview of Level 2, use \(-\nabla J\) (Module 8's "steepest decrease" direction) to take one small step from \((1,1)\) with learning rate \(\alpha=0.1\):
alpha = 0.1
w_new, b_new = np.array([w0, b0]) - alpha * grad_exact(w0, b0)
print("new (w,b):", (w_new, b_new))
print("J before :", J(w0, b0), " J after:", J(w_new, b_new))
Expected output (cost should decrease after the step, confirming we moved in a helpful direction; by hand: \(J(1.3,0.3)=(1.3-3)^2+(0.3+2)^2+1.3(0.3) = 2.89+5.29+0.39 = 8.57\)):
\(J\) dropped from \(14.0\) to \(8.57\) after a single step in the \(-\nabla J\) direction — exactly the mechanism that trains every ML model built on gradient descent, from linear regression to deep neural networks.
How It Actually Works¶
This capstone combines two computational mechanisms from earlier modules, and it's worth being precise about which one does the real work in practice. The finite-difference check you ran in Step 6 evaluates the cost function at \(\theta \pm h\) and divides by \(2h\) — useful as a sanity check specifically because it is a completely independent computational path from the analytic gradient formula, so agreement between the two (to about 6-7 significant digits for float64 with a well-chosen \(h\)) is strong evidence the analytic formula and its code are correct. It is never used as the actual training mechanism, because its cost scales as \(O(n)\) evaluations of \(J\) for \(n\) parameters, and each evaluation carries the truncation/round-off trade-off from Module 06.
The analytic gradient used for the actual descent step, by contrast, is the pattern every real training loop uses: compute it once (here, by hand-derived formula; in Level 3, via reverse-mode autodiff over a computational graph), then apply \(\theta \leftarrow \theta - \alpha\nabla J(\theta)\) — a single vectorized floating-point update per parameter, applied element-wise using the exact IEEE-754 arithmetic and the same BLAS-backed vector operations from Module 02-04. Every subsequent module's "numeric verification" section follows this same two-track pattern: a fast analytic/autodiff path for training, and a slow, independent finite-difference path used only to catch bugs before trusting the fast path.
Exercise (final Level 1 check)¶
Given \(K(w,b) = (2w+1)^2 + 3b^2 - wb\):
- Derive \(\frac{\partial K}{\partial w}\) and \(\frac{\partial K}{\partial b}\) by hand, showing the chain rule step for the squared term.
- Evaluate \(\nabla K\) by hand at \((w,b) = (0, 1)\).
- Implement
K,grad_exact, andgrad_numericin NumPy (following this module's pattern) and confirm all three -- hand, exact formula, and finite difference -- agree at \((0,1)\). - Take one gradient-descent step from \((0,1)\) with \(\alpha=0.1\) and confirm \(K\) decreases.