Machine Learning — Handbook

A complete, example-driven course in machine learning: from what learning even means, through data and features, the core algorithms and how they work mathematically, clustering and ensembles, evaluation and validation, practical strategy, and on to neural networks and deep learning — each idea explained plainly, with the key math, an intuition, and runnable code.

Nearly all ML starts as a table. The columns you learn from are features (X); the column you're trying to predict is the label (y). Each row is one example. Toggle what we're predicting to see X and y change.

The shapes

Features→X Label / target→y

Code

python

import numpy as np
# Features X (inputs) and labels y (the answer to predict)
X = np.array([[1500, 3], [2100, 4], [900, 2.]])   # rows=samples, cols=features
y = np.array([320_000, 410_000, 180_000])          # target (house price)
print("X shape:", X.shape, " y shape:", y.shape)

Train / test split

Why we hold data back

The idea

The golden rule of ML: never judge a model on data it learned from. You split your data — train on most, then test on the held-out part it has never seen. That's the only honest measure of how it'll do in the real world.

Why?

A model can memorize its training data and look perfect — then fail on anything new. The test set catches that lie.

Code

python

import numpy as np
from sklearn.model_selection import train_test_split
X = np.random.randn(100, 3); y = np.random.randint(0, 2, 100)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
print(len(X_tr), "train /", len(X_te), "test")   # never train on the test set

Train / val / test split

The third set

The idea

Train/test isn't enough once you start tuning. Every time you tweak a hyperparameter and check the same set, you quietly fit to it. So we use three: train to fit parameters, validation to tune and choose, test touched exactly once for an honest final number. Slide how many configs you try and watch the cheating gap grow.

The three roles

Train: fit the model's parameters. Validation: tune hyperparameters & pick the best model — looked at many times. Test: the vault: one final, untouched estimate.

Reported vs reality

true skill of chosen model82% if you tuned on TEST87%

Why a third set

Pick the best of many models by their test score and the winner is partly lucky — the more you try, the more its test score overstates reality (the multiple-comparisons trap). The validation set absorbs that optimism so the test set stays an honest proxy for new data.

Code

python

import numpy as np
from sklearn.model_selection import train_test_split
X = np.random.randn(100, 3); y = np.random.randint(0,2,100)
X_tmp, X_te, y_tmp, y_te = train_test_split(X, y, test_size=0.15, random_state=0)
X_tr, X_val, y_tr, y_val = train_test_split(X_tmp, y_tmp, test_size=0.18, random_state=0)
print(len(X_tr), len(X_val), len(X_te))   # train, tune-on-val, report-on-test

The ML workflow

The loop

The idea

Building a model is a cycle, not a straight line. You rarely get it right first pass — you evaluate, learn, and go around again. This loop is the backbone of every project this week. Step through it.

1 · Get & explore data

Load it, look at it, clean it. Understand what you have before modeling. Most real time goes here.

Stats & measurement

Precision vs accuracy

The dartboard

The idea

Two different ideas people mix up. Accuracy = how close you land to the truth (the bullseye). Precision = how consistent you are — hit the same spot every time and you're precise, even if that spot is wrong. Drag the two sliders and fire darts to feel the difference.

Accurate & precise

Code

python

import numpy as np
data = np.arange(100)
simple = np.random.choice(data, 10, replace=False)         # simple random
strata = [data[:50], data[50:]]                             # stratified: sample each
strat = np.r_[np.random.choice(strata[0],5), np.random.choice(strata[1],5)]
print(sorted(strat))

Bootstrapping

Resample for confidence

The idea

How sure are you about an estimate from one sample? The bootstrap answers without any formula: resample your data with replacement hundreds of times, recompute the statistic each time, and the spread of those answers is your uncertainty. Run it and watch a confidence interval emerge.

The math · The idea

Treat your sample as a stand-in for the population. Draw n points from it with replacement (duplicates allowed), compute the mean, repeat: x̄*₁, x̄*₂, … , x̄*B Their distribution ≈ the sampling distribution of the mean.

Result

bootstrap samples0 95% CI—

Why it's powerful

It needs no assumption that the data is normal and works for almost any statistic — medians, correlations, model scores. It's also the engine inside bagging and random forests.

Code

python

import numpy as np
# Bootstrap: resample with replacement to estimate uncertainty
data = np.random.randn(50) + 5
means = [np.random.choice(data, len(data), replace=True).mean() for _ in range(10000)]
lo, hi = np.percentile(means, [2.5, 97.5])
print(f"95% CI for the mean: [{lo:.2f}, {hi:.2f}]")

Code

python

import numpy as np
# Engineer informative features from raw columns
length = np.array([2., 3., 5.]); width = np.array([1., 1., 2.])
area = length * width                       # interaction
ratio = length / width                       # ratio feature
print(np.c_[area, ratio])

Fit on train, apply to test

Learned parameters

The idea

Preprocessing has parameters too — the mean and std a scaler uses, the median an imputer fills with. These must be learned from training data only, then applied unchanged to test and future data. Compute them on everything and you've leaked the test set. Toggle the two ways and watch what the test point sees.

The math · The standardization

z = x − μσ μ used0.395 σ used0.061

Why train-only?

At deployment you score one new point at a time — you simply don't have the test distribution to average over. The scaler must carry the training-time μ and σ and apply them to whatever arrives, even if the new point lands oddly. Fitting on all data lets the test set influence training → optimistic, dishonest scores.

Code

python

import numpy as np
Xtr = np.random.randn(80,3); Xte = np.random.randn(20,3)
mu, sd = Xtr.mean(0), Xtr.std(0)            # fit ONLY on train
Xtr_s = (Xtr-mu)/sd; Xte_s = (Xte-mu)/sd    # reuse train stats on test
print("no leakage: test uses train's mu/sd")

Target / mean encoding

Code

python

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
X = np.random.randn(200, 10); y = (X[:,0]+X[:,1] > 0).astype(int)
sel = SelectKBest(f_classif, k=3).fit(X, y)
print("kept features:", np.where(sel.get_support())[0])

Text features (TF-IDF)

Words → numbers

The idea

Classical models can't read text, so we vectorize it. Bag-of-words just counts terms; TF-IDF is smarter — it boosts words that are frequent in this document but rare across the corpus, and crushes ubiquitous words like “the.” Toggle the two and watch which terms light up.

The math · The math

tfidf(t,d) = tf(t,d) · log Ndf(t) A term in every document has df = N, so log(N/df) = 0 — it's zeroed out. Rare, document-specific terms get the highest weight.

Why it works

“the” appears everywhere and distinguishes nothing; TF-IDF silences it automatically without a stop-word list. The result is a sparse vector per document where the big numbers are the words that actually characterize it — a strong, cheap baseline before embeddings.

Code

python

from sklearn.feature_extraction.text import TfidfVectorizer
docs = ["the cat sat", "the dog ran", "cat and dog"]
tfidf = TfidfVectorizer().fit_transform(docs)
print(tfidf.shape)   # rows=docs, cols=terms; rare-but-present terms weigh more

Multicollinearity & VIF

Correlated predictors

The idea

When two predictors carry nearly the same information, a linear model can't tell which deserves the credit — so its coefficients become wild and unstable, even though predictions stay fine. The variance inflation factor measures it. Crank up the correlation and watch the coefficients swing.

The math · VIF

VIFj = 11 − Rj² VIF1.2 coefficient stabilitystable

What to do

Rule of thumb: VIF > 5–10 signals trouble. Predictions are usually unaffected, but you can't trust the individual coefficients or their p-values. Fix by dropping one of the pair, combining them, or using regularization (ridge) which tolerates correlation. Also check residual plots for non-random patterns.

Code

python

import numpy as np
# VIF flags features that are linear combinations of others
X = np.random.randn(100, 3); X = np.c_[X, X[:,0]+X[:,1]]   # col 4 is redundant
def vif(X, i):
    others = np.delete(X, i, 1); coef,_,_,_ = np.linalg.lstsq(others, X[:,i], rcond=None)
    r2 = 1 - ((X[:,i]-others@coef)**2).sum()/((X[:,i]-X[:,i].mean())**2).sum()
    return 1/(1-r2+1e-9)
print("VIF of redundant col:", round(vif(X,3), 1))   # very high

How models learn

Gradient descent · interactive

The idea

A model “learns” by minimizing error. Picture the error as a hill — the model takes small downhill steps to reach the bottom (lowest error). The step size is the learning rate. Too small = slow; too big = it overshoots. Run it and feel the difference.

Code

python

import numpy as np
# Learning = follow the gradient downhill on a loss
loss = lambda w: (w-3)**2; grad = lambda w: 2*(w-3)
w, lr = 0.0, 0.1
for _ in range(50): w -= lr*grad(w)
print("learned w:", round(w, 3))   # -> 3

Loss surface & LR

Descend the valley

The idea

Gradient descent in 2-D. The contours are a loss “valley”; the model walks downhill toward the minimum. On a stretched valley a big learning rate bounces off the walls instead of flowing down the floor. Click to drop a starting point, pick a rate, and run.

Code

python

import numpy as np
# The learning rate sets the step size on the loss surface
grad = lambda w: 2*w
for lr in (0.05, 0.5, 1.05):
    w = 5.0
    for _ in range(30): w -= lr*grad(w)
    print(f"lr={lr}: w={w:.2e}")   # 1.05 diverges

Overfitting & underfitting

Drag the complexity

The idea

The central struggle of ML. Too simple a model underfits — it misses the real pattern. Too complex and it overfits — it memorizes the noise and fails on new data. Drag the model complexity and watch the fit (and the test error) change.

Underfitting

A straight line can’t capture a curve. High error on both training and test data — the model is too simple.

Code

python

import numpy as np
from numpy.polynomial import polynomial as P
x = np.linspace(0,1,10); y = np.sin(6*x)+0.1*np.random.randn(10)
for deg in (1, 3, 9):
    c = np.polyfit(x, y, deg); tr = np.mean((np.polyval(c,x)-y)**2)
    print(f"degree {deg}: train MSE {tr:.4f}")   # deg 9 fits noise -> overfits

Bias–variance tradeoff

Find the sweet spot

The idea

The same story, drawn as curves. As complexity rises, training error keeps falling — but test error follows a U: it drops, bottoms out, then climbs as the model starts memorizing. The bottom of that U is the model you want. Move the complexity to find it.

Balanced

Near the bottom of the U — low test error. This is the model that generalizes best to new data.

The terms

High bias=underfit (too simple) High variance=overfit (too complex)

Code

python

import numpy as np
# Bias^2 + variance: simple models biased, complex models high-variance
def fit_predict(deg, x0):
    preds = []
    for _ in range(100):
        x=np.sort(np.random.rand(15)); y=np.sin(6*x)+0.2*np.random.randn(15)
        preds.append(np.polyval(np.polyfit(x,y,deg), x0))
    return np.var(preds)
print("variance deg1:", round(fit_predict(1,0.5),3), " deg9:", round(fit_predict(9,0.5),3))

Deep dives (the math)

Underfitting — deep dive

High bias, explained

The idea

An underfit model is too simple to capture the real pattern. It's wrong on the data it trained on and on new data — and no amount of extra data will save it. Press “add new data” and watch the new points miss too.

What you're seeing

Train errorhigh (0.013) Test error (new data)high A straight line can't bend to a curved truth. Both errors are high and similar — the tell-tale signature of underfitting.

The math · The math

Expected test error splits into three parts: Err = Bias² + Var + σ² Underfitting means a large Bias²: the model's average prediction is systematically off because its hypothesis class (here, straight lines) simply cannot represent the true function.

When new data arrives

The new points are predicted poorly — but notice the model was already wrong on the training points. The error is systematic, not random. So more data won't help; the line still can't bend.

How to fix it

1Add capacity: a higher-degree model, more layers, a more flexible algorithm. 2Add or engineer features that expose the real signal. 3Reduce regularization (it's forcing the model too simple).

Code

python

import numpy as np
# Underfitting: model too simple -> high train AND test error
x = np.linspace(0,1,30); y = np.sin(6*x)
c = np.polyfit(x, y, 1)                       # a line can't follow a sine
print("train MSE (linear):", round(np.mean((np.polyval(c,x)-y)**2), 3))

Overfitting — deep dive

Code

python

import numpy as np
sig = lambda z: 1/(1+np.exp(-z))
X = np.c_[np.ones(200), np.random.randn(200,2)]; y = (X[:,1]+X[:,2]>0).astype(float)
w = np.zeros(3)
for _ in range(400): w -= 0.1 * X.T @ (sig(X@w)-y) / len(y)   # grad of log-loss
print(np.round(w, 2))

SVM — the margin objective

Maximize the margin

The idea

An SVM picks the boundary that sits as far as possible from both classes. That “as far as possible” is a precise optimization: maximize the margin width, which equals minimizing ‖w‖². Tilt the boundary and watch the margin shrink away from its maximum.

The math · The objective

Decision function and its margin: f(x) = w·x + b margin = 2‖w‖ Widest margin ⇔ smallest weights, subject to every point staying on its side: min 12‖w‖² s.t. yᵢ(w·xᵢ+b) ≥ 1

This boundary

Margin width— vs maximum—

Why the widest margin?

A fat margin is a buffer: new points near the boundary are more likely to still fall on the correct side. Only the closest points (the support vectors, where the constraint = 1) determine it — the rest are irrelevant.

Code

python

import numpy as np
# Max-margin = minimize ||w|| subject to y(w.x+b) >= 1; hinge for soft margin
hinge = lambda y, s: np.maximum(0, 1 - y*s)
w, b = np.array([1.,-1.]), 0.0
X = np.array([[2,0.],[0,2.],[-1,-1.]]); y = np.array([1,1,-1.])
print("hinge losses:", hinge(y, X@w + b))

Information gain — the math

How a split is chosen

The idea

A decision tree picks the split that most reduces disorder. It measures disorder with entropy, then scores a split by its information gain — how much entropy drops from parent to weighted children. Drag the split line and watch the gain rise and fall; the tree would pick its peak.

The math · The math

Entropy of a node (fraction p of class A): H = −p log₂p − (1−p) log₂(1−p) Information gain of a split: IG = Hparent − Σ |child||parent| Hchild

This split

H(parent)1.000 H(left) · H(right)0.41 · 0.41 Information gain0.586

What the tree does

It tries every feature and threshold, computes IG for each, and greedily takes the highest — then repeats inside each child. (Gini impurity is a near-identical alternative that's cheaper to compute.)

Code

python

import numpy as np
def entropy(y):
    p = np.bincount(y, minlength=2)/len(y); p = p[p>0]
    return -(p*np.log2(p)).sum()
parent = np.array([0,0,1,1,1,1])
left, right = parent[:2], parent[2:]
ig = entropy(parent) - (len(left)/6*entropy(left) + len(right)/6*entropy(right))
print("information gain:", round(ig, 3))

k-Means — the objective

Code

python

import numpy as np
# A cost function scores predictions; training minimizes it
y, yhat = np.array([1.,0.,1.]), np.array([0.8,0.3,0.6])
mse = np.mean((y-yhat)**2)
ce  = -np.mean(y*np.log(yhat)+(1-y)*np.log(1-yhat))
print(f"MSE={mse:.3f}  cross-entropy={ce:.3f}")

Decision tree — the algorithm

Recursive greedy splits

The idea

A tree is built top-down: at each node, try every possible split, keep the one that makes the children purest, then recurse. Grow the depth and watch the space carve into boxes on the left while the actual tree — split conditions and Gini scores — assembles on the right.

The math · The split criterion

Impurity of a node (Gini): G = 1 − Σk pₖ² Pick the split (feature j, threshold t) that minimizes the weighted child impurity: minj,t |L|nGL + |R|nGR

How it runs

1At a node, scan every feature & threshold; score each by child impurity. 2Keep the best split; send points left/right. 3Recurse on each child until pure, max depth, or too few points. 4Predict: walk a point down to a leaf → majority class.

The catch

It's greedy — each split is locally best, not globally optimal. And left unbounded it grows until every leaf is pure (G=0): perfect on training, overfit on new data. Depth, min-samples, and pruning hold it back.

Code

python

import numpy as np
# Greedy tree: pick the split that most reduces impurity, recurse
def gini(y): p=np.bincount(y,minlength=2)/len(y); return 1-(p**2).sum()
x = np.array([1,2,3,4,5,6.]); y = np.array([0,0,0,1,1,1])
gains = {t: gini(y) - (np.mean(x<=t)*gini(y[x<=t])+np.mean(x>t)*gini(y[x>t]))
         for t in (x[:-1]+x[1:])/2}
print("best split:", max(gains, key=gains.get))

Linear regression — least squares

The normal equations

The idea

“Least squares” is literal: the best line minimizes the total area of the squares built on each residual. Adjust the slope and intercept and watch the squares — and the cost — grow and shrink. There's a one-shot formula that lands exactly at the minimum.

The math · The math

ŷ = β₀ + β₁x Cost = mean squared error: MSE = 1n Σ(yᵢ − ŷᵢ)² Setting the gradient to zero gives a closed form — the normal equations: β = (XᵀX)⁻¹ Xᵀy

Current fit

Your MSE0.0034 Best possible (OLS)0.0015

Why squares?

Squaring makes the cost smooth and convex (one global minimum), and punishes big errors more than small ones. That convexity is why a single formula — no iteration needed — finds the optimum.

Code

python

import numpy as np
# Least squares: minimize ||y - X beta||^2 -> normal equations
X = np.c_[np.ones(50), np.random.randn(50)]; y = 2 + 3*X[:,1] + 0.2*np.random.randn(50)
beta = np.linalg.solve(X.T@X, X.T@y)
print("intercept, slope:", np.round(beta, 2))

PCA — eigenvectors

Covariance & variance

The idea

PCA isn't a black box: it's the eigen-decomposition of the data's covariance matrix. The eigenvectors point along the directions of greatest spread; the eigenvalues say how much variance lives along each. Click to add points and watch the covariance, axes, and variance split update live.

The math · The math

Center the data, then form the covariance matrix: Σ = 1n XᵀX = [ … ] Its eigenvectors are the principal components: Σvᵢ = λᵢvᵢ Each λᵢ is the variance captured by component vᵢ.

Eigenvalues → variance

λ₁ (PC1)— λ₂ (PC2)— variance on PC1—

Dimensionality reduction

Keep the top-k eigenvectors (largest λ) and project onto them: you compress the data while preserving the most variance. Drop PC2 here and you've gone 2-D → 1-D with minimal loss.

Code

python

import numpy as np
# PCA: top eigenvectors of the covariance = directions of max variance
X = np.random.randn(300,2) @ [[3,1],[0,1.]]; Xc = X - X.mean(0)
vals, vecs = np.linalg.eigh(np.cov(Xc.T))
order = vals.argsort()[::-1]
print("variance explained:", np.round(vals[order]/vals.sum(), 3))

Naive Bayes — the posterior

Prior × likelihood

The idea

Naive Bayes applies Bayes' theorem directly, with one bold shortcut: it assumes every feature is independent given the class. That “naive” assumption turns a hard joint probability into a simple product — fast, and shockingly effective. Toggle words and watch the posterior assemble term by term.

The math · The math

P(c|x) = P(x|c) P(c)P(x) The naive assumption — features independent given the class — factorizes the likelihood: P(x|c) = ∏j P(xⱼ|c) Predict the class maximizing prior × product of likelihoods (done in log-space to avoid underflow).

Posterior

P(spam | words)40.0%

Why “naive” still works

Words obviously aren't independent (“free” and “winner” co-occur), so the probabilities are miscalibrated — but the ranking of classes is usually still right, which is all you need to classify. Cheap, scalable, a great baseline.

Code

python

import numpy as np
# Naive Bayes posterior ~ prior * product of per-feature likelihoods
prior = np.array([0.5, 0.5])
# likelihoods P(x_i | class) assumed independent -> multiply (sum in log space)
loglik = np.array([[-1.2,-0.8], [-0.5,-2.0]]).sum(0)   # over features, per class
post = np.exp(np.log(prior)+loglik); post/=post.sum()
print("posterior:", np.round(post, 3))

Confusion matrix — the metrics

Precision, recall, F1

The idea

Every classification metric is built from four counts. Drag the threshold and watch the confusion matrix shift — and every metric recompute from its formula. Precision and recall pull against each other; F1 balances them.

The math · The formulas

Accuracy: TP+TNall Precision: TPTP+FP — of those flagged, how many were right Recall: TPTP+FN — of the real positives, how many we caught F1: 2·P·RP+R — harmonic mean

Live metrics

Accuracy82% Precision81% Recall83% F10.82

The tension

Lower the threshold → catch more positives (recall ↑) but more false alarms (precision ↓). Raise it → the reverse. Which you favor depends on the cost of a miss vs a false alarm.

Code

python

import numpy as np
tp, fp, fn, tn = 40, 10, 5, 45
precision = tp/(tp+fp); recall = tp/(tp+fn)
f1 = 2*precision*recall/(precision+recall)
print(f"P={precision:.2f} R={recall:.2f} F1={f1:.2f}")

Softmax & cross-entropy

Scores → probabilities

The idea

For multi-class problems, a network outputs raw scores (logits) — softmax squashes them into a probability distribution that sums to 1, and cross-entropy scores how far that is from the true class. Drag the logits and watch probability mass flow between classes.

The math · The math

pᵢ = ezᵢΣj ezⱼ Loss = categorical cross-entropy (only the true class's term survives): L = −Σi yᵢ log pᵢ = −log ptrue

Output

loss0.266

Why softmax?

Exponentiating makes everything positive; dividing normalizes to a valid distribution. It's the multi-class generalization of the sigmoid — and its gradient with cross-entropy simplifies beautifully to just (p − y).

Code

python

import numpy as np
def softmax(z): e=np.exp(z-z.max()); return e/e.sum()
z = np.array([2.0, 1.0, 0.1])
p = softmax(z)
ce = -np.log(p[0])                       # cross-entropy if true class is 0
print(np.round(p,3), "CE:", round(ce,3))

k-NN & the curse of dimensions

When distance breaks

The idea

k-NN trusts that “nearby” means “similar.” That holds in 2-D — but in high dimensions, every point becomes almost equidistant from every other, so “nearest neighbor” loses meaning. Slide the dimension count and watch the nearest and farthest distances collapse together.

The math · Distance metrics

Euclidean: d = √Σ(xᵢ−qᵢ)² Manhattan: d = Σ|xᵢ−qᵢ|

Distance concentration

nearest0.008 farthest0.670 ratio near/far0.012

The curse

As d grows, that ratio races toward 1 — the nearest and farthest points are nearly the same distance away. “Closest” becomes meaningless, so k-NN (and any distance method) degrades. The fix: reduce dimensions first (PCA) or pick few, good features.

Code

python

import numpy as np
# k-NN: predict by majority of nearest neighbors; curse of dimensionality
X = np.random.randn(200, 2); y = (X[:,0]>0).astype(int)
q = np.array([0.5, 0.0])
d = np.linalg.norm(X - q, axis=1); nn = d.argsort()[:5]
print("predicted class:", np.bincount(y[nn]).argmax())

ROC / AUC — the integral

Code

python

import numpy as np
# MLE: parameters that maximize likelihood = minimize negative log-likelihood
data = np.random.normal(4.0, 2.0, 500)
mu_hat, sigma_hat = data.mean(), data.std()   # closed-form MLE for a Gaussian
print(round(mu_hat,2), round(sigma_hat,2))

Convolution arithmetic

Stride, padding, size

The idea

A CNN layer's output size is fully determined by four numbers: input size, kernel size, stride, and padding. Get them wrong and your tensor shapes won't line up. Tune the dials and watch the kernel step across the (padded) input while the output grid resizes to match the formula.

The math · The output-size formula

out = ⌊ N + 2P − KS ⌋ + 1 = ⌊(7+0−3) / 1⌋ + 1 = 5

Rules of thumb

“Same” padding (P=⌊K/2⌋, S=1) keeps the size unchanged. Stride > 1 downsamples. Each extra filter adds an output channel — depth — so a layer's output is out × out × (#filters).

Code

python

# Conv output size: (W - K + 2P)/S + 1
def out(W,K,P,S): return (W-K+2*P)//S + 1
print(out(28,3,1,1), out(28,3,0,2))   # 28 (same), 13 (downsampled)

Core algorithms

Linear regression playground

Click to add data

The idea

The simplest model: fit a straight line that minimizes the squared error to the points. Click anywhere on the plot to add a data point and watch the best-fit line and the error update instantly. The grey sticks are the residuals it's trying to shrink.

Live fit

Equationy = 0.55x + 0.25 MSE (error)0.0007 Points12

Code

python

import numpy as np
X = np.c_[np.ones(60), np.random.randn(60)]; y = 1 + 2*X[:,1] + 0.3*np.random.randn(60)
beta = np.linalg.lstsq(X, y, rcond=None)[0]
print("intercept, slope:", np.round(beta, 2))

Polynomial regression

Fit the curve

The idea

A straight line can't follow a curve — but a polynomial can. By adding powers of x (x², x³, …) as features, the same linear machinery fits curved trends. The degree is a flexibility dial: too low underfits, too high overfits with wild wiggles. Slide it and watch.

The math · The model

ŷ = w₀ + w₁x + w₂x² + … + wdxd Still linear in the weights — so it's solved exactly like linear regression, just on expanded features. train error0.0016

The flexibility trap

Degree 1 is a line (underfits the curve); a moderate degree captures the real shape; a very high degree threads every point — including the noise — and swings violently between them. That high-degree wiggle is overfitting made visible.

Code

python

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
x = np.linspace(-2,2,40).reshape(-1,1); y = (x.ravel()**2 + 0.2*np.random.randn(40))
Xp = PolynomialFeatures(2).fit_transform(x)
print("R^2:", round(LinearRegression().fit(Xp, y).score(Xp, y), 3))

Elastic Net

Code

python

import numpy as np
# Perceptron: update weights only on misclassified points
X = np.c_[np.random.randn(100,2), np.ones(100)]; y = np.sign(X[:,0]+X[:,1])
w = np.zeros(3)
for _ in range(20):
    for xi, yi in zip(X, y):
        if yi*(w@xi) <= 0: w += yi*xi      # the perceptron rule
print("train acc:", np.mean(np.sign(X@w)==y))

k-Nearest Neighbors

Move the point · change k

The idea

The most intuitive classifier: to label a new point, look at its k closest neighbors and take a majority vote. Move your mouse over the plot to place the query point — its nearest neighbors light up and it gets colored by their vote. Change k and watch the decision flip.

Hover the plot →

Move over the plot to drop a query point. Its k nearest neighbors vote on its class.

Code

python

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
X = np.random.randn(200,2); y = (X[:,0]>0).astype(int)
print("acc:", KNeighborsClassifier(5).fit(X,y).score(X,y))   # lazy: no training

k-NN regression

Code

python

import numpy as np
from sklearn.svm import SVR
X = np.sort(np.random.rand(60,1),0); y = np.sin(6*X).ravel()
m = SVR(kernel='rbf', C=10, epsilon=0.05).fit(X, y)   # epsilon-insensitive tube
print("R^2:", round(m.score(X, y), 3))

Naive Bayes

Toggle the evidence

The idea

A fast probabilistic classifier built on Bayes' theorem. It starts with a prior belief, then multiplies in the evidence from each feature (“naively” assuming they're independent) to land on a posterior probability. Toggle which words appear in an email and watch P(spam) update.

P(spam | words)

Probability40.0% Verdict✅ Not spam

Code

python

import numpy as np
from sklearn.naive_bayes import GaussianNB
X = np.r_[np.random.randn(100,2), np.random.randn(100,2)+3]; y = np.r_[np.zeros(100),np.ones(100)]
print("acc:", GaussianNB().fit(X, y).score(X, y))

One-vs-rest multiclass

Binary → many classes

The idea

A logistic regression only separates two classes — so how do you handle three? Train one binary classifier per class (“this class vs everything else”), then for any point pick the class whose classifier is most confident. Toggle between the individual binary views and the combined decision.

The math · The math

Train K classifiers, each giving P(class k | x). Predict the winner: ŷ = argmaxk hk(x)

vs softmax

One-vs-rest wraps any binary learner into a multiclass one — simple and general, but the K classifiers are trained independently and their scores aren't jointly normalized. Softmax regression instead models all classes together in one shot (probabilities that sum to 1).

Code

python

import numpy as np
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression
X = np.random.randn(300,2); y = np.random.randint(0,3,300)   # 3 classes
clf = OneVsRestClassifier(LogisticRegression()).fit(X, y)    # one binary clf per class
print("classes:", clf.classes_)

Distance & similarity

Distance metrics

Code

python

import numpy as np
a = np.array([1,0,1,1]); b = np.array([1,1,1,0])
hamming = (a != b).mean()
jaccard = (a & b).sum() / (a | b).sum()
print("hamming:", hamming, " jaccard:", round(jaccard,3))

Code

python

import numpy as np
from sklearn.mixture import GaussianMixture
X = np.vstack([np.random.randn(150,2), np.random.randn(150,2)+4])
gmm = GaussianMixture(2).fit(X)            # soft clusters via EM
print("soft memberships:\n", np.round(gmm.predict_proba(X[:2]), 2))

Choosing k

Elbow & silhouette

The idea

k-Means needs you to pick the number of clusters — but how many is right? Two trusted guides: the elbow in the within-cluster error (it drops fast, then flattens once you've captured the real groups) and the silhouette score (how well-separated the clusters are), which peaks at the natural k. Slide k and read both.

Readings

inertia (within-cluster SSE)0.62 silhouette score0.46

Reading the curves

Inertia always falls as k rises (more clusters fit tighter) — so you don't want the minimum, you want the elbow where the gains flatten. Silhouette gives a cleaner signal: it's highest at the k that best balances tight clusters with wide gaps. Here both point to the same natural answer.

Code

python

import numpy as np
from sklearn.cluster import KMeans
X = np.vstack([np.random.randn(100,2)+c for c in ([0,0],[5,5],[0,5])])
inertias = [KMeans(k, n_init=10).fit(X).inertia_ for k in range(1,7)]
print("inertia by k:", np.round(inertias, 0))   # elbow ~ true k

k-medoids (PAM)

Real points as centers

The idea

k-Means centers are averages — fictional points that get yanked around by outliers, and only make sense for Euclidean distance. k-medoids uses actual data points as centers (medoids) and minimizes total distance under any metric. That makes it robust to outliers and usable with the cosine, Hamming, or Mahalanobis distances we just met. Toggle the outlier.

The math · The objective

min Σi d(xᵢ, medoid(xᵢ)) The medoid is the cluster member with the smallest total distance to the others — a real, representative point.

Why it's useful

Add an outlier and a k-means centroid drifts toward it; the medoid stays put on a genuinely central point. And because it only needs a distance function — never coordinates or averages — k-medoids clusters things you can't average, like text or categorical records.

Code

python

import numpy as np
# k-medoids: cluster centers are actual data points (robust to outliers)
X = np.vstack([np.random.randn(30,2), np.random.randn(30,2)+5])
medoids = X[[0, 35]]
for _ in range(10):
    lab = ((X[:,None]-medoids[None])**2).sum(-1).argmin(1)
    for k in range(2):
        pts = X[lab==k]; costs = ((pts[:,None]-pts[None])**2).sum(-1).sum(1)
        medoids[k] = pts[costs.argmin()]
print("medoids:\n", np.round(medoids, 1))

Mean-shift

Climb to the peaks

The idea

Unlike k-means, mean-shift doesn't need you to pick the number of clusters. Treat the points as a density landscape; each point repeatedly hops to the average of its neighbors — climbing uphill toward the nearest peak. Points that arrive at the same peak form a cluster. The number of clusters simply emerges from the bandwidth. Step the climb.

The math · The shift

Move each point to the mean of the neighbors within the bandwidth, and repeat: x ← mean{ xⱼ : ‖xⱼ−x‖ < h } clusters found28

Bandwidth is everything

A small bandwidth finds many fine peaks; a large one merges them into a few. So instead of choosing k directly, you choose a scale — often more natural. Mean-shift handles arbitrary cluster shapes, but it's slower and bandwidth-sensitive. It's also used for image segmentation and tracking.

Code

python

import numpy as np
from sklearn.cluster import MeanShift
X = np.vstack([np.random.randn(80,2), np.random.randn(80,2)+6])
ms = MeanShift(bandwidth=2).fit(X)         # finds modes; k chosen automatically
print("found clusters:", len(set(ms.labels_)))

Spectral clustering

Cluster by graph

The idea

k-Means assumes clusters are round blobs — so it fails on shapes like two interleaving moons, slicing straight through them. Spectral clustering instead builds a similarity graph (connect nearby points), then cuts the graph where connections are sparse. It follows the data's connectivity, capturing curves and rings that distance-to-center methods can't.

The math · The recipe

1Build a graph: connect each point to its near neighbors. 2Form the graph Laplacian and take its smallest eigenvectors. 3Cluster points in that new embedding (k-means there).

Why it wins here

Two points on opposite ends of the same moon are far in straight-line distance but connected through a chain of neighbors — so the graph keeps them together. k-means only sees distance-to-center, so it carves the moons in half. Spectral shines on non-convex, manifold-shaped data.

Code

python

import numpy as np
from sklearn.cluster import SpectralClustering
X = np.vstack([np.random.randn(60,2), np.random.randn(60,2)+5])
labels = SpectralClustering(2, affinity='nearest_neighbors').fit_predict(X)
print("cluster sizes:", np.bincount(labels))   # uses graph Laplacian eigenvectors

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
import numpy as np
X = np.random.randn(300,4); y = (X[:,0]>0).astype(int)
stack = StackingClassifier(
  estimators=[('tree', DecisionTreeClassifier()), ('lr', LogisticRegression())],
  final_estimator=LogisticRegression()).fit(X, y)        # meta-model combines bases
print("acc:", round(stack.score(X, y), 3))

Evaluation

Threshold & confusion matrix

Drag the threshold

The idea

A classifier outputs a score; you pick a threshold to turn it into a yes/no. Where you put that line is a real decision — it trades false alarms against misses. Drag the threshold and watch the confusion matrix and every metric move. There is no free lunch.

Metrics

Accuracy85% Precision83% Recall88% F10.85

Code

python

import numpy as np
from sklearn.metrics import confusion_matrix
y, scores = np.r_[np.zeros(50),np.ones(50)], np.r_[np.random.rand(50), np.random.rand(50)+0.4]
for thr in (0.3, 0.7):
    pred = (scores > thr).astype(int)
    print(f"thr={thr}:", confusion_matrix(y, pred).ravel())   # tn,fp,fn,tp

ROC curve & AUC

Why it matters

When a probability feeds a decision — a medical risk, a loan threshold, expected value — it must mean what it says. Methods like Platt scaling and isotonic regression remap the scores onto the diagonal without changing their ranking (so AUC is unchanged).

Code

python

import numpy as np
p = np.random.rand(5000); y = (np.random.rand(5000) < p).astype(int)
for lo in np.linspace(0,0.8,5):
    m = (p>=lo)&(p<lo+0.2)
    if m.sum(): print(f"pred~{lo+0.1:.1f} actual={y[m].mean():.2f}")

Regression metrics

MAE, RMSE, R²

The idea

Classification has accuracy; regression has a family of error measures that disagree on purpose. MAE treats every miss equally; MSE/RMSE punish big misses quadratically; R² is the fraction of variance explained; MAPE is a percentage. Drag the outlier and watch which metrics panic.

The math · The metrics

MAE0.024 MSE0.001 RMSE0.031 R²0.961 MAPE5.0%

Which to use

MAE when all errors matter equally and outliers are noise. RMSE when big errors are disproportionately bad (it's in the target's units). R² for a unitless "how much better than the mean." MAPE for relative error — but it explodes near zero and is undefined at zero.

Code

python

import numpy as np
y, yhat = np.array([3.,5.,2.,8.]), np.array([2.5,5.5,2.,7.])
mae = np.abs(y-yhat).mean(); rmse = np.sqrt(((y-yhat)**2).mean())
r2 = 1 - ((y-yhat)**2).sum()/((y-y.mean())**2).sum()
print(f"MAE={mae:.2f} RMSE={rmse:.2f} R2={r2:.3f}")

Precision-Recall curve

Better on imbalance

The idea

ROC can look flatteringly good when the positive class is rare, because true negatives swamp the false-positive rate. The precision-recall curve ignores true negatives entirely, so it tells the honest story on imbalanced data. Sweep the threshold and shrink the positive class to watch the gap.

The math · Axes

precision = TPTP+FP recall = TPTP+FN Average precision (area)0.690 baseline (random)0.30

The tell

A random classifier's PR baseline is just the positive rate — so at 5% positives, "good-looking" AUC of 0.9 can hide a PR curve that barely beats 0.05. Always pair PR with ROC when classes are skewed.

Code

python

import numpy as np
from sklearn.metrics import average_precision_score
# PR curve is better than ROC when positives are rare
y = np.r_[np.zeros(950), np.ones(50)]; s = np.r_[np.random.rand(950), np.random.rand(50)+0.4]
print("average precision:", round(average_precision_score(y, s), 3))

Validation strategies

Stratified k-fold

Keep class ratios

The idea

Plain k-fold splits randomly — fine until a class is rare, when some folds may end up with no minority examples at all. Stratified k-fold forces every fold to keep the same class ratio as the whole set. Toggle it and watch the per-fold minority counts even out.

The math · The rule

Each fold's class proportion equals the overall proportion: minorityᶠ|fold| ≈ minorityn

Why it matters

If a validation fold has zero minority cases, your metric for that class is undefined or wildly noisy — and the average misleads you. Stratification gives every fold a fair, comparable test. It's the default for classification.

Code

python

import numpy as np
from sklearn.model_selection import StratifiedKFold
y = np.r_[np.zeros(90), np.ones(10)]       # imbalanced
for tr, te in StratifiedKFold(5).split(np.zeros(len(y)), y):
    print("test positives:", int(y[te].sum()), end='  ')   # keeps class ratio per fold

Leave-one-out CV

Code

python

import numpy as np
from sklearn.model_selection import GroupKFold
X = np.random.randn(12,2); y = np.random.randint(0,2,12)
groups = np.repeat(np.arange(4), 3)        # e.g., same patient
for tr, te in GroupKFold(4).split(X, y, groups):
    print("test groups:", set(groups[te]))   # a group never spans train+test

Nested cross-validation

Tune and test honestly

The idea

If you tune hyperparameters on the same folds you report, the score is optimistic — you've fit to the validation data. Nested CV uses two loops: an inner loop picks the best hyperparameters, an outer loop evaluates on data the tuning never saw. Step through the outer folds.

The math · The two loops

↻Outer: hold out a test fold for an honest estimate. ↺Inner: on the remaining data, run CV to pick hyperparameters. =Score the tuned model on the untouched outer fold. Average over outer folds.

Why bother

Regular "CV for tuning + CV for reporting on the same splits" leaks the test folds into model selection, inflating the number. Nested CV is the gold standard when you must both tune and report — at the cost of training many more models (outer × inner × configs).

Code

python

import numpy as np
from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.svm import SVC
X = np.random.randn(120,4); y = (X[:,0]>0).astype(int)
inner = GridSearchCV(SVC(), {'C':[0.1,1,10]}, cv=3)     # inner: tune
outer = cross_val_score(inner, X, y, cv=4)               # outer: unbiased estimate
print("nested CV score:", round(outer.mean(), 3))

import numpy as np
# Random oversampling duplicates minority examples to balance classes
maj = np.random.randn(90, 2); minr = np.random.randn(10, 2) + 3
idx = np.random.randint(0, len(minr), len(maj))     # resample minority up
balanced = np.vstack([maj, minr[idx]])
print("balanced sizes:", len(maj), len(minr[idx]))

Applied ML & strategy

103

Error analysis

Look at what breaks

The idea

Before optimizing anything, look at the mistakes. Pull ~100 misclassified examples, sort them into categories, and count — the biggest bucket is where your effort pays off most. It's the cheapest, highest-leverage habit in applied ML, and it stops you polishing things that barely matter.

Error rate

starting dev error20.0% after your fixes20.0%

The point

“Blurry” and “lookalikes” are 75% of all errors — fixing either moves the needle far more than chasing the 8% mislabeled or 5% other. Without counting, teams routinely spend weeks on a category worth a fraction of a percent.

Code

python

import numpy as np
# Bucket the misclassified examples to find where the model fails
y, pred = np.random.randint(0,2,200), np.random.randint(0,2,200)
group = np.random.randint(0,3,200)                   # some segment
wrong = y != pred
for g in range(3):
    m = (group==g)
    print(f"segment {g}: error rate {wrong[m].mean():.2f}")

104

Bias & variance — what to do

Diagnose, then act

The idea

The decomposition is only useful if it tells you what to change. Compare three numbers — a human/Bayes baseline, your train error, and your dev error. The gap from baseline to train is avoidable bias; the gap from train to dev is variance. Whichever is bigger decides your next move. Slide them and read the prescription.

Diagnosis

avoidable bias13% variance3% verdicthigh bias (underfitting)

105

Picking an eval metric

One number to chase

The idea

A team that argues over several metrics moves slowly. Pick a single number to optimize so you can rank models instantly. When metrics trade off (accuracy vs speed), split them: satisficing metrics just need to clear a bar, while one optimizing metric is maximized. Set the bar and watch the winner change.

The math · The rule

Among models that meet the bar, maximize the one metric that matters: maximize accuracy s.t. latency ≤ budget

Chosen model

winnerModel B (93.0%, 95ms)

Why not average them?

Averaging accuracy and latency mixes incompatible units and hides real constraints — a model that's 0.5% better but 3× too slow shouldn't “win.” Satisficing encodes the hard requirement; optimizing picks the best of what's left.

106

Train/dev/test mismatch

Match the real world

The idea

Your dev and test sets define the target you're aiming at — so they must look like the data you'll actually face in production. If you train on data from a different distribution, a clever trick separates ordinary overfitting from a genuine data mismatch: a “train-dev” set, drawn from the training distribution but never trained on.

Scenario

Train ≈ dev distribution Turn this off to simulate training on, say, clean web images but deploying on blurry phone photos.

The math · Reading the four levels

human: → train: avoidable bias train: → train-dev: variance train-dev: → dev: data mismatch

Diagnosis

dominant problem—

Why the train-dev set

Without it, you can't tell whether a big train→dev gap is the model overfitting (variance) or the dev data simply being different (mismatch). The train-dev set — same distribution as train, held out — splits the gap. Mismatch calls for different fixes: make training data more like production, or collect real production data.

As a network trains, the distribution of each layer's inputs keeps shifting, which slows everything down. Batch norm re-centers and re-scales each layer's activations to a clean distribution every batch — then lets the network learn its own scale and shift back. Slide the incoming distribution and watch it get normalized.

The math · The transform

x̂ = x − μB√(σ²B+ε) y = γ x̂ + β Normalize to mean 0, variance 1 — then γ, β let the net rescale if it wants.

Why it helps

Stable input distributions let you use higher learning rates and train deeper nets faster. It also has a mild regularizing effect (each batch's statistics add a little noise).

Code

python

import numpy as np
def batchnorm(x, gamma=1., beta=0., eps=1e-5):
    mu, var = x.mean(0), x.var(0)
    return gamma*(x - mu)/np.sqrt(var+eps) + beta       # normalize per feature
h = np.random.randn(8, 4)*5 + 3
print("after BN: mean", np.round(batchnorm(h).mean(0),2))

Deep learning

python

import numpy as np
def step(x, h, Wx, Wh): return np.tanh(x@Wx + h@Wh)
Wx, Wh = np.random.randn(4,8), np.random.randn(8,8)
h = np.zeros(8)
for x in np.random.randn(6, 4): h = step(x, h, Wx, Wh)   # carry state over time
print("final state:", h.shape)

Reinforcement learning

121

Q-learning

Watch an agent learn

The idea

Reinforcement learning has no labeled answers — just rewards. The agent wanders a grid, bumps into a goal (and a pit), and gradually learns which moves pay off. The arrows show its current best-known action per square. Run it and watch a strategy emerge from pure trial and error.

Progress

Episodes0 Last reward—

Code

python

import numpy as np
# Q-learning: Q(s,a) <- Q + alpha*(r + gamma*max Q(s') - Q)
Q = np.zeros((3, 2)); alpha, gamma = 0.5, 0.9
# one update toward reward 1 from state 0, action 1, next state 2
s, a, r, s2 = 0, 1, 1.0, 2
Q[s,a] += alpha*(r + gamma*Q[s2].max() - Q[s,a])
print(np.round(Q, 2))

vs supervised learning

Use this when anomalies are rare and varied — fraud, novel faults — so you can't gather enough positive examples to train a classifier. You model only “normal,” and anything improbable is suspect. With many labelled positives of a consistent type, a supervised classifier is usually better.

Code

python

import numpy as np
# Fit a Gaussian; flag low-probability points as anomalies
X = np.random.randn(500, 2)
mu, cov = X.mean(0), np.cov(X.T); cov_inv = np.linalg.inv(cov)
def nll(x): return (x-mu) @ cov_inv @ (x-mu)
print("anomaly score of [5,5]:", round(nll(np.array([5.,5.])), 1))

127

Recommenders

Users like you also liked

The idea

How does Netflix guess what you'll like? Collaborative filtering finds users who rated things the way you did, then predicts your missing ratings from theirs. No content analysis needed — just the pattern of who-likes-what. Pick the cell to predict and watch it find your taste-twins.

Find taste-twins

Press Predict: the users whose ratings most match yours light up, and their opinion of the unseen movie becomes your predicted rating.

Code

python

import numpy as np
# Collaborative filtering by matrix factorization (R ~ U V^T)
R = np.array([[5,3,0,1],[4,0,0,1],[1,1,0,5],[0,0,5,4.]]); mask = R>0
U, V = np.random.rand(4,2), np.random.rand(4,2)
for _ in range(3000):
    E = mask*(R - U@V.T); U += 0.01*(E@V); V += 0.01*(E.T@U)
print(np.round(U@V.T, 1))

Checking every itemset is exponential. Apriori's trick: if an itemset is rare, every superset is rarer — so prune aggressively. Lift matters most: a high-confidence rule for an already-popular item (high P(Y)) may be worthless (lift ≈ 1).

Code

python

# Apriori idea: find item pairs that co-occur more than chance (lift > 1)
baskets = [{'bread','milk'},{'bread','butter'},{'bread','milk','butter'},{'milk'}]
n = len(baskets)
def support(items): return sum(items <= b for b in baskets)/n
lift = support({'bread','butter'}) / (support({'bread'})*support({'butter'}))
print("lift(bread->butter):", round(lift, 2))

Putting it together

131

Your first model, end to end

The whole loop in code

The idea

Everything today, in one pass. This is the exact sequence you'll run in the lab — from raw data to a tested prediction. Step through the pipeline you'll build by the end of Day 1.

Code

python

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X = np.random.randn(300, 5); y = (X[:,0]+X[:,1] > 0).astype(int)
pipe = make_pipeline(StandardScaler(), LogisticRegression())   # preprocessing + model
print("CV accuracy:", round(cross_val_score(pipe, X, y, cv=5).mean(), 3))