Math & Statistics for Machine Learning
A hands-on tour of the mathematics and statistics that machine learning is built on — probability, linear algebra, calculus, optimization, and inference — each idea derived and run in front of you, not just stated. Three layers: Algorithm derivations that work an ML method end to end, Core results that derive a single key formula, and Concept cards you can drag, type into, and tune to see the math move. A sibling to the ML, Deep-Learning, and Text labs. Press Esc anytime for this menu.
The Gaussian
The normal distribution is the default model for noise and error in machine learning. Two numbers fix it: the mean μ sets its centre, the standard deviation σ sets its width. Drag the sliders and watch the bell move and breathe.
mean μ = 0.0std σ = 1.0
p(x) = (1/σ√2π)·exp(−(x−μ)² / 2σ²).
μ = E[X] (centre), σ² = Var[X] (spread). Area = 1.
Standardize z = (x−μ)/σ ⇒ any normal becomes the standard N(0,1). The 1/σ√2π factor is exactly what makes ∫ p dx = 1.
The default noise/error model: least squares is its MLE. It sets weight-initialization scales, appears in Gaussian mixtures, Bayesian priors, and the reparameterization of VAEs. ±2σ ≈ 95% gives confidence intervals.
Bayes’ theorem
Bayes’ theorem flips a conditional: from how often a test is positive given the disease, it tells you the chance of disease given a positive test. The surprise is how much the base rate matters — a great test can still mean a coin-flip when the disease is rare. Tune all three.
prevalence = 1.0%sensitivity = 90%specificity = 90%
P(D|+) = P(+|D)P(D) / P(+),
P(+) = P(+|D)P(D) + P(+|¬D)P(¬D)
= sens·prev + (1−spec)(1−prev).
Posterior = (likelihood × prior) / evidence. The evidence sums both ways to be positive; rare priors make the (1−spec)(1−prev) term dominate.
Naive Bayes classifiers, posterior inference, and calibration all rest on this. With imbalanced classes the base rate dominates — high accuracy can hide a useless positive predictive value.
Expectation & variance
The expectation E[X] is the probability-weighted average outcome — the balance point of the distribution. The variance measures how far outcomes spread around it. Reshape the distribution by dragging the bars and watch both numbers respond.
E[X] = Σ x·p(x) (the mean).
Var[X] = E[(X−μ)²] = E[X²] − E[X]².
SD = √Var.
Expand E[(X−μ)²] = E[X²] − 2μE[X] + μ² = E[X²] − μ² — the handy "mean of square minus square of mean".
Training minimizes expected loss (risk). The bias–variance decomposition is built from these moments, and averaging gradients over a minibatch is an expectation estimate.
Central limit theorem
Averages behave better than the things they average. Take any distribution, draw n samples, average them, and repeat: the distribution of those averages drifts toward a bell and narrows as n grows — even when the source is lopsided. Raise n and watch it happen.
sample size n = 1
For iid Xᵢ with mean μ, variance σ², the sample mean X̄ₙ satisfies
X̄ₙ → N(μ, σ²/n). Standard error = σ/√n.
E[X̄ₙ] = μ and Var[X̄ₙ] = σ²/n (variances add, then divide by n²). The CLT adds that the shape tends to normal regardless of the source.
A minibatch gradient is a sample mean: its noise shrinks like 1/√n as batch size grows. Also underlies bootstrap confidence intervals and why ensembling averages away variance.
Conditional probability
Conditioning means narrowing the world to the cases where B happened, then asking how often A also holds there. P(A|B) can be wildly different from P(A). Drag the three sliders — the sizes of A, B, and their overlap — and read the conditionals off the Venn diagram.
P(A) = 0.50P(B) = 0.40overlap P(A∩B) = 0.20
P(A|B) = P(A∩B) / P(B).
Chain rule: P(A∩B) = P(A|B)P(B).
Independent ⇔ P(A|B) = P(A).
Conditioning restricts the sample space to B and renormalizes by 1/P(B). Combined with symmetry P(A∩B)=P(B|A)P(A) this gives Bayes’ theorem.
Probabilistic graphical models encode conditional independence; naive Bayes assumes features independent given the class; autoregressive models factor P(x) = ∏ P(xₜ | x_{
Vectors, span & linear combinations
Everything in ML lives in vector spaces: a data point, a weight row, a gradient are all vectors. A linear combination a·v₁ + b·v₂ scales and adds them; the set of all reachable combinations is their span. Drag the vectors and dial the coefficients to sweep out that span.
a = 1.0b = 1.0
w = a·v₁ + b·v₂
span{v₁,v₂} = { a·v₁ + b·v₂ : a,b ∈ ℝ }
Independent ⇔ det[v₁ v₂] ≠ 0 ⇔ span is all of ℝ².
If v₂ = k·v₁ (dependent), then
a·v₁ + b·v₂ = (a + bk)·v₁.
Every combination is a multiple of v₁ — the span collapses from a plane to a line.
A linear layer outputs Wx — each output is a linear combination of input features. The reachable outputs form a span; independent features give a full-rank design matrix, so the normal equations have a unique solution.
A matrix is a transformation
A matrix doesn’t just store numbers — it acts on space. Multiplying A by x maps the vector to a new place, and the whole grid warps with it. The columns of A are exactly where the basis vectors land. Tune the four entries and drag x to feel the map.
c d
Ax = [a b; c d][x₁; x₂] = x₁·col₁ + x₂·col₂
col₁ = A·e₁ = (a,c), col₂ = A·e₂ = (b,d).
det A = ad − bc = area-scaling factor.
Write x = x₁e₁ + x₂e₂. By linearity
Ax = x₁(Ae₁) + x₂(Ae₂) = x₁·col₁ + x₂·col₂.
So knowing where the basis lands determines the whole map.
A dense layer is x ↦ Wx + b: W is a learned transformation of feature space, stacked layers compose them. Singular values of W say how much it stretches each direction (Jacobians, spectral norm, normalizing flows).
Eigenvectors & eigenvalues
Most vectors get knocked off their line when a matrix acts on them. A precious few — the eigenvectors — keep their direction and merely scale by a factor λ, the eigenvalue. Drag the input around the circle; when Av lines up with v, you’ve found one.
A v = λ v.
Nontrivial v exists ⇔ det(A − λI) = 0
⇒ λ² − tr(A)λ + det(A) = 0.
Symmetric A ⇒ real λ, orthogonal eigenvectors.
For [a b; b d]: tr = a+d, det = ad−b²,
λ = (tr/2) ± √((tr/2)² − det).
Each λ gives a direction fixed by A up to scaling.
PCA uses eigenvectors of the covariance matrix — directions of maximum variance, with λ = variance captured. Eigenvalues of the loss Hessian are curvatures; their ratio is the condition number that governs gradient-descent speed. Also spectral clustering and PageRank.
Quadratic forms & definiteness
A quadratic form f(x) = xᵀAx turns a symmetric matrix into a bowl, saddle, or dome. Its level sets are ellipses whose axes are A’s eigenvectors and whose stretch is set by the eigenvalues. Drag x across the contours and reshape A to flip its character.
a d
off-diag b
f(x) = xᵀA x = a·x₁² + 2b·x₁x₂ + d·x₂².
In eigen-coordinates: f = λ₁y₁² + λ₂y₂².
Positive-definite ⇔ λ₁,λ₂ > 0 ⇔ f > 0 for all x ≠ 0.
Symmetric A = QΛQᵀ (orthogonal Q). Substituting y = Qᵀx gives f = yᵀΛy = Σᵢ λᵢ yᵢ² — a sum of squared axes, so the signs of the λ decide the shape.
The loss Hessian is a quadratic form: positive-definite ⇒ a convex bowl with a unique minimum; eigenvalue ratio = condition number (ill-conditioned → slow zig-zagging GD). Covariance matrices are PSD; Mahalanobis distance is xᵀΣ⁻¹x.
Projection & least squares
To approximate a vector b using only directions you have, drop a perpendicular: the projection is the closest reachable point, and the leftover residual is orthogonal to everything you used. That orthogonality is the whole secret of least-squares regression. Drag b and watch.
Project b onto span(a):
t* = aᵀb / aᵀa, p = t*·a.
Residual r = b − p satisfies aᵀr = 0 (⊥).
Minimize ‖b − t·a‖² over t:
d/dt = −2 aᵀ(b − t a) = 0
⇒ t* = aᵀb / aᵀa. Setting the gradient to zero is the orthogonality condition.
Ordinary least squares projects y onto the column space of X: the normal equations XᵀXβ = Xᵀy give β = (XᵀX)⁻¹Xᵀy, and the residual is orthogonal to every feature. This is linear regression, PCA reconstruction, and Gram–Schmidt.
Convexity
A function is convex if the straight chord between any two points sits on or above the curve. Convexity is the property that makes optimization easy: every local minimum is the global minimum. Drag the two points and toggle to a non-convex function to watch the test fail.
Convex: f(λa+(1−λ)b) ≤ λf(a)+(1−λ)f(b), ∀λ∈[0,1].
Twice-differentiable: convex ⇔ f″(x) ≥ 0 everywhere.
The chord at the midpoint has height ½(f(a)+f(b)). If this is ≥ f(midpoint) for all a,b, no “dip” can hide a second valley — so any local min is global (a descent step always reaches it).
Linear/ridge regression, logistic regression, and SVMs have convex losses → a unique optimum, solvable reliably. Deep nets are non-convex — many minima and saddles — but over-parameterization and SGD still find good ones.
Gradient descent & conditioning
On a stretched bowl, gradient descent zig-zags: it keeps pointing across the valley instead of down it. The culprit is the condition number — the ratio of the steepest to the flattest curvature. Stretch the bowl and watch convergence crawl; round it out and it shoots to the bottom.
curv x curv y
η
f(x,y)=½(a·x² + c·y²), ∇f=(a·x, c·y).
Update: (x,y) ← (x,y) − η·∇f.
Condition number κ = max(a,c)/min(a,c).
Axes decouple: x ← (1−ηa)x, y ← (1−ηc)y. Stability needs η < 2/max(a,c); but then the flat axis (small c) moves by only ηc — tiny. Rate ≈ (κ−1)/(κ+1).
Un-scaled features give an ill-conditioned loss — the reason we standardize inputs and use batch/layer norm. Second-order methods (Newton) and Adam rescale by curvature to fight exactly this.
L1 vs L2 regularization
Regularization adds a penalty on the weights, equivalent to constraining them inside a ball. The ball’s shape decides the outcome: the round L2 ball shrinks weights smoothly, while the pointed L1 diamond pulls solutions onto the axes — making them sparse. Drag the target and squeeze the budget.
budget t
min loss(w) + λ‖w‖ ⇔ min loss(w) s.t. ‖w‖ ≤ t.
L2: ‖w‖₂ ≤ t (a disc) · L1: ‖w‖₁ ≤ t (a diamond).
The solution is where the loss contour first touches the ball. L2 shrinks: w ← w*·t/‖w*‖. L1’s corners lie on the axes, so the touch point often has a coordinate exactly 0 — a feature dropped (soft-thresholding).
L2 = ridge / weight decay (smooth shrinkage, the default in deep nets). L1 = lasso (feature selection via sparsity). L2 corresponds to a Gaussian prior on weights, L1 to a Laplace prior (MAP estimation).
Learning rate & convergence
The learning rate is the single most important knob. On a simple quadratic there are three regimes: small steps glide in monotonically, medium steps overshoot and oscillate while still converging, and large steps blow up. The boundary is set entirely by the curvature.
curvature k = 2.0η = 0.30
f(x)=½k·x², gradient k·x.
Update: x ← x − η·k·x = (1 − ηk)·x.
After n steps xₙ = (1−ηk)ⁿ·x₀. Converges ⇔ |1−ηk| < 1 ⇔ 0 < η < 2/k. Monotone if η < 1/k, oscillating if 1/k < η < 2/k, divergent if η > 2/k.
The safe ceiling is 2/(largest curvature) = 2/λ_max of the Hessian. Warmup, decay schedules, and adaptive optimizers all manage η against changing curvature during training.
Estimators & the sampling distribution
An estimator is a recipe that turns a sample into a guess about the population (the sample mean estimates the true mean). Because the sample is random, the estimate is random too — it has its own sampling distribution. Draw samples repeatedly to build that distribution and watch it cluster around the truth.
n = 20
μ̂ = X̄ = (1/n)Σ Xᵢ.
E[X̄] = μ (unbiased), Var[X̄] = σ²/n.
Standard error SE = σ/√n.
E[X̄] = (1/n)Σ E[Xᵢ] = μ. Var[X̄] = (1/n²)Σ Var[Xᵢ] = σ²/n. So the estimator centers on the truth and tightens like 1/√n as n grows.
Every metric you report (validation accuracy, mean loss) is an estimate with a sampling distribution — hence error bars. The same logic gives the variance of a minibatch gradient and underlies the bias–variance view of models.
The bias–variance tradeoff
A too-simple model misses the pattern (high bias, underfitting); a too-flexible one chases the noise (high variance, overfitting). Test error is smallest in between. Raise the polynomial degree and watch the fit go from stiff to wild while train error keeps dropping but test error turns back up.
degree = 3noise
E[(y − f̂(x))²] = Bias[f̂]² + Var[f̂] + σ².
Bias ↓ and Variance ↑ as flexibility grows; σ² is irreducible.
Split the expected test error by adding and subtracting E[f̂(x)]: the systematic miss is bias², the wobble across datasets is variance, and the label noise σ² can never be removed. Minimizing their sum picks the sweet-spot complexity.
The master curve behind model selection, regularization, early stopping, and ensembling (bagging cuts variance, boosting cuts bias). Modern over-parameterized nets revisit this with the “double descent” phenomenon.
Confidence intervals
A 95% confidence interval doesn’t mean “95% chance the truth is in this interval.” It means the procedure traps the true value 95% of the time across repeated samples. Draw many intervals and watch about that fraction cross the true mean — some always miss.
n = 25
CI = X̄ ± z·σ/√n.
z = 1.645 (90%), 1.96 (95%), 2.576 (99%).
Since X̄ ≈ N(μ, σ²/n), the event |X̄ − μ| ≤ z·SE has probability = level. Rearranging puts μ inside [X̄ − z·SE, X̄ + z·SE] with that same probability — a statement about the random interval, not μ.
Report metrics with CIs, not point numbers, before claiming model A beats B. Bootstrapping builds CIs without distributional assumptions; A/B tests and eval harnesses live on this idea.
Hypothesis testing & p-values
A test asks: if nothing were going on (the null), how surprising is what we saw? The p-value is the tail probability of a result at least this extreme under the null. Small p → reject. Increase the true effect or the sample size and watch the test gain power to detect it.
effect
n
z = X̄ / (σ/√n).
p (two-sided) = 2·(1 − Φ(|z|)).
Reject H₀ if p < α, i.e. |z| > z_{α/2}.
Under H₀ the statistic is N(0,1); the critical value z_{α/2} fixes the Type-I rate at α. Power = P(reject | effect real) = Φ(effect·√n − z_{α/2}); a Type-II error (miss) is 1 − power.
A/B tests and “is model A really better?” comparisons. Bigger n shrinks SE and raises power — but with enough data even trivial effects become “significant,” so report effect sizes, and beware multiple-comparison inflation.
MLE vs MAP & the prior
Maximum likelihood trusts only the data; MAP adds a prior belief and lets the two combine. With little data the prior dominates and pulls the estimate toward itself; with lots of data the likelihood wins and MAP → MLE. A Gaussian prior is exactly L2 regularization in disguise. Move the prior and the data.
prior mean
prior strength
data mean
n = 5
MLE: θ̂ = argmax p(D|θ) = X̄.
MAP: θ̂ = argmax p(D|θ)p(θ).
Gaussian prior N(θ₀,τ²): θ_MAP = (nσ⁻²·X̄ + τ⁻²·θ₀)/(nσ⁻² + τ⁻²).
Maximizing log p(D|θ)+log p(θ) for Gaussians gives a precision-weighted average of data and prior. The prior term −(θ−θ₀)²/2τ² is a quadratic penalty — i.e. L2 / weight decay centered at θ₀.
Regularization = a prior. L2 ⇔ Gaussian prior, L1 ⇔ Laplace prior. As data grows the prior’s pull fades (consistency). Bayesian deep learning keeps the whole posterior instead of just its peak.
Linear regression
The simplest learner: fit a straight line that minimizes the total squared vertical gap to the data. There’s a closed-form answer — no iteration needed — because setting the derivative of the squared error to zero gives two linear equations. Drag the points and watch the best-fit line and its residuals respond instantly.
ŷ = wx + b; minimize J = Σ(yᵢ − wxᵢ − b)².
w = Cov(x,y)/Var(x), b = ȳ − w·x̄.
Set ∂J/∂w = 0 and ∂J/∂b = 0 → the two normal equations. In matrix form XᵀXβ = Xᵀy, so β = (XᵀX)⁻¹Xᵀy — exactly projecting y onto the column space of X.
The canonical supervised baseline and the template for GLMs. Ridge adds λ‖β‖² (a Gaussian prior) for stability when XᵀX is near-singular; the same projection idea powers feature regression everywhere.
Logistic regression
Now the labels are classes, not numbers. Logistic regression learns a linear boundary by gradient descent on the cross-entropy loss; the gradient is the same clean (prediction − label)·x. Step the training and watch the boundary swing into place between the two clouds while the loss drops.
p = σ(w·x + b), loss = −Σ[y log p + (1−y)log(1−p)].
∇_w = Σ(pᵢ − yᵢ)xᵢ, ∇_b = Σ(pᵢ − yᵢ).
Using σ′ = σ(1−σ), the per-example gradient of the cross-entropy collapses to (p−y)·x. No closed form (the loss is convex but transcendental), so we descend — but convexity guarantees the global optimum.
The default linear classifier and the output layer of most neural nets. Softmax regression generalizes it to many classes; the (p−y) gradient is exactly what backprop feeds into the rest of the network.
k-means clustering
Unsupervised: group points so each sits near its cluster’s centre. k-means alternates two cheap steps — assign every point to the nearest centroid, then move each centroid to the mean of its members — and repeats. Each step can only lower the total within-cluster distance, so it converges (to a local optimum). Step it and watch it settle.
k = 3
Minimize J = Σᵢ ‖xᵢ − μ_{c(i)}‖².
Assign: c(i) = argmin_c ‖xᵢ − μ_c‖. Update: μ_c = mean of its points.
It’s coordinate descent on J: fixing centroids, nearest-assignment minimizes J; fixing assignments, the mean minimizes Σ‖x−μ‖² (set gradient to 0). Both steps are non-increasing ⇒ convergence; the result depends on initialization.
Fast clustering, vector quantization, and image color compression. It is hard EM for a Gaussian mixture with equal spherical covariances; k-means++ seeding avoids bad local minima.
Principal component analysis
PCA finds the directions along which the data varies most. The first principal component is the axis capturing maximum variance; the next is orthogonal to it, and so on. They are exactly the eigenvectors of the covariance matrix, with eigenvalues equal to the variance each captures. Reshape the cloud and watch the axes track it.
spread x spread y
rotate
Covariance Σ = (1/n) XᵀX (centered).
Σ = VΛVᵀ; columns of V = principal components, Λ = variances.
Project: Z = XV (keep top components).
Maximize the projected variance wᵀΣw subject to ‖w‖ = 1. The Lagrangian gives Σw = λw — so the maximizer is the top eigenvector, and λ is the variance along it.
Dimensionality reduction, denoising, whitening, and visualization (project to 2-D). Closely tied to the SVD of the data matrix; truncating components gives the best low-rank approximation (Eckart–Young).
Maximum-margin classifier
A support vector machine doesn’t just separate the classes — it finds the widest street between them. The boundary is pinned by the few closest points (the support vectors); everything else is irrelevant. Rotate and slide the boundary to widen the margin while keeping the classes apart.
angle
offset
Boundary: w·x + b = 0. Distance of x = |w·x+b|/‖w‖.
Margin = 2/‖w‖. Maximize it ⇔ minimize ‖w‖ s.t. yᵢ(w·xᵢ+b) ≥ 1.
Scale w,b so the nearest points satisfy |w·x+b| = 1. Then each side’s half-width is 1/‖w‖, giving a total street of 2/‖w‖. Widening the street shrinks ‖w‖ — a convex quadratic program.
Only the support vectors matter, so SVMs are memory-efficient and robust. The large margin is a capacity-control prior that often generalizes well; the dual form opens the door to kernels.
Hinge loss & soft margin
Real data isn’t perfectly separable, so the SVM allows violations but charges for them with the hinge loss: zero once a point is correctly classified beyond the margin, then rising linearly. Slide a point’s margin and compare hinge to the logistic loss and the un-trainable 0–1 loss.
m = 0.5
Hinge: L = max(0, 1 − m).
Soft-margin SVM: min ½‖w‖² + C·Σ max(0, 1 − yᵢ f(xᵢ)).
The slack ξᵢ = max(0, 1 − yᵢfᵢ) measures how far inside the margin a point falls. Hinge is the tightest convex upper bound on the 0–1 loss, so minimizing it is a tractable proxy; its subgradient is −y·x when m < 1, else 0.
C trades margin width against violations (small C = wider, more tolerant). Hinge gives sparse support vectors (only margin violators contribute); cross-entropy keeps all points active — the two main linear-classifier losses.
The kernel trick
These two classes — a core and a surrounding ring — cannot be split by any straight line. But add one feature, the squared radius, and they separate trivially: a flat threshold in the lifted space is a circle back in the original space. Slide the threshold and watch a linear cut become a curved boundary.
threshold
Lift: φ(x) = (x₁, x₂, ‖x‖²).
Linear rule w·φ(x)+b > 0 with w=(0,0,1) becomes ‖x‖² > t² — a circle.
Kernel: K(x,x′) = φ(x)·φ(x′).
A separator that is linear in φ maps to a nonlinear surface in x. The trick: algorithms need only inner products φ(x)·φ(x′), which a kernel K computes directly — so you never form the (possibly infinite-dim) φ.
Kernel SVMs, kernel ridge regression, and Gaussian processes all exploit this. The representer theorem says the solution is a weighted sum of kernels on the training points.
The RBF kernel
The radial-basis-function kernel measures similarity as a Gaussian bump: nearby points are similar, far ones aren’t. A classifier sums these bumps over labeled landmarks, carving a smooth nonlinear boundary. The width γ is the key dial — turn it up for wiggly, overfit boundaries; down for smooth ones.
γ = 1.0
K(x, x′) = exp(−γ‖x − x′‖²), γ = 1/(2σ²).
f(x) = Σᵢ αᵢ yᵢ K(x, xᵢ) + b; predict sign(f).
RBF corresponds to an infinite-dimensional feature map, yet K is a single exponential. Large γ → narrow bumps → each point influences only its neighborhood (high variance); small γ → wide, smooth (high bias).
The default SVM kernel and the heart of Gaussian processes and kernel regression. γ (and C) set the bias–variance tradeoff; tuning them is the main RBF-SVM hyperparameter search.
Splitting: entropy & Gini
A decision tree grows by asking yes/no questions that make each side purer. Impurity — entropy or Gini — measures how mixed a group’s labels are; a good split drops the weighted impurity the most. Slide the threshold to find the cut that maximizes information gain.
threshold
Entropy H = −Σ pₖ log₂ pₖ. Gini = 1 − Σ pₖ².
Gain = H(parent) − Σ (n_child/n)·H(child).
Each candidate split partitions the node; the children’s impurities are weighted by their sizes. The tree greedily picks the feature+threshold with the largest gain — the cut that buys the most purity per point.
CART uses Gini, ID3/C4.5 use entropy/gain ratio — they pick nearly the same splits. Recursively repeating this builds the tree; stopping rules (depth, min-samples) control overfitting.
A tree partitions space
Apply splitting recursively and the feature space carves into axis-aligned boxes, each predicting one class. Shallow trees underfit; deep trees can isolate every point — memorizing noise. Raise the depth and watch the rectangles multiply until the boundary turns jagged.
max depth = 2
Each node: split (feature, threshold) minimizing child impurity. Recurse until max depth / pure. Leaf predicts the majority class.
Greedy recursion: the best single split is chosen at each node independently. The result is a partition into hyper-rectangles — expressive but high-variance, since small data changes can flip a split and reshape whole regions.
Single trees are interpretable but unstable and overfit-prone — motivating ensembles (forests, boosting). Depth, min-samples-leaf, and pruning are the main regularizers.
Bagging & random forests
One deep tree is jumpy — retrain it on a slightly different sample and the prediction lurches. Bagging trains many trees on bootstrap resamples and averages them; the wobble cancels out and the curve smooths. Random forests go further by decorrelating the trees. Add trees and watch the variance shrink.
trees = 1
Bagged prediction = (1/B)Σ_b T_b(x), each T_b trained on a bootstrap sample.
Var of average ≈ ρσ² + (1−ρ)σ²/B.
Averaging B identically-distributed predictors keeps the bias but divides the independent part of the variance by B. Trees are correlated (ρ>0), so a floor ρσ² remains — random forests lower ρ by sampling features at each split.
Random forests: a robust, low-tuning default for tabular data. Bagging cuts variance, not bias, so the base trees are grown deep (low bias) and de-correlated by bootstrap + feature subsampling.
Boosting
Boosting builds the model one small step at a time: each new weak learner fits the residual the current ensemble still gets wrong, and is added with a small learning rate. Bias falls round by round. Increase the rounds and watch a sum of tiny stumps converge onto the curve.
rounds M = 1learning rate
F₀ = mean(y). For m = 1..M:
rᵢ = yᵢ − F_{m−1}(xᵢ) (residual = neg. gradient of MSE)
F_m = F_{m−1} + ν·h_m, h_m fits r.
Gradient boosting does stagewise gradient descent in function space: the residual is −∂L/∂F for squared loss, so each weak learner h_m points downhill. The shrinkage ν keeps steps small to avoid overfitting.
XGBoost / LightGBM / CatBoost dominate tabular competitions. Boosting reduces bias (opposite of bagging’s variance focus) but can overfit — controlled by ν, tree depth, and number of rounds (early stopping).
Naïve Bayes
Naïve Bayes classifies by Bayes’ rule plus one bold shortcut: assume the features are independent given the class. Each class becomes a product of simple per-feature densities (here axis-aligned Gaussians). Drag the test point across the boundary and watch the posterior flip.
P(y|x) ∝ P(y)·∏ᵢ P(xᵢ|y).
Gaussian features: log P(y|x) = log P(y) − ½Σᵢ [(xᵢ−μ_{y,i})²/σ²_{y,i} + log σ²_{y,i}] + c.
Bayes gives P(y|x) = P(x|y)P(y)/P(x). The “naïve” assumption factors the likelihood P(x|y)=∏ᵢP(xᵢ|y), turning a hard joint density into easy 1-D fits — hence the axis-aligned ellipses.
A fast, strong baseline for text (bag-of-words → multinomial NB) and spam filtering. The independence assumption is usually false yet often harmless for the argmax; needs little data and trains instantly.
Gaussian mixtures & EM
A Gaussian mixture models data as a blend of bell-shaped clusters — but unlike k-means it assigns points softly, by probability. Expectation–Maximization alternates: estimate each point’s cluster responsibilities (E), then re-fit the Gaussians to those weighted points (M). Step it and watch the ellipses lock on.
K = 2
p(x) = Σₖ πₖ·N(x | μₖ, Σₖ).
E: γ_{ik} = πₖNₖ(xᵢ) / Σⱼ πⱼNⱼ(xᵢ).
M: μₖ = Σᵢγxᵢ / Σᵢγ, similarly Σₖ, πₖ.
EM maximizes a lower bound (the same ELBO idea as VAEs): the E-step fills in soft cluster memberships, the M-step does weighted maximum-likelihood. Each round never decreases the data log-likelihood.
Density estimation and soft clustering; k-means is the hard-assignment, equal-spherical-covariance limit. The same EM machinery handles missing data and latent-variable models broadly.
Markov chains
A Markov chain hops between states with fixed transition probabilities, remembering only where it is now. Multiply the current distribution by the transition matrix repeatedly and it converges to a unique stationary distribution — no matter where you start. Step the distribution and tune how “sticky” the states are.
stay prob
v_{t+1} = v_t·P, Σⱼ P_{ij} = 1.
Stationary π solves π = πP (left eigenvector, eigenvalue 1).
Repeated multiplication is power iteration: components along sub-dominant eigenvectors shrink, leaving the eigenvector with eigenvalue 1. For an irreducible, aperiodic chain that limit π is unique and start-independent.
n-gram language models, PageRank (the web’s stationary distribution), MCMC samplers, and the transition model in MDPs / reinforcement learning. Mixing time governs how fast samples become useful.
Conjugate Bayesian updating
Bayesian learning updates a belief as evidence arrives. For a coin, a Beta prior over the bias meets Bernoulli data and — because they’re conjugate — the posterior is just another Beta with the counts added in. Flip some heads and tails and watch the belief sharpen around the truth.
prior α prior β
Prior θ ~ Beta(α, β). Data: H heads, T tails.
Posterior θ | data ~ Beta(α+H, β+T). mean = (α+H)/(α+β+H+T).
Likelihood θ^H(1−θ)^T times prior θ^{α−1}(1−θ)^{β−1} ∝ θ^{α+H−1}(1−θ)^{β+T−1} — a Beta kernel. Conjugacy means the posterior stays in the same family, so updating is just adding counts.
Online/streaming Bayesian inference, Thompson sampling for bandits, Bayesian A/B testing, and Dirichlet–multinomial topic models. The posterior is a full distribution, not just a point estimate — uncertainty included.
SVD & low-rank approximation
Any matrix factors into rotation·stretch·rotation — the singular value decomposition. Keep only the largest few singular values and you get the best possible low-rank approximation, which is how images and embeddings compress. Slide the rank k and watch a shape rebuild from a handful of components.
rank k = 2
A = UΣVᵀ = Σᵢ σᵢ uᵢ vᵢᵀ.
Rank-k: Aₖ = Σ_{i≤k} σᵢ uᵢ vᵢᵀ (top k terms).
σᵢ are √ of the eigenvalues of AᵀA; the uᵢ, vᵢ are its eigenvectors. Eckart–Young: truncating to the top k singular values is the closest rank-k matrix in Frobenius (and spectral) norm — no better low-rank fit exists.
PCA is the SVD of centered data; latent semantic analysis, recommender systems (low-rank matrix completion), embedding compression, and spectral methods all lean on it. Energy = Σσᵢ² captured.
Whitening
Whitening turns a tilted, stretched cloud into a clean isotropic blob with identity covariance. It runs in three moves: center the data, rotate onto the principal axes, then rescale each axis to unit variance. Step through the stages and watch the covariance ellipse become a circle.
stage
raw data
Center: x̃ = x − μ. Cov Σ = VΛVᵀ.
Whiten: z = Λ^{−1/2} Vᵀ x̃ ⇒ Cov(z) = I.
Vᵀ rotates onto the eigen-axes (decorrelating); dividing each by √λ (its std) equalizes the variances. The result has zero mean and identity covariance — every direction now equally scaled.
Whitening / standardization speeds up optimization (rounds the loss bowl), and PCA-whitening feeds ZCA, decorrelated features, and classical preprocessing. BatchNorm is a learned, per-batch cousin.
Neighbor embeddings (t-SNE / UMAP)
t-SNE and UMAP don’t try to preserve every distance — just who’s near whom. They model neighbors with a similarity kernel and pack the map so local neighborhoods survive. Drag the query and shrink the bandwidth to see how few points really count as “neighbors.”
bandwidth σ
High-D: p_{j|i} ∝ exp(−‖xᵢ−xⱼ‖² / 2σ²).
Low-D (t-SNE): q_{ij} ∝ (1 + ‖yᵢ−yⱼ‖²)⁻¹ (Student-t).
Minimize KL(p ∥ q).
A Gaussian falls off fast, so only nearby points get real weight — that’s the local neighborhood. The heavy-tailed Student-t in the map gives distant points room (fixing the “crowding problem”), and minimizing KL aligns the two.
The go-to tools for visualizing high-dim embeddings (word/image vectors, single-cell data). Caveat: cluster sizes and between-cluster distances aren’t meaningful — only local grouping is.
Random projection (Johnson–Lindenstrauss)
You can crush high-dimensional data into far fewer dimensions with a random matrix and still keep all the pairwise distances almost intact — no training, no data peeking. The Johnson–Lindenstrauss lemma says the needed dimension depends only on the number of points and the tolerance. Raise the target dimension and watch distances snap to the diagonal.
target dim k = 4
z = (1/√k)·R·x, R_{ij} ~ N(0,1), R ∈ ℝ^{k×D}.
(1−ε)‖x−x′‖ ≤ ‖z−z′‖ ≤ (1+ε)‖x−x′‖.
Each random coordinate of z is a Gaussian whose variance equals the squared distance; averaging k of them concentrates around the true value with error ∼ 1/√k. JL: k ≈ 8·ln(N)/ε² suffices — independent of the original D.
Fast dimensionality reduction for huge sparse data, approximate nearest neighbors (LSH), compressed sensing, and feature hashing. Cheaper than PCA when you only need distances, not the principal axes.
Confusion matrix & metrics
A classifier’s score becomes a decision only after you pick a threshold — and that choice trades false positives against false negatives. The confusion matrix tallies the four outcomes; precision, recall, and F1 summarize them. Slide the threshold and watch the trade-off move.
threshold
Precision = TP/(TP+FP) Recall = TP/(TP+FN)
F1 = 2·P·R/(P+R) Accuracy = (TP+TN)/all.
Raising the threshold demands more confidence: FP fall (precision ↑) but FN rise (recall ↓). F1 is the harmonic mean — high only when both are high — making it the go-to single number when classes matter asymmetrically.
The starting point for any classifier report. Pick the threshold for your cost trade-off (medical screening favors recall; spam favors precision); F1 / Fβ weight them, and the matrix generalizes to multi-class.
ROC & AUC
Sweep the threshold across every value and plot true-positive rate against false-positive rate — that’s the ROC curve. The area under it (AUC) is the probability the model ranks a random positive above a random negative: a single, threshold-free score of separability. Pull the classes apart and watch AUC climb toward 1.
separation
threshold
TPR = TP/(TP+FN), FPR = FP/(FP+TN).
AUC = ∫ TPR d(FPR) = P(score₊ > score₋).
Each threshold gives one (FPR, TPR) point; sweeping traces the curve. The area equals the chance a random positive outranks a random negative — so AUC is invariant to the threshold and to class balance, unlike accuracy.
AUC compares rankers independent of any cutoff; 0.5 is random, 1.0 perfect. Under heavy class imbalance, precision–recall curves are more informative than ROC, since FPR hides a flood of true negatives.
Cross-validation
A single train/test split is a noisy estimate of how a model generalizes. k-fold cross-validation rotates the held-out fold k times and averages, using all the data for both training and testing. More folds means more training data per round (less bias) but noisier, costlier estimates. Change k and watch the spread.
k = 5
CV score = (1/k)Σ score(fold_i).
Each fold: train on k−1 parts, test on the held-out part.
Averaging k estimates cuts the variance of the generalization estimate. Larger k → each training set is bigger (lower bias) but folds overlap heavily and tests shrink (higher variance, more compute). k = 5 or 10 is the usual sweet spot.
The standard for honest model selection and hyperparameter tuning. Stratify folds for imbalance; use grouped/time-series splits to avoid leakage. LOOCV (k = n) is nearly unbiased but high-variance and expensive.
Calibration & reliability
A model can rank perfectly yet lie about its confidence — saying “90% sure” when it’s right 70% of the time. A reliability diagram plots predicted probability against observed frequency; the diagonal is perfect calibration. Temperature scaling divides the logits by T to fix overconfidence. Tune T to straighten the curve.
temperature T = 1.0
Calibrated if P(y=1 | p̂ = p) = p for all p.
Temperature scaling: p̂_T = σ(logit / T).
ECE = Σ_b (n_b/n)·|conf_b − acc_b|.
Modern nets are overconfident (logits too large), so points bow below the diagonal. Dividing logits by T > 1 softens them toward the true frequencies without changing the ranking (AUC unchanged) — a one-parameter post-hoc fix.
Calibration matters wherever probabilities drive decisions (risk, cost-sensitive choices, downstream Bayes). Temperature scaling, Platt scaling, and isotonic regression are the standard fixes; ECE is the headline metric.
Collaborative filtering
Collaborative filtering fills in missing ratings using the crowd: to guess how a user would rate an item, look at similar items they have rated and take a weighted average. Similarity comes from the rating patterns themselves — no content needed. Change how many neighbors you trust and watch the prediction shift.
neighbors k = 2
sim(i,j) = cos(col_i, col_j) over shared users.
r̂(u,i) = Σ_{j∈N} sim(i,j)·r(u,j) / Σ_{j∈N} |sim(i,j)|.
Item columns that move together across users are “similar.” A user’s known ratings on those neighbors, weighted by similarity, estimate the unknown one. User-user CF is the transpose; item-item is usually more stable (items have more ratings).
The classic Amazon/Netflix baseline. Strengths: no features required. Weaknesses: sparsity, popularity bias, and cold start (no ratings → no neighbors) — which matrix factorization and hybrids address.
Matrix factorization
Matrix factorization explains the whole rating matrix with a few hidden factors: each user and each item gets a short latent vector, and a rating is their dot product. Fitting these vectors to the observed ratings fills in every blank at once. Train it and watch the reconstruction sharpen and the error fall.
latent d reg λ
R ≈ P Qᵀ, r̂(u,i) = p_u · q_i.
min Σ_{obs} (r − p_u·q_i)² + λ(‖p_u‖² + ‖q_i‖²).
SGD on each observed rating: e = r − p_u·q_i, then p_u += η(e·q_i − λp_u), q_i += η(e·p_u − λq_i). The low rank d forces shared structure, so the model generalizes to unobserved cells instead of memorizing.
The heart of the Netflix-Prize approach (Funk SVD). Add bias terms and you get the standard model; ALS parallelizes it; regularization λ is essential on sparse data. Generalizes to embeddings everywhere.
Implicit feedback & ranking
Most real signals aren’t star ratings — they’re clicks, plays, and purchases: positive when present, but silence doesn’t mean dislike. So instead of predicting a number, we learn to rank seen items above unseen ones. Train the pairwise objective and watch clicked items rise to the top.
BPR: maximize Σ ln σ(x̂_{ui} − x̂_{uj}) for observed i, unobserved j.
Pairwise: rank a clicked item above a non-clicked one.
Absolute values are unknown, but order is informative: a click means i > j for some unseen j. Optimizing many such pairs (BPR / WARP) shapes scores so positives float up — evaluated by ranking metrics, not RMSE.
Implicit feedback dominates production recsys (YouTube, Spotify). Treat unobserved as weak negatives (with confidence weights, as in ALS-implicit), and evaluate with Recall@k / NDCG / MAP rather than rating error.
The cold-start problem
A brand-new user or item has no interactions, so pure collaborative filtering has nothing to go on — it can’t place them in the latent space. The fix is to fall back on content: position newcomers by their features (genre, profile) until behavior accumulates, or just recommend what’s popular. Toggle content features to rescue the cold item.
No interactions → q_new undefined from CF alone.
Hybrid: q_new = f(content features), or back off to a popularity prior.
Matrix factorization learns q_i only from observed ratings; with none, the vector is unconstrained. Mapping features → latent space (a learned regressor) gives an initial position, blending content and collaborative signal.
Hybrid recommenders, content-based fallbacks, and popularity baselines handle cold start; two-tower models learn item embeddings directly from features. Exploration (bandits) gathers the first interactions efficiently.
Standardization & scaling
Many models judge distance or take gradient steps, and both break when features live on wildly different scales — a feature measured in thousands drowns out one in fractions. Standardizing (subtract mean, divide by std) puts every feature on equal footing. Toggle scaling and watch which points count as “nearest.”
z-score: x′ = (x − μ)/σ. min-max: x′ = (x − min)/(max − min).
After z-scoring: each feature has mean 0, variance 1.
Euclidean distance sums squared differences, so a large-range feature dominates the sum and the others barely register. Equalizing variance makes each feature contribute comparably — the same reason it rounds the loss surface for gradient descent.
Essential for k-NN, k-means, SVMs, PCA, and neural nets; fit the scaler on train only (avoid leakage). Tree models are scale-invariant and don’t need it. Standardize vs min-max depends on outliers and bounded ranges.
Categorical encoding
Models eat numbers, not words — so categories must be encoded. Mapping them to 1, 2, 3 sneaks in a false order and false distances (is “cat 3” really three times “cat 1”?). One-hot encoding instead gives each category its own axis, so all of them sit equidistant. Toggle between the two.
categories = 4
Label: catₖ → k (one number, ordered).
One-hot: catₖ → eₖ (unit basis vector). All pairwise distances = √2.
Label encoding embeds categories on a line, so |3−1| = 2 > |2−1| = 1 — a metric the data never implied. One-hot vectors are orthogonal, so every pair is equally (dis)similar, which is the honest default for nominal categories.
One-hot is standard for linear/NN models (watch the dummy-variable trap and high cardinality). Ordinal encoding is fine when order is real; target/frequency encoding and learned embeddings handle many categories compactly.
Polynomial & interaction features
A linear model is only linear in its features — so if you hand it x, x², x³… it can bend into curves while the fitting stays linear and convex. Interaction terms like x₁x₂ let it carve regions a plain linear model can’t. Raise the degree and watch a straight line become a curve (then overfit).
degree = 3
φ(x) = [1, x, x², …, xᵈ]; model = wᵀφ(x).
Interactions (2-D): add x₁x₂, x₁², x₂², …
Mapping x to a higher-dimensional feature vector makes a nonlinear target linear in those features, so ordinary least squares still applies. This is the explicit version of the kernel trick — same idea, computed directly.
Cheap nonlinearity for linear/logistic models; interaction terms encode “A matters only when B.” Degree is a capacity knob — pair it with regularization, since high degrees overfit and explode at the edges.
Binning & discretization
Binning chops a continuous feature into intervals and treats each as a category. A linear model then fits a separate level per bin — a step function that captures nonlinearity and shrugs off outliers, at the cost of throwing away within-bin detail. Change the bin count to trade resolution against smoothing.
bins = 6
Edges split the range; bin(x) = which interval x falls in.
Prediction per bin = mean of y for points in that bin.
One indicator per bin lets a linear model assign an independent level to each interval — a piecewise-constant fit. Equal-width bins are simplest; quantile bins equalize counts. More bins → finer detail but noisier, fewer points each.
Robust to outliers and monotone transforms, and it linearizes nonlinear effects for GLMs (credit scorecards love it). Trade-off: information loss and edge artifacts. Gradient-boosted trees bin internally for speed (histogram methods).
Stationarity & autocorrelation
Time-series methods assume the statistics don’t drift — stationarity. The autocorrelation function (ACF) measures how a series correlates with its own past at each lag. A stationary series’ ACF decays; a trending one stays stubbornly high. Raise the persistence and add a trend to see the ACF refuse to die.
persistence φ = 0.6
ACF(k) = Σ_t (x_t − x̄)(x_{t−k} − x̄) / Σ_t (x_t − x̄)².
Stationary: mean, variance, ACF constant over time.
For AR(1), ACF(k) = φᵏ — a clean geometric decay, faster for small φ. A trend injects a slowly varying mean, so distant points stay correlated and the sample ACF decays barely at all: the tell-tale of non-stationarity.
Check stationarity before ARIMA-type models; difference or detrend to achieve it. The ACF/PACF shapes diagnose model order; spurious regression between two trending series is a classic non-stationarity trap.
Autoregressive (AR) models
An autoregressive model predicts the next value as a weighted sum of recent values plus noise. AR(1) gives smooth persistence or mean-reversion; AR(2) can oscillate. The coefficients decide whether the series settles, cycles, or blows up — stay inside the stationarity region. Tune φ₁ and φ₂.
φ₁
φ₂
AR(p): x_t = φ₁x_{t−1} + … + φ_p x_{t−p} + ε_t.
Stationary if the roots of 1 − φ₁z − φ₂z² lie outside the unit circle.
The series is a linear recurrence driven by noise; its long-run behavior is set by the characteristic roots. Complex roots → damped oscillations (pseudo-cycles); real roots in range → smooth decay; outside the region → the variance diverges.
The “AR” in ARIMA; fit by least squares or Yule–Walker. PACF cuts off at lag p, revealing the order. The same autoregressive principle underlies sequence models that predict the next token from previous ones.
Trend + seasonality decomposition
Many series are a sum of a slow trend, a repeating seasonal pattern, and leftover noise. Classical decomposition pulls them apart: a moving average estimates the trend, averaging by phase gives the season, and what remains is the residual. Adjust the trend and seasonal strength and watch the parts separate.
trend
seasonality
Additive: x_t = T_t + S_t + R_t.
T_t = moving average; S_t = mean of (x−T) by phase; R_t = remainder.
A centered moving average over one season cancels the seasonal swing, leaving the trend. Subtract it and average the leftovers within each phase to get the repeating season; whatever is still unexplained is residual (ideally white noise).
STL/seasonal decomposition for EDA, deseasonalizing before modeling, and anomaly detection on the residual. Multiplicative variants suit series whose seasonal swing scales with level (e.g., sales).
Forecasting & uncertainty
Forecasting projects the pattern forward — and honest forecasts come with widening uncertainty, since errors compound the further out you go. Exponential smoothing tracks the level with a memory knob α, then extrapolates. Tune α and the horizon to see the fit and the fanning prediction interval.
α = 0.3horizon
Level: ℓ_t = α·x_t + (1−α)·ℓ_{t−1}.
Forecast x̂_{t+h} = ℓ_t; interval ∝ σ·√h.
Exponential smoothing is a weighted average with geometrically decaying weights — small α is smooth/sluggish, large α is reactive. Multi-step errors accumulate roughly like a random walk, so the interval grows with √h.
Exponential smoothing / Holt–Winters and ARIMA are strong, cheap baselines that often rival fancy models. Always report intervals, validate with rolling/backtesting splits, and beware leakage from using the future.
Confounding & Simpson’s paradox
Correlation can point the wrong way. A lurking confounder that drives both variables can make an overall trend slope opposite to the trend within every subgroup — Simpson’s paradox. The fix is to condition on the confounder. Toggle the adjustment and watch the misleading overall line flip.
Pooled slope mixes within- and between-group variation.
Adjusted: estimate the X→Y slope within each level of the confounder Z.
If Z raises both X and Y, points from high-Z groups sit up-and-right, faking a positive pooled slope even when X lowers Y inside each group. Stratifying by Z removes the between-group shift, exposing the true relationship.
Why “control for covariates” matters; naive correlations in observational data mislead (hospital data, A/B analysis with mix shifts). The right adjustment set comes from causal reasoning, not just throwing in every variable.
Randomization (RCTs)
The cleanest way to a causal effect is to randomize who gets the treatment. Randomization makes the treated and control groups statistically identical in everything — measured or not — so any outcome difference is the effect. Without it, sicker patients self-select and bias the naive comparison. Toggle randomization.
ATE = E[Y | do(T=1)] − E[Y | do(T=0)].
Randomize ⇒ T ⊥ confounders ⇒ ATE = E[Y|T=1] − E[Y|T=0].
If treatment is assigned by coin flip, the confounder Z has the same distribution in both arms, so the naive mean difference is unbiased for the causal effect. In observational data Z differs across arms, and that imbalance contaminates the estimate.
A/B tests are RCTs — the gold standard for causal product decisions. When randomization is impossible, lean on matching, propensity weighting, or natural experiments — all trying to mimic balance.
Confounding & the backdoor
Causal graphs make the logic explicit. A confounder Z opens a “backdoor” path X ← Z → Y that leaks non-causal association into the X–Y link. The do-operator asks the interventional question, and adjusting for Z blocks the backdoor so the remaining association is the true effect. Toggle the adjustment on the DAG.
Backdoor adjustment: P(Y | do(X)) = Σ_z P(Y | X, Z=z)·P(Z=z).
Valid if Z blocks all backdoor paths and isn’t a descendant of X.
do(X) erases the arrows into X, deleting the Z→X edge and with it the backdoor. Conditioning on Z reproduces that severed graph from observational data, so the conditional association equals the causal effect.
The backdoor criterion tells you exactly which variables to control for — and warns against adjusting for colliders or mediators, which can create bias. Foundational for causal ML, uplift modeling, and policy evaluation.
Instrumental variables
Sometimes the confounder can’t be measured — so adjustment is impossible. An instrument Z is a variable that nudges the treatment X but affects the outcome Y only through X. It carves out variation in X that’s unrelated to the hidden confounder, letting you recover the true effect. Tune the instrument’s strength.
instrument strength
β_IV = Cov(Z, Y) / Cov(Z, X).
Assumes Z→X (relevance) and Z affects Y only via X (exclusion).
Plain OLS of Y on X is biased because the hidden U moves both. Z is independent of U, so projecting Y and X onto Z strips out the confounded part; their ratio isolates the causal slope X→Y.
Used where randomization fails: economics (price/demand), Mendelian randomization (genes as instruments), encouragement designs. Beware weak instruments (tiny Cov(Z,X) → exploding variance) and exclusion violations.
Bayesian inference & the posterior
Bayesian inference treats the unknown as a distribution, not a point. Start with a prior belief, multiply by the likelihood of the data, and you get the posterior — a sharper belief. Here we infer an unknown mean: add data and watch the posterior pull away from the prior toward the sample and tighten.
prior width τ
p(μ | x) ∝ p(x | μ)·p(μ). Conjugate normal:
posterior precision = 1/τ² + n/σ²; mean = weighted avg of prior & data.
Multiplying two Gaussians gives a Gaussian whose precision is the sum of precisions — so each data point adds 1/σ² of certainty. The posterior mean is a precision-weighted blend of the prior mean and the sample mean.
The engine of Bayesian modeling: uncertainty-aware estimates, sequential updating, and regularization (MAP = posterior mode). Conjugate priors give closed forms; otherwise we sample (next cards) or use variational approximations.
MCMC: Metropolis sampling
When the posterior has no closed form, we sample it. The Metropolis algorithm wanders parameter space: propose a nearby step, and accept it with a probability set by the density ratio — always uphill, sometimes downhill. The collected samples trace the target distribution. Step the chain and tune the proposal width.
proposal width
Propose x′ = x + ε. Accept with prob min(1, p(x′)/p(x)).
Stationary distribution of the chain = target p.
The accept rule satisfies detailed balance, so the chain’s long-run distribution is exactly p — even though p is known only up to a constant (the ratio cancels it). Step size trades acceptance against exploration; ~25–50% acceptance is healthy.
The workhorse for Bayesian posteriors without conjugacy. Variants (Hamiltonian Monte Carlo, NUTS in Stan/PyMC) use gradients to move farther per step. Watch for burn-in, autocorrelation, and multimodality (chains can get stuck).
Gibbs sampling
Gibbs sampling is MCMC for multivariate targets: instead of moving in all directions at once, it updates one variable at a time from its conditional distribution given the rest. The chain takes axis-aligned steps that, together, fill out the joint. Step it on a correlated 2-D Gaussian and watch the staircase trace the ellipse.
correlation ρ
Draw x | y ~ p(x | y), then y | x ~ p(y | x), repeat.
For bivariate normal: x | y ~ N(ρy, 1−ρ²).
Each conditional is the exact stationary update for that coordinate, so every move is always accepted (acceptance = 1). The sweep over all coordinates leaves the joint invariant — a special case of Metropolis–Hastings with conditional proposals.
Natural when conditionals are easy but the joint isn’t — LDA topic models, hierarchical Bayesian models, Boltzmann machines. High correlation slows mixing (narrow staircase); blocked or collapsed Gibbs helps.
Posterior predictive & uncertainty
Bayesian regression doesn’t fit one line — it keeps a whole distribution over lines consistent with the data. The posterior predictive averages over them, so its uncertainty band is tight where data is dense and fans out where you extrapolate. Add points and watch the plausible lines converge near the data and spread beyond it.
prior strength α
Posterior over weights: Σ = (αI + βΦᵀΦ)⁻¹, m = βΣΦᵀy.
Predictive variance = 1/β + φ(x)ᵀ Σ φ(x).
With a Gaussian prior and likelihood the weight posterior is Gaussian; each prediction integrates over it, adding the parameter uncertainty φᵀΣφ to the noise 1/β. Far from data φᵀΣφ grows — hence the widening fan.
Honest uncertainty for decision-making and active learning; Gaussian processes are the kernelized limit. Deep ensembles and MC-dropout approximate this predictive spread for neural nets.
Group fairness metrics
A classifier can look accurate overall yet treat groups differently. Fairness is measured by comparing rates across groups: demographic parity wants equal selection rates, equalized odds wants equal true- and false-positive rates. Move the threshold and watch the gaps between two groups (with different base rates) open and close.
threshold
Demographic parity: P(ŷ=1|A) = P(ŷ=1|B).
Equalized odds: equal TPR and FPR across groups.
Predictive parity: equal precision.
When base rates differ between groups, a single threshold gives different selection rates and error rates — so the criteria disagree. Each encodes a different value judgment about what “fair” means; there is no universally correct one.
Choose the criterion from context (lending, hiring, risk). Report multiple metrics and disaggregate by group; aggregate accuracy hides disparities. The next cards show why you can’t satisfy all criteria at once.
The fairness impossibility
A sharp, sobering result: when two groups have different base rates, a classifier cannot be calibrated and have equal false-positive and false-negative rates at the same time — except in trivial cases. Improving one fairness metric necessarily worsens another. Widen the base-rate gap and watch the unavoidable trade-off grow.
base-rate gap
If base rates p_A ≠ p_B, you cannot have calibration AND equal FPR AND equal FNR simultaneously (Kleinberg et al.; Chouldechova).
Calibration ties a score to a true probability; with unequal prevalences the same score implies different precision/error balances per group. Forcing equal error rates then breaks calibration — the constraints are jointly satisfiable only when the base rates match.
This forced the field to pick which fairness notion matters per application, and to be explicit about the cost. It also explains heated debates (e.g., COMPAS): two sides each invoked a different, individually reasonable, criterion.
Where bias comes from
A fair-looking algorithm can still produce unfair outcomes if the data is biased. Under-sampling a group or systematically mislabeling it skews the learned boundary against that group — garbage in, discrimination out. Cycle through bias sources and watch the decision boundary tilt away from the truth.
The model fits P(y|x) in the training distribution — if that distribution is skewed (sampling / label bias), so is the boundary, even with a “neutral” learner.
Under-representation lets the majority group dominate the loss, pulling the boundary toward majority-optimal. Label bias (historical prejudice in y) directly teaches the model the prejudice. Neither is fixed by a better optimizer.
Bias enters via historical labels, sampling, measurement, and proxy features (a redundant encoding of a protected attribute). Auditing data and provenance matters as much as the model; “fairness through unawareness” (just dropping the attribute) usually fails.
Mitigation strategies
Bias can be addressed at three stages: pre-processing (reweight or repair the data), in-processing (add a fairness constraint to training), and post-processing (set group-specific thresholds). Post-processing is simplest: pick a separate cutoff per group so their true-positive rates match. Cycle the method and watch the gap close.
Post-processing: choose t_A, t_B so TPR_A(t_A) = TPR_B(t_B).
In-processing: minimize loss s.t. fairness constraint. Pre: reweight samples.
Because each group has its own score distribution, one threshold yields unequal rates; allowing group-specific cutoffs restores equality on the chosen metric. The cost is usually a small accuracy drop and the legal/ethical question of explicit group-aware thresholds.
Tools: reweighing, adversarial debiasing, constrained optimization (Fairlearn), and threshold optimization. Pick the stage by access and constraints; always re-measure — mitigation on one metric can worsen another (per the impossibility result).
Kernel density estimation
To estimate a distribution without assuming its shape, drop a small bump (kernel) on each data point and add them up — that’s a kernel density estimate. The bandwidth sets the bump width: too narrow and you get spiky noise, too wide and you wash out real structure. Slide the bandwidth to find the sweet spot.
bandwidth h = 0.30
p̂(x) = (1/nh) Σᵢ K((x − xᵢ)/h).
K = Gaussian kernel; h = bandwidth.
Each point contributes a unit of probability mass spread by the kernel; summing gives a smooth estimate. Bandwidth is the bias–variance knob: small h → low bias, high variance (wiggly); large h → smooth but biased. Rules like Silverman’s pick a default.
Nonparametric density for visualization, generative sampling, and anomaly scoring (low density = unusual). Curse of dimensionality limits it in high dimensions; it’s the kernel cousin of histograms and the basis of mean-shift.
Density-based anomalies
An anomaly is a point in a sparse neighborhood — far from its neighbors. Scoring each point by its distance to its k-th nearest neighbor flags the loners. You set the contamination rate: what fraction of the data to treat as anomalous. Slide it and watch the flagged outliers grow inward from the fringe.
contamination = 8%
score(x) = distance to k-th nearest neighbor (or 1/local density).
Flag the top “contamination” fraction by score.
In dense regions neighbors are close, so the k-NN distance is small; isolated points have large distances. Local Outlier Factor refines this by comparing a point’s density to its neighbors’, catching anomalies relative to varying local density.
Fraud, intrusion, defect, and health monitoring. Unsupervised, so set the threshold from a contamination prior or a validation set. Distance methods struggle in high dimensions (everything is far) — reduce dimensionality first.
One-class boundary
When you only have “normal” examples, learn a boundary that wraps around them; anything outside is anomalous. A one-class SVM (or density level-set) draws that envelope, and a parameter ν controls how tightly — effectively, what fraction of training points to leave outside. Tighten ν and watch the envelope shrink and reject more points.
ν (tightness) = 0.10
One-class SVM: separate data from the origin in feature space with max margin; ν bounds the outlier fraction.
Equivalent to a density level-set: keep p(x) > threshold.
Choosing a density threshold so that a ν-fraction of points fall below it carves a level-set — the support of the normal data. With an RBF kernel the boundary hugs the data’s shape; ν trades coverage against tightness.
Novelty detection when only normal data is available (machine health, clean-vs-attack). RBF one-class SVM, SVDD, and deep-SVDD; pick ν from tolerated false-alarm rate. Sensitive to feature scaling and kernel width.
Isolation forest
A clever flip: instead of modeling normal data, isolate anomalies. Split the space with random cuts; outliers, sitting alone, get separated in just a few cuts, while inliers need many. The average number of cuts to isolate a point becomes its anomaly score. Regenerate the random trees and watch the loners light up.
Build random trees with random axis + split. Path length h(x) = depth to isolate x.
score(x) = 2^{− E[h(x)] / c(n)} (near 1 → anomaly).
Random partitioning separates sparse, far-flung points quickly (short paths) and dense-cluster points slowly (long paths). Averaging path length over many random trees gives a stable score — no distance or density model needed, and it scales to large data.
Isolation Forest is a fast, popular unsupervised detector (linear time, low memory), strong in higher dimensions where distance methods fail. Extended Isolation Forest fixes axis-aligned bias with oblique cuts.
Online / streaming learning
When data arrives one example at a time — too much to store — you learn on the fly, updating the model per sample and discarding it. The perceptron is the classic: see a point, and only nudge the weights if you got it wrong. Stream points in and watch the boundary snap toward the right answer as mistakes accumulate then stop.
Predict ŷ = sign(w·x). On a mistake: w ← w + y·x.
Online GD: w ← w − η∇ℓ_t(w) per example.
Each mistake rotates w toward the misclassified point’s correct side. For separable data the perceptron makes at most (R/γ)² mistakes (margin γ, radius R) — a bound independent of how many points stream by.
Streaming/online SGD scales to data that won’t fit in memory and adapts to fresh data; the basis of large-scale and real-time learners. One pass, constant memory — trades some statistical efficiency for huge practical scale.
Active learning
Labels are expensive; not every example is equally worth labeling. Active learning lets the model choose what to ask about — typically the points it’s most uncertain about, sitting right on the decision boundary. Those carry the most information, so accuracy climbs faster per label. Switch strategy and watch uncertainty beat random.
label budget = 8
Uncertainty sampling: query x* = argmin |score(x)| (closest to the boundary), or argmax entropy of p(y|x).
Points near the boundary are where the model is unsure, so their labels most reduce uncertainty about where the boundary lies. Random labeling wastes budget on already-confident points; uncertainty sampling concentrates it where it matters.
Cuts labeling cost in annotation-heavy settings (medical, NLP). Strategies: uncertainty, query-by-committee, expected model change. Caveats: sampling bias and batch diversity — pure uncertainty can pick redundant near-duplicates.
Concept drift
In the real world the thing you’re predicting changes — spam tactics evolve, tastes shift. A model trained once and frozen slowly goes stale as the true boundary moves out from under it. An adaptive model that keeps learning (sliding window or decay) tracks the change. Advance time and watch the static model decay while the adaptive one holds.
time
Distribution P_t(x,y) changes with t. Static model fixed at t=0; adaptive refits on a recent window / with forgetting.
A frozen decision rule is optimal only for the distribution it was trained on; as P_t drifts, its error grows. Re-estimating on recent data (or down-weighting old data) keeps the rule aligned with the current distribution.
Critical in production: monitor for drift, retrain on rolling windows, use forgetting factors, and detect change points (ADWIN, DDM). Trade-off: adapt too fast and you chase noise; too slow and you lag real change.
Regret & online optimization
How do you measure an online learner with no fixed dataset? By regret: your total loss minus that of the best single action you could have committed to in hindsight. A good algorithm is “no-regret” — its regret grows slower than time, so average regret vanishes. Tune the step size and watch the cumulative regret bend sublinearly.
learning rate
Regret_T = Σ_t ℓ_t(x_t) − min_x Σ_t ℓ_t(x).
Online GD achieves Regret_T = O(√T) ⇒ Regret_T/T → 0.
With convex losses and step size η ∝ 1/√T, the per-step suboptimality telescopes to a √T bound. “No-regret” means that in the long run you do as well as the best fixed decision — even against an adversarial sequence.
The theoretical backbone of online learning, bandits, boosting, and game-playing (no-regret dynamics converge to equilibria). Adagrad/Adam descend from online convex optimization with adaptive step sizes.
Gaussian process priors
A Gaussian process is a distribution over functions: pick any set of inputs and the function values are jointly Gaussian, with covariance set by a kernel. Before seeing data, sampling the GP gives random smooth curves whose wiggliness the kernel’s length-scale controls. Resample and stretch the length-scale to feel the prior.
length-scale ℓ = 0.6
f ~ GP(m(·), k(·,·)). RBF kernel: k(x,x′) = exp(−(x−x′)² / 2ℓ²).
Any finite set of f-values is multivariate Gaussian.
Build the covariance matrix K over a grid, Cholesky-factor it (K = LLᵀ), and f = L·z with z ~ N(0,I) is a sample. The kernel’s length-scale sets how fast correlation decays with distance — hence the smoothness.
GPs give flexible nonparametric models with built-in uncertainty. The kernel encodes prior beliefs (smoothness, periodicity); its hyperparameters are tuned by maximizing the marginal likelihood. Cost is O(n³) — the main scaling limit.
GP regression: the posterior
Condition the GP on observations and you get a posterior: a mean curve that passes near the data and an uncertainty band that pinches to zero at the points and balloons between and beyond them. That calibrated “I don’t know here” is the GP’s superpower. Add points and watch the band collapse around them.
length-scale ℓ
mean(x*) = K*ᵀ(K+σ²I)⁻¹ y.
var(x*) = k(x*,x*) − K*ᵀ(K+σ²I)⁻¹ K*.
Joint Gaussianity of training and test values means conditioning is a closed-form Gaussian. The variance drops to (near) zero where data is observed and recovers the prior variance far away — hence the fanning band.
Exact, calibrated regression with uncertainty — ideal for small data, sensor fusion, and as the surrogate in Bayesian optimization. Scaling to big n needs sparse/inducing-point approximations (SVGP).
Kernels encode assumptions
The kernel is where you tell the GP what kind of function to expect. An RBF kernel says “smooth”; a periodic kernel says “repeating”; a linear kernel says “straight lines.” Same machinery, totally different priors. Switch kernels and watch the sampled functions change character.
length-scale ℓ
RBF: exp(−d²/2ℓ²). Periodic: exp(−2 sin²(πd/p)/ℓ²). Linear: x·x′.
Sums/products of kernels combine behaviors.
A valid kernel is any symmetric positive-semidefinite function; it defines the inner product (similarity) in feature space. Its shape dictates which functions get high prior probability — the inductive bias of the GP.
Kernel design is GP modeling: combine RBF + periodic + linear to capture trend, seasonality, and noise (as in Mauna Loa CO₂). The same kernels power SVMs and kernel ridge regression.
Bayesian optimization
To optimize an expensive black-box function with as few evaluations as possible, fit a GP surrogate and let an acquisition function decide where to sample next — balancing exploiting the current best against exploring uncertain regions. Step it and watch the GP home in on the optimum in just a handful of evaluations.
Fit GP to evaluated points. Acquisition (UCB): a(x) = μ(x) + κ·σ(x).
Next point = argmax a(x); evaluate, repeat.
μ(x) drives exploitation, σ(x) drives exploration; κ sets the balance. Sampling where a(x) is largest gathers the most useful information, so the optimum is found in far fewer evaluations than grid or random search.
The standard for hyperparameter tuning and experiment design (expensive objectives). Expected Improvement is a common alternative acquisition; Thompson sampling is the Bayesian-bandit cousin.
Entropy & surprise
Entropy measures average surprise: a rare event carries −log p bits of information, and entropy is the expected value over the distribution. It peaks when outcomes are equally likely (maximum uncertainty) and vanishes when one outcome is certain. Slide a coin’s bias and watch its entropy rise to one bit at 50/50, then fall.
P(heads) = 0.50
surprise(x) = −log₂ p(x) bits.
H(p) = − Σ p(x) log₂ p(x) = E[surprise].
Information should add for independent events and grow as probability shrinks — forcing a logarithm. Averaging −log p over the distribution gives entropy, the minimum bits per symbol needed to encode it (Shannon).
Entropy underlies cross-entropy loss, decision-tree splits (information gain), and exploration bonuses. Maximum-entropy modeling picks the least-committal distribution consistent with constraints.
KL divergence
KL divergence measures how many extra bits you pay by coding data from p using a code optimized for q — the cost of believing the wrong distribution. It’s zero only when q = p, always non-negative, and asymmetric: KL(p∥q) ≠ KL(q∥p). Slide q toward p and watch both divergences fall to zero (at different rates).
q mean q width
KL(p∥q) = Σ p(x) log( p(x)/q(x) ) ≥ 0.
= cross-entropy(p,q) − entropy(p).
Coding p-data with a q-optimal code costs cross-entropy H(p,q) bits; the unavoidable minimum is H(p). The excess H(p,q) − H(p) = KL(p∥q) ≥ 0 by Gibbs’ inequality, with equality iff q = p.
Minimizing cross-entropy = minimizing KL to the data distribution. KL appears in the VAE ELBO, variational inference, policy regularization (RLHF), and distillation. Forward vs reverse KL gives mean-seeking vs mode-seeking fits.
Mutual information
Mutual information is how much knowing one variable reduces uncertainty about another — the shared information between X and Y. It’s zero exactly when they’re independent and grows as they become more dependent, capturing nonlinear relationships that correlation can miss. Crank the dependence and watch the cloud tighten and MI rise.
dependence ρ = 0.0
I(X;Y) = H(X) − H(X|Y) = Σ p(x,y) log( p(x,y)/(p(x)p(y)) ).
= KL( joint ∥ product of marginals ).
If X and Y are independent the joint equals the product of marginals, so the KL (and MI) is 0. Any dependence makes the joint differ, and MI measures it. For a bivariate Gaussian, I = −½ log(1 − ρ²).
Feature selection (keep high-MI features), the information bottleneck, representation learning (InfoNCE maximizes a MI bound), and independence testing. Unlike correlation, MI catches nonlinear dependence.
Coding, compression & MDL
Shannon’s source-coding theorem says entropy is the floor on bits per symbol: give frequent symbols short codes and rare ones long codes, and the best average length approaches the entropy. The same idea — the minimum description length — frames learning as compressing data with the model. Skew the frequencies and watch the optimal code lengths shift.
skew = 0.50
Optimal length(x) ≈ −log₂ p(x). Average length ≥ H(p) (source-coding theorem).
MDL: minimize description(model) + description(data | model).
Assigning −log₂ p(x) bits to each symbol makes the expected length equal to entropy; no prefix code can beat it (Kraft inequality). MDL turns this into model selection — the best model compresses the data most, balancing fit and complexity.
Arithmetic/Huffman coding, the MDL principle for model selection (a formal Occam’s razor), and the deep tie between compression and prediction (a good language model is a good compressor).
Grid vs random search
When only a few hyperparameters actually matter, random search beats grid search at the same budget: a grid wastes trials on a coarse lattice, repeating the same values of the important knob, while random sampling probes many distinct values of it. Slide the budget and watch random cover the important axis more densely.
budget = 16
Grid: k points per dim → only k distinct values each.
Random: n points → up to n distinct values on every dim.
If performance depends mainly on one of D dimensions, a grid of n = kᴰ points still tries only k values of it, while random search tries ≈ n. With effective dimensionality low, random search’s coverage of the relevant axis is far higher per trial.
Bergstra & Bengio’s classic result; random search is a strong baseline before Bayesian optimization. For expensive models, combine with early stopping (next card).
Successive halving & Hyperband
Most hyperparameter configs are bad — so don’t train them all to the end. Successive halving gives every config a little budget, keeps the top half, doubles their budget, and repeats. Promising configs get the compute; losers die early. Step the rounds and watch the field thin while survivors sharpen.
Start with n configs, budget b. Each round: keep top ½, multiply budget ×2. Total cost ≈ n·b·log n.
Spreading a fixed budget thinly first, then concentrating it on survivors, evaluates far more configs than full training would — at the cost of noisy early estimates. Hyperband hedges the unknown n-vs-b trade-off by running several brackets.
The standard for tuning deep nets where each run is expensive; implemented in Optuna, Ray Tune, Keras Tuner. Pairs naturally with random/Bayesian config sampling.
Neural architecture search
Why hand-design architectures? NAS searches the space of architectures automatically — a controller proposes designs, each is scored, and the search climbs toward high-performing regions. Step the search and watch it explore a depth×width landscape and converge on a strong architecture.
max over architectures a: validation accuracy(a).
Methods: RL controller, evolution, or differentiable search (DARTS) over a supernet.
The search space (operations, connections, depth/width) is huge and discrete, so NAS uses proxies — weight sharing, low-fidelity training, or gradient relaxation — to score candidates cheaply and steer sampling toward better designs.
Produced EfficientNet, NASNet, and hardware-aware mobile models. Cost was once enormous; weight-sharing (ENAS) and differentiable NAS (DARTS) made it practical.
Model selection & validation
Tuning is itself a fitting procedure — and you can overfit the validation set. As model complexity rises, training error keeps falling but validation error traces a U: too simple underfits, too complex overfits. Pick the bottom of the U, but report on a held-out test set, since the validation minimum is optimistic. Slide complexity to find it.
complexity = 5
Choose hyperparameter λ = argmin validation error(λ).
Nested CV: inner loop selects λ, outer loop gives an unbiased estimate.
Validation error ≈ bias² + variance: bias falls and variance rises with complexity, producing the U. Selecting on validation and then reporting that same score is optimistically biased — nested CV or a final test set corrects it.
Underlies all hyperparameter tuning and early stopping. Beware leakage and selection bias when the validation set is small or reused many times across experiments.
Censoring & time-to-event
Survival analysis models the time until an event — failure, churn, death — and its defining challenge is censoring: for some subjects the study ends (or they drop out) before the event happens, so you only know their time exceeds some value. Throwing them away biases everything. Toggle censoring to see the information you’d lose.
Observe (Tᵢ, δᵢ): time Tᵢ and indicator δᵢ (1 = event, 0 = censored).
Censored ⇒ true event time > Tᵢ.
Dropping censored subjects, or treating censoring as the event, both bias estimates. Survival methods use the partial information correctly: a censored subject contributes “survived at least this long” to the likelihood.
Customer churn, hardware reliability, clinical trials, credit default — anywhere outcomes unfold over time and follow-up is incomplete. The basis for the Kaplan–Meier and Cox methods that follow.
Kaplan–Meier estimator
The Kaplan–Meier curve estimates the survival function — the probability of surviving past each time — directly from censored data. At every observed event it steps down by the fraction who failed among those still at risk; censored subjects simply leave the risk set without causing a drop. Slide the time cursor along the staircase.
time t = 0.0
Ŝ(t) = ∏_{tᵢ ≤ t} (1 − dᵢ / nᵢ),
dᵢ = events at tᵢ, nᵢ = at risk just before tᵢ.
Surviving past t means surviving each event time up to t; multiplying the conditional survival (1 − d/n) at each gives the product-limit estimator. Censored subjects stay in nᵢ until their censoring time, then exit — using their partial info without biasing the curve.
The standard nonparametric survival estimate; log-rank tests compare two KM curves. A natural target for survival-aware models (random survival forests, DeepSurv).
Hazard & survival functions
The hazard is the instantaneous risk of the event right now, given survival so far. It and the survival function are two views of the same distribution: integrate the hazard and exponentiate the negative to get survival. A rising hazard (wear-out) bends survival down faster over time. Shape the hazard and watch survival respond.
hazard shape = 1.0
h(t) = f(t)/S(t); S(t) = exp(−∫₀ᵗ h(u) du).
Weibull: h(t) = (k/λ)(t/λ)^{k−1}.
The cumulative hazard H(t) = ∫ h accumulates risk; survival is exp(−H). Constant hazard (k = 1) gives exponential survival (memoryless); k > 1 means risk grows with age, steepening the decline.
Parametric survival models choose a hazard family (exponential, Weibull, log-normal); discrete-time models predict per-interval hazards with a neural net. The hazard view also links to point processes.
Cox proportional hazards
The Cox model relates covariates to survival without committing to a hazard shape: each covariate multiplies a shared baseline hazard by exp(βx). The key assumption is proportionality — the hazard ratio between groups is constant over time, so their survival curves never cross. Slide a covariate and watch the curve shift, keeping its shape.
covariate x = 0.0
h(t | x) = h₀(t)·exp(βᵀx). Hazard ratio = exp(βΔx).
S(t | x) = S₀(t)^{exp(βx)}.
Because the baseline h₀(t) factors out, β is estimated from the order of events (partial likelihood) without modeling h₀. The covariate just rescales the cumulative hazard, so survival curves are powers of the baseline — and stay ordered.
The workhorse of clinical and reliability modeling; coefficients read as log-hazard-ratios. Check the proportional-hazards assumption (crossing curves break it); DeepSurv learns βᵀx as a neural net.
Self-training & pseudo-labels
With a handful of labels and a flood of unlabeled data, self-training bootstraps: fit a model on the labels, let it label the unlabeled points it’s most confident about, add those pseudo-labels, and refit. Done carefully it sharpens the boundary using free data; done carelessly it amplifies its own mistakes. Step the rounds and watch confident points get adopted.
1) fit on labeled. 2) pseudo-label x where confidence > τ. 3) add to training set, refit. Repeat.
Confident predictions on unlabeled data are likely correct, so adding them effectively grows the labeled set and pulls the boundary into low-density regions. The risk: confirmation bias — early errors get reinforced, so confidence thresholds and re-checking matter.
Pseudo-labeling and “noisy student” training are simple, strong semi-supervised baselines, especially in vision. FixMatch combines pseudo-labels with consistency (next cards).
Label propagation
If similar points should share labels, build a graph connecting nearby points and let the few known labels diffuse along the edges — like color spreading through a network. Labels flow easily within a dense cluster but barely cross the sparse gap between clusters. Step the propagation and watch the labels flood outward.
Build affinity Wᵢⱼ = exp(−∥xᵢ−xⱼ∥²/σ²). Iterate f ← D⁻¹W f, clamping labeled nodes.
Each node’s label becomes a weighted average of its neighbors’, so labels diffuse fastest where edges are strong (dense regions) and stall across low-density gaps — encoding the cluster/manifold assumption directly in the graph.
Graph-based SSL (label propagation/spreading) is transductive and effective when data forms clear manifolds; it underlies modern graph-SSL and some GNN training. Sensitive to the graph and bandwidth σ.
Consistency regularization
A model’s prediction shouldn’t flip just because you nudged the input — a slightly augmented image is still the same class. Consistency training enforces that on unlabeled data: penalize differences between predictions on two perturbed copies. This pushes the decision boundary out of dense regions into the gaps. Raise the perturbation and watch the boundary settle into the low-density valley.
perturbation = 0.30
Loss += ∥ f(x+ε₁) − f(x+ε₂) ∥² on unlabeled x.
Encourages locally smooth, confident predictions.
Requiring invariance to perturbations forbids the boundary from cutting through clusters (where small nudges would flip labels), so it relaxes into low-density regions — the same low-density-separation principle, enforced softly.
The engine of modern semi-supervised vision: Π-model, Mean Teacher, VAT, FixMatch, MixMatch. Strong augmentation + consistency + pseudo-labels routinely match supervised accuracy with a fraction of labels.
Low-density separation
The principle tying these methods together: a good boundary should pass through regions where data is sparse, not slice through a dense cluster. Unlabeled data reveals where those gaps are. Compare a purely supervised boundary (fit to two labels) against one that uses the unlabeled density to find the true valley.
Prefer boundaries where p(x) is low (the margin lies in a density valley) — the assumption shared by S3VM, entropy minimization, and consistency methods.
Few labels under-determine the boundary; many placements fit them equally. The cluster assumption breaks the tie by favoring the placement that avoids dense regions — which unlabeled data lets you locate, even though it carries no labels itself.
The conceptual core of semi-supervised learning: unlabeled data helps only when its structure (clusters, manifolds) aligns with the labels. When it doesn’t, SSL can hurt — so validate the assumption.