What a neural network actually is

Strip away the mystique and a neural network is just a function with adjustable knobs. You feed in numbers, it multiplies them by weights, bends the result through a nonlinear squashing function, and passes that to the next layer. "Learning" means nudging the weights so the output gets closer to the answer you wanted.

Five expressions are the whole story

Everything below is built from these. A neuron computes a weighted sum, squashes it with an activation, the network chains layers into one function, a loss scores the output, and gradient descent improves the weights.

The squashing function is the key trick: without it, stacking layers would just give another linear function. The sigmoid σ maps any number into (0, 1) — handy for a probability — while its cousin tanh maps into (−1, 1) and is common in hidden layers. Both are smooth, so we can take derivatives and run gradient descent.

A real problem: XOR

We'll teach the network exclusive-or: output 1 when exactly one input is on. It's tiny — four examples — but famous, because no single straight line can separate the 1s from the 0s. That's exactly why we need a hidden layer.

2 inputs → 2 hidden → 1 output

The smallest network that can solve XOR: two inputs, a hidden layer of two tanh neurons, and one sigmoid output that reads as the probability of class 1. That's nine numbers to learn — six weights and three biases.

Compute one prediction, by hand

Let's run the input x = [1, 0] (whose target is 1) through a network with some example weights, doing the actual arithmetic. Step through it:

Backpropagation, by hand

The prediction was 0.80 but should be 1. Backprop answers one question for every weight: which way, and how much, should I move you to reduce the loss? It applies the chain rule from the output backwards. Same example, real numbers:

Stepping downhill

Backprop hands us the gradient — the direction in which the loss increases. Gradient descent does the obvious thing: take a step the opposite way, sized by the learning rate η, and repeat. Here's that idea on a single weight — the curve is the loss, the dot is the weight, the amber arrow is the downhill direction. Roll it, and watch η decide everything.

Watch the boundary form

Now the payoff. Hit Train and the network runs that exact loop — forward, loss, backprop, update — over all four points, thousands of times. The background colour is the network's output across the whole input plane; watch it bend until it carves the XOR pattern, and the loss curve fall. Reset re-randomises the weights so you can see it solve it again from a fresh start.

What the number means, and where this goes

The output 0.80 isn't arbitrary — because of the sigmoid it's a probability: the network's confidence that the answer is class 1. Threshold at 0.5 to decide. Everything you just saw scales directly: real networks use the same weighted-sum-then-nonlinearity neuron, the same loss-and-gradient-descent loop, just with millions of weights, many layers, and specialised structure.

What you saw

A neuron is a weighted sum plus a squash. A network chains them into a function with adjustable weights. A dataset and a loss define "right." The forward pass produces a prediction from real numbers; backprop assigns blame to each weight via the chain rule; gradient descent nudges them all downhill. Repeat, and a network with just two hidden neurons learns a pattern no line can split.

Every word looks at every other word

Older sequence models read words one at a time, passing a memory along. Transformers throw that out: every token looks at all the others at once and decides which are relevant. That mechanism is self-attention. It's parallel (fast to train), it handles long-range links directly, and stacked up it becomes GPT.

A three-word sequence

Our input is "the cat sat" — three tokens at positions 0, 1, 2. We'll use an embedding dimension of just 4, and a single attention head that projects down to 2 dimensions, so every vector and dot-product is small enough to read.

Words become vectors

A lookup table maps each token id to a learned vector. Similar words get similar vectors; here each word is a point in 4-dimensional space. These are the raw material everything else operates on.

Telling the model the order

Attention sees the tokens as a set — it has no built-in sense of order. So we add a positional encoding: a fixed pattern of sines and cosines, different for each position, that stamps "where" onto each vector. The input to attention is x = embedding + position.

Three projections of each token

Three learned matrices turn every token vector x into a query q = x·W_Q, a key k = x·W_K, and a value v = x·W_V. Here they map the 4-dim vectors down to 2-dim. Done for all tokens at once, that's just three matrix multiplies.

Compute one token's context, by hand

Let's compute attention for the query token "cat" (position 1): how much should it draw from "the", "cat", and "sat", and what's the result? Every number is real — step through it.

Explore who attends to whom

Every token runs that same computation, giving a full attention matrix: row = the querying token, column = how much weight it puts on each other token (each row sums to 1). Click a row (or pick a query) to see its weights and the blended output. Toggle the causal mask to forbid looking at future tokens — the rule language models use when predicting left-to-right.

Several attentions at once

One head captures one kind of relationship. Real transformers run several heads in parallel — each with its own W_Q, W_K, W_V projecting into a small subspace — then concatenate the results and mix them with one more matrix. Different heads learn different patterns (one tracks syntax, another nearby words, and so on).

Let's actually do it with two heads for the last token "sat" — the one we'll predict from — combining them with the output matrix W_O:

Attention is half of it

A transformer block wraps attention with two more ideas: a residual connection (add the input back, so information and gradients flow freely) and layer normalization (keep activations well-scaled), then a small per-token feed-forward network (MLP) with its own residual + norm. That whole block is what gets stacked.

Now the arithmetic, continuing with "sat" — every step of the block with real numbers. (LayerNorm here uses scale 1, shift 0 for clarity; real ones learn those two parameters.)

From the last token to a word

To predict what follows "the cat sat", take the processed vector at the last position, map it to one score (logit) per vocabulary word, and softmax into probabilities — using the block's actual output vector for "sat" that we computed just above. The temperature controls how sharp the choice is.

Every formula you just used

Now that you've seen each step with numbers, here are the equations behind them — essentially the whole transformer in nine lines.

And the shape of the data at every stage — the journey of "the cat sat" from words to a probability over the next word:

The loss, the gradient, and training

So far every weight was fixed. Training adjusts them. We score the prediction with cross-entropy loss — how surprised the model is by the correct next word — then nudge the weights to reduce it, exactly the gradient-descent loop from the neural-network dive.

The elegant part is the gradient at the output. For softmax followed by cross-entropy it collapses to one of the simplest expressions in all of deep learning:

That gradient flows backward through W_U, the block, attention, and the embeddings by the chain rule (backpropagation), and every matrix takes a small step downhill: W ← W − η·∂L/∂W. Below we train just the final output layer on this one example — pick a target word and watch the model learn to predict it.

From this to GPT

Everything you just traced — embed, add position, attend, MLP — is one block. Stack dozens to a hundred of them, widen the vectors from 4 to thousands, grow the vocabulary, and train on a large corpus, and the same machinery becomes a large language model.

What you saw

Words became vectors; positions were stamped in; each token produced a query, key, and value. Scores compared queries to keys, softmax turned them into attention weights, and each token's output was a weighted blend of values — its context. Multi-head, residual, norm, and an MLP complete the block; the last token's vector becomes a probability over the next word. That is a transformer.

Slide a small filter everywhere

Instead of one giant weight per pixel, a CNN learns a small filter (here 3×3) and applies the same filter at every location. That's two big wins: far fewer weights, and a pattern learned in one corner is recognized everywhere else.

A tiny image

Our image is 6×6 grayscale — dark on the left, bright on the right, so there's a single vertical edge down the middle. Pixel values are 0 (dark) to 1 (bright). The job of a good filter is to find that edge.

One output pixel, by hand

Place the 3×3 filter over a patch of the image, multiply each filter weight by the pixel beneath it, and add it all up — that single number is one pixel of the feature map. Step through it at the edge:

Now watch the filter slide across the whole image, filling in the feature map one pixel at a time:

Different filters, different patterns

The filter's weights decide what it finds. Swap them and the same image reveals different things — a vertical-edge detector lights up the middle seam; a horizontal-edge detector sees nothing here; a blur smooths; a sharpen exaggerates. Pick one and watch the feature map change.

Colour images and stacks of filters

Real images aren't one grid — a colour image has three channels (red, green, blue). So a filter isn't 3×3, it's 3×3×3: one slice per input channel. You convolve each slice with its channel and add the three results into a single feature-map pixel. And a layer has many filters, each making its own map — so the output is itself a stack whose depth equals the number of filters, ready for the next layer to convolve again.

How big is the output?

A 3×3 filter on a 6×6 image, moved one pixel at a time (stride 1) with no border added (valid padding), can sit in 4×4 positions — so the feature map is 4×4. Larger stride skips positions and shrinks the output; adding a border (same padding) keeps it the same size.

For us: (6 − 3 + 0)/1 + 1 = 4. Each output pixel summarizes a 3×3 patch — its receptive field. Stack layers and that field grows, so deep neurons "see" more of the image.

Keep the strong signals, shrink the rest

Two cheap steps follow convolution. ReLU zeroes negative responses (keep evidence "for" a pattern, drop evidence "against"). Max pooling then slides a 2×2 window and keeps only the largest value in each — shrinking the map, keeping the strongest activations, and making the result robust to small shifts.

Edges → corners → objects

One layer finds edges. Stack a second whose filters convolve the feature maps of the first, and it can combine a vertical and a horizontal edge that meet into a corner detector; a third assembles corners into shapes, and so on. Each deeper layer also sees a wider patch of the original image — a growing receptive field — so early layers handle fine detail while deep layers reason about whole objects. This edges→parts→objects hierarchy is learned, not designed.

From pixels to a label

Put it together: convolve with two filters (a vertical- and a horizontal-edge detector), ReLU, pool each map down to a single strongest response, and feed those two numbers through a softmax to choose a class — "vertical" vs "horizontal". For our image the vertical detector fires hard and the horizontal one barely, so the answer is clear.

Watch the filters become edge detectors

We hand-picked those filters. A real CNN learns them. Starting from random 3×3 weights, we train a tiny CNN (two filters → ReLU → pool → softmax) to tell vertical-edge images from horizontal ones, using the same cross-entropy loss and gradient descent as the other dives. Backprop sends the error to whichever pixel patch won the max-pool. Hit Train and watch the random filters sharpen into edge detectors while accuracy climbs.

First, the gradient for one image — worked by hand. It's the same predicted-minus-target rule, sent backward through the softmax, the max-pool, and the convolution:

Now let the network run that loop automatically over all the images:

Every formula in one place

Convolution is one weighted sum repeated across positions; everything else you've met before.

What you saw

A filter is a small grid of weights; convolution slides it everywhere and records where the pattern matched; ReLU keeps the positive evidence; pooling shrinks and stabilises; a final softmax turns the strongest responses into a label. Train with the same loss-and-gradient loop and the filters discover edges on their own. Stack many such layers and a CNN builds from edges up to whole objects.

A network with a memory

A feed-forward net maps a fixed input to an output and forgets everything between calls. An RNN adds a loop: at each step it combines the new input with its previous hidden state to produce a new hidden state. The same weights are reused at every step, so it can handle sequences of any length and a pattern learned at one position works at any other.

A task that needs memory

To prove the memory does something, we pick a task it can't fake: read a sequence of bits, and at the very end output the first bit you saw. The answer is decided at step 1 but only asked for at the end, so the network has no choice but to carry it the whole way in its hidden state.

Examples: 1011 → 1 · 0010 → 0 · 1000 → 1. The last three bits are distractions; only the first matters.

Watch the memory evolve

Here's a small RNN (2-dimensional hidden state) unrolled across a 5-step sequence. Each cell takes the input bit and the previous state, and produces a new state (shown as two bars, teal positive / purple negative). Toggle the bits and watch how the state — and the final read-out — change. Notice the first bit's long reach.

One step, reused forever

The entire recurrent layer is a single equation applied at each step, plus a read-out at the end.

Wₓ mixes in the new input, Wₕ carries the old state forward, tanh keeps the state bounded in (−1, 1). The same Wₓ, Wₕ, b act at every timestep — that's weight sharing. The final state h_T is squeezed through one more layer into a probability.

Unroll a real sequence

Take the sequence [1, 0, 1] with fixed weights and step through it — each hidden state feeds the next, and the last one decides the answer.

Teaching it to remember

Unrolled, the RNN is just a deep network that happens to reuse its weights — so we train it the same way: forward pass, cross-entropy loss, then backpropagate the error backwards through every timestep (this is backprop-through-time) and step the shared weights downhill. We train on all 16 four-bit sequences, each labelled by its first bit. Hit Train and watch accuracy climb from a coin-flip to perfect recall.

First, the gradient by hand. Backprop-through-time pushes the error from the loss back through every step — and you can watch the signal shrink on the way, the vanishing gradient in real numbers:

Now let the network run that update automatically over all 16 sequences:

Why long memories are hard

Backprop-through-time multiplies a gradient by roughly the same factor at every step it travels back — the recurrent weight times tanh's slope. If that factor is below 1, the gradient shrinks geometrically: after many steps it's effectively zero, so early inputs get almost no learning signal. (Above 1 it explodes instead.) This is why a plain RNN struggles to connect events far apart in time. Drag the per-step factor and watch the signal that survives back to the first step.

A protected path for memory

The Long Short-Term Memory cell adds a second state — the cell state C, a conveyor belt running straight through time. Three gates (each a little sigmoid that outputs values in 0–1) decide what happens to it: the forget gate erases part of the old memory, the input gate writes new candidate information, and the output gate chooses how much of the memory to reveal as this step's hidden state.

Step through one update with real numbers:

Compress, then recreate

An autoencoder is two networks back to back. The encoder maps a data point down to a few latent numbers; the decoder maps those numbers back up to a full data point. Train it to make the output match the input, and the latent code becomes a compact summary — a coordinate in a learned "map" of the data. New points on that map decode to new, plausible data.

Two dimensions, one hidden factor

Our dataset is a cloud of 2-D points lying along a gentle arc. Even though each point has an x and a y, they really vary along a single direction — position on the arc. That hidden 1-D factor is exactly what a good autoencoder with a one-number latent should discover.

Down to one number, back to two

Here the encoder takes the 2-D point through a small hidden layer to a single latent number z; the decoder takes z through another hidden layer back to a 2-D reconstruction. The whole thing is trained to minimise the squared distance between input and output.

One point through the bottleneck

Take the point x = [0.80, 0.60] and push it through a tiny autoencoder (two hidden units, one latent). Watch two numbers collapse to one and expand back to two.

Watch the shape fit the data

Now train the full autoencoder by gradient descent on the reconstruction loss. The line drawn here is the decoded manifold — everything the decoder produces as the latent z sweeps across its range. Start random and it's a tangled stub; as training proceeds it stretches and bends until it threads right through the data cloud. That curve is the 1-D structure the network has discovered.

First, the gradient by hand — continuing the point from chapter 4. The reconstruction error flows back through the decoder, funnels into the single latent, and on into the encoder:

Now let it run that update over all 30 points and watch the manifold settle onto the data:

Generate by choosing a code

Once trained, the decoder is a generator: hand it a latent number and it produces a data point. Drag the slider to move along the latent axis and watch the generated point glide along the learned arc — including spots between the original samples. That's the model inventing plausible new data.

A latent you can sample

To generate from scratch we need to know which latents are valid. A variational autoencoder arranges for the latents to fill a standard bell curve. Its encoder outputs not one z but a mean μ and spread σ; we sample z from that little Gaussian, and a KL term in the loss gently pulls every (μ, σ) toward (0, 1). Sampling is made differentiable by the reparameterization trick.

Why bother? A plain autoencoder scatters its codes wherever is convenient, leaving stretches of the latent axis that decode to nothing — so you can't just pick a random z. Here are the trained autoencoder's actual codes versus the bell the VAE packs them into (train in chapter 5 first to see them spread):

That KL term isn't arbitrary — it's the exact closed form of the gap between the encoder's Gaussian 𝒩(μ, σ²) and the prior 𝒩(0, 1). Here is the integral behind it:

And the reparameterization that lets us sample z while keeping the gradient flowing:

Watch the codes fill the bell

Now train a real VAE. Same decoder as before, but the encoder outputs a mean μ and spread σ for every point; we sample z = μ + σ·ε, reconstruct, and add the KL term to the loss. Hit Train and watch two things at once: the blue manifold settles onto the data (reconstruction), while below, the codes slide inward to fill the standard bell (the KL term doing its job). The amber points are decoded from z drawn out of pure 𝒩(0,1) noise — once the bell is filled, they land cleanly on the data.

First, the two gradients that make a VAE different from a plain autoencoder — the path through the reparameterized sample and the KL term — worked by hand (continuing μ = 0.30, log σ² = −1.20, ε = 0.70 from the previous chapter):

Those are exactly the updates the trainer applies every step. Run it and watch them accumulate — reconstruction sharpening the fit, KL shaping the codes:

And the codes themselves — each point's encoded mean μ — drifting from wherever they started into the shape of the target bell:

What you saw

An autoencoder squeezes data through a bottleneck and learns to rebuild it, and in doing so discovers the data's hidden structure — the decoded manifold that threads through the cloud. Decoding a chosen latent generates new data; the VAE makes that latent space samplable with a probabilistic encoder, the reparameterization trick, and a KL term. From recognition to creation, it's the same machinery — forward pass, a loss, gradient descent — pointed at a new goal.

Learning from reward, not answers

The loop is simple: the agent sees a state, picks an action, and the environment hands back a reward and the next state. Repeat. The agent's job is to choose actions that maximise total reward over time — which forces it to think ahead, since a small reward now can be worth less than reaching a big one later.

A goal, a pit, and two walls

The environment is a 5×5 grid. The agent starts bottom-left (S) and wants the goal top-right (reward +1). One cell is a pit (reward −1) that ends the episode badly; two cells are walls it can't enter. Every ordinary step costs a small −0.02, so wandering is mildly punished. From any cell it can move up, right, down or left; bumping a wall or the edge just leaves it in place. An episode ends at the goal or the pit.

Valuing the future

To choose well the agent must value the future, not just the next reward. The return from a moment is the sum of all rewards after it, with later ones discounted by γ (here 0.9) so nearer reward counts for more:

Worth doing once with numbers — it's exactly why the agent ends up preferring short routes:

It can't know the return ahead of time, so it estimates it. The action-value Q(s, a) is the return the agent expects if it takes action a in state s and then keeps acting well. Given Q, the policy is just: in each state, take the highest-valued action.

One estimate built on the next

Q-learning learns those values straight from experience, one step at a time. After taking action a in state s, landing in s′ with reward r, it nudges Q(s, a) toward a fresher estimate — the reward just seen plus the discounted value of the best action available next:

The bracket is the temporal-difference error δ: the gap between the old estimate and the new one. α (here 0.5) sets how big a correction each experience makes. The trick is that Q(s, a) is improved using Q(s′, ·) — an estimate leaning on another estimate. That bootstrapping is exactly what lets value flow backward from the goal.

Value seeps back from the goal

Two updates with real numbers (γ = 0.9, α = 0.5) show how a cell with no value at all becomes valuable — purely because of what it leads to.

Watch the map and the arrows form

Now let it learn by trial and error. Each cell is tinted by its value (green good, red bad), the arrow is the current best action there, and the amber dot is the agent. Hit Train: value spreads outward from the goal, the arrows organise into a route that skirts the pit, and the return-per-episode curve climbs from negative (falling in the pit, wandering) toward the reward.

Why the agent rolls the dice

If the agent always took its current best guess it would exploit what it knows but never explore — and might lock onto a mediocre route, never discovering a better one. ε-greedy fixes this: with probability ε it ignores its values and tries a random move. Below are the four action-values for one state; greedy always picks the tallest bar, but ε of the time the agent gambles instead.

What you saw

With no labels and only a reward signal, the agent learned to cross the grid. The return discounts future reward by γ; the action-value Q(s, a) estimates that return; the Q-learning update drags each value toward the reward plus the best next value, so value propagates backward from the goal one step at a time; and ε-greedy keeps the agent exploring instead of settling early. The greedy policy — follow the arrows — is what falls out at the end.

Walking downhill, carefully

Backprop gives the gradient — the uphill direction of the loss. Training just steps the opposite way, over and over. Simple in principle, but four choices decide whether it works at all: which update rule, how big a step, how much data per step, and how to stop the model overfitting.

The gradient points the wrong way

Picture the loss as a surface over the parameters; the rings below are its contours, the centre is the minimum. At any point the gradient ∇L points uphill — the steepest way to increase loss. So we step along −∇L, scaled by the learning rate η. That's the whole of gradient descent.

Concretely, this bowl is L = ½(2x² + y²), so ∇L = (2x, y). At the marked point (1.7, 1.25) that’s (3.4, 1.25) — the red arrow — and the orange step heads exactly opposite.

How far is just right?

Take the simplest loss, f(x) = x², and walk it down with real numbers:

The step size η is everything. Drag it below: too small and the ball barely moves; around the sweet spot it drops straight in; push it past the edge and the steps overshoot harder each time — the loss explodes.

Remember where you were going

Plain descent treats every step independently, so in a narrow valley it bounces between the walls and barely crawls along the floor. Momentum fixes this by accumulating a velocity — a running average of past gradients. Consistent directions build up speed; alternating ones cancel out. It's a ball with inertia rolling downhill instead of a dot teleporting.

A learning rate per parameter

One global η is a compromise: directions that need big steps and ones that need small steps get the same. Adam gives each parameter its own effective rate. It keeps a running mean of the gradient (m, like momentum) and a running mean of its square (v, the typical magnitude), then steps by the mean divided by the root of the square — so steep directions are damped and flat ones amplified.

Three optimizers, one awkward valley

Here is the classic test: a loss that's steep in one direction and shallow in the other (the contours are stretched ellipses; here L = ½(18x² + y²), eighteen times more curved across the valley than along it, so no single step size suits both). All three start at the same corner. Watch SGD zig-zag and crawl, momentum build speed along the valley floor, and Adam rescale each axis and drive almost straight to the minimum.

Change the step as you go

η doesn't have to be a fixed number. Early in training you want big steps to cover ground; later you want small ones to settle into the minimum instead of rattling around it. A schedule lowers η over training — most commonly a cosine decay, often with a short warmup at the start so the first steps don't explode while Adam's running averages are still cold.

Why a constant rate leaves a floor: near the minimum a noisy step behaves like x ← (1 − 2η)x − η·ξ, where ξ is the gradient noise (variance s²). The variance settles where it stops shrinking — Var(x) = η²s² ∕ (1 − (1−2η)²) ≈ η s² ∕ 4 for small η — so the expected loss E[x²] is proportional to η. Halve the rate and you roughly halve the floor; drive η → 0 on a schedule and the floor follows it down. With s = 0.7 and η = 0.34 here, the constant floor sits near 0.34·0.49 ∕ 4 ≈ 0.042, while the cosine tail drives η toward 0 and keeps dropping past it.

The bottom panel shows the payoff (log scale): a high constant rate (dashed) drops the loss fast but then bounces on a noise floor it can't get under — each step's randomness is proportional to η. Decaying η lowers that floor as you go, so the loss keeps falling. Pick a schedule:

The real obstacle isn't local minima

The bowl in chapter 2 was a convenient lie. Real loss surfaces are non-convex and high-dimensional, and there the trouble is rarely a bad local minimum — it's saddle points and flat plateaus, regions where the gradient nearly vanishes and plain gradient descent stalls. Below, three optimizers start on the flat shoulder near x = 0 (slope ≈ 0) of a double-well. Watch plain GD inch along the plateau while momentum (carrying velocity) and SGD noise break free and drop into a minimum far sooner.

Which critical point is the trap? The second derivative f″(x) = 3x² − 2 is −2 at x = 0 (a maximum along this slice — the saddle/plateau) and +4 at x = ±√2 ≈ ±1.414 (the two real minima). The hazard isn't the minima; it's the flat top. A gradient step there is f′(0.02) = 0.02³ − 2·0.02 = −0.040, so with η = 0.03 plain GD moves just −η·f′ = +0.0012 per step — a crawl. Down on the wall at x = 1, f′ = 1 − 2 = −1, a step 25× larger. Momentum accumulates those tiny pushes into real velocity; noise jolts x off the ridge directly — both clear the plateau while GD inches.

Why the loss curve is fuzzy

A real gradient is averaged over a batch of examples. The full dataset gives the exact gradient but is slow; a single example (pure stochastic) is fast but noisy, since one example is a rough estimate of the whole. Noise scales as 1∕√batch. Slide the batch size and watch the training curve go from jagged to smooth — small batches are noisier per step, yet cheaper and their noise can even help escape sharp, brittle minima.

Why the square root? The batch gradient is an average of b per-example gradients, and averaging b independent estimates divides their variance by b (a mean of b draws has variance σ²∕b). The standard deviation — the wobble you actually see — is the square root of that, so it falls as 1∕√b. Going from batch 1 to 100 cuts the noise only 10×, which is why ever-larger batches give diminishing returns.

Starting the weights at the right scale

Before training even begins, the initial weight scale decides whether a signal survives a deep stack. Each layer multiplies the typical activation (and gradient) magnitude by roughly a constant factor. Below 1, the signal vanishes layer by layer; above 1 it explodes; right at 1 it holds steady. Slide the per-layer gain and watch the magnitude across 24 layers (log scale):

Each layer scales the typical magnitude by a factor g (set by the weight variance and the fan-in), so down a stack of L the magnitude is the product g·g·…·g = g^L — exponential in depth. That's unforgiving: at L = 24, g = 1.40 gives 1.40²⁴ ≈ 3.2 × 10³ (exploding), g = 0.60 gives 0.60²⁴ ≈ 4.7 × 10⁻⁶ (vanished), and only g = 1.00 holds at exactly 1. He initialization sets Var(W) = 2∕fan-in precisely so each layer's output variance matches its input — pinning g at ≈ 1 so the signal neither dies nor blows up.

This is exactly what Xavier and He initialization solve: they set the weight variance from the layer's fan-in (He uses 2∕fan-in for ReLU) so the gain lands at ≈ 1 and signals propagate intact. And for spikes that still slip through — one huge gradient that would blow up a step — gradient clipping rescales it to a ceiling: g ← g · min(1, c ∕ ‖g‖), capping the norm at c without changing its direction.

Keeping each layer's inputs well-behaved

Even with good initialization, the distribution of each layer's inputs drifts and stretches as the layers below it update — so every layer is forever chasing a moving target. Normalization (batch norm, layer norm) fixes this by standardizing each layer's pre-activations back to zero mean and unit variance, then letting the model rescale them with two learnable parameters γ and β. Toggle it and watch the per-layer distributions:

Worked example: a unit emits activations [2, 4, 6, 8] across a batch. Mean μ = 5; variance σ² = ((−3)² + (−1)² + 1² + 3²) ∕ 4 = 5. Normalize: x̂ = (x − 5) ∕ √5 ≈ [−1.34, −0.45, 0.45, 1.34] — now zero mean, unit variance. Then the learnable γ, β rescale: with γ = 2, β = 1 the output is y = 2x̂ + 1 ≈ [−1.68, 0.11, 1.89, 3.68]. The model can still represent any mean and spread it wants — it just no longer has to chase a drifting input to do it.

Off, the bells march sideways and spread layer by layer; on, every layer sees a clean, standardized input regardless of what's happening below it. That stability lets you train deeper networks at higher learning rates — it both speeds convergence and smooths the loss landscape.

Fitting the signal, not the noise

A model with enough capacity can drive training error to zero by memorising every point — including the noise — and then fails on anything new. Below, a polynomial is fit to noisy samples of a smooth curve. Raise the degree and the fit wiggles through every training point while the held-out (validation) error blows up: overfitting. Then raise weight decay (λ, an L2 penalty on the coefficients) and watch the curve smooth back out — generalisation restored.

That shrink factor isn’t arbitrary: differentiating the penalty gives ∂(λ‖θ‖²)∕∂θ = 2λθ, which adds −η·2λθ to every update. So each step first pulls the weights a little toward zero — by the factor (1 − 2ηλ) — then applies the data gradient. Big weights are taxed, so the fit can’t lean too hard on any one coefficient.

Quit while you're ahead

The simplest regularizer of all is to stop at the right moment. Training loss falls forever — the model keeps fitting the training set harder — but validation loss traces a U: it improves while the model learns real structure, bottoms out, then climbs as the model starts memorising noise. The best model is the one at that bottom. Drag the stopping epoch:

There's nothing to differentiate here — it's a rule, not a formula — but the bookkeeping matters. You track the best validation loss seen so far and the epoch it occurred; each epoch that fails to beat it increments a counter, and when the counter reaches the patience (say 10 epochs) you halt and roll back to that checkpoint. Here validation bottoms out at epoch 22 (val ≈ 0.256); by epoch 48 it has climbed to ≈ 0.564 while training loss kept falling — that widening gap is the overfitting, and the epoch-22 weights are what you actually ship.

What you saw

Training is gradient descent plus a stack of choices. The learning rate sets the step — too large explodes, too small crawls — and a schedule shrinks it over time to settle cleanly. Momentum carries velocity through narrow valleys and off plateaus; Adam gives every parameter its own rate; and on an ill-conditioned surface both leave plain SGD far behind. Batch size trades gradient noise for speed, and that same noise helps escape the saddle points that fill high-dimensional landscapes. Good initialization and normalization keep signals from vanishing or exploding through deep stacks. And capacity without restraint memorises noise — so weight decay, dropout, and early stopping keep the model general. The same machinery as every other dive — a loss and its gradient — but tuned to actually converge.

Learn to undo noise

There are two processes. The forward process is fixed and dumb: it just adds a little Gaussian noise, over and over, until the data is indistinguishable from static. The reverse process is the model: a network that, given a noisy point and a noise level, predicts what noise was added — which is enough to take one step back toward clean data. Run the reverse step from fresh noise enough times and you have generated a brand-new sample.

A ring we can watch

To see the process we use a 2-D toy: 90 points arranged in a ring. It is clearly not a Gaussian — there is a hole in the middle and mass at a fixed radius — so when noise destroys that structure, and when the model rebuilds it, both are obvious to the eye. Everything below is the exact algorithm a real image model uses; only the dimensionality is tiny.

Melting the data into noise

One forward step scales the point down a touch and adds a little Gaussian noise, with the amount set by a schedule β_t:

The √(1−β_t) shrink keeps the variance from growing without bound; the +β_t injects fresh noise. Drag t from 0 to the end and watch the ring dissolve into an isotropic blob — the same blob no matter what the data was:

Jump to any noise level in one shot

Composing many small Gaussian steps is itself one big Gaussian step. Define the surviving signal after t steps as the running product ᾱ_t = ∏_s≤t(1−β_s). Then you can skip the whole loop and sample x_t directly from x₀:

Why it collapses into one step: a single step is x_t = √α_t·x_t−1 + √(1−α_t)·ε. Substitute x_t−1 = √α_t−1·x_t−2 + √(1−α_t−1)·ε′ and the two independent noise terms add:

Independent zero-mean Gaussians merge in quadrature — variances add — so two steps become one: x_t = √(α_tα_t−1)·x_t−2 + √(1 − α_tα_t−1)·ε̃. Induct down to x₀ and the surviving-signal coefficient is just the running product ᾱ_t = ∏ α_s — the closed form above. With α₁ = 0.9, α₂ = 0.8 that gives signal √(0.9·0.8) = √0.72 = 0.849 and noise √(1−0.72) = √0.28 = 0.529 — the very numbers from the by-hand steps below.

That single identity is the engine of the whole method: training never runs the chain, it just picks a random t, draws one ε, and jumps. Below, the green curve √ᾱ_t is how much signal survives and the violet √(1−ᾱ_t) is how much noise has taken over — they cross when the data is half gone.

Two steps, then the shortcut

Worth doing once with real numbers on a single 1-D point, so the closed form stops looking like magic:

Just predict the noise

Here is the part that makes diffusion practical. Because x_t = √ᾱ_t x₀ + √(1−ᾱ_t) ε, the clean point and the noise are interchangeable — recover either and you have the other. So instead of predicting the clean image, the network ε_θ is trained to predict the noise that was added, with a plain mean-squared error:

This is not an ad-hoc trick: the full variational bound on −log p(x₀) unrolls into one denoising term per step, and after the Gaussian algebra each term reduces to exactly this — predict the ε that produced x_t. Dropping the per-step weights (which barely matters in practice) leaves the clean MSE above.

It also has a beautiful interpretation. The score of the noisy distribution — the direction in which data becomes more likely — is ∇_x log q(x_t) = − ε ∕ √(1−ᾱ_t). So predicting the noise is, up to that scale factor, the same as learning the score: the network learns which way to push a point to make it look more like real data. That is literally the arrows in the next panel.

Which way is “more like data”

At a fixed noise level, evaluate the (perfectly trained) denoiser everywhere on the plane and draw −ε_θ, the direction it would nudge each point. Every arrow points back toward the ring: outside it pushes in, inside it pushes out, and on the ring the push vanishes. That field is the learned score. Slide the noise level — at high noise the field is broad and vague, at low noise it sharpens onto the exact ring:

Watch the field be learned

The arrows above came from the perfect denoiser. A real network has to learn that field from data, using exactly the loss from chapter 6: draw a point, noise it to a random level t, and nudge the weights so the predicted ε matches the true ε. Here a small time-conditioned model does precisely that. It starts predicting zero — so its loss is just the variance of the noise, ≈ 2 — and as it trains the arrows swing from random to pointing home. Choose a noise level to inspect and press Train:

The loss falls fast then flattens on a floor well above zero — that floor is real, not a bug. Many different (x₀, ε) pairs collide on the same noisy x_t, so the noise is only partly recoverable; the best possible prediction is the average ε consistent with x_t. That average is exactly the score direction, which is why the learned arrows line up with the ideal field from the previous panel even though the loss never reaches 0.

One step back toward the data

Where does that formula come from? Reverse the chain. The reverse of a single step is intractable in general, but if you also condition on the clean point x₀ it becomes a tractable Gaussian — Bayes on the forward step times the two closed forms:

At sampling time we don’t know x₀ — but ε_θ hands us an estimate of it: x̂₀ = (x_t − √(1−ᾱ_t)·ε_θ) ∕ √ᾱ_t. Substitute that x̂₀ into μ̃_t and the coefficients collapse into the compact update below; the σ_t z term (with σ_t² = β̃_t) puts back exactly the randomness the posterior says belongs there. So the sampling rule isn’t a heuristic — it’s the posterior mean of the true reverse step with x₀ replaced by the model’s guess:

Watch a sample form

Now the payoff. Seventy points start as pure Gaussian noise (t = 39). Each frame applies the reverse step above — predict the noise with the optimal denoiser, nudge toward the clean estimate, add a shrinking dash of randomness — and the cloud walks itself down to t = 0, settling onto the ring it has never been shown directly. This is generation: structure out of static, one denoising step at a time.

From a ring to an image model

Scale this up and almost nothing changes conceptually. The 2-D point becomes a full image (or a compressed latent of one, as in Stable Diffusion, which is cheaper); the toy denoiser becomes a large U-Net or transformer; and the schedule runs over hundreds or thousands of steps (with faster samplers like DDIM cutting that to a few dozen). To make it follow a prompt, the network is also fed a text embedding — conditioning — and classifier-free guidance sharpens prompt adherence by extrapolating between the conditioned and unconditioned predictions.

The network is trained on both the prompt c and an empty prompt ∅ (the conditioning is randomly dropped during training). At sampling you push past the conditioned prediction, away from the unconditioned one; the guidance weight w (typically 5–15) trades diversity for prompt fidelity. And to cut hundreds of steps to a few dozen, deterministic samplers like DDIM drop the random σ_t z and take larger exact steps along the same path — x_t−1 = √ᾱ_t−1·x̂₀ + √(1−ᾱ_t−1)·ε_θ, reusing the same clean-point estimate — same trained model, far fewer evaluations.

The reason diffusion overtook GANs: training is a stable regression instead of a fragile two-player game, it covers the full data distribution instead of collapsing onto a few modes, and the iterative sampler trades compute for quality on demand. The cost is exactly that — many network evaluations per sample — which is what the fast-sampler research keeps chipping away at.

What you saw

A diffusion model is a fixed forward process that drowns data in Gaussian noise, plus a network trained to undo one step of it. The closed form x_t = √ᾱ_t x₀ + √(1−ᾱ_t) ε lets training jump to any noise level in one shot; the network just predicts that ε, which doubles as learning the score — the direction toward more-likely data. You trained a small denoiser from scratch and watched its vector field sharpen toward that score; the reverse step you stepped by hand is the exact posterior mean with x₀ swapped for the model’s guess. Sampling reverses the chain from pure noise, and you watched 70 points do exactly that, reforming a ring they were never directly shown. Same loss-and-gradient machinery as every other dive, pointed at a strange and powerful target.

Four stages, one model

A modern assistant is built in stages. First pretraining: predict the next token on a vast pile of text until the model has absorbed grammar, facts and reasoning patterns. That gives a base model — a brilliant autocomplete that does not yet answer questions. Then fine-tuning and alignment turn that autocomplete into something that follows instructions and prefers helpful, honest answers. Finally decoding chooses, token by token, what to actually emit.

Text becomes a sequence of integers

A model never sees letters — it sees token IDs. Splitting on whole words makes the vocabulary explode and chokes on anything unseen; splitting on single characters makes sequences painfully long. Byte-pair encoding (BPE) finds the middle: start from characters and repeatedly merge the most frequent adjacent pair into a new token, so common chunks like "ing" or "the" become single tokens while rare words still decompose into pieces. Build a tiny BPE vocabulary by hand:

Predict the next token

Pretraining is one task repeated trillions of times: given the tokens so far, put a probability on every token in the vocabulary for what comes next, via a softmax over the model's output logits. Training maximises the probability of the token that actually came next — equivalently, it minimises the cross-entropy, the negative log-probability assigned to the truth:

What does one training example look like? A single sequence is shredded into many next-token problems at once. The tokens "the cat sat ." become four overlapping predictions — given "the" predict "cat", given "the cat" predict "sat", given "the cat sat" predict "." — all scored in a single forward pass. The model's input is the token sequence; its target is the very same sequence shifted left by one position. (This trick, teacher forcing, is what lets a whole document be trained on in parallel rather than one token at a time.)

Perplexity is the friendlier number: it is roughly "how many tokens the model is effectively choosing between." A model that has learned nothing is as confused as the vocabulary is big; a perfect model on a deterministic next token has perplexity 1. Watch the arithmetic:

And the gradient that does the work is famously clean. Differentiating the softmax cross-entropy with respect to the logits gives, for every token i,

— the predicted probability minus the one-hot truth. In the example above (p = [0.50, 0.20, 0.15, 0.10, 0.05], true = "sat"), the gradient on the logits is [0.50−1, 0.20, 0.15, 0.10, 0.05] = [−0.50, 0.20, 0.15, 0.10, 0.05]: one descent step lowers every wrong token's logit and raises "sat"'s, in a single stroke. (At initialisation the model is uniform, p = 1∕V, so L = log V and perplexity = V — exactly the vocabulary size, which is where the live demo below starts.) This is the precise update the trainer runs.

Watch a model learn to talk

Here is a real next-token network — a small two-layer model that sees the previous two tokens and predicts the next, trained by exactly the cross-entropy above on a stream of simple sentences (det → noun → verb → optional adverb → period). It starts with random weights, so its perplexity is near the vocabulary size and its samples are noise. Press Train and watch the perplexity fall and the generated sentences snap into grammar:

Supervision from the data itself

A supervised model needs a human-labelled target for every example. A self-supervised model invents the target from the input: delete a word and the word itself is the label; take two random crops of one photo and "they match" is the label. No annotation required — so you can train on the entire internet. The goal is not the pretext task itself but the representation it forces the network to build: a feature space where similar things land nearby, which then transfers to real tasks.

The pretrain-then-probe bet

The wager of self-supervision: if a network must solve a hard enough pretext task on huge unlabelled data, it has to discover the underlying structure of the data to succeed — and that structure is exactly what downstream tasks need. So you pretrain once, expensively, on a pretext task, then freeze the features and attach a small task-specific head trained on whatever little labelled data you have. The representation does the heavy lifting; the head just reads it out.

This is the engine behind modern foundation models: BERT and GPT pretrain on text, MAE and DINO on images, CLIP on image–text pairs, wav2vec on audio. In every case the labelled fine-tuning is a thin layer on top of representations learned without labels.

Hide it, then predict it

The simplest pretext task: corrupt the input and reconstruct it. BERT masks ~15% of the tokens in a sentence and predicts each from the words on both sides; MAE masks ~75% of the patches in an image and reconstructs them. The loss is the same cross-entropy from the LLM dive — but applied only at the masked positions, and the context is bidirectional, which makes these models superb at understanding (not generation).

There is a subtlety in the data: of the 15% of tokens BERT selects, it replaces 80% with [MASK], 10% with a random token, and leaves 10% unchanged. The reason is that [MASK] never appears at fine-tuning or inference time, so a model trained to expect it everywhere would be mismatched; the random and unchanged cases force it to build a useful representation of every token, not just the masked slots.

Pull together, push apart

Contrastive learning needs no reconstruction at all. Make two augmented views of one example (crop, recolour, blur) — they form a positive pair that should map to nearby points. Every other example in the batch is a negative that should be pushed away. Measure closeness by cosine similarity — the dot product of the unit-normalised embeddings, sim(a, b) = (a·b) ∕ (‖a‖‖b‖) = â·b̂ ∈ [−1, 1] — and the InfoNCE loss is just a softmax cross-entropy whose "correct class" is the positive:

Minimising it means making the anchor far more similar to its positive than to any negative. Work the arithmetic:

And the gradient shows why it works. Writing m_k = softmax_k(sim(a,k)∕τ) for the weight the loss puts on each candidate, the update on the anchor is

— move toward the positive a⁺, and away from the m-weighted average of all candidates. Because m_k is largest for whichever negatives are already closest, the push lands hardest on the "hard negatives." In the worked example, m = [0.965, 0.029, 0.007] is dominated by the positive itself, so a⁺ − Σ m_k·k ≈ 0: an already-aligned anchor barely moves. In the misaligned case (m on the positive only 0.33) that term is large, so the pull toward a⁺ is strong — the force self-limits exactly as alignment improves.

The name is literal: InfoNCE is a lower bound on the mutual information between the two views, and the bound is capped by log(number of candidates). More negatives ⇒ a tighter bound ⇒ better representations — which is precisely why SimCLR trains with very large batches and MoCo keeps a running queue of thousands of negatives.

Clusters from nothing but "same source"

Below are 30 samples — six augmented views of each of five source items — each with a 2-D embedding on the unit circle (a real encoder would compute these from raw input; here they are free parameters so we can watch the forces directly). The only signal is which views came from the same source; no labels, no class names. Press Train and watch InfoNCE pull each source's six views together and shove the five groups apart until clean, evenly-spaced clusters appear:

Learning on nodes and edges

A graph is a set of nodes (each with a feature vector) connected by edges. The catch that breaks ordinary networks: there is no canonical ordering of nodes and every node can have a different number of neighbours, so you cannot just flatten it into a fixed-size vector. The fix is message passing — a local rule, applied identically at every node, that is invariant to how you happen to number things.

An adjacency matrix and a feature matrix

Two arrays describe everything. The feature matrix X stacks one feature vector per node. The adjacency matrix A records the edges: A_ij = 1 if node i connects to node j. Here is a small graph and its adjacency — symmetric, because these edges are undirected:

The one operation everything is built from

Every graph-network layer is the same two-step local rule, run at every node simultaneously: aggregate the neighbours' current vectors with some order-independent function (sum, mean or max), then update the node by combining that aggregate with its own vector through a small learned transform:

The aggregation must be permutation-invariant — a sum does not care what order the neighbours arrive in, and that is exactly what a graph needs. The learned transform (a weight matrix W and bias, shared across all nodes) is the only thing with parameters, which is why one trained network handles graphs of any size or shape.

Formally this makes the layer permutation-equivariant: relabel the nodes by any permutation P and the output is just the same vectors relabelled — GNN(PX, PAP^ᵀ) = P·GNN(X, A). Nothing depends on the arbitrary numbering, which is exactly the property a graph demands.

Averaging a neighbourhood

Take the graph above with a single number on each node and update one of them, so the aggregate-then-transform stops being abstract:

Doing it for all nodes at once

Written as matrices, a whole layer is one line. The graph convolutional network (GCN) uses a particular aggregation: add self-loops (so a node keeps its own features) and normalise by degree symmetrically, so high-degree hubs do not drown everything out:

Here A + I adds the self-loop, D̃ is the degree matrix of A + I, and the D̃^−1/2(·)D̃^−1/2 sandwich weights the edge between nodes i and j by 1∕√(d_i d_j) — gentle on hub nodes. (The by-hand step above used a plain mean for clarity; the symmetric normalisation is the same idea with degree-aware weights.) Multiplying by  mixes each node with its neighbours; multiplying by W transforms the features; σ is a nonlinearity like ReLU.

A concrete entry: suppose node i has 3 neighbours and node j has 2. With self-loops their degrees are d̃_i = 4 and d̃_j = 3, so the weight Â_ij = 1∕√(d̃_i·d̃_j) = 1∕√12 = 0.289, while node i keeps 1∕d̃_i = 0.25 of itself. A degree-50 hub contributes only 1∕√(51·d̃_j) to each neighbour — automatically tiny — which is the whole point of the symmetric normalisation: popular nodes do not shout over everyone.

Each layer is one hop further

One layer lets a node see its immediate neighbours. Stack a second and those neighbours have already absorbed their neighbours — so two layers reach two hops out, three layers reach three, and so on. The number of layers is literally the radius of the receptive field. Pick how many layers and watch the reach of the highlighted node grow:

Labelling a whole graph from a few seeds

The classic graph task is semi-supervised node classification: you know the labels of only a handful of nodes and must infer the rest from the graph structure. Below, a network of two loosely-connected communities has just four labelled seeds (two per colour, ringed). A two-layer GCN is trained on those four alone — every other node starts unlabelled (grey). Press Train and watch labels propagate along the edges until the whole graph is correctly split:

Only the four seeds appear in the loss — yet the gradient trains a classifier for all fourteen. The reason is the two  multiplications: a seed's prediction depends on its 2-hop neighbourhood and on the shared W, so backprop pushes signal into W from the seeds' surroundings, and because W is reused at every node, what it learns there applies everywhere. This is transductive learning: the unlabelled nodes' features and edges are used in the forward pass (they shape the seeds' predictions); only their labels are withheld. The gradient at the output is the familiar p − y, non-zero only on seed rows, then flowed back through  and W.

Not all neighbours matter equally

A GCN weights every neighbour by a fixed function of degree. The graph attention network (GAT) learns the weights instead: it scores each neighbour, softmaxes the scores into attention weights over the neighbourhood, and aggregates accordingly — the same attention idea as transformers, restricted to the graph's edges.

The score concatenates the transformed anchor and neighbour, dots them with a learned vector a, and passes through a LeakyReLU; softmax over the neighbourhood normalises the scores to weights that sum to 1. In practice GAT runs several independent attention heads in parallel and concatenates their outputs — exactly the multi-head trick from transformers — so the node can attend to neighbours in several different ways at once.

Why you cannot just stack more layers

If depth grows the receptive field, why not use fifty layers? Because each layer also averages neighbours, and repeated averaging is diffusion: push it far enough and every node's features blur into the same value, losing all the distinctions the network needed. This is over-smoothing. Slide the depth and watch six initially-distinct node features collapse together:

Why a line and not random noise? Â is an averaging operator whose largest eigenvalue is 1, with eigenvector proportional to √(degree). Repeated multiplication Â^L shrinks every other eigen-component toward zero, so Â^LX converges onto that one direction: every node's feature becomes the same vector, scaled only by √(d_v). Whatever distinguished the nodes is gone — all that survives is a degree pattern.

This is the central tension in graph nets: you need enough layers to reach the relevant neighbourhood, but too many wash out the signal. Most GCNs use just 2–3 layers. Remedies — residual/skip connections, jumping-knowledge, normalisation, or attention that can learn to ignore neighbours — let a few architectures go deeper, but shallow remains the norm.

Three tasks, many flavours

Message passing supports all three graph tasks. Node-level: classify each node (the demo above). Edge-level: predict whether two nodes should be linked — the engine of recommendation and knowledge-graph completion. Graph-level: pool all node vectors into one — a permutation-invariant readout, h_G = Σ_v h_v (or mean/max) — and classify the whole graph, which is how models predict a molecule's properties from its atoms and bonds. (The sum is what keeps the graph label independent of node ordering.) The popular variants differ mainly in the aggregator: GCN (degree-normalised mean), GraphSAGE (sample-and-aggregate for huge graphs), GAT (attention), and GIN (a sum-based form proven as expressive as the classical graph-isomorphism test).

The throughline of this lab holds here too: the loss and gradient are unchanged from every other dive. What is special to graphs is only the layer — an aggregation that respects the irregular, orderless structure of the data — and from that single idea comes drug discovery, traffic forecasting, fraud rings, and recommendation at billion-edge scale.

What you saw

A graph network learns by message passing: every node aggregates its neighbours with a permutation-invariant function and updates through a shared learned transform. You did one GCN layer by hand (aggregate the neighbourhood, transform, activate), saw the whole-graph matrix form  H W, and watched depth extend the receptive field one hop per layer. A two-layer GCN coloured an entire network from four seeds; graph attention learned which neighbours matter; and over-smoothing showed why too much depth collapses every node into the same blur. Same loss, same gradient — a new kind of layer for a new kind of data.

A forger and a detective

Picture a counterfeiter and a bank inspector locked in escalation. The counterfeiter (generator) prints fake notes from random ink; the inspector (discriminator) learns the tells and rejects them; the counterfeiter adapts, the inspector re-learns, and round after round the fakes get better. When the inspector can do no better than a coin flip, the counterfeits are indistinguishable from real money — and that is exactly the generator we wanted.

One value function, two opposite goals

The discriminator D outputs a probability that its input is real. The generator G maps noise z to a sample G(z). They share a single value function — the discriminator tries to maximise it, the generator to minimise it:

The first term rewards D for scoring real data near 1; the second rewards it for scoring fakes near 0. G only touches the second term — and wants the opposite, to make D(G(z)) near 1. Put numbers on it:

The best critic, in closed form

Freeze the generator. What is the best possible discriminator? Because the value function is an average over x, we can maximise it pointwise — and the answer is beautifully simple: at each x, D should report the fraction of probability mass there that is real.

Below, the real data is a fixed bell curve and the generator's distribution is another you can slide. The white curve is the optimal discriminator D* = p_data∕(p_data+p_g). Drag the generator onto the data and watch D* flatten to 0.5 everywhere — the critic can no longer tell them apart:

The game is a divergence

Substitute that optimal D* back into the value function and the algebra collapses into something profound — the generator is minimising the Jensen–Shannon divergence between the fake and real distributions:

JSD is a symmetric, always-finite cousin of the KL divergence: JSD(p ‖ q) = ½ KL(p ‖ m) + ½ KL(q ‖ m) where m = (p + q)∕2 is their average. It is zero only when the two distributions are identical and positive otherwise. So the generator's objective is to drive p_g all the way onto p_data. At that global optimum p_g = p_data, the divergence is zero and D*(x) = ½ everywhere — the detective is reduced to a coin flip.

Why exactly −log4 at the bottom? With D* = ½ every term inside V becomes log ½, so V = E_data[log ½] + E_fake[log ½] = 2·log ½ = −log 4 ≈ −1.386. That is the theoretical promise of a GAN: the unique equilibrium of the game is the true data distribution.

Painting a distribution from noise

Here are two real networks playing the game. The generator (a small MLP) maps 2-D Gaussian noise to points on the plane; the discriminator (another MLP) learns to separate them from the target — a ring. The shaded background is the discriminator's current verdict (green = "looks real," dark = "looks fake"). Press Train and watch the fake cloud, scattered at first, migrate until it covers the ring and the background washes to grey:

The saturating gradient, and the fix

The theory is clean; the training is treacherous. Look at the generator's original term, log(1 − D(G(z))). Early on the generator is terrible, the discriminator rejects its fakes with confidence (D ≈ 0), and exactly there the gradient of log(1 − D) is nearly flat — the generator gets almost no signal precisely when it needs it most. This is the saturating gradient.

The standard fix flips the objective: instead of minimising "the chance D catches me," the generator maximises "the chance D is fooled," log D(G(z)). The optimum is identical, but the gradient is strong exactly when D is winning — so the generator keeps learning even while being beaten. (The live demo uses this non-saturating loss.) Even so, the two-player dynamics can oscillate, stall, or diverge rather than settle; balancing the pair is the central art of training GANs.

Underneath, it is all the familiar gradient. The discriminator minimises a binary cross-entropy, whose gradient on its output logit is the same D − label that trained every classifier in this lab: D − 1 on a real sample (push the score up), D on a fake (push it down). The generator has no labels and never touches a real point — it learns purely by backpropagating through the frozen discriminator: ∂loss∕∂(G's weights) = ∂loss∕∂D · ∂D∕∂(G output) · ∂(G output)∕∂(weights). "Move your samples wherever D thinks looks more real."

The saturating problem lives in that first factor. Under the original loss log(1 − D), the gradient on a fake's logit works out to −D — proportional to D itself, so when the discriminator confidently rejects (D ≈ 0) the signal nearly vanishes. The non-saturating loss −log D instead gives D − 1, proportional to (1 − D), which is largest exactly when D ≈ 0:

Numerically, when the generator is losing badly at D = 0.1: the saturating push is just 0.1 (feeble, right when G most needs to learn), while the non-saturating push is 0.9 (strong). At D = 0.9 they swap — but by then the generator is already winning. Same optimum, far healthier gradient when it counts.

The failure that haunts every GAN

The most infamous failure: the generator discovers one output that reliably fools the discriminator and produces only that, ignoring the rest of the data. Fakes look fine individually but have no variety — a face generator that draws the same face, a digit model stuck on a single "7." Here the target has eight separate clusters; toggle between a healthy generator that covers them all and a collapsed one that has piled all its mass onto a few:

Collapse happens because the generator is only ever asked to fool the discriminator on the current batch, not to match the full diversity of the data — covering one mode perfectly can be a stable trap. Remedies include showing the discriminator several samples at once (minibatch discrimination), unrolling its updates, or changing the distance the game minimises altogether.

Wasserstein and beyond

Jensen–Shannon divergence has a flaw: when the fake and real distributions barely overlap — common early in training — it saturates and its gradient vanishes, starving the generator. The Wasserstein GAN replaces it with the earth-mover (Wasserstein) distance, the minimal "cost" to transport one distribution onto the other, which gives a smooth, informative gradient even when the distributions are far apart:

The discriminator becomes a "critic" that scores realness on an unbounded scale rather than outputting a probability, kept well-behaved by constraining it to be 1-Lipschitz (its output cannot change faster than its input) — the condition that the Kantorovich–Rubinstein duality needs to turn the Wasserstein distance into a maximisation over critics. Weight clipping (as in the live demo above) and the gradient penalty are two ways to enforce that bound. WGANs train far more stably and rarely mode-collapse, which is why the technique became standard.

What GANs built, and what came after

GANs powered a decade of generative leaps: DCGAN made them work with convolutions, StyleGAN produced photorealistic faces with controllable style, conditional and image-to-image variants (pix2pix, CycleGAN) enabled sketch-to-photo and style transfer. Their signature strengths are sharp samples and fast, single-pass generation. The trade-off is the training difficulty you have just seen — instability and mode collapse — which is precisely why diffusion models (their own dive in this lab) have overtaken GANs for many image tasks: they are slower to sample but far more stable to train and cover the full data distribution by construction.

The throughline of the lab holds once more: a differentiable loss and a gradient. What is singular about the GAN is that the loss is not fixed — it is another network, learned in lockstep, so the target moves as the generator chases it. That moving target is both the source of their remarkable sample quality and the reason they are so hard to train.

What you saw

A GAN is a two-player game: a generator turning noise into samples and a discriminator judging them, sharing one value function that one maximises and the other minimises. You scored the game by hand, derived the optimal discriminator D* = p_data∕(p_data+p_g) and the result that the generator is minimising Jensen–Shannon divergence — zero only when the fakes match the data and the critic is reduced to a coin flip. You watched a real generator reconstruct a ring from noise using nothing but the discriminator's gradient, met the saturating-gradient trap and its non-saturating fix, saw mode collapse, and met the Wasserstein distance that tames it. A loss that is itself a learned, adversarial network — same gradient, a moving target.