TRAINING A NETWORK IS AN OPTIMISATION PROBLEM: find the weights that minimise an ERROR (or LOSS, or COST) function. THE STANDARD ERROR FUNCTION for regression is the SUM OF SQUARED ERRORS: E(w) = ½ Σₚ (tₚ − yₚ)² summed over the training examples p. THE FACTOR ½ IS THERE ONLY SO THAT IT CANCELS WHEN DIFFERENTIATED — it has no other purpose, and some texts omit it. WHY SQUARED: it is differentiable everywhere (unlike absolute error at zero), it penalises large errors disproportionately, and for a linear model it gives a single global minimum. THE ERROR SURFACE is E plotted against the weights. For n weights it is a surface in n+1 dimensions, and TRAINING IS THE SEARCH FOR ITS LOWEST POINT. For a linear unit with squared error the surface is a PARABOLOID — convex, with one minimum — which is why the delta rule of the next topic has such clean guarantees. For a multilayer network it is NOT convex, with many local minima, plateaux and saddle points. GRADIENT DESCENT: move each weight in the direction that reduces E most steeply, which is the negative gradient. wᵢ ← wᵢ − η · ∂E/∂wᵢ THE GRADIENT ∇E = (∂E/∂w₁, …, ∂E/∂wₙ) points in the direction of STEEPEST INCREASE, so the minus sign is what makes it descent. η is the LEARNING RATE, controlling the step size. A WORKED SINGLE-VARIABLE EXAMPLE, which shows everything that matters. Minimise f(w) = w², whose derivative is f′(w) = 2w and whose minimum is at w = 0. Start at w = 5. η = 0.1: 5.000 → 4.000 → 3.200 → 2.560 → 2.048 → 1.638 steady progress, converging slowly η = 0.5: 5.000 → 0.000 → 0.000 → … REACHES THE MINIMUM IN ONE STEP. Here 0.5 is exactly optimal, because w − 0.5(2w) = 0 for any w. η = 0.9: 5.000 → −4.000 → 3.200 → −2.560 → 2.048 → −1.638 OSCILLATES ACROSS the minimum, but the magnitude shrinks, so it still converges η = 1.0: 5.000 → −5.000 → 5.000 → −5.000 → … OSCILLATES FOREVER with no progress at all η = 1.1: 5.000 → −6.000 → 7.200 → −8.640 → 10.368 → −12.442 DIVERGES — each step overshoots further than the last FOUR DISTINCT REGIMES FROM ONE PARAMETER: too small is slow, optimal is immediate, moderately large oscillates but converges, and too large diverges. THE BOUNDARY BETWEEN CONVERGENCE AND DIVERGENCE IS SHARP, and in this example it is at η = 1. IN PRACTICE THE OPTIMAL η IS UNKNOWN, because it depends on the curvature of the error surface, which varies from problem to problem and from region to region within one problem.

THE SYMPTOMS, and this diagnostic table is the practical content of the topic: OBSERVED BEHAVIOUR THE LIKELY CAUSE ───────────────────────────────────────────────────────────── the loss decreases very slowly η TOO SMALL the loss decreases then plateaus far above zero η too small, or a local minimum the loss jumps up and down but trends down η slightly too large the loss oscillates with no improvement η at the stability boundary the loss increases, or becomes NaN / infinite η MUCH TOO LARGE — DIVERGENCE ───────────────────────────────────────────────────────────── TYPICAL VALUES: 0.001 to 0.1 for most networks, and the standard practice is to try a geometric sequence — 0.1, 0.01, 0.001 — and observe the loss curve for a few epochs rather than reasoning about it. THE FUNDAMENTAL TRADE-OFF: A LARGE η MAKES FAST PROGRESS FAR FROM THE MINIMUM AND OVERSHOOTS NEAR IT. That observation leads directly to varying η during training. LEARNING RATE SCHEDULES: STEP DECAY multiply η by a factor (say 0.1) every k epochs EXPONENTIAL DECAY η(t) = η₀ · e^(−kt) 1/t DECAY η(t) = η₀ / (1 + kt) COSINE ANNEALING decay following a cosine curve to near zero — widely used WARMUP start SMALL and increase for the first few hundred steps, then decay. Counter-intuitive and genuinely necessary for large models, because the early gradients are large and unreliable while the weights are random. THE ROBBINS-MONRO CONDITIONS give the theoretical requirement for stochastic gradient descent to converge: Σ η(t) = ∞ the steps must not shrink so fast that the total distance travellable is finite Σ η(t)² < ∞ but they must shrink fast enough that the noise averages out A schedule like η(t) = η₀/t satisfies both, which is why 1/t decay appears in the theory. MOMENTUM — the most useful single improvement: v ← β·v + η·∇E w ← w − v with β typically 0.9. The update accumulates a VELOCITY, so consistent gradient directions build speed while oscillating directions cancel. WHY IT HELPS: on a surface that is a long narrow valley — steep across, shallow along — plain gradient descent oscillates across the valley and creeps along it. Momentum damps the oscillation and accelerates the useful direction. With β = 0.9 the effective step in a consistent direction approaches η/(1 − β) = 10η, so momentum gives a tenfold speed-up along a consistent slope without a tenfold learning rate. ADAPTIVE METHODS — a per-weight learning rate: ADAGRAD divides η by the accumulated squared gradient, so frequently updated weights get smaller steps. The accumulation only grows, so η eventually vanishes. RMSPROP uses a decaying average instead of a sum, fixing that. ADAM combines RMSProp's scaling with momentum, and is the standard default in practice. THESE REDUCE THE SENSITIVITY TO η WITHOUT REMOVING IT — Adam still has a learning rate, and it still matters.

HOW MANY EXAMPLES CONTRIBUTE TO ONE WEIGHT UPDATE — three answers, and the difference is practical rather than cosmetic. BATCH (FULL-BATCH) GRADIENT DESCENT Compute the gradient over the ENTIRE training set, then take one step. ✔ the gradient is exact, so the path is smooth and the descent direction is the true steepest one ✔ deterministic and reproducible ✘ ONE UPDATE PER PASS THROUGH THE DATA, so with a million examples one update costs a million forward passes ✘ requires the whole data set in memory ✘ can settle into a local minimum, with no noise to escape it STOCHASTIC GRADIENT DESCENT (SGD) Update after EVERY SINGLE example. ✔ many updates per pass, so progress per unit of computation is far higher ✔ THE NOISE IS USEFUL: a noisy step can escape a shallow local minimum that batch descent would settle into ✔ works with streaming data ✘ the path is erratic and the loss curve noisy ✘ cannot exploit vectorised hardware — one example at a time wastes a GPU MINI-BATCH GRADIENT DESCENT — what is actually used Update after a small batch, typically 32 to 256 examples. ✔ the gradient estimate is much less noisy than SGD's ✔ the batch is ONE MATRIX MULTIPLICATION, so it exploits parallel hardware — the batching argument from the mathematical-model topic ✔ retains enough noise to escape poor minima → THE DEFAULT, and "SGD" in modern usage almost always means mini-batch. THE ARITHMETIC OF THE DIFFERENCE, on 10,000 examples with a batch size of 100: full batch: 1 update per epoch mini-batch: 100 updates per epoch SGD: 10,000 updates per epoch The mini-batch run makes a hundred times as much progress per epoch as full batch, at almost the same computational cost per epoch — because the hundred small matrix products are only slightly less efficient than one large one. TERMINOLOGY, which exams test: EPOCH one complete pass through the training set BATCH SIZE the number of examples per update ITERATION one weight update so iterations per epoch = training set size / batch size THE PROBLEMS OF THE ERROR SURFACE that gradient descent must contend with, and each has a standard response: LOCAL MINIMA a point lower than its neighbourhood but not the global lowest. Response: noise from mini-batching, momentum, several random restarts. PLATEAUX flat regions where the gradient is nearly zero and progress stalls. Response: momentum, adaptive methods. SADDLE POINTS a minimum in some directions and a maximum in others. IN HIGH DIMENSIONS THESE ARE FAR MORE COMMON THAN LOCAL MINIMA — for a point to be a local minimum, the surface must curve upward in EVERY one of thousands of directions, which is improbable. This is a genuinely reassuring result: the classical worry about local minima matters less in large networks than it does in low dimensions. RAVINES long narrow valleys causing oscillation. Response: momentum. VANISHING / the multiplicative decay of the activation EXPLODING functions topic. Response: ReLU, careful GRADIENTS initialisation, gradient clipping, residual connections. THE SUMMARY WORTH KEEPING: GRADIENT DESCENT IS A LOCAL METHOD THAT ONLY KNOWS THE SLOPE WHERE IT STANDS. Everything in this topic — the learning rate, schedules, momentum, adaptive rates, mini-batching — is an attempt to make a purely local rule behave sensibly on a surface it cannot see.