THE PROBLEM WITH THE PERCEPTRON RULE: it uses the STEP output, so it knows only whether the answer was right or wrong — not by how much. And if the data is not linearly separable it never terminates. THE DELTA RULE (also the WIDROW-HOFF rule, or the LEAST MEAN SQUARES rule) fixes both by working with a LINEAR output and minimising a CONTINUOUS error. THE SETUP: output y = net = Σᵢ wᵢxᵢ + b (LINEAR — no step function) error for one E = ½ (t − y)² example over the set E(w) = ½ Σₚ (tₚ − yₚ)² THE DERIVATION, which is short and worth being able to reproduce: ∂E/∂wᵢ = ∂/∂wᵢ [ ½(t − y)² ] = ½ · 2(t − y) · ∂(t − y)/∂wᵢ chain rule = (t − y) · (−∂y/∂wᵢ) = −(t − y) · xᵢ since y = Σwⱼxⱼ, ∂y/∂wᵢ = xᵢ NOTE WHERE THE ½ WENT: it cancelled against the 2 from differentiating the square. That is its only purpose. SUBSTITUTING INTO GRADIENT DESCENT wᵢ ← wᵢ − η ∂E/∂wᵢ: wᵢ ← wᵢ + η (t − y) xᵢ THE DELTA RULE. Writing δ = (t − y) for the error term gives the form the name comes from: Δwᵢ = η · δ · xᵢ COMPARE WITH THE PERCEPTRON RULE, which looks identical: perceptron: Δwᵢ = η (t − y) xᵢ with y from a STEP delta rule: Δwᵢ = η (t − y) xᵢ with y LINEAR THE ALGEBRA IS THE SAME AND THE MEANING IS NOT. In the perceptron y ∈ {0,1}, so (t − y) ∈ {−1, 0, +1} — three possible corrections. In the delta rule y is a real number, so (t − y) is PROPORTIONAL TO HOW WRONG THE ANSWER WAS. A nearly-correct output produces a tiny update; a badly wrong one produces a large one. THAT IS THE ENTIRE DIFFERENCE, AND IT IS DECISIVE — it is what makes the rule a gradient method rather than a correct-the-mistake heuristic. THE ERROR SURFACE for a linear unit with squared error is a PARABOLOID: convex, with a SINGLE GLOBAL MINIMUM and no local minima. So gradient descent on it CANNOT get stuck, and with a suitably small η it converges to the best possible weights whether or not the data is separable.

TRAINING A SINGLE LINEAR UNIT to learn y = 2x, with η = 0.1 and w starting at 0. (No bias, for clarity.) Training set: (x, t) = (1, 2), (2, 4), (3, 6) EPOCH 1, updating after each example (stochastic): x = 1, t = 2: y = 0(1) = 0 δ = 2 − 0 = 2 w ← 0 + 0.1(2)(1) = 0.2 x = 2, t = 4: y = 0.2(2) = 0.4 δ = 4 − 0.4 = 3.6 w ← 0.2 + 0.1(3.6)(2) = 0.92 x = 3, t = 6: y = 0.92(3) = 2.76 δ = 6 − 2.76 = 3.24 w ← 0.92 + 0.1(3.24)(3) = 1.892 END OF EPOCH 1: w = 1.8920, SSE = 0.1633 SUBSEQUENT EPOCHS: epoch 2: w = 1.9942, SSE = 0.000476 epoch 3: w = 1.9997, SSE = 0.000001 epoch 4: w = 2.0000, SSE = 0.000000 CONVERGED TO THE EXACT ANSWER IN FOUR EPOCHS, and note how the error falls: 0.16 → 0.0005 → 0.000001. THE CONVERGENCE IS GEOMETRIC, because each step reduces the remaining error by a roughly constant factor — which is the behaviour of gradient descent on a quadratic. CONTRAST WITH THE PERCEPTRON TRACE of the previous topic, which took six epochs on four examples and WANDERED — reaching a good weight vector, moving away, and returning. THE DELTA RULE APPROACHES MONOTONICALLY, because every step reduces a continuous error rather than merely correcting a classification. THE TWO FORMS OF THE RULE, and the distinction is examinable: BATCH (TRUE GRADIENT DESCENT) accumulate Δwᵢ = η Σₚ δₚ xᵢₚ over the whole training set, then apply it once ✔ this is the TRUE gradient of E, so the descent direction is exact ✔ converges to the global minimum for sufficiently small η ✘ one update per epoch INCREMENTAL / STOCHASTIC (the LMS rule) apply Δwᵢ = η δ xᵢ after each example ✔ many more updates per epoch, so much faster in practice ✔ the noise can help escape poor regions in nonlinear networks ✘ it only APPROXIMATES gradient descent, and does not settle exactly at the minimum — it hovers near it, in a region whose size is proportional to η → this is why the learning rate is often decayed toward the end of training: the hovering shrinks with η. THE WORKED EXAMPLE ABOVE USED THE INCREMENTAL FORM, which is why it converged in four epochs rather than needing many more.

THE THREE LEARNING RULES SO FAR, side by side: PERCEPTRON DELTA RULE ───────────────────────────────────────────────────────────── output step (0/1) LINEAR (real-valued) error used classification the actual difference right/wrong t − y error function none explicit E = ½Σ(t − y)² method error GRADIENT DESCENT correction if separable converges to a converges to the perfect minimum-error solution separator if NOT NEVER CONVERGES to the best separable TERMINATES — linear approximation oscillates forever guarantee finite steps asymptotic; global minimum when separable (the surface is convex) ───────────────────────────────────────────────────────────── THE PRACTICALLY IMPORTANT ROW IS "IF NOT SEPARABLE". Real data is noisy and overlapping, so the perceptron's guarantee is frequently void — and the algorithm's behaviour in that case is to cycle indefinitely without any indication that it will never finish. THE DELTA RULE DEGRADES GRACEFULLY, returning the best available linear fit. THAT IS WHY IT IS THE PRACTICAL RULE. A SUBTLETY WORTH KNOWING: when the data IS separable, the delta rule may produce a boundary that MISCLASSIFIES an example the perceptron would have got right — because it minimises SQUARED ERROR rather than counting mistakes, and a distant correctly classified point can pull the boundary. THE TWO RULES OPTIMISE DIFFERENT THINGS, and neither dominates on every dataset. WHAT THE DELTA RULE ENABLED — its real historical significance: 1. IT MADE LEARNING A GRADIENT-DESCENT PROBLEM. Once the error is a differentiable function of the weights, the whole machinery of optimisation applies — learning rates, momentum, adaptive methods, all from the previous topic. 2. IT REQUIRED A DIFFERENTIABLE ACTIVATION, which is why the step function had to go. That, in turn, is what made the sigmoid the standard hidden activation for decades. 3. IT GENERALISES TO NONLINEAR UNITS. With a differentiable activation f, the derivation gains one factor: ∂E/∂wᵢ = −(t − y) · f′(net) · xᵢ so Δwᵢ = η (t − y) f′(net) xᵢ THAT EXTRA f′(net) IS THE ONLY CHANGE, and it is exactly the term that appears in backpropagation's output layer. The delta rule IS backpropagation for a single-layer network. 4. IT DOES NOT SOLVE THE HIDDEN-LAYER PROBLEM. The rule still needs a target t, and a hidden unit has none. Supplying that missing target — by propagating the output error backward through the weights — is precisely what backpropagation adds, and it is the subject of a later topic in this section. THE ONE-LINE SUMMARY: THE DELTA RULE IS GRADIENT DESCENT ON SQUARED ERROR FOR ONE LAYER. Everything after it in this section is that idea extended — to nonlinear units, then to many layers.