ADALINE (ADAptive LInear NEuron, Widrow and Hoff, 1960) is a single unit that looks almost identical to a perceptron and differs in one decisive place. THE STRUCTURE: inputs x₁ … xₙ, with a bias input x₀ = 1 weights w₀ … wₙ net input net = Σᵢ wᵢxᵢ LINEAR OUTPUT a = net ← used for LEARNING final output y = +1 if a ≥ 0, else −1 ← used for CLASSIFICATION THE CRUCIAL POINT, and it is the whole of the difference from a perceptron: THE PERCEPTRON COMPUTES ITS ERROR FROM THE THRESHOLDED OUTPUT. ADALINE COMPUTES ITS ERROR FROM THE LINEAR OUTPUT, BEFORE THE THRESHOLD. So the error term is (t − a) with a real-valued, rather than (t − y) with y ∈ {−1, +1}. The threshold is still applied to produce the final classification — it is simply not involved in learning. x₀=1 ──w₀──┐ x₁ ────w₁──┤ ┌─────┐ a (linear) ┌──────────┐ x₂ ────w₂──┼───▶│ Σ │───────┬──────────▶│ threshold│──▶ y ⋮ │ └─────┘ │ └──────────┘ xₙ ────wₙ──┘ │ ▼ LEARNING uses a, via δ = t − a ADALINE USES BIPOLAR VALUES (−1, +1) rather than binary (0, 1). This matters for the same reason it did in Hebbian learning: a zero input produces no weight change, so with 0/1 encoding half the weights receive no updates from half the examples. THE LEARNING RULE IS THE DELTA RULE, derived in the previous topic: Δwᵢ = η (t − a) xᵢ also called the WIDROW-HOFF rule or the LMS (least mean squares) rule — the names are interchangeable, and Adaline is where they originate. THE ERROR FUNCTION being minimised: E = ½ Σₚ (tₚ − aₚ)² the MEAN SQUARED ERROR and because a is linear in the weights, THE ERROR SURFACE IS A PARABOLOID with a single global minimum. Gradient descent on it cannot get stuck, which is the guarantee the perceptron lacks.

TRAINING ADALINE ON THE AND FUNCTION with bipolar encoding: inputs and targets in {−1, +1}, η = 0.1, weights starting at 0. Training set (with x₀ = 1 for the bias): (1, −1, −1) → t = −1 (1, −1, +1) → t = −1 (1, +1, −1) → t = −1 (1, +1, +1) → t = +1 EPOCH 1, updating after each example: example 1: a = 0 δ = −1 − 0 = −1 w₀ ← 0 + 0.1(−1)(1) = −0.1 w₁ ← 0 + 0.1(−1)(−1) = +0.1 w₂ ← 0 + 0.1(−1)(−1) = +0.1 w = (−0.1, 0.1, 0.1) example 2: a = −0.1 + 0.1(−1) + 0.1(1) = −0.1 δ = −1 − (−0.1) = −0.9 w₀ ← −0.1 + 0.1(−0.9)(1) = −0.19 w₁ ← 0.1 + 0.1(−0.9)(−1) = +0.19 w₂ ← 0.1 + 0.1(−0.9)(1) = +0.01 w = (−0.19, 0.19, 0.01) example 3: a = −0.19 + 0.19(1) + 0.01(−1) = −0.01 δ = −1 − (−0.01) = −0.99 w₀ ← −0.289, w₁ ← 0.091, w₂ ← 0.109 example 4: a = −0.289 + 0.091 + 0.109 = −0.089 δ = 1 − (−0.089) = 1.089 w₀ ← −0.180, w₁ ← 0.200, w₂ ← 0.218 AFTER ONE EPOCH the weights are already moving toward a sensible solution, and note that EVERY EXAMPLE PRODUCED AN UPDATE — including the ones already classified correctly, because the linear output was not yet close to the target. THAT IS THE DIFFERENCE FROM THE PERCEPTRON IN ONE OBSERVATION: a perceptron would have made no change on a correctly classified example, while Adaline continues to refine the margin. CONTINUING TO CONVERGENCE gives weights near w ≈ (−0.5, 0.5, 0.5) which gives a = −0.5 + 0.5x₁ + 0.5x₂, so (−1,−1): a = −1.5 → y = −1 ✓ (−1,+1): a = −0.5 → y = −1 ✓ (+1,−1): a = −0.5 → y = −1 ✓ (+1,+1): a = +0.5 → y = +1 ✓ ALL FOUR CORRECT, and the linear outputs are symmetric about the boundary rather than sitting arbitrarily close to it. THE COMPARISON — PERCEPTRON versus ADALINE: PERCEPTRON ADALINE ───────────────────────────────────────────────────────────── error from the THRESHOLDED the LINEAR output a output y learning rule error correction DELTA RULE (gradient descent) error function none explicit mean squared error encoding binary 0/1 BIPOLAR −1/+1 updates on a correct example? NO YES, if a ≠ t if not separable never terminates converges to minimum MSE solution quality any separator the least-squares separator, with a more robust margin ───────────────────────────────────────────────────────────── THE ROW ABOUT UPDATING ON CORRECT EXAMPLES IS THE MOST INSTRUCTIVE. Adaline keeps pushing outputs toward the targets even when the classification is already right, which is why its boundaries tend to sit further from the data and generalise better — it is minimising a distance, not counting mistakes.

MADALINE (Many ADALINEs) — the multilayer extension, and the first attempt at training a hidden layer. THE ARCHITECTURE: a layer of Adaline units feeding a fixed combining unit — typically an AND, OR or MAJORITY gate — whose weights are NOT trained. x ──┬──▶ [Adaline 1] ──┐ ├──▶ [Adaline 2] ──┼──▶ [ MAJORITY ] ──▶ y └──▶ [Adaline 3] ──┘ MADALINE CAN SOLVE XOR, using two Adalines and an AND gate: unit 1 learns one line, unit 2 learns another, and the fixed gate combines them so that the region between the two lines is classified differently from the regions outside. TWO LINEAR BOUNDARIES CAN CARVE OUT A NON-LINEARLY-SEPARABLE REGION, which is the geometric content of the hidden-layer idea. THE MADALINE TRAINING RULES: MRI (Madaline Rule I, 1962): the output combiner is fixed, so only the first layer is trained. When the output is wrong, adjust the Adaline whose linear output is CLOSEST TO ZERO — the one whose decision was least confident and therefore cheapest to flip. Adjust it just enough to change its sign, then check whether the output is now correct. THIS IS A HEURISTIC, NOT GRADIENT DESCENT — it uses the "minimum disturbance" principle rather than a derivative, precisely because the threshold blocks the gradient. MRII (1988): extends the trial-and-flip idea to multiple layers. MRIII: uses sigmoid units, at which point it becomes essentially backpropagation. WHY MADALINE MATTERED AND WHY IT WAS SUPERSEDED: IT SHOWED THAT MULTIPLE ADAPTIVE UNITS COULD SOLVE NON-SEPARABLE PROBLEMS, twenty-four years before backpropagation became widely known. But MRI trains only one layer and relies on a fixed combiner, so it does not solve the general hidden-layer problem — it works around it. THE MISSING PIECE WAS STILL A DERIVATIVE, and getting one required replacing the threshold with a differentiable function. ADALINE'S LASTING CONTRIBUTIONS: 1. THE DELTA RULE ITSELF, which is the foundation of gradient-based learning and, with one added factor f′(net), becomes backpropagation's output-layer rule. 2. THE INSIGHT THAT LEARNING SHOULD USE THE PRE-THRESHOLD OUTPUT. This is the step that made the error continuous and therefore differentiable. 3. ADAPTIVE FILTERING as an engineering discipline — echo cancellation, noise cancellation, channel equalisation, all still built on LMS. 4. It established that a network could be trained ONLINE, on a live signal, adapting continuously rather than being trained once — which is what a telephone line requires. THE PROGRESSION, restated with Adaline in place: MP NEURON no learning PERCEPTRON learns; error from the thresholded output; fails if not separable ADALINE learns by GRADIENT DESCENT on the linear output; converges either way; still one layer MADALINE several Adalines plus a fixed gate; solves XOR; trains only one layer MLP + BACKPROP differentiable units throughout, so every layer gets a gradient EACH STEP REMOVES ONE OBSTACLE, and Adaline's contribution was to make the error differentiable — without which none of what follows is possible.