FROM THE PREVIOUS TOPIC: without a nonlinear activation, stacked layers collapse. y = W⁽²⁾(W⁽¹⁾x + b⁽¹⁾) + b⁽²⁾ = (W⁽²⁾W⁽¹⁾)x + (W⁽²⁾b⁽¹⁾ + b⁽²⁾) = W′x + b′ A HUNDRED LINEAR LAYERS ARE ONE LINEAR LAYER. So the activation function is not a refinement — it is the reason depth exists. WHAT AN ACTIVATION FUNCTION MUST PROVIDE: 1. NONLINEARITY, so that composition adds expressive power 2. DIFFERENTIABILITY (or near enough), so gradient-based learning can work — this is why the McCulloch-Pitts step function had to be abandoned 3. A USEFUL DERIVATIVE — not merely nonzero, but not vanishingly small either, as the last section shows 4. CHEAPNESS, since it is evaluated once per unit per example per pass AND ONE PROPERTY THAT IS DESIRABLE RATHER THAN REQUIRED: ZERO-CENTRED OUTPUT. If a layer's outputs are all positive, the gradients for the next layer's weights all share a sign, so the weight vector can only move in certain diagonal directions and convergence zigzags. This is the standard argument for tanh over sigmoid.

STEP (THRESHOLD, HEAVISIDE) f(x) = 1 if x ≥ 0, else 0 derivative: 0 everywhere it exists, undefined at 0 ✔ the original, used by McCulloch-Pitts and the perceptron ✘ NOT USABLE WITH GRADIENT DESCENT — a zero derivative carries no information about which way to move the weights. THIS IS WHY THE PERCEPTRON NEEDED ITS OWN LEARNING RULE rather than gradient descent. LINEAR (IDENTITY) f(x) = x, f′(x) = 1 ✔ used in the OUTPUT layer for REGRESSION, where the output must be an unbounded real number ✘ useless in hidden layers, by the collapse argument SIGMOID (LOGISTIC) f(x) = 1 / (1 + e^−x) range (0, 1) f′(x) = f(x)(1 − f(x)) THE DERIVATIVE IS EXPRESSED IN TERMS OF THE OUTPUT, which is why implementations cache the forward-pass activations — the backward pass needs them and would otherwise recompute. WORKED VALUES: x = −2 → 0.1192 x = 0 → 0.5000 x = 2 → 0.8808 DERIVATIVE VALUES: x = 0 → 0.5(0.5) = 0.2500 ← the MAXIMUM x = 1 → 0.7311(0.2689) = 0.1966 x = 2 → 0.8808(0.1192) = 0.1050 x = 4 → 0.9820(0.0180) = 0.0177 ✔ smooth, bounded, interpretable as a probability ✔ still the standard for a BINARY output unit ✘ VANISHING GRADIENT — the derivative never exceeds 0.25, and approaches zero when the unit saturates ✘ NOT ZERO-CENTRED ✘ the exponential is comparatively expensive TANH (HYPERBOLIC TANGENT) f(x) = (eˣ − e^−x)/(eˣ + e^−x) range (−1, 1) f′(x) = 1 − f(x)² and note tanh(x) = 2·sigmoid(2x) − 1, so it is a rescaled sigmoid WORKED: x = −2 → −0.9640, x = 0 → 0, x = 2 → +0.9640 MAXIMUM DERIVATIVE: 1 − 0² = 1.0 at x = 0 — FOUR TIMES the sigmoid's ✔ ZERO-CENTRED, which helps convergence ✔ a larger maximum gradient than sigmoid ✘ still SATURATES at both ends, so the vanishing gradient problem is reduced and not solved ReLU (RECTIFIED LINEAR UNIT) f(x) = max(0, x) f′(x) = 1 if x > 0, else 0 (undefined at 0; taken as 0) WORKED: x = −2 → 0, x = 0 → 0, x = 0.5 → 0.5, x = 2 → 2 ✔ NO SATURATION for positive input — the gradient is exactly 1, so it neither shrinks nor grows as it propagates. THIS IS THE PROPERTY THAT MADE DEEP NETWORKS TRAINABLE. ✔ extremely cheap: a comparison, no exponential ✔ produces SPARSE activation, since negative inputs give exactly zero ✘ THE DYING ReLU PROBLEM: a unit whose input is always negative has gradient zero forever, so it never recovers and is permanently dead. A large learning rate can kill a substantial fraction of a layer. ✘ not zero-centred, and unbounded above LEAKY ReLU f(x) = x if x > 0, else αx with α small, typically 0.01 WORKED at x = −2, α = 0.01: f = −0.02, and the derivative is 0.01 rather than 0 ✔ FIXES THE DYING ReLU PROBLEM — a dead unit retains a small gradient and can recover (PARAMETRIC ReLU makes α a learned parameter.) ELU (EXPONENTIAL LINEAR UNIT) f(x) = x if x > 0, else α(eˣ − 1) WORKED at x = −2, α = 1: 1(e^−2 − 1) = −0.8647 ✔ smooth, closer to zero-centred than ReLU ✘ the exponential costs more SOFTMAX — for a MULTI-CLASS OUTPUT LAYER f(zᵢ) = e^{zᵢ} / Σⱼ e^{zⱼ} It converts a vector of scores into a PROBABILITY DISTRIBUTION: every output is in (0,1) and they sum to 1. WORKED with z = (2.0, 1.0, 0.1): e^2.0 = 7.3891 e^1.0 = 2.7183 e^0.1 = 1.1052 sum = 11.2125 softmax = (7.3891/11.2125, 2.7183/11.2125, 1.1052/11.2125) = (0.6590, 0.2424, 0.0986) and 0.6590 + 0.2424 + 0.0986 = 1.0000 ✓ NOTE THAT IT IS NOT APPLIED ELEMENTWISE — each output depends on ALL the inputs, which is what makes the outputs compete and sum to one. Every other function in this list is elementwise.

THE VANISHING GRADIENT PROBLEM — the central practical reason the choice of activation matters, and the arithmetic makes it plain. Backpropagation (a later topic) computes a gradient for an early layer by MULTIPLYING the derivatives along the path back from the output. With sigmoid units, each factor is at most 0.25. after 1 layer: 0.25 = 2.5 × 10⁻¹ after 5 layers: 0.25⁵ = 9.77 × 10⁻⁴ after 10 layers: 0.25¹⁰ = 9.54 × 10⁻⁷ after 20 layers: 0.25²⁰ = 9.10 × 10⁻¹³ THE GRADIENT REACHING AN EARLY LAYER IN A 20-LAYER SIGMOID NETWORK IS ABOUT A TRILLIONTH OF THE GRADIENT AT THE OUTPUT. In floating point that is indistinguishable from zero, so the early layers DO NOT LEARN AT ALL — they keep their random initial weights while the last few layers do all the work. AND THAT IS AN OPTIMISTIC CALCULATION, because 0.25 is the MAXIMUM derivative, attained only at x = 0. A saturated unit contributes far less. THIS IS WHY DEEP NETWORKS WERE UNTRAINABLE BEFORE ReLU. The architecture was known — the multilayer perceptron dates from the 1980s — and the gradients simply did not arrive. With ReLU each factor is exactly 1 for active units, so 1²⁰ = 1 and the gradient reaches the first layer undiminished. THE OPPOSITE FAILURE, EXPLODING GRADIENTS, occurs when the factors exceed 1 and the product grows without bound. The standard remedy is GRADIENT CLIPPING — cap the gradient's magnitude — and careful initialisation. OTHER RESPONSES TO THE PROBLEM, worth naming: BATCH NORMALISATION normalise each layer's inputs, keeping units away from their saturated regions RESIDUAL CONNECTIONS add a shortcut path that skips layers, so the gradient has a route back that does not pass through every nonlinearity. This is what permitted networks of hundreds of layers. BETTER INITIALISATION Xavier and He scaling, as the previous topic noted HOW TO CHOOSE — the practical guidance: WHERE USE ───────────────────────────────────────────────────────────── hidden layers, default ReLU (or Leaky ReLU if units are dying) hidden layers, shallow net tanh is acceptable output, binary SIGMOID — gives a probability classification output, multi-class SOFTMAX — gives a distribution classification output, regression LINEAR — unbounded real value ───────────────────────────────────────────────────────────── THE ONE RULE THAT MATTERS MOST: THE OUTPUT ACTIVATION IS DETERMINED BY THE TASK, NOT BY PREFERENCE. A regression network with a sigmoid output cannot predict a value above 1; a multi-class network without softmax produces scores that do not compare. THE HIDDEN ACTIVATIONS ARE A TUNING CHOICE; THE OUTPUT ACTIVATION IS PART OF THE PROBLEM SPECIFICATION. A SUMMARY TABLE OF THE DERIVATIVES, since backpropagation needs them and exams ask for them: f(x) f′(x) ───────────────────────────────────────────── step 0 (undefined at 0) linear x 1 sigmoid σ(x) σ(x)(1 − σ(x)) max 0.25 tanh tanh(x) 1 − tanh²(x) max 1.0 ReLU max(0,x) 1 if x > 0 else 0 leaky x or αx 1 if x > 0 else α ───────────────────────────────────────────── THE SIGMOID AND TANH DERIVATIVES BEING EXPRESSIBLE IN THE FUNCTION'S OWN OUTPUT is a genuine computational convenience — the backward pass reuses the forward pass's stored activations and needs no fresh exponentials.