THE PROBLEM INHERITED FROM THE PERCEPTRON TOPIC: a single layer computes a single hyperplane, so it cannot represent XOR or any other non-linearly-separable function. Minsky and Papert's 1969 proof of this stalled the field for a decade. THE ARCHITECTURE THAT SOLVES IT: INPUT LAYER HIDDEN LAYER OUTPUT LAYER (no computation) (the new part) x₁ ─────┬────────▶ h₁ ────┬───────▶ y₁ ╳ ╳ x₂ ─────┴────────▶ h₂ ────┴───────▶ y₂ ⋮ xₙ ──────────────▶ h_m ───────────▶ y_k FULLY CONNECTED between adjacent layers, FEEDFORWARD (no cycles, no connections within a layer, no skipping backwards). THE COMPUTATION, layer by layer: hidden: h_j = f( Σᵢ v_ji xᵢ + b_j ) output: y_k = g( Σⱼ w_kj h_j + c_k ) THE ABSOLUTELY CRITICAL CONDITION — AND THE MOST COMMON EXAM TRAP: f MUST BE NON-LINEAR. If f is the identity, then y = W(Vx) = (WV)x = W'x and the composition of two linear maps IS A SINGLE LINEAR MAP. A HUNDRED LINEAR LAYERS COLLAPSE TO ONE MATRIX. The network would have more parameters and exactly the same expressive power as a single perceptron. THE NON-LINEARITY IS NOT A DETAIL OF IMPLEMENTATION — IT IS THE ENTIRE REASON DEPTH BUYS ANYTHING. WHAT THE HIDDEN LAYER ACTUALLY DOES, geometrically: EACH HIDDEN UNIT DRAWS ONE HYPERPLANE in the input space, exactly as a perceptron does. The output layer then combines those hyperplanes. · 1 hidden unit → 1 line → 2 regions · 2 hidden units → 2 lines → up to 4 regions · 3 hidden units → 3 lines → up to 7 regions · n hidden units → up to 1 + n + n(n−1)/2 regions in 2D So the hidden layer converts the input into a new representation — one coordinate per hyperplane, recording which side of it the input falls on — AND IN THAT NEW SPACE THE PROBLEM IS LINEARLY SEPARABLE. That is the mechanism in one sentence: THE HIDDEN LAYER RE-REPRESENTS THE DATA UNTIL A LINEAR OUTPUT UNIT CAN FINISH THE JOB.

THE CANONICAL DEMONSTRATION. XOR needs 2 inputs, 2 hidden units and 1 output — NINE PARAMETERS in total (6 weights + 3 biases). THE IDEA: XOR = OR AND NOT(AND) = OR AND NAND. Neither OR nor NAND is hard — both are linearly separable. So let hidden unit h₁ compute OR, hidden unit h₂ compute NAND, and let the output compute AND of the two. THE WEIGHTS (step units, threshold at 0): h₁ (OR): net = 20x₁ + 20x₂ − 10 h₂ (NAND): net = −20x₁ − 20x₂ + 30 y (AND): net = 20h₁ + 20h₂ − 30 THE FULL TRACE, all four cases: x₁ x₂ │ h₁ net h₁ │ h₂ net h₂ │ y net y │ XOR ──────┼────────────┼────────────┼──────────┼───── 0 0 │ −10 0 │ +30 1 │ −10 0 │ 0 ✓ 0 1 │ +10 1 │ +10 1 │ +10 1 │ 1 ✓ 1 0 │ +10 1 │ +10 1 │ +10 1 │ 1 ✓ 1 1 │ +30 1 │ −10 0 │ −10 0 │ 0 ✓ ALL FOUR CORRECT. And note WHY it works: in the hidden space, the four inputs map to (0,1), (1,1), (1,1) and (1,0). THE TWO "TRUE" CASES COLLAPSED ONTO THE SAME POINT (1,1), and a single line now separates (1,1) from the other two. THE HIDDEN LAYER MADE THE PROBLEM SEPARABLE BY MERGING THE TWO CASES THAT NEEDED THE SAME ANSWER. THE SAME WEIGHTS WITH SIGMOID UNITS, since real MLPs are differentiable: (0,0) → h = (0.0000, 1.0000) → y = 0.0000 (0,1) → h = (1.0000, 1.0000) → y = 1.0000 (1,0) → h = (1.0000, 1.0000) → y = 1.0000 (1,1) → h = (1.0000, 0.0000) → y = 0.0000 The large weights (±20) drive the sigmoids deep into saturation, so they behave almost exactly like steps — which shows that the step network is the limiting case of the sigmoid one, and that nothing was lost by making the units differentiable.

THE UNIVERSAL APPROXIMATION THEOREM (Cybenko 1989; Hornik 1991): A FEEDFORWARD NETWORK WITH A SINGLE HIDDEN LAYER CONTAINING A FINITE NUMBER OF UNITS, USING A NON-CONSTANT, BOUNDED, CONTINUOUS ACTIVATION FUNCTION, CAN APPROXIMATE ANY CONTINUOUS FUNCTION ON A CLOSED BOUNDED SUBSET OF ℝⁿ TO ANY DESIRED ACCURACY. READ THE FINE PRINT, because the theorem is routinely overstated: · IT IS AN EXISTENCE RESULT. It says suitable weights EXIST. IT DOES NOT SAY YOU CAN FIND THEM, and backpropagation offers no guarantee of doing so. · "FINITE" MAY BE ASTRONOMICAL. The required width can grow exponentially with the input dimension or with the wiggliness of the target function. · IT SAYS NOTHING ABOUT GENERALISATION. Approximating the training data perfectly is exactly what overfitting is. · CONTINUOUS FUNCTIONS ON A COMPACT SET only. SO WHY GO DEEPER THAN ONE HIDDEN LAYER? Because of a separate result about EFFICIENCY rather than possibility: SOME FUNCTIONS REQUIRE EXPONENTIALLY MANY UNITS IN A SHALLOW NETWORK BUT ONLY POLYNOMIALLY MANY IN A DEEP ONE. Depth lets features COMPOSE: layer 1 learns edges, layer 2 combines edges into corners and textures, layer 3 into object parts, layer 4 into objects. A shallow network must enumerate every combination separately; a deep one reuses each feature everywhere it appears. THAT REUSE IS THE ENTIRE ARGUMENT FOR DEEP LEARNING — not "one layer cannot do it", but "one layer would need absurdly many units to do it". SIZING THE NETWORK — the practical questions: INPUT UNITS: fixed by the data. One per feature; a 28×28 image flattened gives 784. OUTPUT UNITS: fixed by the task. · regression → 1 linear unit · binary classification → 1 sigmoid unit · k-class classification → k SOFTMAX units, so the outputs form a probability distribution summing to 1 HIDDEN LAYERS: one is enough for most problems by the theorem; two or three help when the function has structure worth composing; more only with the modern machinery (batch normalisation, residual connections) that keeps deep gradients alive. HIDDEN UNITS: THIS IS THE REAL DESIGN DECISION. · too few → UNDERFITTING; the network cannot represent the function, and training error stays high · too many → OVERFITTING; the network memorises the training set, so training error is low and validation error rises · rules of thumb, all crude: between the input and output sizes; ⅔ of input size plus output size; ≤ 2× input size · THE HONEST ANSWER IS TO SEARCH — train several sizes and pick by VALIDATION error, never training error. PARAMETER COUNTS, because they grow faster than intuition suggests: 2−2−1 → 6 weights + 3 biases = 9 parameters 784−30−10 → 23,820 + 40 = 23,860 parameters 784−128−10 → 101,632 + 138 = 101,770 parameters A NETWORK WITH 101,770 PARAMETERS TRAINED ON 60,000 MNIST IMAGES HAS MORE PARAMETERS THAN TRAINING EXAMPLES — which is why regularisation, early stopping and validation monitoring are not optional extras but part of the method. THE STANDARD DESIGN CHOICES beyond size: · WEIGHTS MUST BE INITIALISED RANDOMLY AND SMALL. If all weights start equal, every hidden unit computes the same thing and receives the same gradient forever — THE SYMMETRY NEVER BREAKS and the layer behaves as a single unit. This is a favourite exam question. · INPUTS SHOULD BE NORMALISED, so that no feature dominates the initial gradients purely by scale. · HIDDEN ACTIVATION: ReLU by default in modern practice; sigmoid and tanh in the classical treatment, and tanh in preference to sigmoid because it is zero-centred.