THE COMPUTATION OF ONE ARTIFICIAL NEURON, in full: GIVEN inputs x₁, x₂, …, xₙ and WEIGHTS w₁, w₂, …, wₙ and a BIAS b NET INPUT (also the activation potential, or pre-activation): net = w₁x₁ + w₂x₂ + … + wₙxₙ + b = Σᵢ wᵢxᵢ + b = wᵀx + b in vector form OUTPUT: y = f(net) where f is the ACTIVATION FUNCTION. THE ROLE OF EACH PART: THE WEIGHTS determine how much each input matters, and their SIGN determines whether the input is excitatory (positive) or inhibitory (negative). LEARNING MEANS CHANGING THE WEIGHTS — nothing else about the network changes. THE BIAS shifts the threshold. Without it, net = 0 whenever all inputs are 0, so the neuron could not produce a nonzero output for a zero input, and the decision boundary would be forced through the origin. THE BIAS IS WHAT LETS THE BOUNDARY SIT ANYWHERE. THE ACTIVATION FUNCTION introduces NONLINEARITY, and the next topic is devoted to why that matters. THE BIAS AS A WEIGHT — a notational convenience used everywhere: introduce a dummy input x₀ = 1 with weight w₀ = b. Then net = Σᵢ₌₀ⁿ wᵢxᵢ and the bias needs no separate treatment. Every learning rule then updates w₀ exactly as it updates any other weight, WHICH IS WHY IMPLEMENTATIONS DO THIS — it removes a special case from the code. WORKED, one neuron: x = (1, 0, 1), w = (0.5, −0.4, 0.2), b = −0.3 net = (0.5)(1) + (−0.4)(0) + (0.2)(1) + (−0.3) = 0.5 + 0 + 0.2 − 0.3 = 0.4 with a STEP activation (threshold 0): y = 1 with a SIGMOID activation: y = 1/(1 + e^−0.4) = 0.5987 NOTE THAT THE SAME NET INPUT GIVES A DIFFERENT OUTPUT DEPENDING ON f, and that the sigmoid's output is a graded "just above the middle" where the step function commits absolutely. THE GEOMETRIC INTERPRETATION, which explains the whole limitation of a single neuron: The equation wᵀx + b = 0 defines a HYPERPLANE in the input space: in 2 dimensions, a LINE in 3 dimensions, a PLANE in n dimensions, an (n−1)-dimensional hyperplane The neuron outputs one class on one side and the other class on the other side. SO A SINGLE NEURON IS A LINEAR SEPARATOR, and it can only classify correctly when the classes are LINEARLY SEPARABLE. w determines the ORIENTATION of the boundary b determines its DISTANCE FROM THE ORIGIN WORKED, in two dimensions: w = (1, 1), b = −1.5, so the boundary is x₁ + x₂ = 1.5 (0,0): net = −1.5 → class 0 (0,1): net = −0.5 → class 0 (1,0): net = −0.5 → class 0 (1,1): net = +0.5 → class 1 THIS IS THE AND FUNCTION, and the line x₁ + x₂ = 1.5 separates (1,1) from the other three points. FOR XOR NO SUCH LINE EXISTS, which is the geometric form of the proof in the previous topic.

A LAYER of m neurons, each receiving the same n inputs: neuron j computes netⱼ = Σᵢ wⱼᵢ xᵢ + bⱼ yⱼ = f(netⱼ) COLLECT THE WEIGHTS INTO A MATRIX W of shape m × n, where row j holds neuron j's weights. Then the whole layer is: net = W x + b y = f(net) f applied ELEMENTWISE THE SHAPES, and getting these right is most of implementing a network: x is n × 1 W is m × n b is m × 1 net is m × 1 y is m × 1 A MULTILAYER NETWORK is this repeated. For a network with input x, one hidden layer and an output layer: h = f₁(W⁽¹⁾ x + b⁽¹⁾) y = f₂(W⁽²⁾ h + b⁽²⁾) and in general, for layer ℓ: a⁽⁰⁾ = x a⁽ˡ⁾ = f⁽ˡ⁾( W⁽ˡ⁾ a⁽ˡ⁻¹⁾ + b⁽ˡ⁾ ) output = a⁽ᴸ⁾ THAT RECURRENCE IS THE ENTIRE FORWARD PASS OF ANY FEEDFORWARD NETWORK. Everything else in this section — activation functions, learning rules, backpropagation — is either a choice of f or a method for adjusting W. COUNTING PARAMETERS — a standard exam calculation: a layer with n inputs and m outputs has m × n weights + m biases = m(n + 1) parameters WORKED, a network with 4 inputs, a hidden layer of 5, and 3 outputs: layer 1: 5 × (4 + 1) = 25 parameters layer 2: 3 × (5 + 1) = 18 parameters TOTAL 43 parameters A LARGER ONE, 784 inputs (a 28×28 image), hidden layer of 128, 10 outputs: layer 1: 128 × 785 = 100,480 layer 2: 10 × 129 = 1,290 TOTAL 101,770 parameters A MODEST NETWORK ALREADY HAS OVER A HUNDRED THOUSAND PARAMETERS, every one of which is set by training. That figure explains both why networks need data and why they are opaque — there is no reading a hundred thousand numbers. BATCHING — how it is actually computed: Stack B input vectors as the columns of a matrix X of shape n × B. Then NET = W X + b (b broadcast across columns) processes the whole BATCH in one matrix multiplication. THIS IS WHY GPUs MATTER. One large matrix multiplication is vastly more efficient than B small ones, so batching converts a sequence of small operations into a single parallel one — and the batch size becomes a memory-versus-throughput decision.

THE STANDARD TERMINOLOGY, since exam questions use it precisely: INPUT LAYER the inputs themselves. NOT usually counted as a layer, because it performs no computation — so a "two-layer network" normally means one hidden layer plus an output layer. STATE YOUR CONVENTION, because texts differ. HIDDEN LAYER any layer between input and output. "Hidden" because its values are not observed in the training data. OUTPUT LAYER produces the network's answer DEPTH the number of layers with weights WIDTH the number of units in a layer FEEDFORWARD connections go only forward; the network is a directed acyclic graph RECURRENT connections form cycles, so the network has internal state and can process sequences FULLY CONNECTED every unit in a layer connects to every unit (DENSE) in the next THE ARCHITECTURE CHOICES that the mathematical model leaves open, each of which the following topics address: how many layers, and how wide which activation function how the weights are INITIALISED — and this matters more than it appears: initialising all weights to the same value makes every unit in a layer compute the same thing and receive the same update, so THE LAYER NEVER DIFFERENTIATES. Random initialisation breaks that symmetry, and it is why weights are never initialised to zero. how the error is measured how the weights are updated WHAT THE MODEL IMPLIES, and these consequences are worth drawing out: 1. THE NETWORK IS A DIFFERENTIABLE FUNCTION of its parameters, provided f is differentiable. That single property is what makes gradient-based learning possible, and it is why the step function of the McCulloch-Pitts neuron had to be replaced. 2. THE FORWARD PASS IS CHEAP AND PARALLEL — matrix products. Training is expensive because it repeats the forward pass and a backward pass many times over many examples. 3. WITHOUT A NONLINEARITY THE WHOLE NETWORK COLLAPSES. If every f is the identity, then y = W⁽²⁾(W⁽¹⁾x + b⁽¹⁾) + b⁽²⁾ = (W⁽²⁾W⁽¹⁾)x + (W⁽²⁾b⁽¹⁾ + b⁽²⁾) = W′x + b′ — a SINGLE LINEAR LAYER, however many layers were stacked. DEPTH BUYS NOTHING WITHOUT NONLINEARITY. This is the most important consequence of the model and the reason the next topic exists. 4. THE KNOWLEDGE IS IN THE WEIGHTS, distributed across the matrices. There is no symbol, no rule, and nothing to read — which is the opacity the previous topic listed as the principal disadvantage. A CLOSING OBSERVATION ON WHY THIS MATTERS FOR THE REST OF THE SECTION: every remaining topic is a variation on a⁽ˡ⁾ = f( W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾ ) The perceptron fixes f as a step and learns W for one layer; the delta rule makes f differentiable; backpropagation computes the gradient of the error with respect to every W; a Hopfield network makes the connections symmetric and recurrent. ONE EQUATION, MANY SPECIALISATIONS.