THE PRIMARY CLASSIFICATION is by whether the connection graph has cycles. FEEDFORWARD NETWORKS Connections go in ONE DIRECTION, input toward output. The graph is a DIRECTED ACYCLIC GRAPH. · the output depends ONLY on the current input · NO MEMORY of previous inputs · computation is one pass, and always terminates · suited to STATIC pattern classification RECURRENT NETWORKS Connections form CYCLES — a unit's output can feed back to itself or to an earlier layer. · the network has INTERNAL STATE, so the output depends on the HISTORY of inputs · suited to SEQUENCES and to temporal patterns · computation must be iterated, and may or may not settle THE DISTINCTION IS CONSEQUENTIAL, not cosmetic: a feedforward network computes a FUNCTION of its input, while a recurrent network computes a function of its input AND its state. Only the second can recognise "the third A after the last B". THE FEEDFORWARD ARCHITECTURES, in order of capability: 1. SINGLE-LAYER FEEDFORWARD (the single-layer perceptron) An input layer connected directly to an output layer. Only ONE layer of weights, so it is called single-layer even though it is drawn with two rows of units — THE INPUT LAYER IS NOT COUNTED because it computes nothing. input ──W──▶ output · computes a LINEAR SEPARATION, per the geometric argument of the mathematical-model topic · CANNOT compute XOR · trainable by the perceptron rule or the delta rule 2. MULTILAYER FEEDFORWARD (the multilayer perceptron, MLP) One or more HIDDEN layers between input and output. input ──W⁽¹⁾──▶ hidden ──W⁽²⁾──▶ output · can represent NON-LINEARLY-SEPARABLE functions, including XOR · with enough hidden units, approximates any continuous function — the universal approximation theorem · trained by BACKPROPAGATION · FULLY CONNECTED (dense) if every unit connects to every unit in the next layer 3. COMPETITIVE / SELF-ORGANISING NETWORKS Output units COMPETE, and typically one wins. · a WINNER-TAKE-ALL layer, where the unit with the largest response is set to 1 and the rest to 0 · SELF-ORGANISING MAP (SOM, Kohonen network): output units arranged in a grid, trained UNSUPERVISED so that nearby units respond to similar inputs — producing a topology-preserving map of the input space, which is a dimensionality-reduction and clustering method · used for clustering and visualisation, not classification 4. RADIAL BASIS FUNCTION (RBF) NETWORKS One hidden layer whose units respond to the DISTANCE between the input and a stored centre, usually with a Gaussian. · each hidden unit is LOCAL — it responds only near its centre, unlike a sigmoid unit whose response divides the whole space · trains fast, because the output layer is linear given the centres · needs more units to cover a high-dimensional space

THE RECURRENT ARCHITECTURES: 1. SIMPLE RECURRENT NETWORK (Elman network) A hidden layer's output at step t is fed back as an extra input at step t+1, through a CONTEXT layer. · gives short-term memory · trained by BACKPROPAGATION THROUGH TIME — unroll the network over the sequence and apply ordinary backpropagation · suffers badly from vanishing gradients over long sequences, because the unrolled network is deep in exactly the way the activation-functions topic described 2. LSTM and GRU (gated recurrent architectures) Add GATES that learn what to remember, what to forget and what to output, with a cell state that flows through mostly unchanged. · the point is the same as a residual connection: give the gradient a path that does not pass through a squashing nonlinearity at every step · these made sequence learning practical, and dominated NLP before transformers 3. HOPFIELD NETWORK FULLY connected, SYMMETRIC weights (wᵢⱼ = wⱼᵢ), no self-connections, binary units updated asynchronously. · an ASSOCIATIVE MEMORY: it settles into a stored pattern from a partial or noisy version of it · guaranteed to CONVERGE because it minimises an energy function — the subject of the final topic in this section 4. BOLTZMANN MACHINE A Hopfield network with STOCHASTIC units — a unit's state is probabilistic rather than determined. · escapes local minima the way simulated annealing does · the RESTRICTED Boltzmann machine (RBM), with no within-layer connections, was used to pre-train deep networks before better initialisation and ReLU made that unnecessary 5. BIDIRECTIONAL ASSOCIATIVE MEMORY (BAM) Two layers with connections both ways, storing pattern PAIRS so that either recalls the other. THE MODERN SPECIALISED ARCHITECTURES — worth knowing by what assumption each builds in: CONVOLUTIONAL NEURAL NETWORK (CNN) Shared local filters, as derived in the machine vision topic. THE BUILT-IN ASSUMPTIONS: locality, and translation equivariance. For images this is prior knowledge worth roughly the 84,000-fold parameter saving computed there. TRANSFORMER Every element attends to every other, with learned attention weights. THE BUILT-IN ASSUMPTION: any element may depend on any other, with no fixed notion of distance — which is why it handles long-range dependence better than a recurrent network and why it costs O(n²) in sequence length. AUTOENCODER Trained to reproduce its input through a NARROW bottleneck layer, so the bottleneck must learn a compressed representation. Unsupervised, and used for dimensionality reduction, denoising and anomaly detection. GENERATIVE ADVERSARIAL NETWORK (GAN) Two networks trained against each other: a GENERATOR making samples and a DISCRIMINATOR judging whether they are real. The competition is the training signal.

THE SELECTION TABLE — determined by the structure of the data, not by fashion: THE DATA THE ARCHITECTURE ───────────────────────────────────────────────────────────── fixed-length feature vector, MLP (fully connected) no structure among features grid structure — images, CNN spectrograms a sequence where order matters RNN / LSTM, or a transformer long-range dependence in a TRANSFORMER sequence no labels, want structure SOM, autoencoder recall a pattern from a partial HOPFIELD / associative cue memory generate new samples GAN, or another generative model ───────────────────────────────────────────────────────────── THE PRINCIPLE BEHIND THE TABLE: MATCH THE ARCHITECTURE'S BUILT-IN ASSUMPTION TO A TRUE PROPERTY OF THE DATA. A CNN on tabular data assumes an adjacency between columns that does not exist and wastes its advantage; an MLP on images must learn translation invariance from examples, which needs far more data than building it in. SIZING — how many layers and units: TOO FEW → UNDERFITTING; the network cannot represent the function (high bias) TOO MANY → OVERFITTING and wasted computation (high variance) Both terms are from the machine learning topic, and the bias–variance trade-off is what architecture size controls. THE PRACTICAL APPROACH, in the absence of theory: · start from a size known to work on a similar problem · grow until the TRAINING error is acceptable, then apply REGULARISATION to control the test error · use a VALIDATION SET to choose, never the test set · one hidden layer suffices in principle by the universal approximation theorem, but DEEPER NETWORKS OFTEN NEED EXPONENTIALLY FEWER UNITS for the same function — depth is more parameter-efficient than width, which is the practical reason for deep rather than wide networks A WORKED SIZING, using the parameter formula m(n+1) from the mathematical-model topic. Classify 28×28 images into 10 classes: 784 → 128 → 10: 128(785) + 10(129) = 101,770 parameters 784 → 512 → 10: 512(785) + 10(513) = 407,050 784 → 128 → 64 → 10: 128(785) + 64(129) + 10(65) = 100,480 + 8,256 + 650 = 109,386 NOTE THAT ADDING A SECOND HIDDEN LAYER OF 64 UNITS COSTS ONLY ABOUT 7,600 MORE PARAMETERS, while widening the first layer to 512 costs 300,000 more. DEPTH IS CHEAP AND WIDTH IS EXPENSIVE, because the first layer's weight matrix is dominated by the 784 inputs. That asymmetry is a real design consideration. THE ARCHITECTURAL DECISIONS SUMMARISED: the number of LAYERS (depth) the number of UNITS per layer (width) the CONNECTION PATTERN — fully connected, convolutional, recurrent, skip connections the ACTIVATION functions whether weights are SHARED, as in a CNN EVERY ONE OF THESE IS PRIOR KNOWLEDGE, and the reason architecture matters is that prior knowledge substitutes for data.