A HOPFIELD NETWORK IS UNLIKE EVERY NETWORK SO FAR IN THIS SECTION, and the differences are the point: · SINGLE LAYER — there is no input layer, hidden layer or output layer. Every unit is all three at once. · FULLY RECURRENT — every unit connects to every other. The network is a cycle, not a feedforward chain. · SYMMETRIC WEIGHTS — wᵢⱼ = wⱼᵢ. THIS IS ESSENTIAL, and the reason appears below: without it there is no energy function and no convergence guarantee. · NO SELF-CONNECTIONS — wᵢᵢ = 0, or a unit would simply reinforce its own state. · BIPOLAR STATES — sᵢ ∈ {−1, +1}, as in the Hebbian topic. · ASYNCHRONOUS UPDATING — one unit at a time, in random order. s₁ ⟷ s₂ ⤢ ╳ ⤡ every unit connected to every other, s₃ ⟷ s₄ symmetrically THE UPDATE RULE — the same threshold unit as always: sᵢ ← +1 if Σⱼ wᵢⱼ sⱼ ≥ 0 sᵢ ← −1 otherwise Units are updated one at a time until NO UNIT CHANGES on a full sweep. That state is a STABLE STATE, and it is the network's answer. STORING PATTERNS — THE HEBBIAN OUTER-PRODUCT RULE, in ONE PASS with NO ITERATION: wᵢⱼ = Σₚ xᵢᵖ xⱼᵖ for i ≠ j, wᵢᵢ = 0 This is Hebb's rule from earlier in the section, applied to every pair of units and summed over the patterns. THERE IS NO TRAINING LOOP, NO LEARNING RATE AND NO ERROR — the weights are computed directly from the patterns. That alone makes the Hopfield network unusual among everything else here. A WORKED CONSTRUCTION, 4 units, 2 patterns: p¹ = (1, −1, 1, −1) p² = (1, 1, −1, −1) w₁₂ = (1)(−1) + (1)(1) = 0 w₁₃ = (1)(1) + (1)(−1) = 0 w₁₄ = (1)(−1) + (1)(−1) = −2 w₂₃ = (−1)(1) + (1)(−1) = −2 w₂₄ = (−1)(−1) + (1)(−1) = 0 w₃₄ = (1)(−1) + (−1)(−1) = 0 W = ⎡ 0 0 0 −2 ⎤ ⎢ 0 0 −2 0 ⎥ ⎢ 0 −2 0 0 ⎥ ⎣ −2 0 0 0 ⎦ NOTE THAT THE MATRIX IS SYMMETRIC with a zero diagonal, as required, and that several weights CANCELLED TO ZERO because the two patterns disagree about those pairs. That cancellation is the seed of the capacity limit discussed below.

HOPFIELD'S CENTRAL CONTRIBUTION — THE ENERGY FUNCTION, also called the Lyapunov function: E = −½ Σᵢ Σⱼ wᵢⱼ sᵢ sⱼ THE THEOREM: EVERY ASYNCHRONOUS UPDATE EITHER DECREASES E OR LEAVES IT UNCHANGED. E NEVER INCREASES. WHY, and the argument is short enough to reproduce in an exam. Suppose unit i flips from sᵢ to sᵢ′. Only terms involving i change, so ΔE = −(sᵢ′ − sᵢ) · Σⱼ wᵢⱼ sⱼ = −Δsᵢ · netᵢ The update rule sets sᵢ′ = +1 exactly when netᵢ ≥ 0. So if the unit flips upward, Δsᵢ > 0 and netᵢ ≥ 0, making ΔE ≤ 0; if it flips downward, Δsᵢ < 0 and netᵢ < 0, so again ΔE ≤ 0. A FLIP IS ONLY EVER MADE IN THE DIRECTION THAT LOWERS THE ENERGY. And since there are finitely many states (2^N), E is bounded below and must reach a minimum in finite time. THIS IS WHY SYMMETRY IS REQUIRED: the cancellation in ΔE depends on wᵢⱼ = wⱼᵢ. With asymmetric weights E is not a Lyapunov function and the network can OSCILLATE FOREVER. THE PICTURE THAT MAKES IT INTUITIVE: The energy function defines a LANDSCAPE over the 2^N possible states. STORED PATTERNS ARE THE VALLEYS. An input state places a ball somewhere on that landscape, and the update rule rolls it downhill until it settles. RECALL IS THEREFORE RELAXATION TO THE NEAREST MINIMUM — and this is what makes the memory CONTENT-ADDRESSABLE: you retrieve a pattern by supplying a corrupted or partial version of the pattern itself, not an address. THE BASIN OF ATTRACTION around each stored pattern is what gives the network its error tolerance. A TRACED RECALL, with the weights computed above. Present the corrupted state s = (1, −1, −1, −1) — this is pattern p² = (1, 1, −1, −1) with the second bit flipped. E(start) = 0 unit 0: net = w₀₃s₃ = (−2)(−1) = +2 ≥ 0 → s₀ = +1, STABLE unit 1: net = w₁₂s₂ = (−2)(−1) = +2 ≥ 0 → s₁ = +1, FLIPS state → (1, 1, −1, −1) E = −4 unit 2: net = w₂₁s₁ = (−2)(1) = −2 < 0 → s₂ = −1, STABLE unit 3: net = w₃₀s₀ = (−2)(1) = −2 < 0 → s₃ = −1, STABLE SWEEP 2: every unit is stable, so the network has converged. RESULT: (1, 1, −1, −1) = p² EXACTLY, RECOVERED FROM A CORRUPTED INPUT, and the energy fell from 0 to −4, which is the same energy as the stored patterns. THAT IS ASSOCIATIVE MEMORY IN EIGHT LINES OF ARITHMETIC: a damaged input, a downhill walk, and the original pattern restored.

THE STORAGE CAPACITY — the network's principal limitation, and a guaranteed exam question: P_max ≈ 0.138 N (often quoted as 0.15N) where N is the number of units. THE RESULT IS BRUTAL: N = 100 units → about 13 patterns, using 4,950 weights N = 1,000 units → about 138 patterns, using 499,500 weights HALF A MILLION WEIGHTS TO STORE 138 PATTERNS. The capacity grows LINEARLY in N while the weight count grows QUADRATICALLY, so the network becomes less efficient the larger it gets — which is why Hopfield networks are studied for their ideas rather than deployed as storage. AND EXCEEDING THE LIMIT DOES NOT DEGRADE GRACEFULLY. Beyond ≈0.138N the memory undergoes a sudden collapse — a PHASE TRANSITION, in the physics sense — and recall of ALL patterns fails, not just the newest. This is called CATASTROPHIC FORGETTING, and the mechanism is visible in the worked construction above: conflicting patterns cancel each other's weights until nothing coherent remains. THE PATTERNS ALSO NEED TO BE NEAR-ORTHOGONAL. Similar patterns interfere strongly, so the practical capacity for correlated data is far below the theoretical figure. SPURIOUS STATES — stable states that were never stored: 1. THE INVERSE OF EVERY STORED PATTERN is always stable, because E depends on products sᵢsⱼ and flipping every sign leaves them unchanged. In the worked example, E(−p¹) = −4, identical to E(p¹). YOU CANNOT AVOID THIS — it is a structural consequence of the energy function. 2. MIXTURE STATES — combinations of odd numbers of stored patterns, such as sign(p¹ + p² + p³), are often stable. 3. SPIN-GLASS STATES — local minima with no relation to any stored pattern, which multiply as the capacity limit is approached. MITIGATIONS: unlearning (running the network freely and applying anti-Hebbian updates to whatever it settles into), and stochastic updating, which lets the network escape shallow spurious minima. DISCRETE VERSUS CONTINUOUS: The DISCRETE model above uses binary/bipolar states and asynchronous updates, and is used for associative memory. The CONTINUOUS model (Hopfield 1984) uses sigmoid units evolving by differential equations, and is used for OPTIMISATION — most famously an approximate solution to the Travelling Salesman Problem, by encoding the tour cost as the energy function so that relaxation to a minimum corresponds to finding a short tour. IT DOES NOT BEAT SPECIALISED TSP SOLVERS, but the technique — cast the problem as an energy function and let a network descend it — is a genuinely general idea. WHAT CAME AFTER, since the Hopfield network's importance is mostly in what it started: · BOLTZMANN MACHINE — a Hopfield network with STOCHASTIC units, so it can escape local minima by occasionally moving uphill (simulated annealing). Its restricted form, the RBM, was used for the layerwise pre-training that launched the deep learning revival around 2006. · BIDIRECTIONAL ASSOCIATIVE MEMORY (BAM) — the two-layer heteroassociative version. · MODERN HOPFIELD NETWORKS (2020) — a continuous reformulation with EXPONENTIAL rather than linear capacity, and the striking result that its update rule is mathematically EQUIVALENT TO THE ATTENTION MECHANISM in transformers. The 1982 model turns out to describe part of the architecture behind today's language models. THE COMPARISON THAT CLOSES THE SECTION: FEEDFORWARD (MLP) HOPFIELD ───────────────────────────────────────────────────────────── connections one direction FULLY RECURRENT, symmetric layers input/hidden/output ONE — every unit is all three learning BACKPROPAGATION, HEBBIAN OUTER PRODUCT, iterative ONE PASS operation one forward pass ITERATE TO A STABLE STATE what it does maps input→output COMPLETES a corrupted pattern guarantee none CONVERGENCE, via the energy function addressing by input CONTENT-ADDRESSABLE ───────────────────────────────────────────────────────────── THE ROW THAT MATTERS MOST IS THE LAST BUT ONE. A feedforward network offers no guarantee that training converges anywhere useful; the Hopfield network GUARANTEES that recall terminates, because it is descending a bounded function. It buys that guarantee with symmetric weights and a capacity of 0.138N — WHICH IS THE TRADE-OFF THE WHOLE MODEL IS BUILT ON.