HEBB'S POSTULATE (1949), in his own framing: "When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased." THE POPULAR SUMMARY: "CELLS THAT FIRE TOGETHER, WIRE TOGETHER." THE MATHEMATICAL FORM — and it is the simplest rule in the section: Δwᵢ = η · xᵢ · y where xᵢ is the input, y the unit's output, and η the learning rate. WHAT IS NOT IN THAT FORMULA IS THE POINT: THERE IS NO TARGET t, AND THEREFORE NO ERROR TERM. Compare the delta rule, Δwᵢ = η(t − y)xᵢ. Hebbian learning is UNSUPERVISED — it needs no correct answer, only the correlation between input and output. That is what makes it biologically plausible: a synapse has access to the activity at its two ends and to nothing else, certainly not to a global error signal. THE THREE CASES: xᵢ and y both active → the weight INCREASES one active, one not → no change (with binary 0/1 units) both inactive → no change WITH BIPOLAR UNITS (−1 / +1) the rule is more expressive, because the product is negative when the two disagree: xᵢ = +1, y = +1 → Δw = +η strengthen xᵢ = −1, y = −1 → Δw = +η strengthen (both "agree") xᵢ = +1, y = −1 → Δw = −η WEAKEN THIS IS WHY ASSOCIATIVE MEMORIES USE BIPOLAR ENCODING — with 0/1 units, a rule that can only increase weights is far less useful. A WORKED EXAMPLE, storing an association with η = 1 and bipolar units. Weights start at zero. pattern 1: x = (1, −1), y = +1 Δw₁ = 1(1)(1) = +1 w = (1, −1) Δw₂ = 1(−1)(1) = −1 pattern 2: x = (−1, 1), y = −1 Δw₁ = 1(−1)(−1) = +1 w = (2, −2) Δw₂ = 1(1)(−1) = −1 RECALL, presenting x = (1, −1): net = 2(1) + (−2)(−1) = 4 → positive → y = +1 ✓ The association was stored, and note that the two patterns REINFORCED the same weight direction because they are consistent with each other.

THE FATAL FLAW OF THE BASIC RULE: WEIGHTS GROW WITHOUT BOUND. With Δw = η·x·y and a linear unit where y = wx, presenting the same input repeatedly gives Δw = η·x·(wx) = η·w·x² so the change is PROPORTIONAL TO THE CURRENT WEIGHT. That is exponential growth — the weight increases, which increases the output, which increases the next weight change. WORKED, with x = 1, η = 0.1, w starting at 1: w = 1.000 → 1.100 → 1.210 → 1.331 → 1.464 → 1.611 → … after 20 steps: 1.1²⁰ ≈ 6.7 after 100 steps: 1.1¹⁰⁰ ≈ 13,781 after 200 steps: ≈ 1.9 × 10⁸ THERE IS NO EQUILIBRIUM. The rule has no mechanism for weakening a connection, so it cannot forget, cannot discriminate, and eventually saturates or overflows. AND A SECOND PROBLEM: IT CANNOT LEARN TO DISCRIMINATE. Every input that produces an output strengthens its own pathway, so the unit becomes increasingly responsive to everything it has seen rather than selectively responsive to one pattern. THE FIXES — each adds the missing decay: 1. WEIGHT DECAY (forgetting term) Δwᵢ = η·xᵢ·y − γ·wᵢ A constant leak toward zero. Growth stops when the strengthening balances the decay, so the weight settles at a finite value determined by the input's statistics. 2. OJA'S RULE — the principled version Δwᵢ = η·y·(xᵢ − y·wᵢ) The subtracted term grows with y², so it automatically normalises the weight vector to unit length. AND THE RESULT IS REMARKABLE: A SINGLE OJA UNIT CONVERGES TO THE FIRST PRINCIPAL COMPONENT of the input data. So an unsupervised, biologically motivated, purely local rule performs PCA — which connects Hebbian learning directly to the dimensionality reduction of the machine learning topic. 3. GENERALISED HEBBIAN / SANGER'S RULE Extends Oja's rule to several units so that they extract successive principal components rather than all converging on the first. 4. BCM RULE (Bienenstock–Cooper–Munro) Introduces a sliding THRESHOLD on the output: activity above it strengthens the synapse and activity below it weakens it. This models the experimentally observed LONG-TERM DEPRESSION as well as potentiation. 5. ANTI-HEBBIAN LEARNING Δwᵢ = −η·xᵢ·y Used for lateral connections between units, so that units DECORRELATE and learn different features instead of duplicating each other. THE PATTERN ACROSS ALL FIVE: HEBB'S RULE SUPPLIES THE STRENGTHENING AND SOMETHING ELSE MUST SUPPLY THE WEAKENING. A learning rule that can only increase is not a learning rule.

WHERE HEBBIAN LEARNING IS ACTUALLY USED: ASSOCIATIVE MEMORY — its principal application. To store a set of patterns in a fully connected network, set wᵢⱼ = Σₚ xᵢᵖ xⱼᵖ summing the outer product over the patterns. This is the HEBBIAN one-shot storage rule, and it is exactly how a HOPFIELD NETWORK is programmed — the final topic of this section. NOTE THAT NO ITERATION IS REQUIRED: the weights are computed directly from the patterns in one pass, which is unusual among learning rules. BIDIRECTIONAL ASSOCIATIVE MEMORY (BAM) — the same outer-product construction over pattern PAIRS, so either pattern recalls the other. PRINCIPAL COMPONENT ANALYSIS, via Oja's rule, as above. COMPETITIVE LEARNING AND SELF-ORGANISING MAPS — the winning unit's weights move toward the input, which is a Hebbian update restricted to the winner. This is how a Kohonen map from the architectures topic is trained. FEATURE DISCOVERY — unsupervised detection of correlated inputs, which is what the early layers of a sensory system appear to do. THE COMPARISON THAT MATTERS — Hebbian against the supervised rules of the previous topics: HEBBIAN PERCEPTRON / DELTA RULE ───────────────────────────────────────────────────────────── supervision UNSUPERVISED SUPERVISED — needs a target what drives the CORRELATION the ERROR (t − y) the update of x and y biological STRONG — local WEAK for perceptron; NONE for plausibility information backpropagation only stability UNSTABLE stable without decay what it learns the statistical a mapping from input to a structure of specified output the input can it NO — it has no YES discriminate notion of a classes? desired output ───────────────────────────────────────────────────────────── THE DECISIVE DIFFERENCE IS THE THIRD ROW COMBINED WITH THE SIXTH. Hebbian learning is biologically defensible precisely BECAUSE it uses only local information — and it cannot solve a classification task for the same reason, since no local signal can tell a synapse what the network as a whole should have output. THAT IS THE TRADE-OFF AT THE HEART OF THE TOPIC: LOCALITY BUYS BIOLOGICAL PLAUSIBILITY AND COSTS THE ABILITY TO BE TOLD WHAT TO DO. Backpropagation makes the opposite choice, which is why it works and why it is not biological. A NOTE ON THE MODERN POSITION, since the rule is old but not obsolete: · SPIKE-TIMING-DEPENDENT PLASTICITY (STDP) is the contemporary refinement, in which the ORDER of the spikes matters: if the input fires shortly BEFORE the output, the synapse strengthens; if shortly after, it weakens. That directionality gives the rule a notion of CAUSATION that plain Hebbian correlation lacks, and it is what spiking networks use. · CONTRASTIVE and SELF-SUPERVISED methods, which learn by making representations of related inputs agree, are Hebbian in spirit — no labels, learning from co-occurrence — and are among the most active areas of current research. THE 1949 IDEA THAT CO-OCCURRENCE IS ITSELF A TRAINING SIGNAL HAS PROVED DURABLE, even as the specific rule was superseded.