SUPERVISED LEARNING: the training data includes the CORRECT ANSWER for each example. The learner is given pairs ⟨x, y⟩ and must infer the function mapping x to y. training data: {⟨x₁, y₁⟩, ⟨x₂, y₂⟩, …, ⟨xₙ, yₙ⟩} goal: a function h such that h(x) ≈ y on unseen x THE FEEDBACK IS COMPLETE AND IMMEDIATE — for every example you know exactly what should have been predicted, so the error is directly measurable and the learner can be corrected precisely. THAT IS WHY SUPERVISED LEARNING IS THE BEST-DEVELOPED PARADIGM. TWO TASK TYPES: CLASSIFICATION — y is a discrete label spam / not spam · which disease · which digit · creditworthy / not REGRESSION — y is a continuous value house price · tomorrow's temperature · days until failure THE ALGORITHMS, with their inductive bias (from the previous topic): DECISION TREES prefer short trees; axis-aligned splits NAIVE BAYES features conditionally independent given the class k-NEAREST NEIGHBOURS nearby instances share labels LINEAR / LOGISTIC the relationship is linear in the REGRESSION features SUPPORT VECTOR the best boundary is the MACHINES maximum-margin one NEURAL NETWORKS smooth functions of the inputs ENSEMBLES (random many weak models averaged beat one forests, boosting) strong one THE COST, and it is the binding constraint in practice: LABELS. Someone must supply the correct answer for every training example, and that someone is usually an expert. WORKED: a medical imaging model needs, say, 10,000 labelled scans. At 3 minutes of a radiologist's time per scan: 10,000 × 3 min = 30,000 min = 500 hours ≈ 62 working days of specialist time FOR ONE DATASET. That is the knowledge-acquisition bottleneck from ACtE0904, reappearing as a labelling bottleneck — the cost moved from articulating rules to annotating examples, which scales better but is not free. MITIGATIONS worth naming: SEMI-SUPERVISED LEARNING a few labelled examples plus many unlabelled ones TRANSFER LEARNING start from a model trained on a related task, fine-tune on a small labelled set SELF-SUPERVISED LEARNING generate the labels from the data itself — predict a masked word, predict the next frame. THIS IS WHAT MADE LARGE LANGUAGE MODELS POSSIBLE: raw text supplies its own supervision, so the labelling cost vanishes. ACTIVE LEARNING label only the examples the model is most uncertain about

UNSUPERVISED LEARNING: the training data has NO LABELS. The learner must find STRUCTURE in the data itself. training data: {x₁, x₂, …, xₙ} goal: a useful description of the structure THE DIFFICULTY IS THAT THERE IS NO DEFINITION OF CORRECT. With supervised learning you can measure error against the true label; here there is nothing to compare against, so evaluation is intrinsically harder and often requires a human to judge whether the discovered structure is meaningful. THE TASKS: CLUSTERING — group similar instances k-MEANS: choose k, assign each point to the nearest of k centroids, recompute the centroids, repeat until stable. ✔ simple and fast ✘ you must CHOOSE k in advance, it assumes roughly spherical clusters of similar size, and it is sensitive to the initial centroids HIERARCHICAL CLUSTERING: build a tree by repeatedly merging the closest pair (agglomerative) or splitting (divisive). No k required — you cut the tree where you like. DBSCAN: density-based, finds clusters of arbitrary shape and labels sparse points as noise. No k required. WORKED k-MEANS ON ONE DIMENSION, to show the mechanism: data: 2, 3, 4, 10, 11, 12, k = 2 initial centroids: 2 and 3 (a bad choice, deliberately) assign: {2} to c₁=2; {3,4,10,11,12} to c₂=3 recompute: c₁ = 2, c₂ = (3+4+10+11+12)/5 = 8 assign: {2,3,4} to c₁=2; {10,11,12} to c₂=8 recompute: c₁ = 3, c₂ = 11 assign: unchanged → CONVERGED FINAL CLUSTERS {2,3,4} and {10,11,12}, centroids 3 and 11. It recovered the obvious structure despite a poor start — but a different initialisation can converge to a worse answer, which is why k-means is normally run several times. DIMENSIONALITY REDUCTION — find a lower-dimensional representation preserving what matters PRINCIPAL COMPONENT ANALYSIS (PCA) finds the directions of greatest variance and projects onto them. WHY IT MATTERS: the CURSE OF DIMENSIONALITY. As dimensions increase, data becomes sparse and distances become uninformative — every point is roughly equidistant from every other, which destroys any method relying on similarity. ASSOCIATION RULE MINING — find co-occurrence patterns "customers who buy bread and butter also buy milk" Measured by SUPPORT (how often the combination occurs) and CONFIDENCE (how often the consequent follows the antecedent). The Apriori algorithm is the classic method. ANOMALY DETECTION — identify instances unlike the rest Useful precisely where labels are unavailable because the anomalies are rare and varied — fraud, intrusion, equipment failure. DENSITY ESTIMATION — model the distribution the data came from, which then supports generation of new samples.

REINFORCEMENT LEARNING (RL): the learner is an AGENT taking ACTIONS in an ENVIRONMENT and receiving a REWARD. No correct action is ever given — only a signal about how well things went. the agent observes a STATE s takes an ACTION a receives a REWARD r and a new state s′ goal: learn a POLICY π mapping states to actions that maximises CUMULATIVE reward THIS IS EXACTLY THE AGENT OF ACtE0901 WITH LEARNING ADDED, and the components map directly: the reward is the performance measure, the policy is the agent function, and the critic of the learning-agent architecture is what supplies the reward. THE THREE DIFFICULTIES THAT MAKE RL DISTINCT: 1. DELAYED REWARD (the CREDIT ASSIGNMENT problem) The reward may arrive long after the action that caused it. A chess game is won or lost at move 60; which of the earlier moves deserves the credit? THE STANDARD SOLUTION: learn a VALUE FUNCTION V(s) estimating the expected future reward from state s, so intermediate states acquire values and the learner gets a signal at every step rather than only at the end. 2. THE EXPLORATION–EXPLOITATION TRADE-OFF EXPLOIT the best-known action to earn reward now, or EXPLORE an untried action that might be better? Pure exploitation locks in the first adequate strategy found; pure exploration never earns anything. THE STANDARD APPROACH: ε-GREEDY — take the best known action with probability 1−ε, and a random action with probability ε, often decaying ε over time so the agent explores early and exploits later. THIS IS THE PROBLEM GENERATOR of the learning-agent architecture from ACtE0901, given a mechanism. 3. THE ENVIRONMENT MAY BE UNKNOWN MODEL-FREE methods learn the policy or value function directly from experience without ever modelling the environment — Q-learning is the standard example. MODEL-BASED methods learn the transition model and then plan with it. More sample-efficient, and dependent on the model being right. Q-LEARNING, the canonical algorithm, and the update rule is worth knowing: Q(s, a) ← Q(s, a) + α [ r + γ · max_{a′} Q(s′, a′) − Q(s, a) ] where α is the LEARNING RATE and γ the DISCOUNT FACTOR (0 ≤ γ < 1) expressing how much future reward is worth relative to immediate reward. THE TERM IN BRACKETS IS THE TEMPORAL-DIFFERENCE ERROR: the gap between what was predicted and what was observed plus the best future estimate. Learning is driven by surprise. RL'S CHARACTERISTIC WEAKNESS: SAMPLE INEFFICIENCY. It may need millions of trials, which is acceptable in a simulator and prohibitive on physical hardware — the sim-to-real problem from the robotics topic. CHOOSING BETWEEN THE THREE — the decision, and it is determined by the feedback available rather than by preference: WHAT THE DATA GIVES YOU USE ───────────────────────────────────────────────────────────── the correct answer per example SUPERVISED no answers, and you want structure UNSUPERVISED a reward after a sequence of REINFORCEMENT actions a few answers and much unlabelled SEMI-SUPERVISED data answers derivable from the data SELF-SUPERVISED itself ───────────────────────────────────────────────────────────── THE COMPARISON TABLE: SUPERVISED UNSUPERVISED REINFORCEMENT ───────────────────────────────────────────────────────────── feedback the correct none a scalar reward label when it immediate n/a often DELAYED arrives data labelled unlabelled generated by pairs instances interaction goal predict y find structure maximise cumulative reward evaluation error against no ground total reward the label truth typical use classify, cluster, control, games, predict compress sequential decisions main difficulty labelling no definition credit cost of correct assignment, sample cost ───────────────────────────────────────────────────────────── A WORKED SELECTION on one problem — recommending revision topics: SUPERVISED framing: predict, for each topic, whether the student will answer it correctly in the exam. NEEDS exam outcomes per topic per student, which are not collected. UNSUPERVISED framing: cluster students by study pattern to find types. Useful for insight, does not recommend anything. REINFORCEMENT framing: the recommendation is an action, the exam mark is the delayed reward. THE CORRECT FRAMING CONCEPTUALLY — and impractical, because one episode takes a whole semester and you cannot run millions of them. THE PRACTICAL ANSWER: a supervised proxy — predict retention from tick history using data you actually have — plus hand-written rules. THE HONEST CONCLUSION IS OFTEN A SIMPLER PARADIGM THAN THE PROBLEM DESERVES, because the data available decides, not the elegance of the framing.