LOGIC REQUIRES CERTAINTY. Real knowledge does not have it, for three reasons: LAZINESS it is too much work to list every antecedent and consequent — the qualification problem from the KR issues topic THEORETICAL there is no complete theory for the domain IGNORANCE PRACTICAL even with a complete theory, not all the tests IGNORANCE have been run on this patient So instead of asserting a sentence true or false, we assign a DEGREE OF BELIEF between 0 and 1. THE BASIC MACHINERY: P(a) PRIOR or unconditional probability of a P(a | b) POSTERIOR or conditional probability of a GIVEN b P(a, b) JOINT probability of a and b together RANDOM VARIABLE a variable with a domain of values; a BOOLEAN variable has domain {true, false} THE AXIOMS (Kolmogorov): 1. 0 ≤ P(a) ≤ 1 2. P(true) = 1 and P(false) = 0 3. P(a ∨ b) = P(a) + P(b) − P(a ∧ b) INCLUSION-EXCLUSION THE DEFINITION OF CONDITIONAL PROBABILITY, which everything else follows from: P(a | b) = P(a ∧ b) / P(b) whenever P(b) > 0 Rearranged, this is the PRODUCT RULE: P(a ∧ b) = P(a | b) P(b) = P(b | a) P(a) MARGINALISATION — summing out a variable you do not care about: P(Y) = Σ over z of P(Y, z) and CONDITIONING: P(Y) = Σ over z of P(Y | z) P(z) NORMALISATION — the trick that saves work in practice. Because the probabilities of an exhaustive set of alternatives must sum to 1, you can compute unnormalised values and scale at the end: P(X | e) = α P(X, e) where α is whatever makes the distribution sum to 1 THIS MEANS THE DENOMINATOR P(e) OFTEN NEED NOT BE COMPUTED DIRECTLY — a practical simplification used constantly. INDEPENDENCE: a and b are INDEPENDENT if P(a | b) = P(a), equivalently P(a ∧ b) = P(a) P(b). CONDITIONAL INDEPENDENCE is the important one: a and b are conditionally independent GIVEN c if P(a | b, c) = P(a | c) "once c is known, b tells you nothing more about a." CONDITIONAL INDEPENDENCE IS WHAT MAKES PROBABILISTIC REASONING TRACTABLE, and it is the entire basis of the Bayesian networks in the next topic. WHY THE FULL JOINT DISTRIBUTION IS NOT AN OPTION: with n boolean variables, the joint distribution has 2^n entries and 2^n − 1 independent numbers: 5 variables → 31 10 variables → 1,023 20 variables → 1,048,575 30 variables → 1,073,741,823 Specifying a billion numbers by hand is impossible, and no data set would determine them. THE PROBLEM IS THE SAME COMBINATORIAL EXPLOSION AS EVERYWHERE ELSE, and the solution is the same: exploit structure.

FROM THE PRODUCT RULE, written both ways: P(a ∧ b) = P(a | b) P(b) P(a ∧ b) = P(b | a) P(a) Equating and rearranging gives BAYES' RULE: P(b | a) P(a) P(a | b) = ───────────── P(b) IN THE CONDITIONALISED FORM used in diagnosis: P(effect | cause) P(cause) P(cause | effect) = ────────────────────────── P(effect) READ WHAT THIS DOES: it converts CAUSAL knowledge — which is what experts have and what is stable — into DIAGNOSTIC knowledge — which is what you need and what depends on prevalence. P(symptom | disease) is a fact about biology. It does not change when an epidemic arrives. P(disease | symptom) is what the doctor wants, and it CHANGES WITH PREVALENCE. THAT ASYMMETRY IS THE ENTIRE POINT OF BAYES' RULE, and the reason it is worth memorising. THE TERMINOLOGY: P(a) the PRIOR — belief before the evidence P(b | a) the LIKELIHOOD — how well a explains b P(b) the EVIDENCE or normalising constant P(a | b) the POSTERIOR — belief after the evidence AND P(b) EXPANDED BY THE LAW OF TOTAL PROBABILITY: P(b) = P(b | a) P(a) + P(b | ¬a) P(¬a) This is the form to use in calculations, because the two likelihood terms are usually what you are given. THE WORKED EXAMPLE THAT MATTERS — a medical screening test. GIVEN: a disease with PREVALENCE 0.1% P(D) = 0.001 the test detects it 99% of the time P(+|D) = 0.99 (SENSITIVITY) it is negative 99% of the time in the healthy (SPECIFICITY 99%) P(+|¬D)= 0.01 A PATIENT TESTS POSITIVE. What is the probability they have the disease? STEP 1 — the evidence, by total probability: P(+) = P(+|D)P(D) + P(+|¬D)P(¬D) = 0.99 × 0.001 + 0.01 × 0.999 = 0.00099 + 0.00999 = 0.01098 STEP 2 — Bayes' rule: P(D|+) = P(+|D)P(D) / P(+) = 0.00099 / 0.01098 = 0.0902 THE ANSWER IS 9.02%. A 99%-accurate test, a positive result, and the patient is over 90% likely to be HEALTHY. WHY — the intuition, which is worth more than the formula. Take 100,000 people: 100 have the disease (0.1%). Of these, 99 test positive. 99,900 are healthy. Of these, 1% — that is 999 — test positive anyway. TOTAL POSITIVES: 99 + 999 = 1,098 of whom 99 are genuinely ill → 99/1098 = 9.02% THE FALSE POSITIVES OUTNUMBER THE TRUE POSITIVES TEN TO ONE, because there are a thousand times more healthy people to generate them. THE BASE RATE DOMINATES. HOW THE ANSWER CHANGES — the sensitivity of the result: prevalence 0.1% → P(D|+) = 9.02% prevalence 1% → P(D|+) = 50.00% prevalence 10% → P(D|+) = 91.67% SAME TEST, THREE COMPLETELY DIFFERENT CONCLUSIONS. This is why screening the general population differs from testing someone with symptoms: the symptoms raise the prior. AND IMPROVING THE TEST: keep prevalence 0.1%, raise specificity to 99.9% P(+) = 0.99×0.001 + 0.001×0.999 = 0.001989 P(D|+) = 0.00099/0.001989 = 49.77% A TENFOLD REDUCTION IN THE FALSE-POSITIVE RATE took the answer from 9% to 50%. NOTE THAT SPECIFICITY, NOT SENSITIVITY, IS WHAT MATTERS FOR A RARE CONDITION — which is the opposite of most people's instinct, and directly parallels the fraud-detector precision result from the AI applications topic.

MULTIPLE PIECES OF EVIDENCE — and why the naive approach needs a justification. In general, combining evidence requires the full joint distribution: P(cause | e₁, e₂) = α P(e₁, e₂ | cause) P(cause) and P(e₁, e₂ | cause) is a table, not two numbers. THE NAIVE BAYES ASSUMPTION: the pieces of evidence are CONDITIONALLY INDEPENDENT GIVEN THE CAUSE: P(e₁, e₂ | cause) = P(e₁ | cause) P(e₂ | cause) giving P(cause | e₁, e₂) = α P(cause) P(e₁|cause) P(e₂|cause) and in general P(Cause | e₁ … eₙ) = α P(Cause) Π P(eᵢ | Cause) THIS IS THE NAIVE BAYES MODEL, and it reduces the parameter count from exponential to linear: n binary symptoms, full joint per disease: 2^n − 1 numbers naive Bayes: n numbers at n = 20 that is 1,048,575 against 20. IT IS CALLED "NAIVE" BECAUSE THE ASSUMPTION IS USUALLY FALSE — symptoms are correlated — and yet the model works remarkably well in practice, particularly for classification. The reason: classification needs only the ARGMAX to be right, and the errors in the individual probabilities often do not change which class wins. This model returns in the machine learning section. WORKED, two symptoms: P(Flu) = 0.05 P(fever | Flu) = 0.90 P(fever | ¬Flu) = 0.20 P(ache | Flu) = 0.80 P(ache | ¬Flu) = 0.10 A patient has both. P(Flu | fever, ache)? numerator for Flu: 0.05 × 0.90 × 0.80 = 0.0360 numerator for ¬Flu: 0.95 × 0.20 × 0.10 = 0.0190 normalise: 0.0360 / (0.0360 + 0.0190) = 0.6545 ANSWER 65.45%. Note how the prior of 5% was pulled up to 65% by two pieces of evidence — and that normalising made P(fever, ache) unnecessary to compute. THE PRACTICAL LESSONS — what to take away: 1. ALWAYS ASK FOR THE BASE RATE. A conditional probability without a prior is uninterpretable. "The test is 99% accurate" is not enough information to act on. 2. FOR RARE EVENTS, THE FALSE-POSITIVE RATE DOMINATES. Improving specificity matters more than improving sensitivity, because the healthy population is so much larger. 3. ACCURACY IS A USELESS SINGLE NUMBER on imbalanced data — the same conclusion the AI applications topic reached about fraud detection, arrived at from the probability side. 4. TESTING IN SEQUENCE WORKS. A cheap screening test raises the prior for a second, more specific test. Starting from 9.02% rather than 0.1%, a second independent test with the same characteristics gives P(+) = 0.99 × 0.0902 + 0.01 × 0.9098 = 0.09840 P(D | ++) = 0.089298/0.09840 = 90.75% TWO POSITIVES TAKE IT FROM 9% TO 90.75%. This is why real diagnostic protocols are staged, and it is Bayes' rule applied iteratively — each posterior becomes the next prior. 5. BAYES' RULE IS HOW EVIDENCE UPDATES BELIEF, so it is the correct model of learning from data. That framing is the bridge to the rest of the section: a Bayesian network is a structured way to hold the priors and likelihoods for many variables at once, and belief propagation is how the update is computed when there are more than two.