TESTING is the process of executing a program with the intent of FINDING ERRORS. (Myers's definition, and the emphasis is deliberate.) A GOOD TEST CASE is one with a high probability of finding an as-yet-undiscovered error. A SUCCESSFUL TEST is one that uncovers an as-yet-undiscovered error. READ THAT AGAIN, because it inverts the intuition: a test that passes has told you almost nothing; a test that FAILS has found you a defect while it is still cheap to fix. A tester whose tests all pass has either a perfect program or a weak test suite, and the second is far more likely. DIJKSTRA'S OBSERVATION, the fundamental limit: "Testing shows the PRESENCE, not the ABSENCE of defects." Exhaustive testing is impossible for any realistic program — a function taking two 32-bit integers has 2^64 ≈ 1.8 × 10^19 input combinations. At a million tests per second that is 585,000 YEARS. Testing is therefore always SAMPLING, and test design is the art of choosing a sample that finds defects. VERIFICATION vs VALIDATION (recall the requirements topic): VERIFICATION "Are we building the product RIGHT?" Does it conform to its specification? VALIDATION "Are we building the RIGHT product?" Does it meet the customer's actual needs? A system can pass every verification test and fail validation entirely — it correctly implements the wrong thing. ERROR, FAULT, FAILURE — the precise vocabulary: ERROR a human mistake (the developer's misunderstanding) FAULT the resulting defect in the code (also "bug") FAILURE the observable incorrect behaviour when the fault is executed A fault that is never executed causes no failure — which is why faults can lie dormant for years.

1. UNIT TESTING Individual components are tested to ensure they operate correctly. Each component is tested INDEPENDENTLY, without other system components. A unit may be a function, a procedure, a class, or a small module. · normally done by the DEVELOPER who wrote the code · WHITE-BOX oriented — the tester sees the internals · needs DRIVERS (a stub main program that calls the unit and supplies test data) and STUBS (dummy replacements for the units this unit calls) · fast, cheap, precise: when a unit test fails you know exactly where the fault is · WHAT IT CANNOT FIND: any defect in the interaction between units, which is where a large share of real defects live 2. INTEGRATION / MODULE TESTING Testing of a collection of dependent components. Exposes faults in the INTERFACES and in the interaction between integrated components. THREE STRATEGIES: TOP-DOWN start with the high-level control module, stub out the modules below, and replace stubs progressively downward. ✔ major design faults found early ✔ an early skeleton of the system can be demonstrated ✘ needs many stubs; low-level utilities are tested last, and they are heavily used BOTTOM-UP start with the lowest-level utilities, using drivers, and work upward. ✔ no stubs needed; utilities well tested ✘ the control structure — where design faults concentrate — is tested LAST ✘ no working program exists until near the end SANDWICH / both at once, meeting in the middle at a BIG-BANG target layer. BIG-BANG (integrate everything at once) is named for completeness and is almost always a mistake: when it fails, you have no idea which interface is at fault. REGRESSION TESTING belongs here too: re-running existing tests after a change, to confirm the change broke nothing. 3. SYSTEM TESTING Testing of the complete system prior to delivery, as an integrated whole. Concerned with validating that the system meets its functional AND non-functional requirements. BLACK-BOX oriented, done by an independent test team. Includes the specialised kinds: FUNCTION TESTING does it do what the spec says? PERFORMANCE / STRESS behaviour at and beyond the specified load. Stress testing deliberately pushes past the design limit to see HOW it fails — gracefully, or catastrophically? SECURITY TESTING can protection be breached? RECOVERY TESTING forces failure and verifies recovery and data integrity CONFIGURATION TESTING across supported platforms COMPATIBILITY / with other systems it must work INTEROPERABILITY alongside USABILITY TESTING with real users 4. ACCEPTANCE TESTING Testing by the CUSTOMER to decide whether to accept the system. Uses REAL customer data, not test data — and this regularly finds defects everything else missed, because real data contains cases nobody imagined. ALPHA TESTING conducted at the developer's site with the customer present; used for bespoke systems BETA TESTING the software is released to a set of real users at their own sites, who report problems; used for generic products The output is an accept/reject decision, so this level is contractual rather than merely technical. A FIFTH LEVEL, sometimes listed: RELEASE / DEPLOYMENT TESTING — testing a particular release in its production-like environment, including installation, upgrade and rollback.

THE V-MODEL — the standard picture of how the levels correspond to development phases. Each development phase has a matching test level, and the tests for a phase are designed AS the phase is done, not afterwards. requirements ─────────────────▶ acceptance testing ╲ ╱ system spec ──────────────▶ system testing ╲ ╱ architecture ────────▶ integration testing ╲ ╱ detailed design ─▶ unit testing ╲ ╱ implementation THE V-MODEL'S REAL INSIGHT: the acceptance test criteria come from the REQUIREMENTS, not from the code. Writing the acceptance tests during requirements analysis is what catches untestable requirements — which is exactly the "test-case generation" validation technique from the requirements topic. WHERE THE EFFORT AND COST GO — with the numbers. EFFORT DISTRIBUTION in a typical project: testing accounts for 30–40% of total project effort, and for safety-critical systems can exceed 50%. THE COST OF FINDING A DEFECT AT EACH LEVEL — this is why the levels are ordered as they are. Using the Boehm ratios from the design topic, with a nominal 1 unit at requirements: level found relative cost in a suite of 10,000 tests, time per run ───────────────────────────────────────────────────────── unit test 20× seconds integration test 30× a few minutes system test 50× tens of minutes acceptance test 75× hours to days after release 100× plus reputation WORKED — why the pyramid shape is economic, not aesthetic. Suppose a suite with these per-test costs and run times: unit 0.002 s per test integration 0.4 s per test end-to-end 8 s per test A "PYRAMID" suite: 5,000 unit, 500 integration, 50 e2e 5000 × 0.002 = 10 s 500 × 0.4 = 200 s 50 × 8 = 400 s ─────── 610 s ≈ 10.2 minutes An "INVERTED PYRAMID" (ice-cream cone) with the same total test count, 5,550, but weighted to e2e: 50 unit, 500 integration, 5,000 e2e 50 × 0.002 = 0.1 s 500 × 0.4 = 200 s 5000 × 8 = 40,000 s ───────── 40,200 s ≈ 11.2 HOURS SAME NUMBER OF TESTS. 66× the wall-clock time. And the inverted suite is also FLAKIER (e2e tests fail for environmental reasons) and LESS PRECISE (a failure tells you "checkout is broken", not which line). This arithmetic is the entire argument for the test pyramid. THE PRACTICAL CONSEQUENCE: a suite that takes 11 hours is run nightly at best, so a developer learns about a break the next morning — by which time they have moved on and the context is gone. A 10-minute suite runs on every push. The FEEDBACK LATENCY, not the test count, is what determines whether tests actually prevent defects.