Testing is an EXPENSIVE process phase. Testing WORKBENCHES — integrated sets of tools — provide a range of tools to reduce the time required and the total testing cost. THE ARGUMENT FOR AUTOMATION, stated properly: · a manual test is run once; an automated test is run thousands of times. The cost is paid once and amortised. · REGRESSION TESTING is impossible manually at scale. Every change requires re-running everything, and no team can manually re-execute 5,000 tests per commit. · humans are unreliable at repetitive checking, and get worse with fatigue and familiarity · automated tests are EXECUTABLE DOCUMENTATION — they state precisely what the system is supposed to do, and unlike a document they cannot silently go out of date · they give FAST FEEDBACK, which is the property that actually prevents defects (see the levels topic) WHAT SHOULD NOT BE AUTOMATED: · exploratory testing — deliberately unscripted investigation, which is where a skilled tester's judgement finds what no script would · usability and aesthetic judgement · tests that will be run once (a one-off migration check) · anything so unstable that maintaining the test costs more than running it manually AUTOMATION IS NOT A REPLACEMENT FOR TESTERS. It replaces the repetitive part of their work so their judgement goes where it is needed. COMPONENTS OF A TESTING WORKBENCH (Sommerville's list): TEST MANAGER manages the running of program tests; keeps track of test data, expected results and program facilities tested TEST DATA GENERATOR generates test data for the program under test, by selecting from a database or using patterns ORACLE generates predictions of expected test results. May be a previous program version or a prototype. FILE COMPARATOR compares the results of program tests with previous results and reports differences REPORT GENERATOR provides report definition and generation facilities for test results DYNAMIC ANALYSER adds code to a program to count the number of times each statement has been executed — i.e. a COVERAGE tool SIMULATOR simulates the machine, the target environment or the users, where the real thing is unavailable or unsafe THE ORACLE PROBLEM — worth understanding, because it is the genuinely hard part of automation. To automate a test you must know the EXPECTED result. For a calculation that is easy; for "is this rendered chart correct?", "is this translation good?" or "is this recommendation sensible?" there may be no mechanical oracle at all. Partial answers: · a REFERENCE IMPLEMENTATION or a previous version · METAMORPHIC RELATIONS — properties that must hold between outputs even when no single output is known (sorting a list twice gives the same result; searching for "a b" and "b a" should return the same set) · GOLDEN FILES / snapshot testing — record the current output, and flag any change for human review · INVARIANTS — the total after a transfer equals the total before

EVERY AUTOMATED TEST HAS FOUR PHASES — the "arrange, act, assert" pattern (sometimes "given, when, then"): ARRANGE / GIVEN set up the system and the test data ACT / WHEN perform the single operation under test ASSERT / THEN check the outcome against the expectation TEARDOWN restore state so the next test is unaffected A worked unit test of the fee validator from the previous topic: test_fee_below_minimum_is_rejected(): # arrange app = Application(fee=499, age=30, district="01") # act result = validate(app) # assert assertEquals(ERR_FEE, result.code) assertEquals("fee below minimum", result.message) PROPERTIES OF A GOOD AUTOMATED TEST — the FIRST acronym: FAST milliseconds, so the suite can run constantly INDEPENDENT no test depends on another's outcome or on execution order REPEATABLE same result every run, in any environment SELF- passes or fails with no human interpretation VALIDATING TIMELY written with (or before) the code, not months after TEST DOUBLES — the standard vocabulary for fake collaborators, needed because a unit test must isolate the unit: DUMMY passed but never used; fills a parameter slot STUB returns canned answers to calls SPY a stub that also records how it was called MOCK pre-programmed with expectations about the calls it should receive; FAILS if they do not occur FAKE a working but simplified implementation — an in-memory database standing in for the real one THE STUB/MOCK DISTINCTION MATTERS: a stub verifies STATE ("after this, the balance is 500"); a mock verifies BEHAVIOUR ("the mailer was called exactly once with this address"). Over-using mocks couples tests to the implementation, so the tests break on every refactor — which is a common reason teams come to resent their own suite. THE ANTI-PATTERNS — the reasons automated suites get abandoned: FLAKY TESTS — pass sometimes, fail sometimes, with no code change. Causes: timing and race conditions, dependence on wall-clock time or timezone, test order dependence, shared mutable state, real network calls, unseeded randomness. WHY THEY ARE FATAL: once a suite is known to fail randomly, every red build is assumed to be flakiness, and real failures are ignored. A flaky test is worse than no test, because it destroys the signal. Fixes: fix or delete — never "retry until green" as a policy; inject the clock instead of calling it; seed randomness; run in random order deliberately to expose order dependence. SLOW TESTS — see the pyramid arithmetic in the levels topic. TESTING THE IMPLEMENTATION rather than the behaviour — asserting on private internals, so every refactor turns the suite red without any bug being introduced. ASSERTION-FREE TESTS — code that runs the system and asserts nothing. It contributes coverage and detects nothing, which is exactly why coverage alone is a poor quality signal. ERRATIC / INTERDEPENDENT TESTS — test B passes only because test A ran first and left data behind. Runs fine locally, fails in parallel CI.

WHEN IS AUTOMATING A TEST WORTH IT? A concrete model. For one test case: M = manual execution time A = time to automate it (one-off) E = automated execution time (negligible, but included) Mn = maintenance cost per run cycle of the automated test N = number of times the test will be run manual total = N × M automated total = A + N × (E + Mn) BREAK-EVEN at N* = A / (M − E − Mn) WORKED — a regression test for the licence issuance flow: M = 6 minutes of a tester's time A = 90 minutes to write and stabilise the automated test E = 0.2 minutes to execute Mn = 0.3 minutes of amortised maintenance per cycle N* = 90 / (6 − 0.2 − 0.3) = 90 / 5.5 = 16.4 → automation pays back after about 17 runs. IS 17 RUNS REALISTIC? With continuous integration, the suite runs on every push. A team of 6 pushing 4 times a day each runs it 24 times PER DAY. Break-even is reached on DAY ONE. · over one year (say 250 working days × 24 runs = 6,000 runs) manual = 6,000 × 6 = 36,000 min = 600 hours automated = 90 + 6,000 × 0.5 = 3,090 min = 51.5 hours → a saving of 548 hours on ONE test case, and the manual figure assumes a human is available to run it 24 times a day, which no team is. NOW THE OTHER DIRECTION — when automation LOSES: a test run only at each quarterly release, N = 4/year M = 6, A = 90, so N* = 16.4 > 4 → over a year, manual = 24 min, automated = 92 min. Automating it is a net LOSS unless it will live several years or the manual test is error-prone. THE FLAKINESS PENALTY — the term everyone forgets. Add a probability f that a test fails spuriously, each costing I minutes to investigate: automated total = A + N × (E + Mn + f × I) With f = 2% and I = 20 minutes: per-run cost rises from 0.5 to 0.5 + 0.02 × 20 = 0.9 min N* = 90 / (6 − 0.9) = 17.6 — barely worse. But at f = 20% (a genuinely flaky test) and I = 20: per-run cost = 0.5 + 4.0 = 4.5 min N* = 90 / (6 − 4.5) = 60 runs And the hidden cost is not in this formula at all: the credibility of the WHOLE suite. THAT is why flaky tests are treated as a priority-one defect rather than an annoyance. CONTINUOUS INTEGRATION — the practice that makes the economics work: · every commit triggers a build and the automated suite · the build must be kept green; a broken build is fixed before new work continues · results are visible to everyone · CONTINUOUS DELIVERY extends this to an automated, always-ready release pipeline The point is not the tooling but the FEEDBACK LATENCY: a defect found 5 minutes after it was written is fixed by the person who wrote it, who still remembers why. The same defect found 5 weeks later is an investigation.