Testing metrics fall into three families, answering three different questions. 1. COVERAGE METRICS — how much of the system has been exercised? statement coverage = statements executed / total branch coverage = branches taken / total branches condition coverage = conditions evaluated both ways / total function coverage = functions called / total requirements = requirements with at least one coverage test / total requirements path coverage = paths executed / total (rarely computable) 2. DEFECT METRICS — what have we found, and what is left? DEFECT DENSITY = defects found / size (per KLOC, or per function point) DEFECT REMOVAL DRE = E / (E + D) EFFICIENCY where E = errors found BEFORE delivery and D = defects found AFTER delivery → DRE = 1 is the ideal. This is probably the single most useful process metric in this topic, because it measures the thing you actually care about: did we catch it, or did the customer? DEFECT LEAKAGE defects that escaped from one phase to the next defect arrival rate, and the DEFECT DISCOVERY CURVE (below) MTTD / MTTR mean time to detect / to repair 3. EFFORT AND PROGRESS METRICS test cases planned / executed / passed / failed / blocked test execution rate (tests per day) automation percentage test effort as a fraction of total project effort mean time between failures (MTBF), for reliability RELIABILITY METRICS, since they are what a customer actually experiences: POFOD probability of failure on demand — the likelihood the system fails when a service request is made. Right for systems where each demand matters (a safety shutdown). ROCOF rate of occurrence of failures — failures per unit time. Right for transaction systems. MTTF mean time to failure. The reciprocal of ROCOF, for long-running systems. AVAIL availability = MTBF / (MTBF + MTTR) — the metric from the architecture topics.

A REAL-SHAPED SET OF NUMBERS from a system test cycle on the licence system. total test cases planned 860 executed 742 passed 651 failed 74 blocked (cannot run — a defect prevents reaching the case) 17 not yet executed 118 defects found this cycle 91 defects fixed and verified 58 defects open 33 code size 32.4 KLOC statement coverage 78% branch coverage 61% DERIVED METRICS — compute them all: TEST EXECUTION PROGRESS = executed / planned = 742/860 = 86.3% TEST PASS RATE = passed / executed = 651/742 = 87.7% TEST EFFECTIVENESS (defects per executed test) = 91/742 = 0.123 → about 1 defect per 8 tests DEFECT DENSITY = 91 / 32.4 = 2.81 per KLOC DEFECT FIX RATE = 58/91 = 63.7% BLOCKED RATE = 17/742 = 2.3% NOW READ THEM, because the numbers only matter interpreted: · 86.3% executed with 118 cases outstanding: the cycle is NOT complete, so the pass rate is provisional. Untested areas are where the remaining defects are. · 87.7% pass rate sounds healthy, but 74 failures against 91 defects means most failures are distinct defects rather than repeats of one — the failures are SPREAD, not clustered, which suggests systemic rather than local problems. · THE MOST IMPORTANT NUMBER IS THE COVERAGE GAP: 78% statement but only 61% branch. That 17-point spread says that many decisions have been executed one way only — the error paths, precisely where untested code hides. Recall from the previous topic that this is exactly the pattern that leaves off-by-one and error-handling defects alive. · 2.81 defects/KLOC found in system test is a plausible figure; published industry ranges are roughly 1–25 per KLOC delivered depending on rigour, with high-maturity organisations well under 1. But a LOW density can mean either good code OR weak testing, and this metric alone cannot tell you which — always read it beside coverage. · 33 open defects and 17 blocked cases interact: fixing the blockers may unblock cases that then find more defects. The defect count has not stabilised. IS IT READY TO RELEASE? On these numbers, no — and the argument is not the pass rate but that (a) 14% of cases have never been run, (b) branch coverage is 61% so the error paths are largely unexercised, and (c) the defect discovery curve has not flattened. DEFECT REMOVAL EFFICIENCY, WORKED ACROSS PHASES: found in review 42 found in unit test 67 found in integration 38 found in system test 91 ───────────────────────────── total pre-delivery E = 238 found by customers in the first 6 months D = 19 DRE = 238 / (238 + 19) = 238/257 = 92.6% INTERPRETATION: 7.4% of defects reached the customer. World class is often quoted above 95%, and above 99% for safety-critical work. The phase breakdown also tells you where to invest: 42 found in review is a good return on a cheap activity, and 91 in system test is expensive — those are defects that got all the way through unit and integration testing before being caught. PHASE CONTAINMENT EFFECTIVENESS, per phase: PCE = defects found in the phase they were INTRODUCED / total defects introduced in that phase A low PCE for design means design defects are escaping to be caught in testing — at the 5×-to-50× cost multiplier from the design topic.

THE DEFECT DISCOVERY (RELIABILITY GROWTH) CURVE — cumulative defects found, plotted against testing effort or time. cumulative defects │ ┌──────── plateau: the │ ┌───┘ curve FLATTENS │ ┌────┘ │ ┌────┘ │ ┌───┘ │ ┌─┘ └───────────────────────────── testing time → HOW TO READ IT — this is the practical release-readiness judgement: STILL RISING STEEPLY keep testing. There are plenty more defects; stopping now ships them. FLATTENING you are finding fewer per unit effort. Either the code is getting clean, OR your tests have stopped exploring new territory — CHECK COVERAGE to tell which. FLAT WITH HIGH genuinely approaching readiness COVERAGE THE AMBIGUITY IS THE CRUX: a flat curve means either "few defects remain" or "we have stopped looking in new places". Coverage is what disambiguates, which is why the two families of metric must always be read together. DEFECT CLUSTERING (the Pareto observation): defects are not uniformly distributed. Commonly ~80% of defects are found in ~20% of modules. THE ACTIONABLE CONSEQUENCE: a module with a high defect count is not "nearly fixed" — it is the most likely place for the NEXT defect too, and deserves more testing, not less. Complexity metrics (V(G) from the heuristics topic) predict these modules in advance. THE PESTICIDE PARADOX: if the same tests are repeated, they stop finding new defects — the code becomes immune to that test set. Test suites must be reviewed and extended, not just re-run. HOW EACH METRIC IS GAMED — know these, because being able to criticise a metric is what examiners reward: COVERAGE Gamed by writing tests with no assertions, or tests that call everything and check nothing. 100% coverage with zero assertions is achievable and worthless. → measure MUTATION SCORE alongside, or review test quality DEFECT COUNT Gamed in both directions. Rewarding testers for defects found produces many trivial reports; penalising developers for defects introduced produces defects reclassified as "requirements changes" and defects fixed without being logged. → never attach individual incentives to defect counts PASS RATE Gamed by deleting or disabling failing tests, or by writing easy tests. A rising pass rate with a falling defect discovery rate can mean the suite got weaker. TEST CASE COUNT Gamed by splitting one test into five. Counting tests measures typing, not coverage of risk. VELOCITY OF TEST EXECUTION Gamed by running the fast, shallow tests first. THE GENERAL PRINCIPLE: use metrics to ASK QUESTIONS, never as targets attached to individuals. A metric's value is in the anomaly it reveals — "why did branch coverage fall this sprint?" — not in the number itself. This is Goodhart's law applied to software, and it is the most important professional judgement in this topic. WHEN TO STOP TESTING — the honest answer combines several signals, no one of which suffices: · all planned test cases executed, with an acceptable pass rate · coverage targets met, including BRANCH coverage · the defect discovery curve has flattened while coverage is still growing · no open defects above an agreed severity · reliability measured against the requirement (MTBF, POFOD) · the remaining risk is accepted explicitly by the customer Testing is never "finished"; it is STOPPED, as a documented risk decision. Saying that plainly is more professional than pretending completeness.