The simplest distributed system: many processors, one logical system.
🌍 Where this lives: the phone in your pocket has 6 to 8 cores, and its performance depends entirely on whether the software can use them. Since around 2005, single-core clock speeds have essentially stopped rising — chips got wider instead of faster. That means the only route to more performance is parallelism, and the only thing standing between your program and that parallelism is architecture. This is why concurrency stopped being a specialist skill and became a baseline one. Search "end of frequency scaling multicore free lunch is over".
The model
A MULTIPROCESSOR SYSTEM is the simplest DISTRIBUTED SYSTEM
model. It is composed of a number of PROCESSES that may (but
need not) execute on different processors.
ARCHITECTURALLY, this model is very common in large real-time
systems. These systems collect information, make decisions
using that information, and send signals to actuators.
Logically, distributing processes to processors may be
PRE-ORDERED (fixed at design time) or may be under the control
of a DISPATCHER that decides at run time which process runs
where.
THE DEFINING CHARACTERISTIC: the processes form ONE LOGICAL
SYSTEM. Unlike client–server or distributed-object
architectures, the parts are not independently useful services
— they are pieces of one program that happen to run on
different processors.
A WORKED EXAMPLE — a traffic control system (the standard
textbook one):
┌──────────────┐ ┌───────────────┐ ┌──────────────┐
│ Sensor │ │ Traffic │ │ Light │
│ processes │──▶│ flow │─▶│ control │
│ (many, one │ │ processor │ │ processes │
│ per sensor) │ │ (decisions) │ │ (actuators) │
└──────────────┘ └───────────────┘ └──────────────┘
│ │ │
└────────────────────┴─────────────────┘
shared/passed traffic data
· sensor processes collect traffic-flow information and
buffer it for processing
· the traffic-flow processor computes the system model and
passes commands to the light-control processes
· light-control processes convert commands to signals for the
lights themselves
· each of these may run on its own processor — or several may
share one — and the SOFTWARE STRUCTURE IS THE SAME EITHER
WAY. That location-independence is the architectural
point.
WHY MULTIPROCESSOR ARCHITECTURE EXISTS — three distinct
motivations, often confused:
PERFORMANCE do more work in the same wall-clock time
RESPONSIVENESS keep a control loop meeting its deadline
regardless of what else is running
AVAILABILITY survive the loss of one processor
Shared memory versus distributed memory
THE FUNDAMENTAL SPLIT, which determines how the processes
communicate.
SHARED-MEMORY (TIGHTLY COUPLED) MULTIPROCESSORS
All processors access one common address space.
· communication is by READING AND WRITING SHARED VARIABLES
· fast — memory bandwidth speeds, no message overhead
· needs SYNCHRONISATION (locks, semaphores, monitors) to
prevent race conditions
· limited scalability: the memory bus becomes the
bottleneck as processors are added, and cache coherence
traffic grows
· this is your laptop: SMP (symmetric multiprocessing),
typically up to tens of cores
· NUMA (non-uniform memory access) is the large-scale
variant — memory is still shared, but access cost depends
on which processor's memory you touch, so placement
matters
DISTRIBUTED-MEMORY (LOOSELY COUPLED) MULTIPROCESSORS
Each processor has private memory; there is no shared address
space.
· communication is by MESSAGE PASSING over an interconnect
· slower per interaction — a message costs microseconds
where a memory read costs nanoseconds
· NO race conditions on data, because nothing is shared —
which removes an entire bug class
· scales to thousands of nodes; this is how supercomputers
and clusters are built
· the programming model is MPI, or in modern practice a
cluster of machines
THE TRADE-OFF IN ONE LINE: shared memory is easy to program
and hard to scale; distributed memory is hard to program and
easy to scale.
FLYNN'S TAXONOMY — the standard classification, worth knowing
because it names the shapes:
SISD single instruction, single data — a classic
uniprocessor
SIMD single instruction, multiple data — one operation
applied to many data items at once. This is a GPU,
and it is also vector instructions (SSE, AVX,
NEON) in an ordinary CPU.
MISD multiple instruction, single data — rare; some
fault-tolerant designs run the same data through
different units and compare
MIMD multiple instruction, multiple data — independent
processors on independent data. This is a
multiprocessor or a cluster, and it is the category
this topic is about.
How much does parallelism actually buy? — Amdahl's law
THE MOST IMPORTANT NUMERICAL RESULT IN THIS TOPIC, and a very
common exam calculation.
AMDAHL'S LAW. If a fraction p of a program's work can be
parallelised and (1 − p) is inherently serial, then with N
processors:
speedup S(N) = 1 / ( (1 − p) + p/N )
and as N → ∞ the ceiling is
S(∞) = 1 / (1 − p)
WORKED — a program that is 90% parallelisable (p = 0.9):
N = 1 S = 1 / (0.1 + 0.9/1) = 1.00
N = 2 S = 1 / (0.1 + 0.45) = 1.82
N = 4 S = 1 / (0.1 + 0.225) = 3.08
N = 8 S = 1 / (0.1 + 0.1125) = 4.71
N = 16 S = 1 / (0.1 + 0.05625) = 6.40
N = 32 S = 1 / (0.1 + 0.028125)= 7.80
N = 100 S = 1 / (0.1 + 0.009) = 9.17
N = ∞ S = 1 / 0.1 = 10.00
READ THE LAST TWO LINES TOGETHER. Going from 32 to 100
processors — TRIPLING the hardware — buys you 7.80 → 9.17,
about 18%. And infinite processors never beat 10×.
WORKED — the same program at different serial fractions,
with N = 16:
p = 0.50 S = 1/(0.50 + 0.03125) = 1.88
p = 0.75 S = 1/(0.25 + 0.046875)= 3.37
p = 0.90 S = 1/(0.10 + 0.05625) = 6.40
p = 0.95 S = 1/(0.05 + 0.059375)= 9.14
p = 0.99 S = 1/(0.01 + 0.061875)= 13.91
THE LESSON: the serial fraction dominates completely.
Halving the serial fraction from 10% to 5% is worth more
(6.40 → 9.14) than going from 16 processors to 100 at 10%
(6.40 → 9.17). ELIMINATING SERIAL WORK BEATS ADDING
PROCESSORS, almost always.
WHAT THE LAW IGNORES, and why real results are often worse:
· COMMUNICATION AND SYNCHRONISATION OVERHEAD, which GROWS
with N — so real curves bend down rather than
flattening
· LOAD IMBALANCE — the slowest processor sets the pace
· CONTENTION for memory bandwidth, locks, I/O
GUSTAFSON'S LAW is the optimistic counterpart, and it is not a
contradiction — it answers a different question. Amdahl asks
"how much faster is a FIXED problem on more processors?"
Gustafson asks "how much BIGGER a problem can I solve in the
same time?" — and there the answer scales much better, because
the parallel part grows with the problem while the serial part
usually does not. Weather forecasting exploits exactly this: a
finer grid, not a faster run.
scaled speedup S = N − (1 − p)(N − 1)
at p = 0.9, N = 16: S = 16 − 0.1 × 15 = 14.5
compare Amdahl's 6.40 for the same p and N.
WHICH LAW APPLIES IS A QUESTION ABOUT YOUR PROBLEM, not about
arithmetic: is the workload fixed, or does it grow with the
resources available?
If you take one number from this topic, take the p = 0.9 ceiling: a program that is 90% parallel can never exceed a 10× speedup, no matter how many processors you buy. Performance work therefore starts with finding and removing serial bottlenecks, not with adding hardware — which is the opposite of most people's instinct.
🌍 Go further: the practical modern version of Amdahl's law is the Universal Scalability Law, which adds a second penalty term for coherency — the cost of processors keeping each other consistent. Because that term is quadratic in N, the resulting curve does not merely flatten, it peaks and then declines: past some number of processors, adding more makes the system slower. Anyone who has watched a database get slower after adding connection-pool threads has met this curve, and it explains why "add more workers" sometimes makes throughput fall. Search "Universal Scalability Law Gunther coherency".
💡 Exam angle: describe the multiprocessor architecture as the simplest distributed system model, draw the traffic control example with sensor processes, a traffic-flow processor and light-control processes, and note that distribution may be pre-ordered or dispatcher-controlled. Distinguish shared-memory (tightly coupled) from distributed-memory (loosely coupled) with the communication mechanism and scalability of each, and give Flynn's taxonomy (SISD, SIMD, MISD, MIMD). Expect to compute a speedup with Amdahl's law — memorise S(N) = 1/((1−p) + p/N) and the ceiling 1/(1−p).
Syllabus points
Multiprocessor design
Create a free account to tick topics off, take notes as you read, watch the video lessons and get a day-by-day study plan built around your exam date.