Where does the memory live? UMA, NUMA, DSM, clusters and programming models

Shared or private: what an address means

Start with two instructions. Processors P0 and P1 both hold R1 = 0x2000. P0 executes SD F2, 0(R1) with F2 = 5; a moment later P1 executes LD F4, 0(R1). Does P1 get the 5?

It depends on the address space: "the set of memory addresses that a processor can use in its instructions" (p. 425). In a multiprocessor the question is whether all processors share one global address space or each has a private one of its own (Fig. 1).

In a shared address space all processors see one single logical memory: 0x2000 is the same location for everybody, and P1 reads the 5. Communication is implicit — a store by one processor, a load by another, no "send" — and a parallel program looks like a uniprocessor program with several threads (Pthreads, OpenMP): the shared-memory model.

With private address spaces each processor has its own separate memory space: P0's 0x2000 is not P1's, and no load or store crosses from one processor to another. P1 reads its own 0x2000; to get P0's value it must receive a message that P0 sends explicitly, with MPI_Send() and MPI_Recv() in MPI: the message-passing model (p. 425).

; P0, R1 = 0x2000
SD  F2, 0(R1)
; P1, R1 = 0x2000
LD  F4, 0(R1)

; P0
SD  F2, 0(R1)
MPI_Send(X, dest=P1)
; P1
MPI_Recv(X, source=P0)
LD  F4, 0(R1)

Each model buys what the other pays for (pp. 425–426). A shared address space is easy to program, copes with irregular communication, moves small items without message overhead, keeps the memory model that compilers already optimize for, and lets caches cut remote traffic automatically. It pays with hardware that is hard to build — a scalable, coherent shared memory — with locks and barriers, and with little control over where shared data sits. Private address spaces need simple hardware with no coherence protocol, make communication explicit, synchronize at every send/receive, give control over placement and isolate faults. They pay with a harder programming model, a fixed overhead on every small message, and the work of distributing the data by hand.

Shared memory is easier for the programmer and harder for the hardware, which needs cache coherence; message passing is the other way round. Both can express any algorithm; they differ in latency, control and scalability (p. 426).

Where the memory sits: UMA, NUMA and the multicore chip

The address space is about names. The second question is about wires: where the memory is physically placed, and how the latency behaves (p. 427). Fig. 2 redraws the UMA and NUMA pictures of the notes (Figs. 34 and 36), which, the notes warn, "could happen on the exam". The second one did: with a directory added to every node, it is the figure of the directory question in all three official papers.

In a UMA (Uniform Memory Access) machine all processors share one centralized physical memory "with the same access latency and bandwidth, regardless of which processor initiates the access" (p. 427); such machines are also called Symmetric Multiprocessors (SMPs) or Centralized Shared-Memory Multiprocessors. Each processor has private caches, sometimes a shared L3, and a snooping protocol keeps them coherent.

Uniform latency makes UMA simple: nobody asks whether data is local or remote, and sequential code ports naturally (p. 428). The price is scalability. Every miss of every processor goes to the same memory through the same interconnect, so demand grows with the processors while supply does not. The result is contention: "two or more processors (or components) try to use the same resource at the same time", and some must wait (p. 432, footnote 12). Classic SMPs stop at about 32 processors (p. 428); lesson 33 meets the same wall from the side of the bus, which saturates at about 36 (p. 420).

The way out is to move the memory into the nodes. In a NUMA (Non-Uniform Memory Access) machine each node is "Processor(s) + private caches + local memory module", a network (mesh, torus, crossbar) joins the nodes, and "the union of all local memories is one global shared address space" (p. 429). P0 reading MM0 stays inside its node; P0 reading MM3 crosses the network and waits longer.

NUMA buys scalability, to dozens or hundreds of nodes, and bandwidth, since every memory module adds its own. It costs programming effort, because performance now depends on where the data is placed, and hardware: a scalable network and, since snooping a whole network does not scale, a directory-based coherence protocol (p. 430).

The multicore chip is a small SMP

A multicore with a handful of cores is an SMP on one chip (p. 432) — the "single-chip dual-core shared-memory architecture" of the June 2025 paper's Question 2. Private L1 and L2 caches hold each core's working set and keep it off the shared resources; a larger, slower L3, shared by all cores, cuts the traffic to DRAM and is where shared data meets (Fig. 3, left). Each cache controller snoops the shared bus and invalidates or updates its copy when another core writes, which a few cores can afford (p. 432).

Beyond about eight cores, one shared L3 and one memory controller become the bottleneck (p. 433). The L3 is cut into banks placed around the chip, which serve requests in parallel, and the bus becomes a network-on-chip (NoC): a mesh, a ring or a crossbar.

DSM: one address space over four memories

The notes explain a DSM with a university library system (pp. 434–435). Every faculty (a node) has its own library (a memory module); all the libraries together form one global catalogue (the address space); a student (a processor) can request any book (an address). A book from the own faculty comes quickly, one from another faculty rides the campus shuttle (the interconnect) and takes longer, and a central catalogue (the directory) records who borrowed which copy, so that everybody reads the latest version.

Distributed Shared Memory (DSM) is that machine: physical memory distributed among the nodes, one global shared address space, any location addressable by any processor — "distributed physically, shared logically" (p. 434). It is also called Cache-Coherent NUMA (cc-NUMA).

How can one address space live in four memories? The notes partition it (p. 436): Node 0 owns 0x0000–0x0FFF, Node 1 0x1000–0x1FFF, Node 2 0x2000–0x2FFF, and so on, so with four nodes Node 3 owns 0x3000–0x3FFF. Each node's memory controller serves its own range, and the owner of an address can be read off its top bits (Fig. 4).

Now follow the notes' example, a processor in Node 0 that accesses 0x2200:

; on P0, R1 = 0x2200
LD  F2, 0(R1)

1. P0 looks for 0x2200 in its cache C0 and misses. The request leaves the node and goes to the interconnect.

2. The interconnect routes it to Node 2, which owns 0x2000–0x2FFF: the top two bits of the address are 10.

3. Node 2's memory module MM2 reads the block at offset 0x200 and sends the data back across the network into C0; the load completes into F2.

4. Change R1 to 0x0200. The instruction is the same, but MM0 owns the address: the miss is served inside Node 0 and the network is never used. The program cannot tell the two cases apart; only the time differs.

The node whose memory holds an address is its home node; the node whose processor issues the access is the local node — here N2 and N0. The July 2024 paper asks it word for word, "What is the definition of home node?", about the NUMA picture of Fig. 2 with a directory D0 … D3 beside every memory, on an "Interconnection Network".

The second half of the widget is the formula behind "performance depends on data placement" (p. 430). If a fraction \(f\) of a loop's misses find their block in the local memory:

With 70% local misses, 0.7 × 80 + 0.3 × 240 = 56 + 72 = 128 ns; with 95%, 76 + 12 = 88 ns. Placement alone makes the average 45% longer.

Why reach into another node's memory at all? Because parallel programs share data (p. 437): an array that Node 0 initializes and Node 1 processes in place, a lock in Node 3's memory that Node 0 must acquire, the border cells of a simulation grid spread over the nodes. "If no processor could access remote memory, it wouldn't be ‘shared memory’ anymore; it would be pure message passing." With private memories the programmer would orchestrate every transfer — send(array, Node 1) here, receive(array, Node 0) there — while in a DSM "a simple load/store suffices" (p. 437): easier to program, harder to build.

Two pieces of hardware make it work. The interconnect moves cache lines between nodes and routes each request to the memory module that owns the block: a crossbar in small systems, a 2-D mesh, a torus or a fat tree in larger ones, a NoC inside a chip (p. 436). And since a DSM has no single bus to snoop, and broadcasting every write would not scale, a directory records which caches hold a copy of each block. It is stored alongside each memory block, at its home, and only the caches it lists are notified (p. 435); lessons 37 and 38 run it message by message.

In a DSM the address decides the home node; local or remote is a relation between the processor and that address. The program is the same either way; only the latency changes.

Clusters: separate computers that talk by messages

Now take four ordinary servers in a rack. Each has its own processors, often multicore, its own private memory, a local disk and its own instance of the operating system; Ethernet, InfiniBand or another high-speed fabric joins them. That is a cluster (p. 438). There is no shared address space: P0's load of 0x2200 reads P0's own memory, whatever Node 2 keeps at that number.

Nodes communicate by explicit message passing: a process in Node A calls send(message, Node B), a process in Node B calls receive(message, Node A) (p. 438). The receive waits until the message has arrived (Fig. 5), so every send/receive pair is a synchronization point, and the network latency is paid in full.

The software is MPI (Message Passing Interface), the de facto standard of high-performance computing: not a language but a specification plus libraries, for C, C++ and Fortran and with Python bindings (p. 438). Processes that share no memory must exchange messages to cooperate; a standard interface makes their programs portable across machines and vendors.

Clusters win on scale and cost (pp. 438–439). They grow from a few nodes to tens of thousands, and memory grows linearly with them, since every node brings its own DRAM. They are built from off-the-shelf servers and network gear, cheaper than custom supercomputers, and they run scientific computing, data analytics and AI training at scale; they are the dominant architecture of cloud datacenters, where each server is one node. They also isolate faults: one node can fail without crashing the system, because nothing is shared with it.

The bill goes to the programmer (p. 439): the data must be partitioned explicitly and every transfer orchestrated — "who owns the data", "when to send it". The network latency is much larger than a local memory access, the bandwidth between nodes lower than to the local DRAM, and every send/receive pair costs coordination.

A DSM and a cluster can be built from the same boxes. The difference is the address space: one address space and one operating system, reached by loads and stores, or one of each per node, reached by messages.

Three ways to program the machine

A programming model is the same choice seen from the software: "the software-level abstraction that maps onto the underlying hardware organization" (p. 440). It fixes how processors communicate — through a common memory, by explicit messages, or by all running the same program on different pieces of data — and the primitives that express parallelism: threads, send/receive, barriers.

Shared memory

Threads communicate through shared variables — Thread 0 writes x = 42, Thread 1 later reads x — on any machine with one address space, UMA or NUMA (p. 440). Locks, semaphores, barriers and atomic operations prevent race conditions; the tools are OpenMP (pragmas for parallel loops), POSIX threads, Java or C# threads. It is the natural model for SMPs and multicores; coherence traffic, placement on NUMA and synchronization limit it at scale (p. 441).

Message passing

Process A calls MPI_Send(data, dest=B), Process B calls MPI_Recv(data, source=A); groups use collective operations — broadcast, scatter/gather, reduce, barriers (p. 441). The model scales to thousands of nodes with no coherence problem and explicit control of locality, at the price of complex programming, network overhead and waiting at the send/receive points (pp. 441–442).

Data parallel: one program, many slices

The third model starts from the data: the same operation on different elements of a large data set. All processors run the same program, each on its own portion — SPMD, Single Program, Multiple Data (p. 442). The notes add two 1000-element vectors A and B on four processors: Proc 0 takes indices 0–249, Proc 1 250–499, Proc 2 500–749, Proc 3 750–999, each with the same loop.

/* on every processor p = 0 … 3 */
for (i = 250*p; i < 250*(p+1); i++)
    C[i] = A[i] + B[i];
barrier();

Data-parallel programs move in bulk-synchronous phases: local computation on local data, then global communication or a barrier, then the next phase — the BSP model formalized by Leslie Valiant (p. 442). The barrier is also the weak spot: a phase lasts as long as its slowest processor (Fig. 6).

Data parallelism runs on SIMD and vector machines, on GPUs where CUDA or OpenCL launch thousands of threads running the same kernel, on multicores with #pragma omp parallel for, and on MapReduce or Spark clusters (pp. 442–443). It is simple, scales when the data are large and split evenly, and keeps GPUs and vector units busy; it suffers from load imbalance and stragglers, and suits regular, array-like work, not trees or graphs (p. 443).

Two axes, four machines

Put the two questions side by side. The address-space model, shared or private, and the physical memory organization, centralized or distributed, are orthogonal: "choosing one does not force the other" (p. 431). All four combinations exist (p. 430).

The only unfamiliar cell is private and centralized: MPI processes on one SMP box, each in its own address space, exchanging copies through the one memory — "rare, but possible in small message-passing systems" (p. 430). The January 2026 quiz, though, gives a picture, not a name.

Back to the quiz of the hook. The column first: every node has its own memory module and there is no central one, so the memory is distributed, and Answers 1 and 2 are out. Then the row: every module is labelled "P0, P1, P2, P3", so every processor can load and store every module — a single logically shared address space. That rules out Answer 4, in which each processor reaches only its own memory and communicates by send/receive. The answer is 3: distributed memory, single logically shared address space, a DSM. Give each memory only its own processor's label, and the same drawing becomes a cluster: Answer 4.

"Shared memory" is about addresses, not about where the chips are: the number of memory modules sets the column, the processors that can address them set the row.