Mechanical Sympathy

Jackie Stewart, three times Formula One world champion, liked to say that you don’t need to be an engineer to be a great racing driver. But you do need mechanical sympathy: a feel for the machine underneath you. How the tyres are wearing. When the brakes are getting tired. What the car can give, and what it can’t.

Decades later a software engineer called Martin Thompson borrowed the phrase, because he’d noticed something about the fastest programs in the world. The people who wrote them weren’t using cleverer languages or more elaborate designs. They simply had a feel for the machine. They knew, roughly, what the hardware underneath their code was actually doing, and they wrote with it rather than against it.

This brick is about that feel. It builds on the first one, so here’s the one-line recap: every computer system is made of three acts — remembering, changing and sending. Today we look more closely at the first of those, and discover something the first brick quietly skipped.

The street was a lie

In the first brick, I described memory as a long street of numbered boxes, with a clerk (the processor) at the end, reading and writing. That picture is true in the way a map is true. But it hides one enormous fact:

Remembering has a distance. Some boxes are right beside the clerk. Others are very, very far away.

In your code, every box looks the same. x = basket[3] reads a box. But depending on where that box physically is when the clerk asks for it, the answer might come back almost instantly, or the clerk might wait a very long time.

The same day, at human scale

The numbers involved are so small that they mean nothing to us: billionths of a second. So let’s do what engineers sometimes do to make them felt. Take the fastest thing a processor can remember — reading from its own tiny, closest memory, about half a nanosecond — and pretend it takes one second. Then scale everything else the same way.

Using the well-known rough figures that Peter Norvig and Jeff Dean popularised:

• The clerk’s own desk drawer (the L1 cache): 1 second.
• The filing cabinet across the room (the L2 cache): about 14 seconds.
• The warehouse across town (main memory, the RAM we called “the whiteboard”): about 3 minutes.
• The archive in another city (reading from a fast SSD): about 3½ days.
• The old archive abroad (a spinning hard disk finding its place): about 8 months.
• A letter to Amsterdam and back (one network round trip from California to the Netherlands): about 9½ years.

Read that list again slowly. A processor that has to fetch something from main memory, instead of from its own drawer, is a person who has to drive across town and back for every single item. A program that waits on the network is someone who sends a letter and then sits by the door, doing nothing, for nearly a decade.

The figures are old and approximate, and modern machines vary. But the shape has barely changed in decades, and the shape is what matters: every step outward is at least ten times slower than the last, and some steps are a thousand.

Caches: keep it close

Because of this, every modern processor keeps small, fast memories close to itself, called caches, and quietly copies into them whatever it thinks you’ll need next. When the thing you ask for is already in the drawer, that’s a cache hit: one second. When it isn’t, that’s a cache miss, and the clerk drives across town.

How does the processor guess what you’ll need? It relies on two habits that almost every program has:

• What you used recently, you’ll probably use again soon. So it keeps recently used things in the drawer.
• What you need next is probably next to what you needed last. So it never fetches just one box. It fetches the whole small shelf of neighbouring boxes at once, a cache line, typically 64 bytes. If you then read the neighbours, they’re already in the drawer, for free.

This leads to the single most practical piece of mechanical sympathy there is:

Data that’s read together should live together.

A list of numbers stored side by side, read from start to end, is a gift to the processor: every trip across town brings back a whole shelf of things you’re about to use. The same numbers scattered around memory, each pointing to where the next one lives, force a drive across town for almost every item. Two programs can do identical work, line for line, and one can be many times faster than the other, purely because of where things sit.

The fastest exchange in the room

Here’s what this looks like at its most extreme, in a world you may one day work in.

In 2011, Martin Fowler wrote about LMAX, a financial exchange in London that had to process trades at enormous speed. You might expect a system like that to use dozens of threads, clusters of machines, an elaborate architecture. What the team actually built, at its heart, was a single thread, running entirely in memory, processing around six million orders a second.

They’d tried the more complicated way first. They found that their many threads spent more time passing messages and waiting on each other — fighting over the same memory, knocking each other’s data out of the drawer — than doing real work. So they removed almost all of it. One clerk, one desk, everything they needed kept close, never waiting for anyone. They called the approach mechanical sympathy.

There’s a very old lesson hiding in there. The fastest thing in the system was also the simplest. Less coordination, less sending, less distance: more done.

The aha: your kitchen is already a cache

While writing this, I noticed something that made me laugh, because it connects to almost every other piece in this journal.

You already know all of this. You’ve designed caches your whole life.

In a well-run kitchen, the knife and the chopping board are in your hands: the drawer. The salt, oil and spices sit beside the stove: the filing cabinet. The pantry is across the room. The supermarket is across town. And no sensible cook drives to the shop for each onion; they bring back a whole bag, because what you need next is usually next to what you needed last.

It goes further. The Phone at the End of the Lane is a caching decision in reverse: deliberately putting something far away so that reaching it costs you a walk. A Ladder of Small Movements puts the walking shoes by the door, close, so movement becomes a cache hit. Death cleaning is clearing the drawer of things you never use, so the things you do use can live within reach.

Engineers call it locality. Minimalists call it a well-arranged home. It’s the same idea: put what you use most, closest; and what you want less of, further away. The processor, the kitchen and the life are all arranged by the same law.

Sympathy for the machine you live in

Jackie Stewart’s point wasn’t really about engineering. It was about attention: about feeling what the machine under you was telling you, and not asking it to do what it couldn’t.

That’s a good way to treat a body, too. The ladder of movements works because it respects how bodies actually adapt: slowly, rung by rung, with rest in between. Pushing past the body’s signals is like writing code that ignores the cache — it works, sort of, until it suddenly doesn’t.

So mechanical sympathy is a habit worth carrying everywhere: before you ask a thing to work harder, understand how it actually works.

What this brick adds to the house

• Remembering has a distance. The same instruction can take a second or a decade at human scale, depending on where the data lives.
• Caches keep the likely-needed close, betting on recent use and on neighbours.
• Data read together should live together. Locality is the cheapest speed there is.
• Simplicity is often speed. Less coordination and less sending can beat more machines.

Where the next bricks go

• Boxes inside boxes: abstraction — and the price it sometimes hides.
• The post office: what’s actually inside that nine-and-a-half-year letter, and why networks are built the way they are.
• The careful librarian: how databases use these same distances, keeping hot data in memory and the rest on disk.
• Letters between villages: what happens when the slowest step, sending, can also fail.

Sources

• Martin Fowler, Mechanical Sympathy, on Jackie Stewart’s phrase and Martin Thompson’s use of it
• Martin Fowler, The LMAX Architecture (2011)
• Latency numbers every programmer should know, after Peter Norvig and Jeff Dean (figures c. 2012; human-scale conversions are mine, multiplying everything by two billion)