Lesson 22: What Erlang's scaling limits reveal about the difference between a good abstraction and a complete one
In a previous issue, I wrote about the hidden assumption baked into Erlang's actor model: that process isolation and message passing give you fault tolerance almost for free, provided you accept the model's terms. That post focused on what the model does guarantee. This one is about where that guarantee stops.
The origin story is worth revisiting briefly, because the design decisions make more sense when you remember the original constraint. In 1986, Joe Armstrong at Ericsson faced a requirement that most software engineers never encounter: telephone switches that could never stop. No maintenance windows. No scheduled downtime. The answer he arrived at — let processes crash and restart under supervision — was counterintuitive enough that it looked like surrender. It was actually a design philosophy: treat failure as the normal case and build the system around that assumption rather than against it.
The result, Erlang, reportedly achieved nine nines of availability in Ericsson's AXD301 switch — roughly 31 milliseconds of downtime per year. WhatsApp later ran messaging for 450 million people on Erlang with around 35 engineers before Facebook acquired it for $19 billion. These are real numbers, and they're genuinely impressive. They're also not the whole story.
The Abstraction Is Leaky at Scale
Here's the thing about elegant abstractions: they tend to hold until the hardware disagrees with them.
A paper by Phil Trinder and colleagues (arXiv:1704.07234) examined Erlang's scalability across two case studies — a distributed hash table-based Orbit calculation and an Ant Colony Optimisation benchmark. In the ACO benchmarks involving over 80 hosts, the researchers found that maintaining global recovery data dramatically limits scalability. The actor model didn't break — but the global state that lived alongside it did.
The Colony's analysis of this research puts the diagnosis cleanly: the problem isn't the actors, it's the hidden cost of global state. When a system is large enough, "global" becomes a lie. If your recovery data is global, your network traffic becomes a bottleneck that no amount of actor-based reasoning can solve.
The fix the researchers introduced — Scalable Distributed (SD) Erlang libraries and VM improvements — works by partitioning the network to reduce traffic. Even for programs with no global recovery data to maintain, SD Erlang's network partitioning improves performance. That's a meaningful result: the improvement isn't just about fixing a specific bottleneck, it's about acknowledging the physical topology of the cluster.
What this tells you is that the actor model is a programming model, not an infrastructure strategy. It gives you a clean way to reason about concurrency and failure. It does not give you a free pass on thinking about how your processes map to machines, network links, and memory topology.
What Happens When You Port the Model Elsewhere
One way to test a design philosophy is to transplant it and see what survives the move.
CloudMicroHaskell is a recent research project that brings Erlang-style distributed programming to Haskell — lightweight processes, message passing, monitors, supervisors, the whole OTP-inspired stack. The motivation is interesting: Haskell's strong type system lets you describe message-passing protocols at compile time in ways Erlang, being dynamically typed, cannot enforce statically.
The tradeoff is instructive. Cloud Haskell's source-level API requires remote processes to be expressed as static closures, and messages must satisfy serialization constraints. CloudMicroHaskell sidesteps this by serializing the runtime combinator graph directly, which means process bodies can capture variables from their surrounding scope naturally. But that directness comes at a cost: some guarantees enforced by Cloud Haskell's source-level types become dynamic checks, and programmers must be aware of laziness and runtime-owned resources when moving graphs between nodes.
This is the same lesson in a different key. Every time you make the actor model more ergonomic — easier to write, more expressive, less ceremony — you push some of the complexity somewhere else. In Erlang's case, it lands in the gap between the actor abstraction and the physical cluster. In Cloud Haskell's case, it lands in the type system. In CloudMicroHaskell's case, it lands in runtime behavior that the compiler can no longer see.
None of this is a flaw. It's the nature of abstractions.
The Lesson That Transfers
The actor model's real contribution to programming isn't a recipe for infinite scalability. It's a discipline for reasoning about failure. Processes are isolated. Crashes are local. Supervisors handle recovery. That discipline is genuinely valuable — it's why supervision trees, circuit breakers, and crash-only design have spread far beyond Erlang into systems that never touch the BEAM.
But the discipline only holds if you don't smuggle global state back in through the side door. The moment you have recovery data, coordination state, or shared configuration that spans your entire cluster, you've reintroduced exactly the coupling the actor model was designed to eliminate. The actors are still isolated. The state they depend on isn't.
The practical question for any system built on actor-model principles — whether in Erlang, Elixir, Akka, or something newer — is: where is your global state hiding? Not whether you have actors, but what those actors are secretly coordinating around.
That's the question the SD Erlang research forces into the open. And it's worth asking before you hit 80 hosts, not after.
