Kotan Code 枯淡コード

In search of simple, elegant code

Menu Close

Latency vs Throughput

Lately I’ve been surrounded by discussions involving things such as high performance and people asking questions like “Well, how fast is it?” Not too long ago, I would’ve been inclined to simply answer that question but it’s really not that simple.

The new answer is, “That depends on what you want.” You might be ready to throw me to the wolves but bear with me, both enlightening analogy and video game screenshots are coming forthwith.

This brings me to two words I’ve been hearing an awful lot of lately: latency and throughput.

Latency, from a network standpoint, is a measure of delay or how long it takes for something to start in one place and arrive in another. Throughput, in this context, can be thought of as the amount of “stuff” processed within a given period of time. So, if we’re on a network talking about messages then throughput can be measured in messages over time whereas latency might be the time it takes for a single message to be sent and delivered.

The reason I wanted to write this blog post is that “fast” is actually a fairly nebulous term. You can have a system where it takes very little time to deliver a message but the number of messages processed over time is actually pretty poor and vise versa. The interesting thing is that quite often people have to make decisions or tune systems where the smallest latency isn’t always preferable because sometimes it doesn’t offer the maximum throughput.

Take, for example, the concept of playing a real-time strategy game like one of my favorites, Warcraft 3. In this game, playing as orcs, you have a resource such as gold. This resource comes from gold mines and you have to send little orc miners out to harvest this resource.

Orcs Mining Gold in Warcraft 3

Orcs Mining Gold in Warcraft 3

What’s interesting here is how closely this maps to network application performance. If you have one orc miner on your team, and he can carry 10 gold per trip and a trip takes 20 seconds to a particular gold mine, you have a latency of 40 seconds (the orc has to go out to fetch the gold and then come back). Your throughput here is the amount of resource processing over time that can be done, e.g. gold per minute. 10 gold every 40 seconds is .25 gold per second or 15 gold per minute.

More Orcs pulling gold from a mine

More Orcs pulling gold from a mine

So now you decide that 15 gold per minute just plain sucks. You need more cash to fund your world domination plans, so you spawn new orc miners. Let’s say you create 10 of them. Now you still have a latency of 40 seconds but your throughput is now 150 gold per minute, assuming the mine can sustain that kind of load.

This isn’t part of the actual game but, for the sake of argument now assume that there’s only room for 4 miners at a time inside the mine. At a latency of 40 seconds (determined by the walking speed of the orcs and distance to the mine), at any given time you’re now going to have 6 idle orcs outside the mine.

Now you have a very real-world situation where no matter how many more orcs you throw at this mine, your throughput stays the same. If the mine is closer to your base, then the waiting orcs spend less time getting to and from the mine but really, do you want those guys idle or would they be better spent making a longer walk to a more distant mine to avoid the cost of leaving them idle?

All this babble about orcs and mines and gold really boils down to the core point of this blog post: The next time someone asks you how fast your system is, resist the urge to give them an answer. Instead, ask them what they want to know – throughput or latency. Or better yet, give them three values – latency/throughput at max latency, latency/throughput at max throughput, and some number in the middle.

Hidden beneath that question is the search for the “sweet spot” where all your resources are maximized in their pursuit of speed and you’re probably giving up some latency in order to push more resources through the system.

The second point of this blog post: Everything I know about network optimization I learned from playing resource-oriented strategy games.