97 Things Every SRE Should Know

Chapter 11. Why You Should Understand (a Little) About TCP

Julia Evans

Wizard Zines

I’d like to convince you that understanding a little bit about TCP (like how packets work and what an ACK is) is important, even if you only have systems that are making regular boring HTTP requests. Let’s start with a mystery I ran into at work: the case of the extra 40 milliseconds.

One day, someone mentioned in Slack, “Hey, I’m publishing messages to NSQ and it’s taking 40 ms each time.” A little background: NSQ is a queue. The way you publish a message is to make an HTTP request on localhost. It really should not take 40 milliseconds to send an HTTP request to localhost. Something was terribly wrong. The NSQ daemon wasn’t under high CPU load, it wasn’t using a lot of memory, it didn’t seem to be in a garbage collection pause. Help!

Then I remembered an article I’d read a week before, called, “In Search of Performance: How We Shaved 200 ms Off Every POST Request.” That article described how the combination of two TCP features (delayed ACKs and Nagle’s algorithm) conspired to add a lot of extra time to every POST request.

Here’s how delayed ACKs plus Nagle’s algorithm can make your HTTP requests slow. I’ll tell you what was happening in the blog post I read. First, some background about their setup:

• They had an application making requests to HAProxy.

• Their HTTP library (Ruby’s Net::HTTP) was ...