The Solved Problem of Thundering Herds

[EDIT: As if fate decided to slap me in the face, the strangest entry decided to get an enormous influx of traffic right as I was running an XCloner backup while updating the wording of an entry, trying to make myself sound like a bit less of a jerk. Egg on my face as this poor little Micro server did exactly what I crowed about it not doing…dying. Mumble mumble excuse diversion]

This blog runs on an Amazon AWS EC2 t1.micro instance (~600MB of RAM and a single, heavily throttled vCPU), an intentionally restrictive choice I made years ago in an effort to practice something I oft preach: That it is unacceptable that sites fall over and die under the slightest bit of attention, failing at what should be their moment of glory.

I’ve had a number of very heavy burst days since with absolutely no issues at all, if even a perceptible slowdown for users.

Last evening into today, for instance, saw a spike of traffic directed here from Hacker News, Reddit, and various Twitter referrals. In all there were some 50,000+ page impressions in the past 20 hours.

Spread out that is hardly impressive (under a page impression a second), however traffic was very bursty, with extended periods exceeding 60 page impressions per second.

That isn’t huge by any measure of the imagination, but this is exactly the same sort of situation that sees the dreaded “Database connection not available” error on so many sites.

Yet there were zero database connection errors. No error 500s. Everything running smoothly without the slightest hint of trouble. Absolutely nothing was stressed at all. A run of top usually showed top or sshd as the top consumer of resources.

On this miserably tiny server. Running WordPress!

The reason, of course, is caching. With W3 Total Cache the vast majority of requests are served as if they are static requests, with the smallest dynamic wrapper to match the request with the corresponding static resource. You could take it a step further and actually generate static file resources, eliminating anything dynamic above the instance of nginx or Apache, as I did with the Name Navigator, however that is optimizing the edges and can be an optimization too far.

The Thundering Herd problem is a solved issue at normal scales, without reactively firing up an army of AWS instances because you made it to the front page of a social news site. It is very unfortunate when sites die under marginal loads, the time of users wasted, and authors of interesting content deprived of their moment of exposure.

As an aside, one of the most rewarding experiences is when I see people who come here via posts on social news sites, but instead of simply bouncing back they continue on to read through various other pages. That it was interesting enough on a net of endless information is very gratifying.

WordPress and Comment Spam

For years I blogged on a self-created blog platform. It was very secure, very efficient, and could serve up massive page render counts on miserably weak machines.

It was deficient on the actual content creation side, however (the iPad of blogging…rimshot), making authoring new posts an onerous task. That was one of those “to be completed” things that always got de-prioritized.

I allowed comments. If someone disagreed with something I’d written, I’d happily host their counterpoint. Security through obscurity helped me pre-filter, most spammers unwilling to bother for one relatively small technical blog with a very unique infrastructure.

Every now and then someone would create a script just for me and I’d slightly change-up the API to thwart their attempts.

Not the case with WordPress however. I’m racking up 1400+ spam comments a week, after filtering through available spam plugins. That is untenable, and sadly I can’t spend the time filtering through for the one or two legitimate comments.

So comments are no more, killed by the popularity of WordPress and its prolific and well-targeted comment API. I still hope that people who disagree or have information to add will take to their own content systems to make their voices heard as well.

Sidenote: Many popular blogs killed comments long ago. The reason is simple: When people lose the option of arguing a position they disbelieve in the comments, they are more likely to take it somewhere else, providing some delicious PageRank juice in the process. That is not and has never been a concern of mine, made evident by the fact that I keep tossing out all PageRank goodness every time I decide to completely revamp URLs and domains.