Drinking From the Fire Hose #2 – the China Syndrome

One of the big complaints that the anti-Wordpress chorus croons is that the vaunted blogging platform doesn’t scale.  Certainly we’ve all seen sites brought low by the “Slashdot Effect” or the “Digg Effect” but my experience tells me that WordPress is getting a bad rap for poor server setup, poor plugin choice, etc.

How do I know?  Well, one of the sites I work with last week experienced the “China Syndrome” or potentially “The Great Fire Hose”.  The site, TheTruthAboutCars.com posted a story that the Chinese might buy GM, and that opened the flood gates for traffic.  The problem is, while we watch for excess traffic from Digg, or Slashdot, we don’t watch the Chinese sites that offer similar service.  In a matter of a couple hours, the traffic surged to 10 or 15 times its normal levels(and that’s conservative, once we max out server connections, we have no way of knowing how much is actually refused).  Our system administrator alerted me and I quickly through the SuperCache plugin into lockdown mode, ensuring that the site rendered virtually all its content as flat html, rather than going to the database every time.  

A strange thing happened.  The server remained up, but literally we swamped the number of available connections to Apache.  If you got a connection, you could surf the site fairly well.  If you didn’t, you got a fail to connect message.  So after some general hand wringing (and too much time), we reconfigured the site to work in our load balanced cluster.  Problem solved, but not without a lot of pain for those of us who are responsible for making sure that kind of thing doesn’t happen.

This is the kind of thing that is the down side of success.  The editor of TheTruthAboutCars.com has been a voice in the wilderness for years, screaming that the emperor had no clothes, with respect to the Auto Industry, particularly that they were in deeper financial trouble than anyone realized.  The past couple weeks have revealed The Real Truth About Cars – and that could be summed up as “Robert Farago was right.”  In retrospect, we should have expected a big traffic surge, it’s just that I don’t think we would have expected all of China to visit, at one time.

Lessons learned:

  • China is an Internet powerhouse.
  • All hail the Super Cache Plugin - it rules!
  • Server configuration is key – always think about the next step you might have to take to add capacity, and most importantly, think “how do I take the next step when I’m under fire…”
  • Make sure you have traffic warnings in place so you are alerted BEFORE you are in trouble with excess traffic.  We use Akamai DSA which makes this much easier.
  • If you get a massive traffic spike, take action immediately. Don’t wait, because it won’t get better.  Go to lock down mode, and add an extra server or two if you can (yeah, I know, not all of us have the luxury…).
  • Consider having the editors (if there is more than one) schedule their posts for the top of every hour, rather than just hitting the publish button.  This will keep Super Cache from being flushed every time a post goes live.
  • Again, live stats plugins rear their ugly head - This site was using a plugin that counted every time a post was viewed.  This got ugly.  If you really need that type of functionality perhaps you ought to use something like MyBlogLog.com and get the pro edition.  It offers live tracking, but it does it off server.  Any thing that wants to open a database connection every time someone views a page is downright dangerous.
Plan for success, and be sure you don’t set yourself up for failure. Answer the question “what would I do if…” right now.

4 thoughts on “Drinking From the Fire Hose #2 – the China Syndrome”

  1. Right, it was amazing to see. Usually when a wordpress site gets taken out, it kills the database by opening too many connections. Simply repointing the load balancer to include another server did the trick.

    Of course, as I note, not everyone has their site sitting around on a single server that’s getting served through a load balancer, with a couple of other machines that they could easily point traffic at. This is part of the joy of working with a top notch team of system administrators.

  2. Have you noticed rendering issues on IE6.0 when using super-cache plugin? We are having scalability issues with our site, and have looked into using super cache but it seems to have issues with IE6.0. Any other pointers for improving scalability?

    1. Ah, the curse of success. I haven’t seen an issue with rendering in IE 6 and the supercache plugin, and would have expected it to have cropped up before this. It could be that you’ve got an error in your css and for some reason the plugin is exposing it, but it’d be a long shot.

      Shoot me an email explaining what type of problems you’re having and we can maybe catch up via IM…happy to help as much as I can

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>