That’s Magic!

If you don’t understand stuff, it’s magic. And if you’re relying on something that’s magic, your platform can disappear in a puff of smoke. This especially true of newer things – I don’t understand MySQL but it’s long in the tooth enough I can (mostly) trust it. Some of the newer NoSQL techs do not have that lineage…

Open Source allows you to get under the hood of all these things, to look behind the curtain and reverse-engineer what is going on. You invariably have to as the documentation is a TODO item. This means that when you do hit these extreme edge cases situations you can fix them, eventually.

But that’s only once you’ve really understood the problem. In black-box situations it’s all too easy to pull the levers you have until it seems the problem has gone away, but all you’ve done is masked, displaced, or deferred it. You have to understand the whole stack and not just “your bit”. (This reminded me a bit of a conversation with a friend who does network security, where decisions not to collect some data for “safety” actually made potential targets more obvious)

There are no gremlins

My favourite point was this: Computers are (mostly) deterministic.

We talk about bugs, issues, intermittent and transient faults – almost resigning ourselves to sometimes “things just happen”.

As Artur points out, computers are deterministic state machines, this randomness doesn’t really exist. Yes, the complex interplay of our interconnected systems can give the appearance of a random system, but that is just the appearance.

There is pattern in there, and when find it, you can fix it. How? Lots of monitoring, lots of measuring, and good old-fashioned investigation.

Stop throwing boxes & sharding at things

The easy availability of horizontal scale-out makes us lazy and complacent: “we’ll just throw another amazon instance at the problem”. That can be a valid approach, but only when your existing instances are actually spending all of their time doing meaningful work and not stuck queuing on some random service. If you’re site is sluggish because of poor code, database performance or tuning, you’re not really solving the problems.

Latency is even more critical(Google PDF), and scaling out a broken system may just let more people use it slowly – not make it faster.

Post-Cloud Call to Arms?

Scaling was hard: ordering servers took ages and it was all confusing. CDNs cost lots of money, were hard to use and only for the big boys.

Then “The Cloud” appeared: people like amazon and others made stuff cheaper and faster to get machines from. For a while we could ignore the complexity and just throw money at it.

But latency isn’t as simple as capacity, and we’re back to the situation that isn’t always about throwing more boxes into the battle.

I really like WordPress. I like the interface and the available plug-ins. It’s impressive how much you can do without coding, just configuring things. I use WPTouch to provide an iPhone/mobile devices theme. However, as a PHP application WordPress famously doesn’t scale to being Fireballed, Slashdotted, Dugg when you’re running on a small server. Even if you’re not too worried about being popular, by the time you’ve installed a few plugins, your site might be feeling a bit sluggish.

Like anyone with the naive belief I could write just such a post, but sensible enough to know it’ll too late to fix if I do, I’ve always had a caching plugin installed. Firstly it was WP Supercache, but now I use WP Total Cache (W3TC).

Short of using a caching front-end server such as Varnish (which is my next side project), WP Supercache/WPTC’s “disk (enhanced)” Page-Caching is the fastest way to serve your pages, because it’s done an Apache level. PHP isn’t loaded, databases aren’t queried, static files are served off disk.

W3TC also does JS/CSS minification, compression, and CDN distribution. These also can accelerate your site, but again won’t always work with all the plug-ins that you have installed. I would suggest you start off with basic caching, enable modules selectively, and test many times.

Caching, and things like themes don’t really co-exist that well, especially if you try and share the same URL for one bit of content rather than having the m.mysite.com thing. The normal install guides for WPtouch and caching plug-ins involve preventing page caching for all mobile user-agents. While it works, it’s not the most efficient use of your server, especially given people clicking on links from Twitter clients on mobile devices may prove part of the onslaught.

Rather than preventing all mobile traffic being cached, what you can do instead is request that pages are cached until “wptouch_toggle_cookie” is set. This is the cookie which allows users to jump back to the desktop view of the site on a mobile device. In this way you’re caching the first hit to your site, and all hits until they change theme. That should cover a lot of visitors.

You have two parallel caching silos: The normal site, and the mobile site. You direct to the correct one of these based on user-agent, and when the user manually flips, the cookie is set, and you’re no longer doing page level caching. You still getting the benefit of the Database and Object caching that WP Total Cache does, but it’s much slower. If you are lucky enough to be suddenly very popular, hopefully the vastly reduced load of the majority of users should mean that even non-cached users get the right content, and not a 500 error.

I’m not going to cover installing W3TC because the plugin itself covers this quite well, and other places. If you’re on Debian I’ll flag up that you can install all the additional libraries like APC/memcached for caching, Curl for CDN uploads etc from packages, and avoid compiling anything.

WARNING: this is potentially brittle. If the cookie name changes you will be leaving mobile users stranded in the mobile interface, with a “turn mobile off” button that doesn’t work. That said, if you have any mismatches between user-agents in the variety of lists they can appear, that also can cause problems. Check this all works after you update any of WordPress, WP Touch, WP Total Cache.

How to do it

WARNING: When you edit this group, WP Turbo Cache is editing .htaccess files. It doesn’t not escape them automatically, it’s VERY easy to break your website, including your admin interface… Don’t attempt to do this without file access, access to apache error logs, and ideally SSH access. Backup before you start: Your WordPress files, your wordpress database and also the WPTC config (there’s a download button on the first page).

To enable this hack you’ll have to do two things, one create the user agent group that will get the mobile touch theme.

1) Define the WP Touch User Agent Group

What we’re doing here is defining a list of the agents that should be redirected to the alternative cache pool, that shows the mobile theme.

My current WP Touch plugin claims on the admin interface to support “Android, CUPCAKE, bada, blackberry 9800, blackberry9500, blackberry9520, blackberry9530, blackberry9550, dream, iPhone, iPod, incognito, s8000, webOS, webmate”

The list in W3 Total Cache must match what WPTouch expects, if it doesn’t then you could well have mobile served to desktops, and vice versa. The nicest way of doing this would be if your mobile theme was in fact a standalone theme, as W3TC can serve different themes to different User Agent Groups, and you’d only be doing the switch once. This isn’t possible though.

Select “User Agent Groups” from the dropdown in the W3TC admin panel

Copying, pasting and editing from the existing User Agent groups, I created a group with the following. When you save this page, you can easily cause 500 errors on your site, until you manually edit/remove various .htaccess files in your website directory. If this happens you’ll need to read the apache error log to track down the offending files.

android
blackberry9500
blackberry9520
blackberry9530
blackberry9550
blackberry9800
cupcake
dream
incognito
iphone
ipod
samsung\-s8000
webos
webmate

Delete the other groups that WP Total Cache has put in place by default. That keeps the set of mod_rewrite rules used by Apache smaller.

2) Add the cookie exemption

On the Page Cache Settings, add “wptouch_switch_toggle” (which used to be “wptouch_switch_cookie”) to the rejected cookies list.

Save, Deploy & Flush all the caches and test

After you’ve done that, you’re good to go. Apply the config, purge all your caches, then try to access the site on a “clean” desktop browser. Verify that you’re getting the desktop version, and the comments in the source show you’re being page-cached. (compare to a browser where you’re logged in where you should see something about ‘cookie rejected’)

The try on your nearest iPod/iPhone etc. You may will want to install a Javascript source viewer to debug and verify you’re indeed getting a cached copy, then comparing what happens as soon as you turn the mobile version off. You should the desktop site, with the “change to mobile” button. Expect to do a lot of user-agent faking, and cookie clearing to verify you’re really seeing what you should be getting.

Finally, hit reload lots, and if you’re cached, your server shouldn’t really notice and speed should be quite fast.

At this point you’ll also want to ask a few friends just to see what they are seeing on the site, just to check for consistency.