Secret Cinema’s PR Car-crash

Secret Cinema showed how not to communicate after the opening night of their latest event was cancelled.

Lots of modern knowledge-based skills are like Search Engine Optimisation: the first 80% of SEO is “build a decent website” and the last 20% is the ever-changing dark-magic that few people really understand.

I’m adding “communications in a crisis” to this list.

Secret cinema have cancelled their opening shows of Back To The Future, the first show cancelled about 2 hours before it was due to start. The comments on that post are just about as awful for the company as you’d expect.

The company is replying, but with a statement usually of the lines “please address your concerns to us at this email”. Unsurprisingly this isn’t meeting with much understanding from their customers.

As I type this on Friday evening, they’ve just cancelled the weekend shows, and the “situations beyond their control” appear to be the council aren’t satisfied the venue is safe.

Predictably, their Facebook wall has been carnage. People explaining how they’ve travelled far for this event, and are feeling let down. Now if you travel to a faraway place for a pop-up event, by a company who have cancelled opening nights before1, caveat emptor comes to mind. I’m not saying I don’t have sympathy, but I doubt I’d travel myself in the circumstances…

Crisis comms are hard

There are companies who charge you an awful lot of money for just this. The ones you call when things are really bad: like when your product kills people.  But much like SEO, companies can do the simple things to get the first part themselves.

4 Basic Steps to Delivery, You’ll Never Guess What Happens When You Don’t Do Them:

  1. Project Management is your friend: If they didn’t know until the first day that they had these problems, they don’t have a decent project/production management team. This isn’t a hobby, this is a company that take a lot of money from people, they need a decent delivery function that could warn ahead of time.
  2. Honestly within the company: can you delivery team tell you that there are possible problems, or are you stuck in an organisation where the status report has to be green? Or worse, are you in an organisation that denies possible problems until they’ve actually happened.
  3. Run Pilot events. This is the kind of thing you probably want a few preview nights, beyond rehearsing with the cast, but rehearsing with audiences there so you can check things work. You can set expectations for these nights better, with lower tickets prices, and framed as a community rather than a customer experience. Scratch that, apparently their preview on Wednesday was also cancelled.
  4. Prioritise: there will have been things here key to the experience, and things that were icing. Build and get approval for the main stuff first. If you can’t do the other bits that’s a shame, and the pilot/early nights might be impaired. But at least they can run.

The 6 Secrets to Basic Crisis Comms Techniques They Don’t Want You To Know:

  1. Don’t Weasel Word: Be very careful about the phrase “beyond our control”. I watched a documentary about Crossrail last week. The crane they needed one weekend didn’t turn up because only 2 of them are in the country, and the one they’d booked was delayed. That is “beyond their control”.  I say this with no insider knowledge, beyond the news articles, but  Secret Cinema were in control of applying and meeting council safety approvals. Saying it’s “beyond your control” makes an organisation look like it’s in denial.
  2. Appear Open:  They should have published their compensation policy and directed people to that. Telling people to “address concerns” privately makes it look like the organisation has something to hide.
  3. Appear Honest: This isn’t an outage of a complex system that takes time to diagnose. Saying you’ll post “more information later” just makes it look like an organisation in disarray.
  4. Take the Hits upfront: They could have cancelled more shows upfront, still disappointed people, but put them in control earlier. Drip-feeding cancellations just continues the uncertainty, again adding to the appearance of disarray.
  5. Finally, you’ve broken promises: Don’t make any other promises you can’t keep. It seems so minor, but saying you’ll update at 11am and failing to post anything until after 12 just continues the appearance of the organisation in crisis and denial.

I suspect this incident will be a case-study for crisis PR for years to come.

The Meaning of Silence

Marco’s new podcast app Overcast can remove the silences. Does our relentless demand to make everything more efficient sometimes remove more than is desired?

Marco Arment, formerly of Tumblr, Instapaper,  and The Magazine, has released his podcast app Overcast. It’s generally very nice, and already seems to annoy me less than Apple’s own app.

As well as the standard playback speed settings, Overcast offers the option to shorten silences. This speeds up your podcast playing without distorting the audio. It’s an optional setting, and one that correctly you can set per-podcast.

Now… I can.. appreciate… how this might.. erm… help if you’re listening to a podcast by someone who has awful delivery. Most of mine are from radio shows from members of Big Media: there isn’t a lot of silence to be culled.

Some of the tweets have been very, for want of a better edited phrase, Techno-utopian-efficiency-fetishizing. Comments along the lines of “Already saved 30 minutes using SmartSpeed” and “Can you add up and display all the time I’ve saved?”

My issue is that well-meaning pauses are just as much part of good oratory as the words.

Take them away and things can go hilariously wrong:

This isn’t a criticism of the app, or the author. The feature has its place. I’d prefer to think about it being used to fix deficient audio, rather than eke every possible minute possible out of listening.1

I just tire of the endless demand for evermore efficiency in everything.

Yes I want my banking to be easier.  Of course I’d rather type data into systems directly rather than sitting on the phone, as someone enters it for me…

But when the need for faster/cheaper/better detracts from the experience, that’s when it starts annoying me. When it’s the kind of mindset that thinks that chewing food is a chore.

Not everything needs to be efficient, not everything needs to be a measured.2

  1. I’d prefer people to produce better audio in the first place, but it turns out producing decent audio takes time…who knew? ↩
  2. And on that note my FuelBand is nagging me to get moving ↩

Falsehoods Smart-Device people believe about Home Networks

A few years ago someone posted a great article about the bad assumptions programmers make about names; here’s a similar list about assumptions about home networks and smart devices.

We all remember the excellent Falsehoods people believe about names don’t we?

Having lived with a few smart devices sharing my network for a while, I thought we need a similar one about smart devices and home networking.

Items marked with a * contributed or inspired by @davidmoss

  • The WiFi is always available
  • The WiFi is continuously connected to the internet
  • The WiFi network isn’t hidden
  • The WiFi network isn’t restricted by MAC address so they can be hidden from the user
  • The WiFi network doesn’t use strong authentication like WPA2
  • The WiFi network definitely doesn’t use authentication mentioning the word ‘Enterprise’
  • The user knows the exact authentication type is use for the WiFi, so no need to auto-detect it*
  • There is only a single WiFi network
  • The name of the WiFi network is ASCII*
  • There is only a single access point for the WiFi network
  • Any device connected to the home-network is trusted to control the smart devices on it
  • Smart devices and their controllers are on the same network
  • Devices on the network can connect directly to each other
  • The network is simple, and doesn’t use other technologies such as powerline1
  • All networks have a PC type device to install/configure/upgrade devices (and that device is running Windows)*
  • There is always a DHCP Server*
  • Devices will always get the same IP address on the internal network from the DHCP server
  • DHCP device names don’t have to be explanatory, because nobody ever sees them
  • Devices can have inbound connections from the internet 2
  • The network is reliable without packet loss
  • The connectivity is sufficient for all devices on the network
  • The performance characteristics of the network is constant and doesn’t change across time
  • The Internet connectivity isn’t metered, and there’s no problem downloading lots of data
  • Encryption of traffic is an overhead that isn’t needed on embedded devices
  • Predictable IDs like Serial-Numbers are good default security tokens
  • Unchangeable IDs like Serial-Numbers are acceptable security tokens
  • The device won’t be used as a platform for attacks, so doesn’t need hardened from threats internal and external to the network. 3
  • Devices can be shipped and abandoned. They won’t be used for years, as so any future software vulnerabilities can be ignored
  • IPv6 is for the future, and doesn’t need to be supported4

What have I missed?

  1. These should be layer 2 transparent, but they can disrupt Multicast which can break bonjour ↩
  2. aside from security implications, ISPs are moving to a carrier-grade NAT to work around IPv4 address exhaustion, so inbound ports may not be possible ↩
  3. many devices have a pretty complete Linux stack, at least complete enough for attackers to use ↩
  4. Chicken and Egg this one ↩

Security is hard, but the easy bits aren’t

The hard bits of security are hard, but the easy bits aren’t. As infrastructure gets more dynamic, we need to make sure it isn’t everyone else redefining it.

Another week, another story about security.

Actually multiple stories about security.

And what’s upsetting with these ones are the fact that the fixes for them are already available.

I don’t cut-code anymore. I’m not a particular adept coder, and I think my code is a bit ugly. But I still know what bad practice smells like and what upsets me is how often we have repeat the mistakes of old. 1 2

Yes there are always deadlines, but if we’re working with advanced software defined infrastructures, then we have to restrict who can redefine those.

If you’re in a Product Manager role, don’t be afraid to ask what you’re doing for security, or what response plans are if something is compromised. Be mindful of the risk to your reputation or risk if you don’t give developers time to improve security instead of piling ever more features on. The mitigations for the most obvious attacks are documented, and usually relatively easy to implement.

And now to the details

Code Spaces had all their data wiped, we don’t know all the details but it sounds like:

  • They hadn’t enabled 2factor auth on their AWS account
  • Their backups weren’t to a different AWS account,  or better still to another provider.

If you’re running a production service, and you’re hosting data for anyone else, then your backups need to be rock solid. Backing up to the same provider, in the same account, is like copying all the files from your desktop into a folder called “backup”.  Sure you’ve two copies but when that disk goes bang they’re both gone.

And yes, 2 Factor is a pain when you’re logging into services, but if you’re hosting customer data that’s a pain you need to cope with. Providers usually let you set up many secondary accounts with reduced privileges, so use those tools to protect your services, and let people do just what they need in order to do their jobs.

On a similar theme people are leaving their AWS keys in android apps. Amazon offers a ticket granting service that’s ideal for this, but that’s more work, but work that you should be doing.

Some people aren’t even using those permissioning tools to embed keys with limited access, which just to reiterate, you shouldn’t be doing anyway. Instead they are embedding their main access key pair, which means that attackers could access and delete all data, and spin up thousands of instances just for fun/profit.

Security is hard, the recent problems found in libraries like OpenSSL are hard for an individual coder to work around, but decent libraries are still better than going it alone.

The 80:20 rule is ever present, will you ever make your app fully secure; unlikely. Can you prevent the most obvious attacks with application of best practices, which many programming languages can do for you; yes.

Don’t leave keys lying around, give apps or services any more permissions than they need, or use predictable IDs for sensitive data…

Do sanitise data you’re given, protect from XSS attacks, turn on 2-Factor Authentication for anything serious and always keep decent backups hosted on separate infrastructure…

These lists go on, but they not new: Best practice years ago, is still best practice now.

  1. Don’t get me started on file-moving scripts that don’t use incoming and outgoing folders to avoid race-conditions ↩
  2. Or when we tolerate software from vendors that can’t run as anything other than root or Administrator ↩

Cloud, the cost and value of everything

Forget scalability, speed to change and flexibility, I think the single most important thing about cloud hosting is putting an explicit cost on everything…

Last night I gave a lightning talk at the newly tweaked #metabeertalks, these guys are great friends of mine, and their topic was “is realtime Fashion or Fad?”.

Modern hosting approaches, aka “the cloud” have many advantages: They scale trivially, encourage you to use best practices in how you architect and deploy, and are flexible to change as your application does.

The single most powerful thing though, is that it puts a cost on every element of your application. We can debate if it’s cheaper, more expensive, or about the same as hosting on tin: but you know what your components cost.

Your application isn’t being bundled up with a load of others on a server, with your IT team complaining they have to install a new one with about 1 months lead time every 3 months.

You host your application on instance the right size for it, be it small or huge, single or a fleet of 20. You use the storage you need, when you need it, without playing that impossible game of “how much storage will we need by the time the storage system actually arrives”.

And all this comes with transparency: set your system up with the right tags, and all the costs of an application are known.

Knowing those, you can start flexing: If you need 10 machines to keep up with realtime analysis, they’re yours. Or if you don’t want to pay that, bid for some cheaper instances and batch the work overnight.

Within reason, you can do anything, if you can afford it. So you can take a call about which bits of information are valuable enough to justify being realtime.

When Netflix launched House of Cards series two, you can hear them talking about the “Play Start” messages coming in. That kind of realtime information is amazingly helpful for debugging.

The deeper stats of how many people watched, and how many episodes they binged on, that information could probably wait a few hours to batch…

My take on realtime: do it where it’s valuable, and where you can justify the cost.

Which is exactly the same for all elements of cloud hosting.

Two Apps Better Than One?

Mixing Subscription and Transactional VOD in the same application can give a bit of a confusing user experience.

When Sky launched Sky Store, which lets you rent films, it felt unnecessary alongside their existing subscription services: Sky Go (for Sky TV subscribers) and Now TV (for everyone else). It seemed little more than an attempt to get extra space on Smart TV menus.

Since then though, LOVEFiLM has finally acknowledged its longtime parent Amazon. I churned from LOVEFiLM a while back1, but I’ve a few months left on Amazon Prime. This now gives me access to “Prime Instant Video” via the “Amazon Instant Video” app2.

It’s good because thanks to all the exclusive content deals Amazon made 3, I’m able to catch on the series that weren’t previously available to me.  4.

But, the Amazon Instant Video app mixes stuff that’s ‘free’, with stuff I have to buy or rent.  There are categories designed to help me filter; but if I search for a series directly, I’m back to the jeopardy of “free or not” after seeing a search result.

Netflix doesn’t have that: if I see it, I can play it. The logic of Sky Store becomes clear.

Yes, NowTV has three subscription tiers of Movies, Entertainment and Sport: but those are really clear facets. I know which of those I’ve paid for, so I search for an entertainment show, knowing I can watch it.

Multiple apps may be the online equivalent of grabbing extra shelf-space, but I can see the UX benefit in separating subscription from purchase & rental.

  1. And their come-back emails would not let me forget this ↩
  2. Brand recognition since the rebrand is apparently poor ↩
  3. Alongside all the non-exclusive deals both Amazon and Netflix have ↩
  4. I’ve yet to figure out the rights-deal that’s made the BBC series Miranda appear with 4OD branding in Amazon ↩

Articles like this are why people think the cloud is oversold

When Malaysian 370 went missing, someone suggested “The Cloud” could solve all the problems.

The cloud can solve many problems, and is rightly seen as one of the easiest ways to launch web services. But it isn’t magical, and articles like this  are why people think the cloud is being oversold: The cloud is not the solution to finding missing Malaysian flight 370:

But if MH370 had been fitted with technology that made use of the cloud it may never have been lost in the first place. The cloud is a cluster of computers that provides reliable computing and storage as a service to large numbers of requests from computers with limited capabilities, such as those on board a plane or inside a mobile phone.

What the author says is really “planes should dial-back to a server with their telemetry”

This may be true, but as a comment on the article points out: that doesn’t need the cloud.

It needs a server in a data-centre. Now you may choose to deploy that server as a virtualised box in the cloud, but this is not an application where you need the main virtues of cloud type platforms.

Over and above machine virtualisation, I tend to think about ‘cloud’ meaning some combination of these  things:

  1. You scale your resources when you need them, not ahead of time. The best example of this is storage: you don’t have to pre-size your storage allocation in Amazon or Azure. 1
  2. Your application is making use of the two main scaling patterns, incoming load balancers2 and asynchronous message passing3, to dynamically change the amount of processing capacity that you have.
  3. Not a technical thing, but your costs should be scaling in line with your usage. Having the incentive to save money by doing as little as possible when you’re idle will encourage you to properly scale.
  4. You start treating your servers as livestock and not pets. If  virtualisation separates your instances from the physical hardware,  cloud deployment should separate your application from the instances.
  5. Your deployment should be cheap. It should take minutes, and be painless, and shouldn’t make your ops team bite their nails in fear. It needs to be a routine, accepted, automated process. This also requires you to have your config held in more durable places than a file on an instance, which could disappear at any moment.

The dial-back type solution that could help us find missing planes doesn’t really need many of these characteristics. The data formats would be relatively static, and the loading wouldn’t peak to such levels that you needed to place it all behind a massive loadbalancer. You’d care about reliability, but I don’t see masses of room for flexing things here.

Yes dial-back is a good opportunity to improve visibility (the ACARS data from Air France flight 440 provided a  trace of the accident), but what really could have helped us in the case of Malaysian 370, would have been that something had continued to report back position information after ACARS was disabled.

We don’t know if ACARS was disabled manually by the flight crew, or by a result of electrical systems being de-powered due to a fire. We do know the Inmarsat satellite modem was still functioning for some time, and responded to a network level ping. This only gave us a confirmation that the modem was still in range of a satellite beam, and unfortunately it was a large satellite beam which covers a wide area.

Had the plane been fitted with a newer Inmarsat system, it would have been connecting to a satellite beams with smaller footprints, which could have narrowed the search area.

What might have helped  would have been if there was another GPS receiver integrated with the satellite modem, so that even without the main ACARS system, at least position could still be reported.

That isn’t in the cloud however, that’s on the plane.

Better reporting back could have helped the investigators here, but no, that is not another solution in search of “the cloud”.

  1. Much to the cheers of capacity planners ↩
  2. When more people hit your website, you launch more servers ↩
  3. Instead of doing an operation when a request comes in, you put it on a queue. When the queues start growing too big, you start additional instances ↩

Good Luck Microsoft

Microsoft have appointed Satya Nadella as their new CEO. He’s an internal hire, but from the services bit which includes Azure.

Microsoft have appointed Satya Nadella as their new CEO. He’s an internal hire, but from the services bit which includes Azure. Although everybody is playing catch up to Amazon Web Services, Azure has a number of features that are interesting: getting that cloud computing isn’t just about easy access to disposable servers.

Microsoft today is like the uncle who’s was great when you were a kid, got you interested in stuff, and has now fallen on hard times.

Maybe I’m just biased because I like Office (which makes me a minority I know), but I don’t want a world where there isn’t Microsoft. Google Docs is great for sharing or collaboration documents, Apples iWork is great for simple documents, and well I’m sure OpenOffice is good for something.

Microsoft Research produces so many good ideas, or clever ideas, or just the plain “hey we had a random idea” ideas. They don’t many to use that many of these, so many of them are impractical with current tech. But the ideas are there, at some level the company still tries to innovate.

That innovation doesn’t come easily however, as Windows 8 and attempts for a converged desktop/mobile/tablet interface have shown. The company doesn’t have that Apple confidence of “this is the way we scroll now”. Appeasing the fans of the legacy will not help them move on. Perhaps when the company has a better idea of what the “new” Microsoft is, selling those ideas will be a bit easier.

I may well be a Mac and iOS user now, but I think if I was going to switch phones, it would be for a Windows Phone. A bit like the Palm Pre, or Blackberry’s ultimately doomed Blackberry 10 operating system, Windows Phone didn’t feel like it started off with the requirement “be like iOS”. Android and iOS are really converging in many ways, features hopping from one to the other.

For that reason alone, I would like Microsoft to do well in the future: much like the Shuttle’s fifth computer, I think we need a strong third platform in the mobile market.

 

Management Tips from Astronauts

What former Astronaut Chris Hadfield can tell us about managing things down on earth.

Being on the International Space Station (ISS) for a few months is a pretty unique experience. I’m pretty sure that nobody reading this will get to do it. Chris Hadfield, the first Canadian astronaut, did spend some time there, and I’m sure you remember his Bowie cover Space Oddity.

Anyway, he has a book out (Amusingly at Christmas I bought it for my Dad, while he bought it for me). The book manages to make space travel both more alluring, and yet in many parts tediously mundane. It’s seemingly a lot of study, luck, and waiting for your mission. Also sounds hazardous to your marriage if you’ve anything less than the most understanding of spouses.

Alongside this he highlights a few management things, from his employers, or himself, that are worth remembering here on earth.

Confessing to Near-Misses

NASA, like every safety-critical system (or at least like they should) place great emphasis on being able to speak about near-misses. About the times that something nearly went wrong, so that changes can be made before it actually happens. (I’m not aware of how much of this was in place before the Challenger Launch Decision was made.)

I don’t work in safety critical systems, I work in computers and websites. Although much less severe, we do face  similar challenges. Do you have that random configuration utility that if you feed it incomplete or invalid configuration details, will honour those and wipe out an environment?

You shouldn’t.

In an ideal culture you should be able to say “I was messing about on stage, and noticed that I could break the system with the config tool” and that the reaction should be “Oh, great, let’s figure out if we can easily fix that, and if we can pop it in a sprint” and not the sometimes standard reaction from developers, inwardly judging the operator for using the tool wrongly, while outwardly declaring “Well then you should be more careful with that tool”.

These kind of things matter: You’re not always the ‘you’ in the office.  11am at the ideal caffeination level ‘you’.  At 3am, roughly extracted from sleep by PagerDuty, you’re a lesser ‘you’.

At those points, you’re flying on instincts and adrenaline.

Systems need to be idiot proof because we can all be idiots. (And thanks to a neanderthal leftovers, I think that sometimes the smartest people can be the best idiots).

What’s the BOLDFACE for this?

Documentation and procedures are another ongoing theme. Unsurprisingly every procedure and task in space are heavily documented, because you don’t want this to go wrong when you encounter problems. To paraphrase “you should always know what the next most likely thing is that can kill you, and how to go about stopping it”

The BOLDFACE bits are the critical bits of documentation that keep you alive. Again, IT is not life or death, but your run-books and documentation should have this similar level of priority.

No operator probably needs to know everything of every system, but they should know the procedures which if done incorrectly, (or the ones that done correctly) cause data-loss or system outages.

Some years ago, I was personally stung by changes between software versions: the version before didn’t, the version after wouldn’t, but the current version had some horrible behaviour, and I managed to cause a significant outage.

So on top of your documentation, when the operations become more or less dangerous than they were, make sure that people know about the changes.

Being a Zero

This is perhaps the best way I’ve ever heard anyone talk about the problems of being the new person on an existing team. Being a zero basically means “do no harm, make nothing worse”.

Mr Hadfield correctly states that everyone wants to be a plus-one. We want to do good, think we’re doing good and be seen to be doing good. At the start you’re eager, but that comes with impulsiveness which causes problems.

He talks about some times that in that eagerness, he ended up being a minus-one, someone who made things worse. That isn’t a good first impression on earth, let at alone on the ISS where you’re about to be stuck with those people for 3 months.

His philosophy is that aiming to be a plus-one will only turn you into a minus-one, so aim to be a zero and wait until you’re more certain before you start trying to add something.

Having seen people launch themselves into teams only to fail, this is one I entirely intend to live by.