Amazon Prime Video recently wrote about how changing away from managed services and writing a more integrated application saved them money. Despite being a few months old, this appeared to blow-up this week, and predictably has caused some cries of “SEE, SEE YOU SHOULD JUST RUN EVERYTHING YOURSELF”.
But to those of use who have been building on AWS (and other providers) for many years, it’s not a surprise, and we all have stories where we’ve done similar.
I say this as someone who is an annoying cheerleader for serverless compute and managed services, but despite that, I have home-rolled things, when it made sense.
How do you solve a problem
When you’re solving a problem, you look at what the managed services that you have available, considering factors like:
- Your teams experience with the service
- Limitations on the service, and what it was intended for, against what you’re doing
- What quotas may apply that you hit
- How the pricing model works
While pricing for managed-services is generally based on usage, sometimes specific costs will apply more to your workload, e.g. if you’re serving small images, you’ll be more impacted by the per-request cost than the bandwidth charges.
I would be surprised if an experienced architect hasn’t faced a situation where “Service X would be perfect for this, if only it didn’t have this restriction Y, or wasn’t priced for Z”.
My example
We’d built out a system that was performing 3 distinct processing steps on large files.
The system had built out incrementally, and we had the 3 steps on three different auto-scale groups, fed by queues.
While some of the requests could be processed from S3 as a stream, one task required downloading the file to a filesystem, and that download took time.
The users wanted to reduce the end-to-end processing time. Some of the tasks were predicated on passing prior steps, and so we didn’t want to make the steps parallel.
Attempt 1: “EFS looks good”
We used the ‘relatively’ new Elastic File System service from AWS… The first component downloaded the file, subsequent operations used this download.
This also had the advantage that the since the ‘smallest’ service was first, you paid for that download on the cheapest instance, and the more expensive instances didn’t have to download it.
We developed, deployed, and for the first morning it was really quick… until we discovered that we were using burst quota, and spent the afternoon rolling back.
Filesystem throughput was allocated based on the amount stored on the filesystem, but as this was a transient process, we didn’t replenish it quickly enough, and didn’t like the idea of just making large random files to earn it.
Now you can just pay for provisioned throughput, perhaps in a small part because of a conversation we had with the account managers.
Attempt 2: “Sod it, just combine them”
The processes varied in complexity, there was effectively a trivial, a medium, and a high complexity task… So the second solution we approached was combining all the tasks onto a single service… the computing power for the highest task would zoom through the other two tasks, and so we combined them into what I jokingly called “the microlith”.
We didn’t touch the other parts of the pipeline, or the database, they remained in other services, but combining the 3 steps worked.
What did we gain
The system was faster, and even more usefully to operators, more predictable.
Once processing had started you could tell, based on the file size, when the items would be ready…
Much like “lower p90 but higher maximum” feels better for user experience, this consistency was great.
What did we lose
Two of the three components had external dependancies, and this did mean this component was one of the less ‘safe’ to deploy, and while tests built up to defend against that… the impact of failed deploy was larger than you’d want.
In Conclusion
There are always times when breaking your patterns makes sense, the key is knowing what you’re both gaining and losing, and taking decisions for the right reasons at the right times.
Prime video refining an architecture to better meet scaling and cost models, making it less “Pure”, isn’t the gotcha against these services that some people would have you believe.
“Pure” design doesn’t win prizes.
Suitable design does.