AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338)

What people said

talawahtech · 2020-11 · Recommendation

Multi-AZ doesn't protect against a software/OS issue like this, Multi-AZ would be relevant if it was an infrastructure failure (e.g. underlying EC2 instances or networking). The relevant resiliency pattern in this case would be what they refer to as cell-based architecture, where within an AZ services are broken down into smaller independent cells to minimize the blast radius. They specifically mention in the write-up that this was a gap they plan to address, the "backend" portion of Kinesis was already cellularized but that step had not yet been completed on the "frontend". Celluarization in combination with workload partitioning would have helped, e.g. don't run Cloudwatch, Cognito and Customer workloads on the same set of cells. It is also important to note that celluarization only helps in this case if they limit code deployment to a limited number of cells at a time. This YouTube video[1] of a re:invent presentation does a great job of explaining it. The cell-based stuff, starts around minute 20. 1. [VIDEO]

Read the original HN comment ↗

ignoramous · 2019-03 · Recommendation

This is a great outline. Martin Kleppmann's Designing Data Intensive Applications [0] is very much in the same domain as what's in these notes. Speaking about scale and distributed systems: AWS specific resources I find interesting from their EdgeEngineering/NetEng teams (I haven't seen many other service teams in AWS openly share as much about design as them): - [LINK] series of articles on Route53's 100% data-plane availability architecture [1]. - [LINK] colmmacc [2] (seems to be the eng behind AWS HyperPlane [3][4]?) on 10 design patterns for building resilient systems. - [VIDEO] Peter Vosshall (co-creator of Dynamo [5]) presenting "cell-based" design in-use at AWS. ---- [0] [LINK] [1] [LINK] [2] [LINK] [3] [LINK] [4] [LINK] [5] Decandia, G.; Hastorun, D.; Jampani, M.; Kakulapati, G.; Lakshman, A.; Pilchin, A.; Sivasubramanian, S.; Vosshall, P.; Vogels, W. (2007). "Dynamo: Amazon's Highly Available Key-value Store". [6] Bonus: The SRE Book [LINK]

Read the original HN comment ↗

ignoramous · 2019-03 · Recommendation

From what you describe the 'AWS cells' are very different than Borg's. My view is that at a high level aws cells are isolated, fuzzy-partitioned, migratable, heterogeneous instances of a complete service stack that are managed such that they always fail independently of each other; contact points among these cell instances, if required, are established via highly available intermediate services (like ReplicatedStateMachinesOrStorage); the clients are routed to assigned cells by a highly available routing layer (like NetworkTrafficShapers/DNS), which can also help facilitate migration of clients from one/more cell to another. I encourage you to view the re:invent presentation on it [0] if resiliency in distributed systems is something that interests you. [0] [VIDEO]

Read the original HN comment ↗