NYC Tech Talk Series: How Google Backs Up the Internet

What people said

carapace · 2019-09 · Recommendation

FWIW... "Backups are a tax you pay for the luxury of restore" "How Google Backs Up the Internet" [VIDEO] Detailed notes: [LINK]

Read the original HN comment ↗

twotwotwo · 2021-06 · Reference

The "petabyte for a century" problem statement he mentions near the top is fun: can you preserve 1 PB with better-than-even odds of all the bits coming back right in a century? He wrote something about how he saw the durability situation in 2010: [LINK] . He seems to define the problem to allow for maintenance, e.g. check-and-repair operations are discussed. Broadening a little, I read it as "if you're trying to be more cautious than you usually want for commercial/academic online storage or even backup, what do you do? And would it work?" A lot (but not all) of the author's Queue piece talks about stats from online storage, which doesn't have some wins you can get if you're entirely about long-term durability. In online systems, heavy ECC seems out of favor compared to replication for performance reasons, but additional LDPC or RS at the app layer can absorb a substantial % of your volumes having problems, or a ton of random bad blocks. (In a more near-line context, Backblaze uses RS: [LINK] ) Same for tape -- offline LTO is slow and drives are costly, but the cost/TB and the rated lifespan seem like advantages over HDDs for this specific goal, at least in a narrow engineering sense. A pile of cryptographic hashes that fits on one device can help you check for and localize errors without sending everything over a network/doing the full ECC dance. If the hash being broken and data tampered with is in your threat model (hey, weird things can happen in a century), you can also

Read the original HN comment ↗

stegro32 · 2016-11 · Reference

I also remember that talk. I haven't verified by watching again, but if anyone's interested I suspect this is the one: [VIDEO]

Read the original HN comment ↗