HackerTube.dev

Dev Deletes Entire Production Database, Chaos Ensues

Kevin Fang · 3.6M views · 2023-04-26 · 10:20

Open on YouTube ↗

If playback is restricted in your region, try opening the original video. Use Mark watched to add it to your marked videos.

Overview

Kevin Fang comically retells GitLab's 2017 incident in which an engineer deleted the production database, based on the company's postmortems. It walks through the confusion and failed backups.

On Hacker News

Shared as a funny, accurate retelling of a classic outage, especially on a bad ops day.

Recommended by 1 HN people across 1 months · HN mentions 2023–2024

Topics

Cloud, DevOps & reliabilityDatabases & storageSatire & parody
Search by tags
gitlabpostmortemproduction databasedevopsincident

What people said

xyst · 2024-08 · Recommendation

Every time GL is mentioned, I am reminded of the 2017 incident where a person 'rm -rf' the primary pg data logs. The reenactment by Kevin Fang is always a good source of comedy on a bad day. [VIDEO] It’s a shame they are selling out, but I suppose it was bound to happen sooner or later. GH is vacuuming up all of the market share. I kind of liked the ability to self host my own GL. But for small/personal use cases. It was a bit overkill.

Read the original HN comment ↗
renegade-otter · 2024-04 · Joke / meme

If the primary fails, will the replica fail over? How do you know? Are the backups working? Who's checking? Can anyone delete a directory when tired at 6PM on a Friday? [VIDEO]

Read the original HN comment ↗
laserDinosaur · 2023-09 · Reference

"When a program has nothing surprising to say, it should say nothing." Which sounds good in theory, until you run into issues like this: [VIDEO] TLDW: Most of the Gitlabs outage where they deleted the primary prod and backup prod database could have been solved if they just waited for the command line op to complete - but because of a lack of feedback they thought it had frozen, leading to several other plans which made the situation worse.

Read the original HN comment ↗

Sources

Video by Kevin Fang, played through YouTube.

Data and selection notes

HN summaries and labels may contain AI errors. HN dates refer to discussions, not video publication.

HN Library: HackerTube export (2026-10-31) and open-index/hacker-news. Upstream summaries and classifications may contain AI errors.

Views, duration and playback status refresh separately from the published catalog. When an update fails, existing data remains visible.

Selection checks
  • SaaS & AI candidates: Topic keywords do not match
  • Editor-approved picks: Editorial review is required

Rule matches identify candidates. They do not guarantee video quality or financial results.

About the library and its sources