GitLab Reliability: Lessons From the 2017 Database Incident

The incident that everyone learned from

On January 31, 2017, a GitLab engineer accidentally deleted a production database directory during an attempt to deal with a spam-driven load spike. The team then discovered that five different backup and replication mechanisms were either not working or not configured as believed, and recovery ultimately relied on a lucky, out-of-date snapshot — with some production data permanently lost.

What made the incident legendary was not the mistake but the response: GitLab live-streamed the recovery, posted running updates, and published an unusually candid postmortem. It turned an embarrassing failure into one of the most valuable public lessons in operations — coverage in TechTarget and the trade press cemented it as a teaching case.

The reliability lessons that stuck

The core lesson is blunt: an untested backup is a hypothesis, not a safety net. GitLab had multiple backup strategies on paper and none that worked when needed. Regularly exercising restores — actually recovering from your backups into a real environment — is the only way to know they work.

Secondary lessons include guarding destructive operations against tired engineers working an incident at 11pm, reducing the number of places a single fat-finger can cause irreversible harm, and separating the person mitigating from the systems that can delete production state.

Where GitLab reliability stands today

GitLab today operates GitLab.com at scale and publishes a public status page, availability reporting, and a strong culture of public postmortems — a direct legacy of 2017. For teams choosing between GitLab and alternatives, that transparency is a genuine reliability asset: it lets you reason about how the provider fails and recovers, rather than trusting a marketing uptime number.

The practical takeaway for any team: adopt GitLab’s post-2017 discipline. Test restores on a schedule, run blameless postmortems, and treat transparency as part of reliability, not a reputational risk.

Frequently asked questions

What happened in the 2017 GitLab database incident?

An engineer accidentally deleted a production database directory during a load spike, and five backup/replication methods were found not to work. Recovery relied on an out-of-date snapshot, and some data was permanently lost.

Why is the GitLab incident so widely cited?

Because of the response: GitLab live-streamed the recovery and published a candid, detailed postmortem, turning a serious failure into a widely-referenced lesson about tested backups and blameless incident culture.

What is the main takeaway from the GitLab incident?

Untested backups are not backups. Exercise your restores regularly, guard destructive operations, and treat transparent postmortems as part of reliability.