Datadog Reliability: Why Your Monitoring Must Not Share Fate With Your App

When the thing that watches everything goes dark

Observability platforms like Datadog ingest metrics, traces, and logs from across your stack and turn them into dashboards and alerts. That makes them uniquely load-bearing during an incident: they are the tool you reach for first when something breaks. If the observability layer is also down, you are debugging blind at the worst possible moment.

This is a specific instance of a general reliability principle: a monitoring system that can fail together with the thing it monitors provides false comfort. The value of monitoring is highest precisely when your primary systems are unhealthy, so the monitoring must be at least as independent and resilient as what it watches.

The March 2023 outage and what it revealed

On March 8, 2023, Datadog experienced a major outage that affected multiple regions at once and, for some services, stretched beyond 24 hours. Datadog published a detailed postmortem attributing the trigger to a routine update to the underlying operating system on a large number of hosts that caused a systemd change to unexpectedly sever network connectivity, cascading across services and regions. It was widely covered in the trade press as one of the more significant SaaS incidents of that year.

Two things stand out. First, the trigger was again a change, not a hardware failure, consistent with the pattern that configuration and change error dominates major incidents. Second, the multi-region scope undercut the assumption that regional separation alone guarantees isolation: a globally rolled-out change can defeat regional redundancy if the change itself is the fault.

Datadog restored service, published a candid writeup, and issued service credits to affected customers. The transparency is a positive reliability signal, but the operational lesson for customers is independent of any one vendor.

Building monitoring that does not share fate

Treat your observability provider as a dependency that can itself fail. Keep a lightweight, independent health check and alerting path (for example a simple external uptime monitor or a second, separate alerting channel) that does not rely on the same platform, so a provider outage cannot silence every alarm at once. Make sure your on-call escalation can still page a human if the primary tool is down.

Retain enough local or independent signal to triage without the full platform: basic synthetic checks from outside your infrastructure, and critical alerts delivered through a channel that is not co-located with the monitored system. Subscribe to the Datadog status page and route it into a separate channel, and rehearse the scenario where your monitoring is unavailable so the team is not improvising during a real event.

Frequently asked questions

What happened in the March 2023 Datadog outage?

On March 8, 2023, Datadog suffered a multi-region outage that lasted more than a day for some services. Its postmortem traced the trigger to a routine operating-system update that caused a systemd change to unexpectedly cut network connectivity, which then cascaded across services and regions.

Why is observability-platform reliability so important?

Because monitoring is most valuable exactly when your own systems are failing. If the observability platform goes down at the same time as an incident, you lose the visibility you need to diagnose and respond, so the monitoring must not share fate with what it monitors.

How do I keep monitoring resilient to a provider outage?

Keep an independent, lightweight health-check and alerting path that does not depend on the same platform, ensure on-call can still be paged if the primary tool is down, run external synthetic checks, and subscribe to the provider status page in a separate channel.

Does regional redundancy protect against this kind of outage?

Not on its own. The March 2023 incident spanned multiple regions because a globally rolled-out change was the fault. Regional isolation helps against localized failures but cannot stop a defective change that ships everywhere at once.