Monitoring even a modestly-sized systems infrastructure quickly becomes untenable without automated alerting. For many metrics it is nontrivial to define ahead of time what constitutes “normal” versus “abnormal” values. This is especially true for metrics whose baseline value fluctuates over time. To make this problem more tractable, Datadog provides outlier detection functionality to automatically identify any host (or group of hosts) that is behaving abnormally compared to its peers.

In this talk I will discuss the algorithms we use for outlier detection, and show how easy they are to implement using Python. I will discuss the lessons we've learned from using outlier detection on our own systems, along with some real-life examples on how to avoid false positives and negatives.