Datadog integration

Datadog Alert Triage Agent

+1

Turns a Datadog page into one Slack note that says whether it is one host or the whole fleet, carries the metric numbers around the alert window, and names the first place to look. A Datadog page tells you a threshold moved and almost nothing else, so the first four minutes of every alert go on the same question: is this one sick host or is it everything. This agent answers that before a person opens a dashboard, reading the event and the monitor, counting what is actually alerting, pulling the metric around the window and any open incident on the service, then posting one short note to Slack. It suggests where to look next and touches none of your monitors, so the judgement stays with whoever is on call. ## What people use it for - **One host or the fleet** - The scope line comes first, with the host or tag count behind it. A single bad node reads very differently from thirty four pods, and that difference is what decides whether anyone gets out of bed. - **Numbers, not adjectives** - The metric in the monitor's own query is pulled across the alert window and the hour before it, so the note carries real values and the times they were taken instead of the word elevated. - **Know if somebody already has it** - Open incidents on the same service are checked before the message goes out, so the third alert of an ongoing outage arrives labelled as such rather than starting a parallel investigation in another channel. - **Quiet monitors stay quiet** - Your severity rules decide what earns a channel post, so warning level noise and the monitors you have made peace with never reach Slack, and the ones that do are worth reading. ## Before you fork **Which Datadog access does the agent need?** An API key and an application key that can read events, monitors, metrics, logs and incidents, plus the Slack channel you want the notes in. Nothing it does needs write scope, so a read only application key works in full. Your severity rules go in as a variable at setup. **Will it post on every warning we have configured?** Only on the monitors you route into the trigger, and then only when your severity rules say that level earns a channel post. Everything else is read, judged and dropped in silence. Start by pointing one noisy service at it for a day and read what it would have said. **Can it mute a monitor that is flapping at 3am?** No, and that is deliberate. It cannot mute, resolve or edit a monitor, or open or close an incident, so the most it can get wrong is a note you disagree with. Muting is a decision about what you are willing to miss, and it belongs to the person on call rather than to an agent reading a single event.