Honeycomb Alert Scribe

A fired trigger tells you an aggregate crossed a line, which is the least useful shape that fact ever takes: nobody wants to know that p99 is high, they want to know whose p99 it is. This agent runs the trigger's own query again around the alert window, runs it once more grouped by the field you care about, and posts one Slack note with the threshold, the real value and the two or three groups holding it. It never says why, because the query cannot, and it leaves every trigger exactly as you wrote it.

What people use it for

• The group by the alert skipped - A trigger fires on one number for an entire dataset. The note arrives with the breakdown already underneath it, so the first thing anybody reads says which route, service or tenant is carrying the change.
• When a trigger goes quiet - If the query returns no rows, the note says so and shows what the same window gave before. A service that stopped emitting looks identical to a service with nothing to do, and only one of those is acceptable.
• Numbers a thread can use - Threshold against measured value, plus the leading groups and their share, taken from a result that finished computing rather than from the figure in the notification, which is one evaluation old by the time a human reads it.
• Your triggers come back untouched - It creates a scratch query to answer the alert, which is what a person does in the interface anyway, and edits nothing. No trigger is disabled, and no saved query, board or SLO is altered.

Before you fork

What does it need access to?

A Honeycomb key that can read triggers and run queries in the environment you point it at, the trigger notifications routed into this workflow, and a Slack channel. Queries go through the same API your team already uses, two per alert, one for the window and one for the breakdown. Your triage notes go in as a variable.

Does creating queries clutter our environment?

The queries it creates are the same throwaway objects the interface creates every time somebody presses run, and they are never saved into a board or a collection. Nothing existing is modified, no trigger, saved query, SLO or derived column, which also means the worst mistake available to it is a note people scroll past.

How is this better than the notification Honeycomb already sends?

The notification carries the number that crossed and a link into the interface. This carries the number as it stands when the note is written, the group holding it, and the first check your own notes prescribe, delivered where the team already is. It genuinely cannot tell you a cause, so treat it as the first two minutes of an investigation done for you rather than as an answer.