Databricks Job Failure Agent

A failed Databricks run tells you almost nothing on its own, and the question that actually decides your morning is never in the alert: has this job failed before. This agent pulls the run output, reads the last runs of the same job, and writes one Linear issue that says first failure, flaky, or broken since Tuesday, with the error text copied out exactly and the downstream tables you told it about. Whoever picks it up starts from evidence rather than from a red square on a dashboard.

What people use it for

• First failure or the usual - Run history is read first, so a job that has quietly failed every night since a schema change is labelled as such, and the one that just broke for the first time is not lost among them.
• The error, character for character - The exception class and message are copied out of the run output unedited, which is what makes an issue searchable later and what lets the next person match it against a vendor bug report.
• Downstream owners named upfront - Your runbook notes attach tables, dashboards and owners to each job, so the issue says what is now stale instead of leaving that to be discovered by whoever opens the dashboard at nine.
• One issue per broken job - Linear is searched for the job and the exception before filing, so an hourly job failing all afternoon leaves one issue and a pattern, not fourteen tickets somebody has to close by hand.

Before you fork

Does it need permission to run or restart jobs in Databricks?

No. It reads runs, run output and job definitions, and that is the whole of its Databricks access, so a token scoped to read is enough. The writing all happens in Linear and Slack. If you later want automatic retries, that is a separate workflow and a deliberate decision rather than a side effect of triage.

We run hundreds of jobs a night. How often does this actually fire?

Once per failed run, not per run, so cost tracks your failure rate and not your job count. A platform with a few failures a night is a few short runs a night. Jobs you have already accepted as flaky can be marked low stakes in the runbook notes, so they still file an issue without pinging anyone.

What stops it filing the same issue every hour?

Every run begins with a Linear search on the job name and the exception class, and an open match stops it filing. The history check also gives the failure a name, so a persistent break reads as one problem with a date on it rather than as a fresh incident each hour. You can rehearse it against the staged failures before it writes to your real Linear team.