BigQuery Metrics Digest

Every company with a warehouse has a dashboard that was supposed to answer this, and nobody opens it. Hand this agent the five or six SELECTs you actually care about and at 08:00 it runs exactly those, drops the values into Slack with the day on day move, and names any query that broke with the error BigQuery gave it. Because a scan is billed by the byte it runs each query once and never retries, so a bad query costs you one failed run instead of a surprise at the end of the month.

What people use it for

• One scan a day - Each query runs once every morning, which makes the BigQuery side of this a number you can work out before you start. No retries, no exploratory queries widening a date range behind your back.
• Catch a pipeline that never landed - When last night's partition is missing the metric comes back as zero and the digest prints zero next to yesterday's real number. That contrast is usually the fastest warning you get that an upstream job died.
• Stop being the human query endpoint - Analysts get asked for the same six numbers every morning by the same four people. Once those numbers arrive on their own the DMs stop, and the analyst gets their mornings back.
• Dull enough to trust - Same queries, same order, same names, every day, and a query that failed is stated rather than hidden behind a blank. A number nobody has to double check is the only kind worth posting.

Before you fork

What access does it need?

A Google Cloud credential with BigQuery Data Viewer on the datasets your queries read and Job User on the project that will run them, plus one Slack channel. Then you paste in your query list with the name each metric should carry, and describe the digest. If you want a hard ceiling, point it at a project with a custom query quota set.

What will this actually cost?

Thirty odd agent runs a month, one each morning, and on the BigQuery side whatever your queries scan. Six well partitioned queries against summary tables is pennies, and the same six against a year of an unpartitioned events table is not, exactly as it would be if a person ran them. A query that fails to compile is not billed, and since the agent never retries, a broken query costs one attempt rather than a loop.

What happens when one of my queries breaks?

The digest still goes out. The broken query gets its own line under QUERIES THAT DID NOT RUN with the BigQuery error copied word for word, including the location it points at, and the owner you named gets tagged. It will not attempt a fix, guess at a replacement table, or quietly leave the metric out, and since it holds only a query tool and a table listing tool, it cannot write anything back to correct itself either.