Cloud Storage Object Watch

Data deliveries fail quietly. The partner's job exports zero rows, or lands at 05:10 instead of 04:00, or begins writing to a path with the date in a different order, and you find out at ten in the morning when a dashboard is wrong and nobody can say for how long. This agent checks each object as it arrives against the prefixes, sizes and timings you describe, compares it with the same prefix on the days before, and puts one line in the data channel when something is off.

What people use it for

• The overnight drop, checked overnight - A partner file due at four in the morning is read at four in the morning. A missing or half sized delivery reaches the channel before the first ingest job has run on top of it.
• Zero bytes is not a delivery - An empty file uploads successfully, satisfies every existence check ever written, and breaks whatever reads it next. The size goes into the note beside what that same path weighed on the days around it.
• A prefix nobody agreed to - When an export starts writing one directory across, or with the date arranged differently, downstream jobs read an empty prefix and report no problem at all. Every object gets its path compared with what the rules expect.
• It only ever reads - No object is moved, rewritten or deleted, no lifecycle rule or bucket setting is changed, and nothing upstream is rerun. Reprocessing remains a decision for the people who own the pipeline.

Before you fork

What do I connect, and what permission does it need?

A Google Cloud service account that can list objects and read object metadata on the bucket, the object notification wired into this workflow, plus a Slack channel the data team reads. List and read is the whole requirement, since it never writes, and granting nothing beyond that is the cleanest way to keep it harmless. What a good delivery looks like goes in as a variable, in your own words.

Our loads write hundreds of part files a night, is that hundreds of runs?

Yes, one run per object, which is why most teams scope the trigger instead of watching a whole bucket. Point it at the prefixes where a late or missing delivery actually costs you something, or at the manifest object that completes a drop, and let the parts land unwatched. A bucket taking twelve deliveries a day costs about twelve short runs.

Can it tell me whether the data inside is right?

No, and it states which checks it made rather than implying more. It sees path, name, size, content type and write time, plus every neighbouring object, so it catches late, missing, empty, oddly sized, badly named and misfiled deliveries. Row counts, schema drift and duplicated batches live inside the file, which puts them in your pipeline's own validation step.