Google Cloud streaming analytics

When to use it

This is the common Google Cloud pipeline for clickstream, IoT or application events that should land in a warehouse within seconds. Producers publish to a Pub/Sub topic, which buffers bursts and decouples them from processing. A Dataflow streaming job, written with Apache Beam, reads the subscription, validates and enriches each event, windows and aggregates where needed, and writes to partitioned BigQuery tables with the Storage Write API. Records that fail validation go to Cloud Storage as dead letters, along with a raw archive you can replay later. Looker reads BigQuery for dashboards.

Dataflow handles autoscaling and exactly-once processing for you, but a streaming job runs and bills around the clock. If you only need raw events in BigQuery without transformation, a BigQuery subscription on Pub/Sub skips Dataflow entirely. Alert on subscription backlog and pipeline lag in Cloud Monitoring so you notice when the pipeline falls behind producers.