BigQuery for Data Warehousing: Managed Data Analysis in the Google Cloud

© Mark Mucchetti 2020

M. MucchettiBigQuery for Data Warehousinghttps://doi.org/10.1007/978-1-4842-6186-6_7

7. Dataflow

Mark Mucchetti¹ 

(1)

Santa Monica, CA, USA

 

Once you have your data warehouse built, its schemas defined, and all of your external and internal data migrated into BigQuery, it’s time to start thinking about your data pipeline architecture and how you can enable your organization to accept stream or batch processing into the warehouse.

Your organization may already have a well-established system of relational or NoSQL databases collecting and distributing data to end users. The trick is to get that data into BigQuery as efficiently as possible so you can begin analysis. If your organization has or requires big data pipelines, at some ...