July 2018
Intermediate to advanced
506 pages
16h 2m
English
Content preview from Google Cloud Platform for Developers
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
Start your free trial
Sources and sinks
In almost every case, Cloud Dataflow pipelines will read data from one or more external sources, perform processing tasks on that data, and write the results to an external target, or sink. In the Dataflow/Beam model, such operations are treated as any other data processing operation, and hence are done through the use of transforms. For many common read/write operations, developers simply leverage one of the built-in Read and Write I/O Transforms included in the Cloud Dataflow and Apache Beam SDKs.
These transforms provide a clear interface for interacting with external sources and sinks, with support for many Google Cloud services, including the following:
- BigQuery
- Cloud Storage
- Cloud Bigtable
- Cloud Pub/Sub
- Cloud Datastore ...