O'Reilly logo

Learning Google BigQuery by Eric Brown, Thirukkumaran Haridass

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Importing message data into BigQuery

To import messages into BigQuery in batch mode, you can use the gcloud utility to log all the messages retrieved from the Cloud Pub/Sub to a file. Then you can upload the file to Google Cloud Storage and import data from the file to the BigQuery table. The following section provides a walkthrough of this scenario. This section uses the same Pub/Sub topic used in the previous section and uses the same Pub/Sub subscriber for pulling messages. Publish some sample messages to the Pub/Sub topic before running the commands given in this section.

Schedule a job to call the following command at specified intervals to retrieve the message from Cloud Pub/Sub, and append it to a local file. Replace the subscriber, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required