Snowflake: The Definitive Guide

Chapter 6. Data Loading and Unloading

Data engineers are responsible for managing the ingestion and transformation of raw data sets from various disparate sources to deliver the data in a state that end users need in order to gain actionable insights. This chapter provides the foundational knowledge anyone can use to learn more about how to achieve the best data engineering results on the Snowflake cloud data platform.

The chapter begins with a summary of basic concepts of Snowflake data loading and unloading, including data types, compression methods, file format types, and Snowflake stages. In the data loading tools section, we’ll take a deep dive into learning how to use SQL insert statements in the Snowflake worksheet to load both structured and semi-structured data. We’ll also learn how to load data using the web UI Load Data wizard and the SnowSQL command-line interface (CLI). For a more automated approach to data loading, we can use data pipelines, connectors, and third-party tools. We’ll explore some of these automated data loading options later in the chapter. In addition, we’ll look at some alternatives to data loading, such as creating materialized views on external stages and accessing shared data. Finally, we’ll discuss unloading data files from Snowflake.

For our hands-on examples, we’ll be completing most of the work in Snowsight, the default web UI. However, we’ll need to switch to the Snowflake Classic Console briefly to use the Load Data wizard and when we need ...