book

Analytics Engineering with SQL and dbt

by Rui Pedro Machado, Helder Russa

December 2023

Intermediate to advanced

324 pages

8h 23m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Why We Wrote This BookWho This Book Is ForHow This Book Is OrganizedConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Databases and Their Impact on Analytics EngineeringCloud Computing and Its Impact on Analytics EngineeringThe Data Analytics LifecycleThe New Role of Analytics EngineerResponsibilities of an Analytics EngineerEnabling Analytics in a Data MeshData Productsdbt as a Data Mesh EnablerThe Heart of Analytics EngineeringThe Legacy ProcessesUsing SQL and Stored Procedures for ETL/ELTUsing ETL ToolsThe dbt RevolutionSummary
A Brief on Data ModelingThe Conceptual Phase of ModelingThe Logical Phase of ModelingThe Physical Phase of ModelingThe Data Normalization ProcessDimensional Data ModelingModeling with the Star SchemaModeling with the Snowflake SchemaModeling with Data VaultMonolith Data ModelingBuilding Modular Data ModelsEnabling Modular Data Models with dbtTesting Your Data ModelsGenerating Data DocumentationDebugging and Optimizing Data ModelsMedallion Architecture PatternSummary
The Resiliency of SQLDatabase FundamentalsTypes of DatabasesDatabase Management System“Speaking” with a DatabaseCreating and Managing Your Data Structures with DDLManipulating Data with DMLInserting Data with INSERTSelecting Data with SELECTUpdating Data with UPDATEDeleting Data with DELETEStoring Queries as ViewsCommon Table ExpressionsWindow FunctionsSQL for Distributed Data ProcessingData Manipulation with DuckDBData Manipulation with PolarsData Manipulation with FugueSQLBonus: Training Machine Learning Models with SQLSummary
dbt Design Philosophydbt Data Flowdbt CloudSetting Up dbt Cloud with BigQuery and GitHubUsing the dbt Cloud UIUsing the dbt Cloud IDEStructure of a dbt ProjectJaffle Shop DatabaseYAML FilesModelsSourcesTestsAnalysesSeedsDocumentationdbt Commands and Selection SyntaxJobs and DeploymentSummary
Model MaterializationsTables, Views, and Ephemeral ModelsIncremental ModelsMaterialized ViewsSnapshotsDynamic SQL with JinjaUsing SQL Macrosdbt PackagesInstalling PackagesExploring the dbt_utils PackageUsing Packages Inside Macros and Modelsdbt Semantic LayerSummary
Problem Definition: An Omnichannel Analytics CaseOperational Data ModelingConceptual ModelLogical ModelPhysical ModelHigh-Level Data ArchitectureAnalytical Data ModelingIdentify the Business ProcessesIdentify Facts and Dimensions in the Dimensional Data ModelIdentify the Attributes for DimensionsDefine the Granularity for Business FactsCreating Our Data Warehouse with dbtTests, Documentation, and Deployment with dbtData Analytics with SQLConclusion

Content preview from Analytics Engineering with SQL and dbt

Chapter 1. Analytics Engineering

The historical development of analytics includes significant milestones and technologies that have shaped the field into what it is today. It began with the advent of data warehousing in the 1980s, which created the foundational framework for organizing and analyzing business data. Bill Inmon, a computer scientist who continued to publish throughout the 1980s and 1990s, is widely regarded as providing the first solid theoretical foundation for data warehousing.

A subsequent wave of development occurred when Ralph Kimball, another leading contributor to data warehousing and business intelligence (BI), published his influential work, The Data Warehouse Toolkit, in 1996. Kimball’s work laid the foundation for dimensional modeling, marking another crucial milestone in the evolution of analytics. Together, the contributions of Inmon and Kimball, spanning the late 20th century, played pivotal roles in shaping the landscape of data warehousing and analytics.

In the early 2000s, the emergence of tech giants like Google and Amazon created the need for more advanced solutions for processing massive amounts of data, leading to the release of the Google File System and Apache Hadoop. This marked the era of Big Data Engineering, in which professionals used the Hadoop framework to process large amounts of data.

The rise of public cloud providers like Amazon Web Services (AWS) revolutionized the way software and data applications were developed and deployed. ...