Chapter 7. Enterprise Power Query Best Practices
This chapter focuses on enterprise-scale best practice design. We will work through a hands-on exercise, importing a substantial volume of data from a set of text files. The solution will transition to a SQL Server connector and a Fabric lakehouse to compare performance and scalability options.
Power Query and Dataflows
Power Query is a data transformation tool and query engine that is incorporated into multiple Microsoft products and services due to its flexibility and ease of use. These include:
-
Excel
-
Azure Data Factory
-
Azure Synapse Analytics
-
Dataverse
-
Power Apps
-
Dynamics 365 Insights and Finance and Operations
-
Power BI Dataflow Gen1
-
Fabric Dataflow Gen2
-
Fabric Data Warehouse visual query
-
Power BI Desktop
The specific features and capabilities of these implementations vary a little based on the version and capabilities of the product, but the core Power Query functionality is the same.
The Importance of Query Folding
When working with enterprise data, query folding is one of the most important features to understand and utilize, particularly with large quantities of data. Query folding refers to the process of translating Power Query transformations into a native query that the data source can execute. This allows Power Query to push as much of the data processing workload as possible to the data source, rather than performing all transformations on the local machine (or wherever Power Query is running). ...
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access