O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

SQL Server 2017 Integration Services Cookbook

Book Description

Harness the power of SQL Server 2017 Integration Services to build your data integration solutions with ease

About This Book

  • Acquaint yourself with all the newly introduced features in SQL Server 2017 Integration Services
  • Program and extend your packages to enhance their functionality
  • This detailed, step-by-step guide covers everything you need to develop efficient data integration and data transformation solutions for your organization

Who This Book Is For

This book is ideal for software engineers, DW/ETL architects, and ETL developers who need to create a new, or enhance an existing, ETL implementation with SQL Server 2017 Integration Services. This book would also be good for individuals who develop ETL solutions that use SSIS and are keen to learn the new features and capabilities in SSIS 2017.

What You Will Learn

  • Understand the key components of an ETL solution using SQL Server 2016-2017 Integration Services
  • Design the architecture of a modern ETL solution
  • Have a good knowledge of the new capabilities and features added to Integration Services
  • Implement ETL solutions using Integration Services for both on-premises and Azure data
  • Improve the performance and scalability of an ETL solution
  • Enhance the ETL solution using a custom framework
  • Be able to work on the ETL solution with many other developers and have common design paradigms or techniques
  • Effectively use scripting to solve complex data issues

In Detail

SQL Server Integration Services is a tool that facilitates data extraction, consolidation, and loading options (ETL), SQL Server coding enhancements, data warehousing, and customizations. With the help of the recipes in this book, you’ll gain complete hands-on experience of SSIS 2017 as well as the 2016 new features, design and development improvements including SCD, Tuning, and Customizations.

At the start, you’ll learn to install and set up SSIS as well other SQL Server resources to make optimal use of this Business Intelligence tools. We’ll begin by taking you through the new features in SSIS 2016/2017 and implementing the necessary features to get a modern scalable ETL solution that fits the modern data warehouse.

Through the course of chapters, you will learn how to design and build SSIS data warehouses packages using SQL Server Data Tools. Additionally, you’ll learn to develop SSIS packages designed to maintain a data warehouse using the Data Flow and other control flow tasks. You’ll also be demonstrated many recipes on cleansing data and how to get the end result after applying different transformations. Some real-world scenarios that you might face are also covered and how to handle various issues that you might face when designing your packages.

At the end of this book, you’ll get to know all the key concepts to perform data integration and transformation. You’ll have explored on-premises Big Data integration processes to create a classic data warehouse, and will know how to extend the toolbox with custom tasks and transforms.

Style and approach

This cookbook follows a problem-solution approach and tackles all kinds of data integration scenarios by using the capabilities of SQL Server 2016 Integration Services. This book is well supplemented with screenshots, tips, and tricks. Each recipe focuses on a particular task and is written in a very easy-to-follow manner.

Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

Table of Contents

  1. Preface
    1. What this book covers
    2. What you need for this book
    3. Who this book is for
    4. Sections
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
      5. See also
    5. Conventions
    6. Reader feedback
    7. Customer support
      1. Downloading the example code
      2. Downloading the color images of this book
      3. Errata
      4. Piracy
      5. Questions
  2. SSIS Setup
    1. Introduction
    2. SQL Server 2016 download
      1. Getting ready
      2. How to do it...
    3. Installing JRE for PolyBase
      1. Getting ready
      2. How to do it...
      3. How it works...
    4. Installing SQL Server 2016
      1. Getting ready
      2. How to do it...
    5. SQL Server Management Studio installation
      1. Getting ready
      2. How to do it...
    6. SQL Server Data Tools installation
      1. Getting ready
      2. How to do it...
    7. Testing SQL Server connectivity
      1. Getting ready
      2. How to do it...
  3. What Is New in SSIS 2016
    1. Introduction
    2. Creating SSIS Catalog
      1. Getting ready
      2. How to do it...
    3. Custom logging
      1. Getting ready
      2. How to do it...
      3. How it works...
      4. There's more...
        1. Create a database
        2. Create a simple project
        3. Testing the custom logging level
      5. See also
    4. Azure tasks and transforms
      1. Getting ready
      2. How to do it...
      3. See also
    5. Incremental package deployment
      1. Getting ready
      2. How to do it...
      3. There's more...
    6. Multiple version support
      1. Getting ready
      2. How to do it...
      3. There's more...
    7. Error column name
      1. Getting ready
      2. How to do it...
    8. Control Flow templates
      1. Getting ready
      2. How to do it...
  4. Key Components of a Modern ETL Solution
    1. Introduction
    2. Installing the sample solution
      1. Getting ready
      2. How to do it...
      3. There's more...
    3. Deploying the source database with its data
      1. Getting ready
      2. How to do it...
      3. There's more...
    4. Deploying the target database
      1. Getting ready
      2. How to do it...
    5. SSIS projects
      1. Getting ready
      2. How to do it...
    6. Framework calls in EP_Staging.dtsx
      1. Getting ready
      2. How to do it...
      3. There's more...
  5. Data Warehouse Loading Techniques
    1. Introduction
    2. Designing patterns to load dimensions of a data warehouse
      1. Getting ready
      2. How to do it...
      3. There's more...
    3. Loading the data warehouse using the framework
      1. Getting ready
      2. How to do it...
    4. Near real-time and on-demand loads
      1. Getting ready
      2. How to do it...
      3. There's more...
    5. Using parallelism
      1. Getting ready
      2. How to do it...
      3. There's more...
  6. Dealing with Data Quality
    1. Introduction
    2. Profiling data with SSIS
      1. Getting ready
      2. How to do it...
    3. Creating a DQS knowledge base
      1. Getting ready
      2. How to do it...
    4. Data cleansing with DQS
      1. Getting ready
      2. How to do it...
    5. Creating a MDS model
      1. Getting ready
      2. How to do it...
    6. Matching with DQS
      1. Getting ready
      2. How to do it...
    7. Using SSIS fuzzy components
      1. Getting ready
      2. How to do it...
  7. SSIS Performance and Scalability
    1. Introduction
    2. Using SQL Server Management Studio to execute an SSIS package
      1. Getting ready
      2. How to do it...
      3. How it works...
    3. Using T-SQL to execute an SSIS package
      1. How to do it...
      2. How it works...
    4. Using the DTExec command-line utility to execute an SSIS package
      1. How to do it...
      2. How it works...
      3. There's more...
    5. Scheduling an SSIS package execution
      1. Getting ready
      2. How to do it...
      3. How it works...
    6. Using the cascading lookup pattern
      1. How to do it...
      2. How it works...
    7. Using the lookup cache
      1. How to do it...
      2. How it works...
    8. Using lookup expressions
      1. How to do it...
      2. How it works...
    9. Determining the maximum number of worker threads in a data flow
      1. How to do it...
      2. How it works...
    10. Using the master package concept
      1. How to do it...
      2. How it works...
    11. Requesting an execution tree in SSDT
      1. How to do it...
      2. How it works...
    12. Monitoring SSIS performance
    13. Establishing a performance monitor session
      1. How to do it...
      2. How it works...
    14. Configuring a performance monitor data collector set
      1. How to do it...
      2. How it works....
  8. Unleash the Power of SSIS Script Task and Component
    1. Introduction
    2. Using variables in SSIS Script task
      1. Getting ready
      2. How to do it...
    3. Execute complex filesystem operations with the Script task
      1. Getting ready
      2. How to do it...
    4. Reading data profiling XML results with the Script task
      1. Getting ready
      2. How to do it...
    5. Correcting data with the Script component
      1. Getting ready
      2. How to do it...
    6. Validating data using regular expressions in a Script component
      1. Getting ready
      2. How to do it...
    7. Using the Script component as a source
      1. How to do it...
      2. How it works...
    8. Using the Script component as a destination
      1. Getting ready
      2. How to do it...
      3. How it works...
  9. SSIS and Advanced Analytics
    1. Introduction
    2. Splitting a dataset into a training and test set
      1. Getting ready
      2. How to do it...
    3. Testing the randomness of the split with a SSAS decision trees model
      1. Getting ready
      2. How to do it...
    4. Preparing a Naive Bayes SSAS data mining model
      1. Getting ready
      2. How to do it...
    5. Querying the SSAS data mining model with the data mining query transformation
      1. Getting ready
      2. How to do it...
    6. Creating an R data mining model
      1. Getting ready
      2. How to do it...
    7. Using the R data mining model in SSIS
      1. Getting ready
      2. How to do it...
    8. Text mining with term extraction and term lookup transformations
      1. Getting ready
      2. How to do it...
  10. On-Premises and Azure Big Data Integration
    1. Introduction
    2. Azure Blob storage data management
      1. Getting ready
      2. How to do it...
    3. Installing a Hortonworks cluster
      1. Getting ready
      2. How to do it...
    4. Copying data to an on-premises cluster
      1. Getting ready
      2. How to do it...
    5. Using Hive – creating a database
      1. Getting ready
      2. How to do it...
      3. There's more...
    6. Transforming the data with Hive
      1. Getting ready
      2. How to do it...
      3. There's more...
    7. Transferring data between Hadoop and Azure
      1. Getting ready
      2. How to do it...
    8. Leveraging a HDInsight big data cluster
      1. Getting ready
      2. How to do it...
      3. There's more...
    9. Managing data with Pig Latin
      1. Getting ready
      2. How to do it...
      3. There's more...
    10. Importing Azure Blob storage data
      1. Getting ready
      2. How to do it...
      3. There's more...
        1. Azure Data Factory and SSIS 
  11. Extending SSIS Custom Tasks and Transformations
    1. Introduction
    2. Designing a custom task
      1. Getting ready
      2. How to do it...
      3. How it works...
    3. Designing a custom transformation
      1. How to do it...
      2. How it works...
    4. Managing custom component versions
      1. Getting ready
      2. How to do it...
      3. How it works...
  12. Scale Out with SSIS 2017
    1. Introduction
    2. SQL Server 2017 download and setup
      1. Getting ready
      2. How to do it...
      3. There's more...
    3. SQL Server client tools setup
      1. Getting ready
      2. How to do it...
    4. Configuring SSIS for scale out executions
      1. Getting ready
      2. How to do it...
      3. There's more...
    5. Executing a package using scale out functionality
      1. Getting ready
      2. How to do it...