Mastering Hadoop 3

Amazon Data Pipeline

AWS Data Pipeline is a web service and can be used to automate the movement and transformation of data from source to destination. The Data Pipeline can be used to define workflows that indicate task flow from source to destination. The intermediate tasks are dependent on completion of its previous task. It allows us to configure and manage this workflow as a pipeline. AWS Data Pipeline manages and handles the details of scheduling and ensures that the data dependencies are met for the applications to focus on processing the data.

Data Pipeline provides an implementation of a task runner called AWS Data Pipeline Task Runner. AWS Data Pipeline has some of the default task runners for performing common tasks, such as executing ...