Cloud Native Development Patterns and Best Practices

Example – Data Lake consumer component

On AWS, the Kinesis Firehose service provides the essential components for processing a stream and storing the events in an S3 bucket, an Elasticsearch instance, or a Redshift instance. The following fragment from a serverless.yml file shows an example of connecting a Kinesis stream to an S3 bucket via Firehose. The entire solution can be configured declaratively as CloudFormation resources. In this example, a basic S3 bucket is defined to hold the data lake. Life cycle management and replication configurations are excluded for brevity. Next, a Firehose delivery stream is defined and connected to the Kinesis stream source and the S3 bucket destination. All of the necessary roles and permissions are also ...