Hands-On AWS CDK

Chapter 4. Integrating CDK Constructs

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 4th chapter of the final book. Please note that the GitHub repo will be made active later on.

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

In this chapter you will integrate multiple CDK constructs to create a dynamic greeting by uploading data to your S3 bucket. This will be the final modification to your “Hello, World” CDK application. You will chain together an S3 bucket, an event notification, and a Lambda function to return a dynamic greeting. We will later modify this event-driven architecture to accept utility data and calculate utility usage patterns.

An S3 event notification is a way to trigger an AWS Lambda function when an object is uploaded to an S3 bucket. This is a common pattern for building event-driven architectures.

You can think of an S3 event notication as a universal remote control with buttons that trigger actions in different applications. When you press a button, a signal goes to the remote control hub. The hub then sends commands to take appropriate actions in the linked apps and services for that button.

An event notification makes it easier to build serverless architectures. Services can generate events when something happens, and rules can route them to serverless compute like Lambda to take action on those events. This decouples and scales applications by avoiding direct dependencies. Event notifications handle all the event ingestion, buffering, routing, and delivery reliability. It eliminates complexity so you can focus on writing code to react to events.

In our application we will use S3 event notifications to fire events when data is uploaded to our S3 bucket. To start this will just be a simple text file. This will be a major upgrade to our application since it will allow us to greet anyone by name! Later in our project we will modify this a bit to kick off a data pipeline that will perform calculations.

Project Architecture - Dynamic Lambda Greeting

In previous projects, we hard coded the name for the greeting. In this version, we set up the structure so that the name can be dynamic. Addmittedly, this is still a simple use case, but this illustrates how we an use event-driven architecture and Amazon Lambda to customerize the application.

Figure 4-1 shows the logial architecture for this part of the project. A user will put raw data into an S3 bucket which will send a notificatin to Eventbridge. That will, in turn, trigger the “Hello, CDK” function which will trigger a Lambda function to print "Hello, {your name}".

Figure 4-1. Architecutre for this part of the project

This architcture is the scaffolding for our bigger project. Once we know how to do this, we can start adding more triggers, more functions, and more nifty features.

Add an EventBridge notification

The code section below creates the EventBridge notification. Our goal is to use CDK to design and build applications that follow sustainable architecture principles, such as designing for scalability, reliability, and performance efficiency. In future chapters we will add additional CDK constructs to implement cloud-native architectural patterns and microservices. This approach is more sustainable in terms of resource utilization, energy consumption, and environmental impact than a monolithic application.

Example 4-1. Hello CDK S3 Bucket

import * as cdk from "aws-cdk-lib";
import { aws_s3 as s3 } from "aws-cdk-lib"; 
import { aws_lambda as lambda } from "aws-cdk-lib";
// <-- NEW CODE STARTS HERE -->
import { aws_s3_notifications as s3n } from "aws-cdk-lib";
// <-- NEW CODE ENDS HERE -->
import * as path from "path";

export class HelloCdkStack extends cdk.Stack { 
  /**
   * Constructor for the stack
   * @param {cdk.App} scope - The CDK application scope
   * @param {string} id - Stack ID
   * @param {cdk.StackProps} props - Optional stack properties
   */
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) { 
    super(scope, id, props);

    const helloCdkS3Bucket = new s3.Bucket(this, "HelloCdkS3Bucket", { 
      // <-- NEW CODE STARTS HERE -->
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
      lifecycleRules: [
        {
          expiration: cdk.Duration.days(1),
        },
      ],
      // <-- NEW CODE ENDS HERE -->
    });

    const helloCdkLambdaFunction = new lambda.Function(this, "HelloCdkLambda", { 
      description:
        "Lambda function generates a dynamic greeting by retrieving the text from an S3 object and when triggered by S3 event",
      runtime: lambda.Runtime.NODEJS_18_X,
      handler: "index.main",
      code: lambda.Code.fromAsset(
        path.join(__dirname, "./lambda/lambda-hello-cdk")
      ),
    });

    // <-- NEW CODE STARTS HERE -->
    helloCdkS3Bucket.grantRead(helloCdkLambdaFunction); 

    helloCdkS3Bucket.addEventNotification(
      s3.EventType.OBJECT_CREATED,
      new s3n.LambdaDestination(helloCdkLambdaFunction),
      { suffix: ".txt" }
    );

    new cdk.CfnOutput(this, "bucketName", {
      value: helloCdkS3Bucket.bucketName,
    }); 
    // <-- NEW CODE ENDS HERE -->
  }
}

Let’s break down what this code does to augment our previous version of the HelloCdk stack. We import AWS CDK and Node.js Core Libraries modules like cdk, s3, and lambda from the aws-cdk-lib. We set the path module from Node.js core libraries assists with file paths.

Import S3 Notifications Library imports the S3 Notifications library, which helps in setting up notifications, e.g., when an object is added to an S3 bucket.

The HelloCdkStack class extends the base CDK Stack. It represents cloud resources as a single deployable unit.

The constructor accepts the CDK app’s scope, a stack ID, and optional stack properties.

We create an S3 Bucket named HelloCdkS3Bucket to upload greetings. As part of the new bucket, we add configurations to destroy the bucket when the stack is destroyed and auto-deleting objects upon stack destruction. We also set a lifecycle policy to expire objects after 1 day.

We create a Lambda Function named HelloCdkLambda and set configurations like runtime environment (Node.js 18.x), handler method, and its code location. This also includes a description, which is a great feature that makes it easier for others to understand the purpose of your Lambda function. We need to make sure Lambda has Read Access to the HelloCdkS3Bucket which we accomplish with the grantRead method.

Defines Lambda Invocation on S3 Event when a new .txt object is created in the S3 bucket.

Output S3 Bucket Name, which can be helpful for referencing it after the stack is deployed.

In simple terms we are adding an event that will fire each time an object is uploaded to our S3 bucket as long as that object is a .txt file. We are giving our Lambda function to access the bucket and the contained file, and we are reading the contents of the object and printing a dynamic greeting accordingly.

Removal policies and destroy on delete

Most L2 and L3 CDK constructs that deal with data will default to keeping that data even if you destroy, or delete, your CDK stacks. That’s because it shouldn’t be easy to accidentally delete a bunch of data in an application just because you made a change. This can get a little messy in the early stages of CDK development, because as you inevitably deploy, destroy, and revise stacks you can end up with tens or even hundreds of S3 buckets, database tables, etc.

Because we are just getting started, and because our application so far is an event-driven architecture that doesn’t yet require data persistence, we are not going to persist any data. Later in the book we will walk you through several of the standard options to destroy, keep, or take snap shots on delete. By the end of the book you will have applied a CDK Aspect that destroys data stores in development, but persists them in production. Think of the CDK aspect as a kind of filter that is applied while your stack synthesizes. More on that later.

Time to test our application out! See if you can remember how to synthesize and deploy your application. The great thing about the CDK Toolkit is that if you do something wrong you will get immediate feedback. Try to do it from memory, and if you get stuck go back a little earlier in the chapter to review the steps.

Open the Cloudformation service in the AWS console. You should notice a pattern here as well. We still have our same stack, but now there is an additional resource in there. Let’s test our Lambda function in a different way this time. We are going to upload a few files and see what happens:

1. Open a text editor and create a file called test.txt.

2. Add your name to the file and save it locally. The file should look like this:

   Firstname Lastname

In the S3 console, locate the bucket called HelloCdkBucket. Then, click Upload and select the test.txt file. Soon you will upload the file and see the greeting in the Lambda function logs. But don’t upload your file yet! First we want to look at each part of the architecture so that we can trace the event from the S3 bucket to the Lambda function output.

Figure 4-2. Hello CDK Bucket Upload Part 1

Figure 4-3. Hello CDK Bucket Upload Part 2

Now open the Lambda console and find the HelloCdkLambda function. It will be called something like HelloCdkStack-HelloCdkLambdaAABB123-abc123 except the letters and numbers at the end will be different. More on this in the note below.

Figure 4-4. Hello CDK Lambda Function

At this point, switch to the Monitor tab and expand the metrics. You will want to take note of the current metric values. Since you haven’t triggered your function yet you should see that the invocations metric is at 0.

Figure 4-5. Hello CDK Lambda Function Metrics

Now return to the Lambda Function in the console. Here you will see a visual depiction of the S3 event notificaiton that is connected to your Lambda Function.

Figure 4-6. Hello CDK Lambda S3 Event

If you click on the S3 event notification you will see the details of the event. This is a great way to troubleshoot your application if it isn’t working as expected. By looking at the event details you can verify the configuration of the event notification and see if the event is working as expected. This is, in general, a good practice when troubleshooting event-driven applications. In most cases you can find your way into the AWS console or at least CloudWatch to see the event details. If for some reason your event isn’t firing the way you expect you can go and trace each step of the event to see where the issue is.

Figure 4-7. Hello CDK Lambda S3 Event Details

Great! Now you are ready to upload your file. Go back to the S3 console and upload the test.txt file. Once you have uploaded the file, go back to the Lambda console and check the metrics. You should see that the invocations metric has increased. This means that the function has been triggered. Go to CloudWatch Logs and view the log group for the function. Look for a recent log stream and inspect the contents. At this point you need to check that the “Hello, CDK!” message printed out from the function executing. This confirms the S3 event properly invoked the Lambda function via Event Notifications.

Figure 4-8. Hello CDK Lambda Function Logs

Try the same steps, but this time try creating a new file called test.md, a markdown file. Put your name in the markdown file and upload it using similar steps. What do you expect to happen? Why does this happen? Is there a way you can modify the function to handle situations like this?

Have you noticed the naming conventions used with CDK resources yet? There are multiple types of identifiers and names. The two we will focus on are Physical IDs and Logical IDs. Physical IDs are used to reference resources across applications where the resource name needs to be predictible in all situations. For instance: if application A needs to reference an S3 bucket in application B you might need to name it VeryImportantS3Bucket and no matter how many times you deploy, you need the name to stay the same.

This would matter a lot if application B always had to reference VeryImportantS3Bucket. The problem with this type of identifier is the other side of it’s strength — it is always the same. This means that if you make changes to your application and/or create new buckets the names can collide, resulting in accidental references, failed deployments, and all kinds of other issues.

Logical IDs, on the other hand, are dynamically constructed each time a new resource is deployed. If there are considerable changes to that resource it may get a new Logical ID, which will be passed to all of the resources that rely on it in an update, too. In this example your S3 bucket will follow the naming pattern <stack>-<resource><unique-identifier>. So your S3 bucket might be called VeryImportantStack-VeryImportantS3BucketAABB123-abc123. This dramatically reduces the chance that resources conflict, collide, or are duplicated and is considered the most stable naming convention, which is why CDK does this by default.

Tip

How to let CDK name resources for you:

• Avoid changing the logical ID of an AWS CloudFormation resource after creation, as it may result in resource deletion, service interruptions, or data loss.

• When using AWS CDK, avoid explicitly setting resource names; if you do, changing an immutable configuration could gridlock your stack. Instead, either rename the resource or delete and re-deploy the stack to move forward.

• Allow CDK to auto-generate resource names to avoid issues during updates, but explicitly set resource names when referring to resources from another application.

Using Parameters and Overrides

Parameters in CloudFormation allow you to customize the behaviour of your stacks at deployment time. CDK provides a convenient way to define these parameters, making it easier to manage configurations without changing your code.

You can define parameters directly in your CDK stack. When you deploy your stack, you can provide parameter values using the CLI. Using parameters in this way adds flexibility to your infrastructure as code.

Overrides are another feature that allows you to modify specific properties of resources at deployment time. This can be especially useful if you change a property based on the deployment environment (e.g., development vs. production). To use overrides, you can set properties on your resources after they have been defined. Overrides can be helpful when integrating existing resources or making small adjustments without redefining your entire stack.

When defining parameters, you can provide default values to streamline the deployment process. This is particularly useful in development environments where standard configurations are used. By providing defaults, users can deploy without specifying every parameter, reducing the chance of errors.

AWS CloudFormation allows you to specify constraints on parameters, such as minimum or maximum values for numerical inputs or allowed values for string inputs. By implementing validation directly in your CDK parameters, you can catch configuration errors early in the deployment process.

In more complex applications, you might have parameters that depend on each other. For example, the type of instance you want to deploy may depend on the selected environment. You can implement logic in your CDK code to handle these dependencies and validate that the input parameters make sense together.

Getting Unstuck

I’ve deployed my stack, but I don’t see any EventBridge rules in the AWS console. What could be the issue?

There are a few things to check to resolve this. First, make sure you’ve actually added the EventBridge notification to your S3 bucket in your CDK code. Look for something like helloCdkS3Bucket.addEventNotification() in your stack. You can check that you’ve deployed the latest version of your stack by running cdk diff to see if there are any pending changes. Go ahead and check your IAM role to make sure it the necessary permissions to create EventBridge rules. As always, you can check if there are any error messages in the CloudFormation console or in the CDK deployment output. Finally, if none of these solve the issue, try destroying and redeploying your stack.

My Lambda function isn’t being triggered when I upload a file to the S3 bucket. How can I troubleshoot this?

Check if the file you’re uploading matches the suffix specified in the event notification (.txt in this case). Verify that the Lambda function has the necessary permissions to be invoked by S3. Look for the resource-based policy on your Lambda function. In the S3 console, look at the properties of your bucket and check if the event notifications are correctly set up. You can try uploading a file manually through the AWS console to rule out any issues with your upload process. Check the CloudWatch Logs for your Lambda function to see if there are any error messages.

I’m getting a “circular dependency detected” error when I try to deploy my stack. What’s causing this and how can I fix it?

Circular dependencies occur when two or more resources depend on each other. For starters, review your code to identify where the circular dependency might be. It’s often between the S3 bucket and the Lambda function. Next, try creating the Lambda function first, then the S3 bucket, and finally add the event notification to the bucket. If the circular dependency persists, you might need to split your stack into multiple stacks to break the dependency cycle. You can use Fn.importValue() to import values from one stack to another, which can help break circular references. Finally, consider using a custom resource to create the event notification after both the S3 bucket and Lambda function have been created.

My Lambda function is being triggered, but it’s not reading the content of the uploaded file. What could be wrong?

There are several potential issues to check. First, make sure you’ve granted the necessary permissions for the Lambda function to read from the S3 bucket. Look for helloCdkS3Bucket.grantRead(helloCdkLambdaFunction) in your stack. Then, check if your Lambda function code is correctly retrieving the S3 object. It should use the AWS SDK to get the object content. Next, verify that the Lambda function’s execution role has the necessary IAM permissions to read from S3. As always, review the CloudWatch Logs - they are super useful for debugging. Finally, try increasing the Lambda function’s timeout if the file is large and might be taking too long to read.

In the next chapter we will add more complexity to our application by adding a new trigger and a new function. This will allow us to start building a data pipeline that will calculate utility usage patterns. This is a common use case for event-driven architectures and will give you a good foundation for building more complex applications.

Chapter Synth

In this chapter you learned:

1. How to deploy an Eventbridge notification

2. How to pass parameters between CDK constructs

3. How to integrate multiple CDK constructs

4. How to use CDK to create an event-driven architecture

5. The difference between physical and logical IDs in CDK

Discussion

1. In this chapter you strung together the inputs and outputs of several CDK constructs. How would you add more user interactivity using these constructs?

2. What is the difference between an event-driven application and a polling application? What are the benefits?

3. Describe the method for passing data between CDK constructs. At what point does this happen in the synthesis process?

4. How would you modify the application to accept multiple files and print a greeting for each file uploaded?

5. How would you modify the application to accept a file with multiple names and print a greeting for each name in the file?

6. Articulate the difference between a physical ID and a logical ID in CDK. How does CDK use these IDs to manage resources?

Extension activities

Now that you have completed this chapter, try to modify the application to:

1. Print the name in the greeting from a list

2. Print a greeting based on the time of day

3. Accept files in different formats and/or reject files if they do not meet the format specification

4. Use a hard-coded greeting in the Lambda function and see if you can get it to print out a greeting without using the S3 bucket

5. Integrate with an external API to tell the user the current weather based on location

6. Select an additional api to integrate with and add a new greeting based on the data from that API

7. Add a simple notification service notification so that any errors in the Lambda function are sent to an email address