Learn Python by Building Data Science Applications

Building a serverless function as a data pipeline

So far, we have only used serverless functions as API endpoints, but they can serve in many other ways as well. For example, they can be triggered to run for each new file uploaded to a specific folder on S3, or scheduled to run at a specific time. 

Let's create one more application for data collection. We can specify that we need the requests library in requirements.txt. We can also copy and paste the _get_data function from Chapter 15, Packaging and Testing with Poetry and PyTest, along with the resource and time columns. One part of the code that we are still missing is that for uploading data to S3. Here is the code:

def _upload_json(obj, filename, bucket, key): S3 = boto3.client('s3', ...