October 2019
Intermediate to advanced
426 pages
10h 8m
English
Now we will create a Glue job using Boto3, which is the AWS SDK for Python. This SDK allows Python developers to create, configure, and manage AWS services.
Let's create a Glue job by providing the following specifications:
response = glue_client.create_job( Name=job_name, Description='PySpark job to featurize the ABC News Headlines dataset', Role=role, ExecutionProperty={ 'MaxConcurrentRuns': 1 },
In the preceding code block, we call the create_job() method of the AWS Glue client by passing in the job name, description, and role. We also specify how many concurrent we want to execute.
Now let's look at the command that's sent by Glue to the Spark cluster:
Command={'Name': 'glueetl','ScriptLocation': script_location ...Read now
Unlock full access