Cloudformation Setup Below is not behaving as I expected. The following variables are not being set with the the template below. When the variables are set manually the job runs successfully.
- IAM Role
- Type
- Language
Description: "AWS Glue Job Test"
Resources:
MyJobRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "glue.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
-
PolicyName: "root"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action: "*"
Resource: "*"
MyJob:
Type: AWS::Glue::Job
Properties:
Name: cf-job1
Command:
Name: test-etl1
ScriptLocation: "s3://project_bucket/releases/latest/mixpanel_job.py"
PythonVersion: "3"
Description: "Testing setup config"
ExecutionProperty:
MaxConcurrentRuns: 2
MaxRetries: 2
GlueVersion: "3.0"
WorkerType: "G.1X"
NumberOfWorkers: 2
Timeout: 2880
DefaultArguments:
"--class": "GlueApp"
"--enable-continuous-cloudwatch-log": "true"
"--enable-job-insights": "true"
"--enable-metrics": "true"
"--enable-spark-ui": "true"
"--extra-jars": "s3://project_bucket/releases/latest/jars/delta-core_2.12-1.0.1.jar"
"--extra-py-files": "s3://project_bucket/releases/latest/lib.zip"
"--job-bookmark-option": "job-bookmark-disable"
"--job-language": "python"
"--spark-event-logs-path": "s3://project_bucket/logs/"
Role: !Ref MyJobRole
CodePudding user response:
This is a known issue with AWS Cloudformation on the Glue Team.
A default IAM role to run the job can only be SET with a physical ARN string value pointing to another ROLE
--job-language parameter doesn't set correctly which is a known issue. This must be done manually via the Glue Console
Type parameter in the console still doesn't have a cloudformation parameter & must be set manually.
True automation can not be achieved at this time.