How to use CloudFormation to update AWS Glue Jobs-CodePudding

We have many AWS Glue jobs and we are only updating the job code, which are scripts stored in S3. The problem is CloudFormation couldn't tell when and when not to update our Glue jobs because all CloudFormation template parameters remain the same after script changes, even the script location is pointing to the same S3 object.

CodePudding user response：

As this is similar to Lambda code package, you can use parameterize the Glue script path and then have different versions of Glue script file

Glue CFT has "Command" parameter taking in "JobCommand" Type value which includes "ScriptLocation" attribute, make this as a CFT parameter and have the script dynamic

{
  "Name" : String,
  "PythonVersion" : String,
  "ScriptLocation" : String
}

You can probably setup a CI/CD pipeline using AWS CodePipeline or your 3rd party CI/CD tool with the below steps

Pull the new code from your SCM like Github to deploy S3
Update CloudFormation stack with new S3 script path (with versions like v1, v2 etc...)

CodePudding user response：

You can use the CloudFormation package command. This enables you to reference local files in your git repository as scripts for Glue Jobs. Every time before you deploy to CloudFormation you just run the package command.