AWS Glue job ignores logging settings-CodePudding

I have a glue job in CDK that looks like this:

const job = new Job(scope, `${appName}${stageName}GlueJobCloudTrailLogs`, {
    executable: JobExecutable.pythonEtl({
        glueVersion: GlueVersion.V3_0,
        pythonVersion: PythonVersion.THREE,
        script: Code.fromAsset(path.join(__dirname, 'cloud_trail_logs.py')),
    }),
    role: glueRole,
    jobName: `${appName}${stageName}GlueJobCloudTrailLogs`,
    continuousLogging: {
        enabled: true,
        logGroup,
    },
});

When I go to cloudwatch to see the custom log group that was passed, it is empty, and when I see a job run of it, I see this:

Then, the trigger I have is:

new CfnTrigger(scope, `${appName}${stageName}GlueJobCloudTrailLogsTrigger`, {
    type: 'SCHEDULED',
    schedule: 'cron(* * * * ? *)',
    actions: [
        {
            jobName: job.jobName,
            arguments: jobDefaultArguments,
        },

    ],
    startOnCreation: true,
});

So, is the setting on the job definition being overwritten by something? Why if I have enabled the logging it still shows as disabled in all job runs?

CodePudding user response：

Were you able to trouble shoot the issue ? I'm facing the same problem..

CodePudding user response：

Your suspicion about overwriting is correct. CfnTrigger's arguments are overwriting the job's.

The CfnTrigger's arguments are "the job arguments used when this trigger fires. For this job run, they replace the default arguments set in the job definition itself." Arguments are optional in the trigger.

The trigger arguments overwrites the CDK L2 Job construct's logging configuration, which are defaultArguments. You can see this in the Job construct's github source or in the CDK's CloudFormation template output in your cdk.out directory:

// MyStack.template.json
"MyJobE62EB972": {
  ...
  "DefaultArguments": {
    "--job-language": "python",
    "--enable-continuous-cloudwatch-log": "true",
    "--enable-continuous-log-filter": "true"
  },
  ...