Boto3 API for EMR doesn't support 'OnDemandPrice'-CodePudding

Having some trouble with creating an EMR spark cluster using Boto3. The error I'm getting is as follows:

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the RunJobFlow operation: Attempted to set bid price for on demand instance group.

I specified this configuration as follows:

 'Name': 'Core - 2',
 'InstanceRole': 'CORE',
 'InstanceType': 'i3.2xlarge',
 'InstanceCount': 50,
 'BidPrice': 'OnDemandPrice',

It seems the Boto3 API doesn't allow this and I don't want to use the AWS Cli either. Is there a workaround to specify spot instances through Boto3?

CodePudding user response：

There's an open issue about this in the boto3 repo. For the time being, if you specify "Market": "SPOT", BidPriceAsPercentageOfOnDemandPrice will default to 100%, aka the on-demand price.

Here's an example using boto3==1.18.42.

import boto3

client = boto3.client("emr", region_name="us-east-1")

response = client.run_job_flow(
    Name="Boto3 test cluster",
    ReleaseLabel="emr-5.33.0",
    Instances={
        "KeepJobFlowAliveWhenNoSteps": True,
        "TerminationProtected": False,
        "InstanceGroups": [
            {
                "InstanceRole": "MASTER",
                "InstanceCount": 1,
                "InstanceType": "i3.2xlarge",
                "Name": "Master",
            },
            {
                "InstanceRole": "CORE",
                "InstanceCount": 2,
                "InstanceType": "i3.2xlarge",
                "Name": "Core",
                "Market": "SPOT",
            },
        ],
    },
    VisibleToAllUsers=True,
    JobFlowRole="EMR_EC2_DefaultRole",
    ServiceRole="EMR_DefaultRole",
)