Is there data for AWS spot interruption rate over time?-CodePudding

We are running an EMR cluster with spot instances as task nodes. The EMR cluster is executing spark jobs which sometimes run for several hours. Interruptions of spot instances can cause the failure of the spark job which then requires us to restart the job entirely.

I can see that there is some basic information on the "Frequency of interruption" on AWS Spot Advisor - However, this data seems to be very generic, I can't see historic trends and I also miss the probability of interruption based on how long the spot instance is running (which should have a significant impact on the probability of interruption).

Is this data available somewhere? Or are there other data points that can be used as proxy?

CodePudding user response：

I found this Github issue which provides a link to this JSON file in Spot Advisor S3 bucket that includes interruption rates.

https://spot-bid-advisor.s3.amazonaws.com/spot-advisor-data.json

CodePudding user response：

AWS Spot Instances are spare computing capacity in the AWS Cloud that is available at discounted prices. These instances can be interrupted by Amazon EC2 with two minutes of notification when the capacity is needed elsewhere.

AWS provides data on the Spot Instance interruption rate as a percentage of the total number of instances in the region. This data is available in the AWS Management Console and can be accessed by going to the EC2 dashboard and selecting "Spot Request History" under the "Spot Requests" tab.

You can also use the AWS CLI or the AWS SDKs to retrieve this data programmatically. The describe-spot-instance-request-history command in the AWS CLI can be used to retrieve the Spot Instance interruption rate for a specific region and time period.

Here is an example of how you can use the AWS CLI to retrieve the Spot Instance interruption rate for the us-east-1 region for the past week:

aws ec2 describe-spot-instance-request-history --region us-east-1 --start-time $(date -v-7d ' %Y-%m-%dT%T')

Note that the Spot Instance interruption rate can vary over time depending on the demand for computing capacity in the region and the availability of spare capacity. It is also important to note that the interruption rate is not guaranteed and can vary based on the instance type, Availability Zone, and other factors.