GitLab Runner suddenly fails to run jobs using Docker Machine and AWS Autoscaling-CodePudding

I use GitLab Runner for running CI jobs on AWS EC2 spot instances, using its autoscaling feature with Docker Machine.

All of a sudden, today GitLab CI failed to run jobs an shows me the following job output for all jobs that I want to start:

Running with gitlab-runner 14.9.1 (f188edd7)
  on AWS EC2 runner ...
Preparing the "docker machine" executor
10:05
ERROR: Preparation failed: exit status 1
Will be retried in 3s ...
ERROR: Preparation failed: exit status 1
Will be retried in 3s ...
ERROR: Preparation failed: exit status 1
Will be retried in 3s ...
ERROR: Job failed (system failure): exit status 1

I see in AWS that the EC2 instances do get created, but the instances always get stopped immediately by GitLab Runner again.

The GitLab Runner system logs show me the following errors:

ERROR: Machine creation failed                      error=exit status 1 name=runner-eauzytys-gitlab-ci-1651050768-f84b471e time=1m2.409578844s
ERROR: Error creating machine: Error running provisioning: error installing docker:   driver=amazonec2 name=runner-xxxxxxxx-gitlab-ci-1651050768-f84b471e operation=create

So the error seams somehow to be related to Docker machine. Upgrading GitLab Runner as well as GitLab's Docker Machine fork to the newest versions do not fix the error. I'm using GitLab 14.8 and tried GitLab Runner 14.9 and 14.10.

What can be the reason for this?

CodePudding user response：

Workaround/fix:

Explicitly specify the AMI in your runner configuration and do not rely on the default one anymore, e.g. add "amazonec2-ami=ami-02584c1c9d05efa69" to your MachineOptions:

MachineOptions = [
  "amazonec2-access-key=xxx",
  "amazonec2-secret-key=xxx",
  "amazonec2-region=eu-central-1",
  "amazonec2-vpc-id=vpc-xxx",
  "amazonec2-subnet-id=subnet-xxx",
  "amazonec2-use-private-address=true",
  "amazonec2-tags=runner-manager-name,gitlab-aws-autoscaler,gitlab,true,gitlab-runner-autoscale,true",
  "amazonec2-security-group=ci-runners",
  "amazonec2-instance-type=m5.large",
  "amazonec2-ami=ami-02584c1c9d05efa69",  # do not use default AMI
  "amazonec2-request-spot-instance=true",
  "amazonec2-spot-price=0.045"
]

You can get a list of Ubuntu AMI IDs here.

Explanation:

The default install script for Docker, which is available on https://get.docker.com/ and which Docker Machine relies on, seems to have stopped supporting Ubuntu 16.04, but Ubuntu 16.04 is the default AMI that GitLab Runner / the EC2 driver for Docker Machine use. Thus, the installation of Docker fails on the EC2 instance and the job cannot run.