How to cache Docker files across self-hosted runners in GitHub Actions?-CodePudding

I have several GitHub Actions jobs that use a matrix on self-hosted runners. As part of this they require database services for testing, etc. The job definition looks something like this.

  my-test-job:
    name: test ${{ matrix.app }}
    runs-on: self-hosted
    strategy:
      fail-fast: false
      matrix:
        app:
          - alpha
          - beta
          - charlie
          ...
    services:
      postgres:
        image: postgres:12.8
        ...
      mysql:
        image: mysql:5.7
        ...
      redis:
        image: redis:5.0
        ...

What I have found is that all of the Docker layers are getting pulled down with each matrix run. So if I have 3 jobs, 5 matrix entries, 3 services and each service image requires pulling 10 layers then each each run results in 450 pull requests. This then ultimately causes me to hit our Docker Hub pull limit.

Is there anyway to share the Docker cache across the self-hosted runners or some better way of structuring this while still using self-hosted runners?

I'm not sure if it is relevant or not but the self-hosted runners are EC2 instances that were configured using this module https://github.com/philips-labs/terraform-aws-github-runner. I don't seem to run into this issue using the GitHub hosted runner (ubuntu-latest).

Thanks!

CodePudding user response：

Seeing as you're already using AWS and you're running into rate limits on Docker Hub for just one GitHub workflow, you might just want to look into hosting the public images yourself. You can set up a publicly accessible AWS Elastic Container Registry instance pretty inexpensively if you're only transferring data out within your AWS region. You might also want to consider pulling your images from the Amazon ECR Public Gallery.