error configuring S3 Backend: no valid credential sources for S3 Backend found-CodePudding

I've been trying to add CI/CD pipeline circleci to my AWS project written in Terraform. The problem is, terraform init plan apply works in my local machine, but it throws this error in CircleCI. Error -

Initializing the backend...
╷
│ Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.
│ 
│ Please see https://www.terraform.io/docs/language/settings/backends/s3.html
│ for more information about providing credentials.
│ 
│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.
│   For verbose messaging see aws.Config.CredentialsChainVerboseErrors

My circleCi config is this -

version: 2.1

orbs:
  python: circleci/[email protected]
  # terraform: circleci/[email protected]

jobs:
  build: 
    #  will use a python 3.10.2 container
    docker:
      - image: cimg/python:3.10.2
    working_directory: ~/project
    steps:
      - checkout


      - run:
          name: Check pyton version
          command: python --version
          
      - run:
          name: get current dir
          command: pwd

      - run:
          name: list of things in that
          command: ls -a

      - run:
          name: Install terraform
          command: bash scripts/install_tf.sh

      - run:
          name: Init infrastructure
          command: bash scripts/init.sh dev



# Invoke jobs via workflows
workflows:
  .......

And my init.sh is -

cd ./Terraform
echo "arg: $1"

if [[ "$1" == "dev" || "$1" == "stage" || "$1" == "prod" ]]; 
    then
        echo "environement: $1"
        terraform init -migrate-state -backend-config=backend.$1.conf -var-file=terraform.$1.tfvars
    else
        echo "Wrong Argument"
        echo "Pass 'dev', 'stage' or 'prod' only."
fi

My main.tf is -

provider "aws" {
    profile = "${var.profile}"
    region = "${var.region}"
}

terraform {
  backend "s3" {

  }
}

And `backend.dev.conf is -

bucket = "bucket-name"
key    = "mystate.tfstate"
region = "ap-south-1"
profile = "dev"

Also, my terraform.dev.tfvars is -

region = "ap-south-1"
profile = "dev"

These work perfectly with in my local unix (mac m1), but throws error in circleCI for backend. Yes, I've added environment variables with my aws_secret_access_key and aws_access_key_id, still it doesn't work. I've seen so many tutorials and nothing seems to solve this, I don't want to write aws credentials in my code. any idea how I can solve this?

Update: I have updated my pipeline to this -

version: 2.1

orbs:
  python: circleci/[email protected]
  aws-cli: circleci/[email protected]

jobs:
  build: 
    #  will use a python 3.10.2 container
    docker:
      - image: cimg/python:3.10.2
    working_directory: ~/project

    # Checkout the code as the first step. This is a dedicated 
    steps:
      - checkout
  
      - run:
          name: Check pyton version
          command: python --version
          
      - run:
          name: get current dir
          command: pwd

      - run:
          name: list of things in that
          command: ls -a

  aws-cli-cred-setup:
    executor: aws-cli/default
    steps:
      - aws-cli/setup:
          aws-access-key-id: aws_access_key_id
          aws-secret-access-key: aws_secret_access_key
          aws-region: region
      
      - run:
          name: get aws acc info
          command: aws sts get-caller-identity

  terraform-setup:
    executor: aws-cli/default
    working_directory: ~/project
    steps:
      - checkout
      - run:
          name: Install terraform
          command: bash scripts/install_tf.sh
      
      - run:
          name: Init infrastructure
          command: bash scripts/init.sh dev
          context: terraform


# Invoke jobs via workflows
workflows:
  dev_workflow: 
    jobs:
      - build:
        filters:
          branches:
            only: main

      - aws-cli-cred-setup
        # context: aws
      
      - terraform-setup:
          requires:
            - aws-cli-cred-setup

But it still throws the same error.

CodePudding user response：

You have probably added the aws_secret_access_key and aws_access_key_id to your project settings. But I don't see them being used in your pipeline configuration. You should do something like, so they are known during runtime:

version: 2.1

orbs:
  python: circleci/[email protected]

jobs:
  build: 
    docker:
      - image: cimg/python:3.10.2
    working_directory: ~/project
    environment:
      AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
    steps:
      - run:
          name: Check python version
          command: python --version
      ...

I would advise you read about environment variables in the documentation.

CodePudding user response：

Ok I managed to fix this issue. You have to remove profile from provider and other .tf files files. So my main.tf file is -

provider "aws" {
    region = "${var.region}"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.30"
    }
  }
  backend "s3" {

  }
}

And backend.dev.conf is -

bucket = "bucket"
key    = "dev/xxx.tfstate"
region = "ap-south-1"

And most importantly, You have to put acccess key, access key id and region inside circleci-> your project -> environment variable, And you have to setup AWS CLI on circleci, apparently inside a job config.yml-

version: 2.1

orbs:
  python: circleci/[email protected]
  aws-cli: circleci/[email protected]

jobs:
  build: 
    #  will use a python 3.10.2 container
    docker:
      - image: cimg/python:3.10.2
    working_directory: ~/project

plan-apply:
    executor: aws-cli/default
    docker:
      - image: docker.mirror.hashicorp.services/hashicorp/terraform:light
    working_directory: ~/project
    steps:
      - checkout

      - aws-cli/setup:
          aws-access-key-id: aws_access_key_id
          aws-secret-access-key: aws_secret_access_key
          aws-region: region

      - run:
          name: get aws acc info
          command: aws sts get-caller-identity
      
      - run:
          name: Init infrastructure
          command: sh scripts/init.sh dev

      - run:
          name: Plan infrastructure
          command: sh scripts/plan.sh dev

      - run:
          name: Apply infrastructure
          command: sh scripts/apply.sh dev
.....
.....

This solved the issue. But you have to init, plan and apply inside the job where you set up aws cli. I might be wrong to do setup and plan inside same job, but I'm learning now and this did the job. API changed and old tutorials don't work nowadays. Comment me your suggestions if any.