I use Azure DevOps Pipelines to run terraform code. The terraform code and modules stored in connected GitHub repo to Azure Devops Pipelines. If I not use TF modules - all works fine. But If I try to use modules which stored in the same repo - I got error during Terrafrom init (when modules are downloaded):

Initializing modules...
Downloading git::https://github.com/username/terraform.git?ref=v0.0.1 for storage...
╷
│ Error: Failed to download module
│ 
│ Could not download module "storage" (modules.tf:1) source code from
│ "git::https://github.com/username/terraform.git?ref=v0.0.1": error
│ downloading 'https://github.com/username/terraform.git?ref=v0.0.1':
│ /usr/bin/git exited with 128: Cloning into '.terraform/modules/storage'...
│ fatal: could not read Username for 'https://github.com': terminal prompts
│ disabled

For some reason - service connection to GitHub are not enought to download modules. But checout repo, or tf code without modules works fine.

My repo looks like this:

• azure-pipeline.yml
• environments:
  • prod
  • dev
• modules:
  • module_1
  • module_2

For Terraform install, init etc I use provided plugins.

Example of part of pipeline.yml

stages:
  - stage: prepare
    displayName: "TF - Init and Plan"
    jobs:
      - job: prepare
        displayName: "Prepare"
        steps:
          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
            displayName: 'Install Terraform 1.3.6'
            inputs:
              terraformVersion: 1.3.7
              
          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV3@3
            displayName: 'Terraform : init'
            inputs:
              workingDirectory: '$(System.DefaultWorkingDirectory)/environments/dev'
              backendServiceArm: '<subscription_id>'
              backendAzureRmResourceGroupName: <backend_rg_name>
              backendAzureRmStorageAccountName: <backend_storage>
              backendAzureRmContainerName: <backend_container>
              backendAzureRmKey: dev/terrafrom.state

          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV3@3
            displayName: 'Terraform : plan'
            inputs:
              command: plan
              workingDirectory: '$(System.DefaultWorkingDirectory)/environments/dev'
              environmentServiceNameAzureRM: '<subscription_id>'
              backendServiceArm: '<subscription_id>'
              backendAzureRmResourceGroupName: <backend_rg_name>
              backendAzureRmStorageAccountName: <backend_storage>
              backendAzureRmContainerName: <backend_container>
              backendAzureRmKey: dev/terrafrom.state

I already tried - Solution 1 and Solution 2

But looks like it works only for code which stored in Azure Repos.

I hope exist a way how I can use Azure DevOps with code and modules in GitHub.

CodePudding user response：

As this is a private GitHub repository by the error message it seems that the authorization to the GitHub repo is missing.

As per the Hashicorp Modules Sources: GitHub documentation, If using the HTTP/HTTPS protocol, or any other protocol that uses username/password credentials, configure Git Credentials Storage to select a suitable source of credentials for your environment.

Also, you can authenticate via SSH keys with GitHub. The best would be to use Deploy Keys, which is an SSH key that grants access to a single repository.

Steps:

• Create a Deploy Key in your Private repo containing the modules.
• Use this GitHub Documentation on how to create a deploy key for reference.

Info: Generating a new SSH key

• Use Azure DevOps Task InstallSSHKey@0 to use the key during execution.

          # Install SSH Key for private repo modules {config valid for all private Github Repos with a valid deploy key} ]
          - task: InstallSSHKey@0
            displayName: "Install SSH key for <repo_name> Repo"
            inputs:
              knownHostsEntry: $(knownHostsEntry)
              sshPublicKey: $(sshPublicKey)
              sshKeySecureFile: $(name_of_secure_file_in_library_group) # where private SSH key was upladed

# in Variable Group 
knownHostsEntry = github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB weqqUUmpaaasXVal72J UX2B 2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7 wC604ydGXA8VJiS5ap43JXiUFFAaQ==

sshPublicKey = "SSH Public Key generated in the `Generating a new SSH key` step"

• Modify your terraform source URLs from https to git

module "module_name" {
  source = "github.com/username/terraform//modules/storage?ref=v0.0.4"
  ## CHANGE THIS TO ##
  source = "[email protected]:username/terraform.git//modules/storage?ref=v0.5.1"
}

Still having doubts: Please use this step-by-step tutorial with screenshots.

https://www.codewithadam.com/using-terraform-modules-from-github-in-azure-devops/