Background:

• I'm trying to deploy a Django app to the Google App Engine (GAE) standard environment in the python39 runtime
• The database configuration is stored in a Secret Manager secret version, similar to Google's GAE Django tutorial (link)
• The app is run as a user-managed service account [email protected], which has the appropriate permissions to access the secret, as can be confirmed using gcloud secret versions access

Problem:

• In the Django settings.py module, when I try to access the secret using google.cloud.secretmanager.SecretManagerServiceClient.access_secret_version(...), I get the following CONSUMER_INVALID error:

google.api_core.exceptions.PermissionDenied: 403 Permission denied on resource project myproject. [links {
    description: "Google developer console API key"
    url: "https://console.developers.google.com/project/myproject/apiui/credential"
  }
  , reason: "CONSUMER_INVALID"
  domain: "googleapis.com"
  metadata {
    key: "service"
    value: "secretmanager.googleapis.com"
  }
  metadata {
    key: "consumer"
    value: "projects/myproject"
  }

My Debugging

• I cannot reproduce the error above outside of GAE;
  • I can confirm that the SA can access the secret:

gcloud secrets versions access latest --secret=server_env --project myproject \
  --impersonate-service-account=server@myproject.iam.gserviceaccount.com

WARNING: This command is using service account impersonation. All API calls will be executed as [[email protected]].

DATABASE_URL='postgres://django:...'
SECRET_KEY='...'

• I've also confirmed I run the django app locally with service account impersonation and make the above access_secret_version(...) calls

• In desperation I even created an API key for the project and hardcoded it into my settings.py file, and this also raises the same error

• I've confirmed the following settings in the project:

  • the app is running with using the correct user-managed SA
  • the call to access_secret_version is being made with the correct SA (ie that the credentials are being pulled from the GAE environment correctly)
  • the project has the secretmanager.googleapis.com service enabled, and has billing enabled and the billing account is active

If you have any suggestions for a configuration or method to help debug this, I'd much appreciate it!

Relevant Code Snippets

app.yaml

service_account: [email protected]

runtime: python39

handlers:
# This configures Google App Engine to serve the files in the app's static
# directory.
- url: /_static
  static_dir: _static/

# This handler routes all requests not caught above to your main app. It is
# required when static routes are defined, but can be omitted (along with
# the entire handlers section) when there are no static files defined.
- url: /.*
  script: auto

env_variables:
  ...

inbound_services:
  - mail
  - mail_bounce

app_engine_apis: true

Service Account Creation & Permissions

• The SA is created with Terraform as below
• (The SA doesn't have the role roles/secretmanager.secretAccessor, but has an IAM binding directly on the secret itself)

resource "google_service_account" "frontend_server" {
  project      = google_project.project.project_id
  account_id   = "server"
  display_name = "Frontend Server Service Account"
}

resource "google_project_iam_member" "frontend_server" {
  depends_on = [
    google_service_account.frontend_server,
  ]
  for_each = toset([
    "roles/appengine.serviceAgent",
    "roles/cloudsql.client",
    "roles/cloudsql.instanceUser",
    "roles/secretmanager.viewer",
    "roles/storage.objectViewer",
  ])
  project  = google_project.project.project_id
  role     = each.key
  member   = "serviceAccount:${google_service_account.frontend_server.email}"
}

Django settings.py

The relevant sections of the app settings.py are shown below; the access_secret_version raises the

import logging
import environ
from google.cloud import secretmanager
import google.auth

# Load secrets from secret manager; the client is auth'd by SA IAM policies
credentials, project = google.auth.default(
  scopes=['https://www.googleapis.com/auth/cloud-platform']
)
secretmanager_client = secretmanager.SecretManagerServiceClient(credentials=credentials)

# Load the database connection string into the environment
secrets = [
  f"projects/{GOOGLE_CLOUD_PROJECT}/secrets/server_env/versions/latest",
]
for name in secrets:
  try:
    logging.info(f"Reading secret {name} into django settings module...")
    payload = secretmanager_client.access_secret_version(name=name).payload.data.decode("UTF-8")
    env.read_env(io.StringIO(payload))
  except Exception as e:
    logging.error(f"Encountered error when accessing secret {name}: {e}")
    logging.error(f"Client credentials during error: {secretmanager_client._transport._credentials.__dict__}")
    raise e from None

CodePudding user response：

You have granted the incorrect role. Have a look to that documentation page.

• Secret Viewer role allows you to view the secret and versions but NOT the content.
• Secret Accessor role allows you to access to secret version content.

CodePudding user response：

Sigh. This was a terrible case of a hard-to-read typo in the app.yaml file. The project had a mnemonic substring with very similar letters that I had mistyped and just couldn't see.

FWIW, if anyone is running into a similar flub, you can at least avoid this one source of error:

• I was passing in a project prefix string and an environment string through the app.yaml file, and then settings.py file I concatenated these strings to make the project
• When running gcloud app deploy the (correct) concatenated project string also existed in my shell $GOOGLE_CLOUD_PROJECT variable, so the deployment happened correctly with the right project id
• However I removed the concatenation code in settings.py in favour of the GOOGLE_CLOUD_PROJECT env variable that is always present in GAE (docs)

TLDR: DRY is good..