gcloud.dataflow.flex-template.run unrecognized arguments: --temp-location-CodePudding

I'm providing both a staging and temp location via the gcloud dataflow flex-template run CLI. These are both valid optional flags (as per docs) and there isn't any mention that you should or shouldn't provide both.

Why else would this error come up?

ERROR: (gcloud.dataflow.flex-template.run) unrecognized arguments: --temp-location (did you mean '--staging-location'?) gs://gcs-bucket-name

Edit: adding context as requested

The process is executing within a Buildkite CI/CD pipeline, so generally speaking a Buildkite agent/step calls a gcloud Docker container which runs a bash script. I'm also able to run this command and container combo locally just fine -- the error is only present while running in the pipeline

Dockerfile

FROM gcr.io/google.com/cloudsdktool/cloud-sdk

COPY docker/scripts/gcloud-deploy-flex-template.sh /app/gcloud-deploy-flex-template.sh

WORKDIR /app

# RUN sudo apt-get install google-cloud-sdk <-- threw error

ENTRYPOINT "/app/gcloud-deploy-flex-template.sh"

gcloud-deploy-flex-template.sh

gcloud dataflow flex-template run ${JOB_NAME} \
    --template-file-gcs-location ${TEMPLATE_PATH}.json \
    --region us-central1 \
    --staging-location ${GCS_PATH}/staging/${JOB_NAME} \
    --temp-location ${GCS_PATH}/temp \
        --parameters requirements_file=requirements.txt \
        --parameters input_subscription=${INPUT_SUBSCRIPTION} \
        --parameters output_table=${OUTPUT_TABLE} \
        --parameters subject=${SUBJECT} \
        --parameters schema_registry_url=${SCHEMA_REGISTRY_URL} \
    --subnetwork=${SUBNETWORK} \
    --service-account-email=${SERVICE_ACCOUNT_EMAIL}

CodePudding user response：

As I proposed in my comment, you can use the offical google sdk image with the latest version :

FROM google/cloud-sdk:412.0.0

COPY docker/scripts/gcloud-deploy-flex-template.sh /app/gcloud-deploy-flex-template.sh

WORKDIR /app

# RUN sudo apt-get install google-cloud-sdk <-- threw error

ENTRYPOINT "/app/gcloud-deploy-flex-template.sh"

FROM gcr.io/google.com/cloudsdktool/cloud-sdk:408.0.1

COPY docker/scripts/gcloud-deploy-flex-template.sh /app/gcloud-deploy-flex-template.sh

WORKDIR /app

# RUN sudo apt-get install google-cloud-sdk <-- threw error

ENTRYPOINT "/app/gcloud-deploy-flex-template.sh"