Is there a clean way to provide env var overrides using defaults, configmap, secrets, and command-li-CodePudding

We have some services that can be installed in multiple locations with differing configurations. We've been asked to support multi-level configuration options using environment variables set with defaults, configmaps, secrets, and command-line options passed in via helm install --set. The following works, but is very cumbersome as the number of parameters for some of the services are numerous and the Values dot-notation goes a few levels deeper.

env:
  # Set default values
  - name: MY_VAR
    value: default-value
  - name: OTHER_VAR
    value: default-other-value

  # Allow configmap to override
  - name: MY_VAR
    valueFrom:
      configMapKeyRef:
        name: env-configmap
        key: MY_VAR
        optional: true
  - name: OTHER_VAR
    valueFrom:
      configMapKeyRef:
        name: env-configmap
        key: OTHER_VAR
        optional: true

  # Allow secrets to override
  - name: MY_VAR
    valueFrom:
      secretsKeyRef:
        name: env-secrets
        key: MY_VAR
        optional: true
  - name: OTHER_VAR
    valueFrom:
      secretsKeyRef:
        name: env-secrets
        key: OTHER_VAR
        optional: true

  # Allow 'helm install --set' to override
  {{- if .Values.env }}
  {{- if .Values.env.my }}
  {{- if .Values.env.my.var }}
  - name: MY_VAR
    value: {{ .Values.env.my.var }}
  {{- end }}
  {{- end }}
  {{- if .Values.env.other }}
  {{- if .Values.env.other.var }}
  - name: OTHER_VAR
    value: {{ .Values.env.other.var }}
  {{- end }}
  {{- end }}
  {{- end }}

Using envFrom for the ConfigMap and Secrets would be nice, but tests and docs show this would not allow the command-line override, since env: and envFrom: doesn't mix in the way that's needed. As the v1.9 and v2.1 Kubernetes API states:

envFrom: List of sources to populate environment variables in the container. The keys defined within a source must be a C_IDENTIFIER. All invalid keys will be reported as an event when the container is starting. When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence. Cannot be updated.

Is there a better way to provide this default->configmap->secrets->cmd-line override precedence?

CodePudding user response：

all you need to do is to use helmfile which do exactly what you need. in short using helmfile you can create dynamic parameters in all your configuration files and create values files per environment like values-prd values-tst and so on.

https://github.com/roboll/helmfile

CodePudding user response：

I found a solution that I mostly like. My issue was caused by giving too much weight to the "Values defined by an Env with a duplicate key will take precedence" comment in the docs, and thinking I needed to exclusively use Env. The defined precedence is exactly what I needed.

Here's the helm chart files for my current solution.

configmap/templates/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-configmap
data:
  {{- if .Values.env }}
    {{- toYaml $.Values.env | nindent 2 }}
  {{- end }}
  {{- if .Values.applicationYaml }}
  application.yml: |
    {{- toYaml $.Values.applicationYaml | nindent 4 }}
  {{- end }}

secrets/templates/secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: my-secrets
type: Opaque
data:
  {{- range $key, $val := .Values.env }}
  {{ $key }}: {{ $val | b64enc }}
  {{- end }}
stringData:
  {{- if .Values.applicationYaml }}
  application.yml: |
    {{- toYaml $.Values.applicationYaml | nindent 4 }}
  {{- end }}

deployment.yaml

apiVersion: apps/v1
kind: Deployment
  ...
spec:
  ...
      containers:                                                                                     
      - name: my-deployment
        {{- if .Values.env }}
        env:
        {{- range $key, $val := .Values.env }}
        - name: {{ $key }}
          value: {{ $val }}
        {{- end }}
        {{- end }}
        envFrom:
        - configMapRef:
           name: my-configmap
        - secretRef:
           name: my-secrets
        volumeMounts:
        - name: configmap-application-config
          mountPath: /application/config/configmap/
        - name: secrets-application-config
          mountPath: /application/config/secrets/
      volumes:
        - name: configmap-application-config
          configMap:
             name: my-configmap
             optional: true
        - name: secrets-application-config
          secret:
             secretName: my-secrets
             optional: true

Since this is a Spring Boot app, I used volumeMounts to allow the application.yml default values to be overridden in the ConfigMap and Secrets. The order of precedence from lowest to highest is:

the application's application.yml (v1 in following examples)
the configmap's applicationYaml (v2)
the secret's applicationYaml (v3)
the configmap env (v4)
the secret env (v5)
the helm install/uninstall --set (v6)

To complete the example, here's test values yaml files and the command-line.

app/src/main/resources/application.yml

applicationYaml:
  test:
    v1: set-from-this-value
    v2: overridden
    v3: overridden
    v4: overridden
    v5: overridden
    v6: overridden

configmap/values.yaml

applicationYaml:
  test:
    v2: set-from-this-value
    v3: overridden
    v4: overridden
    v5: overridden
    v6: overridden

env:
   TEST_V4: set-from-this-value
   TEST_V5: overridden
   TEST_V6: overridden

secrets/values.yaml

applicationYaml:
  test:
    v3: set-from-this-value
    v4: overridden
    v5: overridden
    v6: overridden

env:
   TEST_V5: set-from-this-value
   TEST_V6: overridden

command-line

helm install --set env.TEST_V6=set-from-this-value ...

Ideally, I'd like to be able to use dot-notation instead of TEST_V6 in the env and --set fields, but I'm not finding a way in helm to operate only on the leaves of yaml. In other words, I'd like something like range $key, $val, but where the key is equal to "test.v6". If that was possible, the key could be internally converted to an environment variable name with {{ $key | upper | replace "-" "_" | replace "." "_" }}.