I am new to helm and I am trying to generate different topics for kafka with a range function to not have a yaml file for each topic:

I have different topics (topic1, topic2, topic3,...) and the only difference they have is the retention in ms of the topic and the name, some topics have 3600000 and the others 540000, this is my values file:

KafkaTopics:
  shortRetentionTopics:
    name:
    - topic1
    - topic2
    - topic3
    - topic4
    spec:
      config: 
        retention.ms: 540000
      topicName:
      - topic1logs
      - topic2logs
      - topic3logs
      - topic4logs
  longRetentionTopics:
    name:
    - topic34
    - topic35
    spec:
      config: 
        retention.ms: 3600000
      topicName:
      - topic34logs
      - topic34logs

And I would like to set the name, topicName and retention.ms on this template doing a for loop from the values file:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  labels:
    strimzi.io/cluster: kafka
  name: (here the name of the topic)
  namespace: default
spec:
  config:
    retention.ms: (here the retention of the topic)
  partitions: 12
  replicas: 1
  topicName: (here the topicName of the topic)

Or if you have any suggestion to change the structure of the values file to make it easier to parse the values to the template I'm interested as well.

CodePudding user response：

Here's an example

$ cat values.yaml
KafkaTopics:
  shortRetentionTopics:
    name:
    - topic1
    - topic2
    - topic3
    - topic4
    spec:
      config:
        retention.ms: 540000
  longRetentionTopics:
    name:
    - topic34
    - topic35
    spec:
      config:
        retention.ms: 3600000

$ cat templates/topics.yml
{{- with .Values.KafkaTopics.shortRetentionTopics }}{{- range .name }}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  labels:
    strimzi.io/cluster: kafka
  name: {{ $.Release.Name }}-{{ . }}
  namespace: default
spec:
{{- toYaml $.Values.KafkaTopics.shortRetentionTopics.spec | nindent 2 }}
  partitions: 12
  replicas: 1
  topicName: {{.}}logs
{{- end}}{{- end}}

Repeat for the long retention topics, or use separate template files.

Sample debug output - helm template topics ./topic-example --debug

---
# Source: topic-example/templates/topics.yml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  labels:
    strimzi.io/cluster: kafka
  name: topics-topic1
  namespace: default
spec:
  config:
    retention.ms: 540000
  partitions: 12
  replicas: 1
  topicName: topic1logs
---
# Source: topic-example/templates/topics.yml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  labels:
    strimzi.io/cluster: kafka
  name: topics-topic2
  namespace: default
spec:
  config:
    retention.ms: 540000
  partitions: 12
  replicas: 1
  topicName: topic2logs
---
# Source: topic-example/templates/topics.yml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  labels:
    strimzi.io/cluster: kafka
  name: topics-topic3
  namespace: default
spec:
  config:
    retention.ms: 540000
  partitions: 12
  replicas: 1
  topicName: topic3logs
---
# Source: topic-example/templates/topics.yml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  labels:
    strimzi.io/cluster: kafka
  name: topics-topic4
  namespace: default
spec:
  config:
    retention.ms: 540000
  partitions: 12
  replicas: 1
  topicName: topic4logs

CodePudding user response：

At the end I ended up doing this:

{{- range $topics := .Values.kafkaTopicList }}
{{ $spec := default dict $topics.spec }}
{{ $config := default dict $spec.config }}
{{ $retention := default dict $config.retentionMs }}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  labels:
    strimzi.io/cluster: kafka
  name: {{ $topics.name }}
  namespace: default
spec:
  config:
    retention.ms: {{ $retention | default "540000" }}
  partitions: 12
  replicas: 1
  topicName: {{ $topics.name | replace "-" "." }}
{{- end}}

Values file:

kafkaTopicList:
  topic1:
    name: events-1
  topic2:
    name: events-2
  topic3:
    name: events-3
  topic4:
    name: events-4
  topic5:
    name: events-5
  topic6:
    name: events-6
  topic7:
    name: events-7
    spec:
      config: 
        retentionMs: 3600000