I'm working on setting up a 6 replica redis database cluster where 3 nodes are primary and 3 are secondary (fallback). Everything is set up on 3 nodes / machines.

Is there a way for me to specify where each primary and secondary node should go? I believe good set of rules for this case would be

1. Each primary pod should be on it's own node
2. Each secondary pod should be on it's own node not on the same node as it's primary pod companion (i.e. if node goes down, we don't want secondary pod to also go down with master)

Would appreciate any pointers to docs / examples

CodePudding user response：

Regarding the first point, distributing redis masters on different nodes, you may do this with the following:

kind: StatefulSet
metadata:
  name: lemon-kube-primary
spec:
  replicas: 3
  matchLabels:
    redis-set: lemon-kube-primary
  template:
    metadata:
      labels:
        redis-set: lemon-kube-primary
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: redis-set
                operator: In
                values:
                - lemon-kube-primary
            topologyKey: kubernetes.io/hostname
# pods with label redis-set=lemon-kube-primary must be scheduled on
# nodes with distinct kubernetes.io/hostname label

While the second point is not obvious. We would not have a label to look for, giving you pod index within statefulset.

However we could rely on pod-name labels, somehow ... Assuming we do not use a StatefulSet for those secondary nodes. Instead, we could create Nx ReplicaSet, setting proper affinity rules in each of them:

kind: ReplicaSet
metadata:
  name: lemon-kube-secondary-0
spec:
  replicas: 1
  matchLabels:
    redis-set: lemon-kube-secondary
    redis-id: "0"
  template:
    metadata:
      labels:
        redis-set: lemon-kube-secondary
        redis-id: "0"
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: redis-set
                operator: In
                values:
                - lemon-kube-secondary
            topologyKey: kubernetes.io/hostname
          - labelSelector:
              matchExpressions:
              - key: statefulset.kubernetes.io/pod-name
                operator: In
                values:
                - lemon-kube-primary-0
            topologyKey: kubernetes.io/hostname
[...]
kind: ReplicaSet
metadata:
  name: lemon-kube-secondary-N
spec:
  replicas: 1
  matchLabels:
    redis-set: lemon-kube-secondary
    redis-id: "N"
  template:
    metadata:
      labels:
        redis-set: lemon-kube-secondary
        redis-id: "N"
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: redis-set
                operator: In
                values:
                - lemon-kube-secondary
            topologyKey: kubernetes.io/hostname
          - labelSelector:
              matchExpressions:
              - key: statefulset.kubernetes.io/pod-name
                operator: In
                values:
                - lemon-kube-primary-N
            topologyKey: kubernetes.io/hostname