Home > other >  Best practices for sharded services with varying RAM consumption in kubernetes
Best practices for sharded services with varying RAM consumption in kubernetes

Time:10-16

I have sharded services, where each one have to have its ordinal. Also, each service consume different amount RAM, for example one - 4Gib, another - 20Gib. I would like to roll out these services on kubernates cluster.

I thought of using Stateful Set, where each service gets its ordinal from pod name. But there is the problem relating to resourse limits. As I wrote above, RAM consumption varies between shards, and as far as I know, high resourse limits may lead to less effective service distribution on nodes by scheduler.

Which solution suits to my problem or is the suggested one acceptable?

Requirements:

  • each pod have to get unique number (that may be acquired inside service) starting from 0
  • easy to increase shard number; new shard must get a new unique number
  • easy to deploy new service changes
  • there are dns entries for each pod, where the differences between them are only pod numbers
  • possibility to set limits for each pod (this requirement stems from the fact that each shard consumes different amount of RAM)

CodePudding user response:

The standard Kubernetes Deployment and StatefulSet objects require the replicas to be more or less the same. A StatefulSet has more specific rules on how the Pod name is constructed and the order Pods are created and creates a PersistentVolumeClaim for each replica, but in particular every Pod has the same resource constraints.

That means you don't have a choice but to create a separate Deployment for each of your shards. This lines up will with your requirement that the Pods be individually addressable, though, since that requirement also means you probably need a separate Service for each shard.


If you're deploying in a Helm chart then you could use a range loop to construct the Deployments and Services, and a well-constructed chart should be able to meet your other requirements as well. For example, you could write a Helm YAML values file containing something like

# values-prod.yaml
registry: registry.example.com
image: myimage
tag: a1b2c3
shards:
  - memory: 4GiB
  - memory: 20GiB

You could update this file to change the tag, or add shards, and run helm update -f values-prod.yaml ... to redeploy it.

Inside the chart, you'd have to create Deployments and Services in a loop, which would look like

{{- $image := printf "%s:%s/%s" .Values.registry .Values.image .Values.tag }}
{{- range $index, $shard := range .Values.shards -}}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shard-{{ $index }}
  ...
spec:
  template:
    spec:
      containers:
        - name: shard
          image: {{ $image }}
          resources:
            limits:
              memory: {{ $shard.memory }}
---
apiVersion: v1
kind: Service
...
{{ end }}
  • Related