Health check of Postgres database in separate pod from springboot application pod to ensure database-CodePudding

as a training in my company I have to deploy my simple microservices app in minikube Kubernetes local cluster.

My app is as follows:

Post microservice
User microservice both communicate with eachother throught RestTemplate services uses postgres databases in separate containers build on official postgres image - each have its own database run i a docker container.

I have both services run i a containers alongside with 2 databases. In docker-compose.yml I could simply add 'depends_on' to make sure that application containers will run only after containers with database are ready and running.

How can I achive the same thing in Kubernetes? I must run 8 pods (2 for microservices and 2 for databases all*2 instances) all pods in the same namespace. I must make sure that pods with databases will run first and only after those are ready, pods with microservices will run.

All should work with command

kubectl apply -f .

So shortly, pod with application should start after pod with database.

I have tried using InitContainer, but it doesnt work.

my post-manifest.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: post
  namespace: kubernetes-microservices-task2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: post
  template:
    metadata:
      labels:
        app: post
    spec:
      containers:
        - name: post
          image: norbertpedyk/post-image:varsion8
          envFrom:
            - configMapRef:
                name: post-config
            - secretRef:
                name: secret
          ports:
            - containerPort: 8081
      initContainers:
        - name: check-db
          image: busybox
          command: [ "sh", "-c", "until nslookup post-db; do echo waiting for post-db; sleep 2; done" ]

---

apiVersion: v1
kind: Service
metadata:
  name: post
  namespace: kubernetes-microservices-task2
spec:
  selector:
    app: post
  ports:
    - name: post
      port: 8081
      targetPort: 8081

my post-db-manifest.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: post-db
  namespace: kubernetes-microservices-task2
spec:
  replicas: 2
  selector:
    matchLabels:
      app: post-db
  template:
    metadata:
      labels:
        app: post-db
    spec:
      containers:
        - name: post-db
          image: postgres
          env:
            - name: POSTGRES_DB
              value: "posts"
            - name: POSTGRES_USER
              value: "pedyk"
            - name: POSTGRES_PASSWORD
              value: "password"
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: postgres-post-data
              mountPath: /data/postgres
      volumes:
        - name: postgres-post-data
          emptyDir: { }

---

apiVersion: v1
kind: Service
metadata:
  name: post-db
  namespace: kubernetes-microservices-task2
spec:
  selector:
    app: post-db
  ports:
    - name: http
      port: 5433
      targetPort: 5432

Ehat I got is: enter image description here

This is an example only for Post service, the same is with User service

CodePudding user response：

I must make sure that pods with databases will run first and only after those are ready, pods with microservices will run.

You shouldn't rely on a specific startup order in order for your application to function. You really want your application to wait until the database is ready; your initContainer idea is heading in the right direction, but you're not checking to see if the database is ready; you're simply checking to see if the hostname resolves.

An easy way to check if postgres is ready is to use psql to run a simple query (SELECT 1) against it until it succeeds. We could do something like the following. In this example, I assume that we have a secret named postgres-credentials that looks like:

apiVersion: v1
kind: Secret
metadata:
  labels:
    app: wait-for-db
    component: postgres
  name: postgres-credentials-b8mfmtc4g6
  namespace: postgres
type: Opaque
stringData:
  POSTGRES_PASSWORD: secret
  POSTGRES_USER: postgres
  POSTGRES_DB: postgres

That secret is used to configure the postgres image; we can use the same secret to set up environment variables for psql:

initContainers:
- name: wait-for-db
  image: docker.io/postgres:14
  env:
  - name: PGHOST
    value: postgres
  - name: PGUSER
    valueFrom:
      secretKeyRef:
        key: POSTGRES_USER
        name: postgres-credentials-b8mfmtc4g6
  - name: PGPASSWORD
    valueFrom:
      secretKeyRef:
        key: POSTGRES_PASSWORD
        name: postgres-credentials-b8mfmtc4g6
  - name: PGDATABASE
    valueFrom:
      secretKeyRef:
        key: POSTGRES_DB
        name: postgres-credentials-b8mfmtc4g6
  command:
  - /bin/sh
  - -c
  - |
    while ! psql -c 'select 1' > /dev/null 2>&1; do
      echo "waiting for database"
      sleep 1
    done
    echo "database is ready!"

This initContainer runs psql -c 'select 1' until it succeeds, at which point the application pod can come up.

You can find a complete example deployment here (that uses a PgAdmin pod as an example application).

It's worth noting that this is really only half the picture: in an ideal world, you wouldn't need the initContainer because your application would already have the necessary logic to (a) wait for database availability at startup and (b) handle database interruptions at runtime. In other words, you want to be able to restart your database (or upgrade it, or whatever) while your application is running without having to restart your application.