Volumes in k8s pod/Container ... loosing some dirs from the Conatiner Image-CodePudding

I have a custom Container Image for postgresql and try to run this as a Stateful kubernetes application

The image knows 2 Volumes which are mounted into

/opt/db/data/postgres/data (the $PGDATA directory of my postgres intallation)
/opt/db/backup

the backup Volume contains a deeper folder structure which is defined in the Dockerfile

Dockerfile

(excerpts)

...
...
# Environment variables required for this build (do NOT change)
# -------------------------------------------------------------
...
ENV PGDATA=/opt/db/data/postgres/data
ENV PGBASE=/opt/db/postgres
...
ENV PGBACK=/opt/db/backup/postgres/backups
ENV PGARCH=/opt/db/backup/postgres/archives


# Set up user and directories
# ---------------------------
RUN mkdir -p $PGBASE $PGBIN $PGDATA $PGBACK $PGARCH && \
    useradd -d /home/postgres -m -s /bin/bash --no-log-init --uid 1001 --user-group postgres && \
    chown -R postgres:postgres $PGBASE $PGDATA $PGBACK $PGARCH && \
    chmod a xr $PGBASE

# set up user env
# ---------------
USER postgres
...
...
# bindings
# --------
VOLUME ["$PGDATA", "$DBBASE/backup"]
...
...
# Define default command to start Database.
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["postgres", "-D", "/opt/db/data/postgres/data"]

When I run this as a container or single pod in kubernetes without any volumeMounts all is good and the folder structure looks like it should

find /opt/db/backup -ls
2246982      4 drwxr-xr-x   3 root     root         4096 Feb 18 09:00 /opt/db/backup
2246985      4 drwxr-xr-x   4 root     root         4096 Feb 18 09:00 /opt/db/backup/postgres
2246987      4 drwxr-xr-x   2 postgres postgres     4096 Feb 11 14:59 /opt/db/backup/postgres/backups
2246986      4 drwxr-xr-x   2 postgres postgres     4096 Feb 11 14:59 /opt/db/backup/postgres/archives

However once I run this based on the Statefulset below (which mounts to Volumes into the pod @ /opt/db/data/postgres/data & /opt/db/backup (which includes folder structure that goes deeper then the mount point, as listed above) this is not bein carried out as intended

find /opt/db/backup –ls
2      4 drwxr-xr-x   3 postgres postgres     4096 Feb 17 16:40 /opt/db/backup
11     16 drwx------   2 postgres postgres    16384 Feb 17 16:40 /opt/db/backup/lost found

/opt/db/backup/postgres/backups & /opt/db/backup/postgres/archives, inherit in the Image are gone.

Can anybody point me to where to start looking for a solution in this?

Statefulset

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: postgres-stateful
  name: postgres-stateful
  labels:
    app.kubernetes.io/name: postgres
    app.kubernetes.io/component: database
    app.kubernetes.io/part-of: postgres
    app: postgres
spec:
  serviceName: "postgres"
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: postgres
      app.kubernetes.io/component: database
      app.kubernetes.io/part-of: postgres
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
        app.kubernetes.io/name: postgres
        app.kubernetes.io/component: database
        app.kubernetes.io/part-of: postgres
    spec:
      serviceAccountName: default
      initContainers:           # Give `postgres` user (id 1001) permissions to mounted volumes
      - name: take-volume-mounts-ownership
        image: dev-local.dev.dlz.net/postgresql:14.2-deb11
        securityContext:
          readOnlyRootFilesystem: true
        env:
          - name: "PGDATA"
            value: "/opt/db/data/postgres/data"
        command: [ "/bin/sh" ]
        args: ["-c", "chown -R 1001:1001 /opt/db/data/postgres /opt/db/backup /tmp" ]
        volumeMounts:
        - name: pv-data
          mountPath: /opt/db/data/postgres
        - name: pv-backup
          mountPath: /opt/db/backup     # /opt/db/backup/postgres
        - name: emptydir-tmp
          mountPath: /tmp
      containers:
      - name: postgres
        image: dev-local.dev.dlz.net/postgresql:14.2-deb11
        imagePullPolicy: Always
        readinessProbe:
          exec:
            command: ["pg_isready", "-q"]
          periodSeconds: 10
          initialDelaySeconds: 7
          timeoutSeconds: 2
        livenessProbe:
          exec:
            command: ["psql", "-q", "-c", "SELECT 1 WHERE 1=0"]
          periodSeconds: 15
          initialDelaySeconds: 20
          timeoutSeconds: 2
        env:
          - name: "PGDATA"
            value: "/opt/db/data/postgres/data"
        envFrom:
        - configMapRef:
            name: postgres-configuration
        ports:
        - containerPort: 5432
          name: postgresdb
        resources:
          requests:
            memory: "256Mi"
            cpu: "50m"
          limits:
            memory: "1Gi"
            cpu: "1"
        securityContext:
          privileged: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1001
          allowPrivilegeEscalation: false
        volumeMounts:
        - name: pv-data
          mountPath: /opt/db/data/postgres/data   # /var/lib/postgresql/data
        - name: pv-backup
          mountPath: /opt/db/backup     # /opt/db/backup/postgres
        - name: emptydir-tmp
          mountPath: /tmp
      volumes:
      - name: pv-data
        persistentVolumeClaim:
          claimName: pgdata-ina-pvc
      - name: pv-backup
        persistentVolumeClaim:
          claimName: pgbackup-ina-pvc
      - name: emptydir-tmp
        emptyDir: {}

CodePudding user response：

What you have encountered is one of the many different behaviours between Docker and Kubernetes. In this case, it is how Docker vs Kubernetes treat the original contents of the image at a particular path when a volume is mounted there.

For Docker, the original contents will be copied to the volume if it is a new volume. So the four subfolders you see in the Docker container are pre-existing from the image (created by RUN mkdir Dockerfile line).

As for Kubernetes, however, you are on your own. Kubernetes simply mounts the empty PVC to the parent path overshadowing the original image content and just leave you at that. That's why for your specific case, you are missing those folders.

A common method is to use initContainer that mounts the PVC on another path and performs the copy first before the actual container starts and mounts the PVC with the copied contents on the intended path.

CodePudding user response：

The directory which you had created in Dockerfile will be overlaid when you mount persistent volume to the same path. You can re-construct the directory structure in your "take-volume-mounts-ownership" container:

...
initContainers:
- name: take-volume-mounts-ownership
  ...
  env:
  - name: PGDATA
    value: /opt/db/data/postgres/data
  - name: PGBASE
    value: /opt/db/postgres
  - name: PGBACK
    value: /opt/db/backup/postgres/backups
  - name: PGARCH
    value: /opt/db/backup/postgres/archives
  ...
  args: ["-c", "mkdir -p $PGBASE $PGBIN $PGDATA $PGBACK $PGARCH && chown -R 1001:1001 /opt/db/data/postgres /opt/db/backup /tmp" ]
  ...

CodePudding user response：

This is expected behaviour. The goal is to persist data, after all.

Imagine it would work the way you want. The container populates the volume on startup, and you get your folder structure. Now, after 1 hour, the container adds a backup to this folder structure. What do you expect to happen if the container restarts? The new container doesn't have a backup in its initial file system. Should it override the volume and purge the backup, or should the backup remain? The answer is the volume is the source of truth and not the container's file system.

That said, with docker it's actually possible. See https://docs.docker.com/storage/volumes/#populate-a-volume-using-a-container. That requires the volume to be created through the run instruction.

In Kubernetes, you need to work with entrypoint or init containers to create your desired folder structure if it doesn't exist.