Home > Software engineering >  Unable to connect to mysql pod from app pod with in the same cluster and same namespace
Unable to connect to mysql pod from app pod with in the same cluster and same namespace

Time:12-30

I am using minikube to deploy mysql and spring bootapp in the same namespace. For some reason I am unable to connect to the database from the app pod. Posting it here as I don't know where to go from here. Appreciate any help.

Can connect from my Laptop

I am able to connect to same mysql on minikube from my laptop both through MySQL workbench and through mysql command. I can run the app on my machine and the app is able to apply liquibase migrations to the database in the pod.

App cannot connect

When I try to apply deployment.yaml file, spring throws a huge stacktrace but these two lines summarise it all I think.

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

Caused by: java.net.ConnectException: Connection refused

Cannot connect even through MySQL client installed on the app pod

I installed MySQL client in the app pod using apt-get install -y mysql-client and then tried to connect to the MySQL db using the following commmands. I've also tried capture the dns log kubedns using kubectl logs -n kube-system -l k8s-app=kube-dns. The output of that command is attached below each mysql command.

Service name

mysql -u root -h syms-mysql -p
Enter password:
ERROR 2003 (HY000): Can't connect to MySQL server on 'syms-mysql:3306' (111)

log

[INFO] 172.17.0.1:60559 - 27285 "AAAA IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000323333s
[INFO] 172.17.0.1:60559 - 31377 "A IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000450412s
[INFO] 172.17.0.1:1440 - 42659 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000298098s
[INFO] 172.17.0.1:1440 - 17839 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000576619s
[INFO] 172.17.0.1:65385 - 46870 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000476572s
[INFO] 172.17.0.1:65385 - 39937 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000647759s
[INFO] 172.17.0.1:3786 - 45586 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000515706s
[INFO] 172.17.0.1:3786 - 60960 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.00063402s
[INFO] 172.17.0.1:7212 - 14821 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000255618s
[INFO] 172.17.0.1:7212 - 53452 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000563956s

DNS name

bash-4.4# mysql -u root -h syms-mysql.syms.svc.cluster.local -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on 'syms-mysql.syms.svc.cluster.local:3306' (111)

log

[INFO] 172.17.0.1:7212 - 14821 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000255618s
[INFO] 172.17.0.1:7212 - 53452 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000563956s
[INFO] 172.17.0.1:45103 - 8087 "AAAA IN syms-mysql.syms.svc.cluster.local.syms.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 167 0.001098626s
[INFO] 172.17.0.1:45103 - 25752 "A IN syms-mysql.syms.svc.cluster.local.syms.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 167 0.000285655s
[INFO] 172.17.0.1:14956 - 9219 "AAAA IN syms-mysql.syms.svc.cluster.local.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000281106s
[INFO] 172.17.0.1:14956 - 1818 "A IN syms-mysql.syms.svc.cluster.local.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000927891s
[INFO] 172.17.0.1:14683 - 58630 "AAAA IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000198551s
[INFO] 172.17.0.1:14683 - 7423 "A IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000984013s
[INFO] 172.17.0.1:2028 - 8676 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000984814s
[INFO] 172.17.0.1:2028 - 20460 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.001115167s

Cluster IP

bash-4.4# mysql -u root -h 10.110.201.105 -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on '10.110.201.105:3306' (111)

log

[INFO] 172.17.0.1:7212 - 14821 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.000255618s
[INFO] 172.17.0.1:7212 - 53452 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000563956s
[INFO] 172.17.0.1:45103 - 8087 "AAAA IN syms-mysql.syms.svc.cluster.local.syms.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 167 0.001098626s
[INFO] 172.17.0.1:45103 - 25752 "A IN syms-mysql.syms.svc.cluster.local.syms.svc.cluster.local. udp 74 false 512" NXDOMAIN qr,aa,rd 167 0.000285655s
[INFO] 172.17.0.1:14956 - 9219 "AAAA IN syms-mysql.syms.svc.cluster.local.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000281106s
[INFO] 172.17.0.1:14956 - 1818 "A IN syms-mysql.syms.svc.cluster.local.svc.cluster.local. udp 69 false 512" NXDOMAIN qr,aa,rd 162 0.000927891s
[INFO] 172.17.0.1:14683 - 58630 "AAAA IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000198551s
[INFO] 172.17.0.1:14683 - 7423 "A IN syms-mysql.syms.svc.cluster.local.cluster.local. udp 65 false 512" NXDOMAIN qr,aa,rd 158 0.000984013s
[INFO] 172.17.0.1:2028 - 8676 "A IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 100 0.000984814s
[INFO] 172.17.0.1:2028 - 20460 "AAAA IN syms-mysql.syms.svc.cluster.local. udp 51 false 512" NOERROR qr,aa,rd 144 0.001115167s

MySQL Manifests

Service

apiVersion: v1
kind: Service
metadata:
  name: syms-mysql
spec:
  ports:
  - protocol: TCP
    port: 3306
    targetPort: 3306
  selector:
    app: syms-mysql-db

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: syms-mysql
  labels:
    app: syms-mysql-db
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: mysql:8.0.31
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: syms-mysql-secret
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: syms-mysql-pvc

Persistent Volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: syms-mysql-pv
  namespace: syms
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

Persistent Volume Claim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: syms-mysql-pvc
  namespace: syms
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

App Manifests

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: syms-api
  namespace: syms
spec:
  selector:
    matchLabels:
      app: syms-api
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: syms-api
    spec:
      containers:
      - image: syms-api:0.0.1-SNAPSHOT
        name: syms-api
        env:
        - name: SPRING_DATASOURCE_PASSWORD
          value: test1234
        - name: SPRING_DATASOURCE_URL
          value: jdbc:mysql://syms-mysql:3306/syms?createDatabaseIfNotExist=true&useSSL=false&allowPublicKeyRetrieval=true
        - name: SPRING_DATASOURCE_USERNAME
          value: root
        ports:
        - containerPort: 8090
          name: syms-api

Service

apiVersion: v1
kind: Service
metadata:
  name: syms-api
  namespace: syms
spec:
  ports:
  - port: 8090
  selector:
    app: syms-api

Edit 1

Attached below is the app docker file I am using.

FROM adoptopenjdk/openjdk16

WORKDIR /code

ENV PORT 80

RUN java -version

RUN apt-get update && \
    apt-get install -y mysql-client

ENTRYPOINT ["tail", "-f", "/dev/null"]

Network Policies

I wasn't sure if I needed to create any network policies, but this is what the documentation says.

By default, if no policies exist in a namespace, then all ingress and egress traffic is allowed to and from pods in that namespace. The following examples let you change the default behavior in that namespace.

https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-policies

CodePudding user response:

Please update mysql deployment's selector "app: mysql" and template labels to match the one specified in the service.

  • Related