can't get access to UI Grafana Prometheus on Kubernetes Baremetall-CodePudding

Could you help me with resolving issue with accessing grafana UI on Kubernetes baremetall. I have one master node with two worker nodes:

NAME          STATUS   ROLES           AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION           CONTAINER-RUNTIME
master-node   Ready    control-plane   6d17h   v1.25.2   212.112.123.114   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.8
w-node1       Ready    worker          6d17h   v1.25.2   212.112.123.115   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.8
w-node2       Ready    worker          6d17h   v1.25.2   212.112.123.117   <none>        CentOS Linux 7 (Core)   3.10.0-1160.el7.x86_64   containerd://1.6.8

I have 3 static IP addresses for Nodes : 212.112.123.114, 212.112.123.115, 212.112.123.117 I have installed metallB on Kubernetes and Nginx Ingress

I have installed prometheus grafana:

k get pods -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Running   0          15h
alertmanager-main-1                    2/2     Running   0          15h
alertmanager-main-2                    2/2     Running   0          15h
blackbox-exporter-569d5d4bb7-wc8rv     3/3     Running   0          15h
grafana-7f9bdfd769-l7cvs               1/1     Running   0          15h
kube-state-metrics-5bd9d9bf68-dkm6f    3/3     Running   0          15h
node-exporter-lnjhv                    2/2     Running   0          15h
node-exporter-wcc5r                    2/2     Running   0          15h
node-exporter-xh2xn                    2/2     Running   0          15h
prometheus-adapter-867598997b-f94x5    1/1     Running   0          15h
prometheus-adapter-867598997b-pnq6x    1/1     Running   0          15h
prometheus-k8s-0                       2/2     Running   0          15h
prometheus-k8s-1                       2/2     Running   0          15h
prometheus-operator-7b64d465b9-pd6cj   2/2     Running   0          15h

k get svc -n monitoring
NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                         AGE
alertmanager-main       LoadBalancer   10.97.81.41     192.168.1.12   9093:31134/TCP,8080:31016/TCP   15h
alertmanager-operated   ClusterIP      None            <none>         9093/TCP,9094/TCP,9094/UDP      15h
blackbox-exporter       ClusterIP      10.110.121.18   <none>         9115/TCP,19115/TCP              15h
grafana                 LoadBalancer   10.103.36.246   192.168.1.13   3000:31657/TCP                  15h
kube-state-metrics      ClusterIP      None            <none>         8443/TCP,9443/TCP               15h
node-exporter           ClusterIP      None            <none>         9100/TCP                        15h
prometheus-adapter      ClusterIP      10.102.62.25    <none>         443/TCP                         15h
prometheus-k8s          LoadBalancer   10.100.34.90    192.168.1.11   9090:32619/TCP,8080:30045/TCP   15h
prometheus-operated     ClusterIP      None            <none>         9090/TCP                        15h
prometheus-operator     ClusterIP      None            <none>         8443/TCP                        15h

but when i tried to check grafana with nodeIP:LBPort it stuck, also i tried to create nginx ingress but the same issue:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: ingress-monitoring
  namespace: monitoring
spec:
  ingressClassName: nginx-monitoring
  rules:
  - http:
      paths:
      - backend:
          service:
            name: grafana
            port:
              number: 31657
        path: /monitoring(/|$)(.*)
        pathType: Prefix

it's always stuck, but if i create any pod or deployment and then create nginx ingress for that it works successfully

CodePudding user response：

Try doing port forwarding to your local to confirm that your grafana is working and accepting traffic. You can use below command for port forwarding:

kubectl -n monitoring port-forward pods/grafana-7f9bdfd769-l7cvs 3000:3000

In your local, open browser and go to http://127.0.0.1:3000

If grafana pod is working and ports are open, then you should see the grafana UI. If not, then you should start with confirming the port access. For that you can use netcat or any tools to check whether the port is blocked or not.

If its all good, you should check the ELB port mapping to your app. For reference: https://community.grafana.com/t/cant-access-grafana-web-interface-trought-port-3000/16661

Regarding the ingress not working for grafana, change the port to 3000 as shown below

- backend:
    service:
      name: grafana
      port:
        number: 3000

Hope it helps!

CodePudding user response：

problem solved! The firewall rules on the virtual machine were blocking access to the web browser. So with the sudo iptables -F command, I cleared all rules and now I’m able to access Grafana through the link