ECONNREFUSED: Can't connect to RabbitMQ pod on default port within the cluster-CodePudding

I have a pod running RabbitMQ inside my local cluster. I have configured it like:

apiVersion: v1
kind: Service
metadata:
  name: service-rabbitmq
spec:
  selector:
    app: service-rabbitmq
  ports:
    - name: rabbitmq-amqp
      port: 5672
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-rabbitmq
spec:
  selector:
    matchLabels:
      app: statefulset-rabbitmq
  serviceName: service-rabbitmq
  template:
    metadata:
      labels:
        app: statefulset-rabbitmq
    spec:
      containers:
        - name: rabbitmq
          image: rabbitmq:latest
          volumeMounts:
            - name: rabbitmq-data-volume
              mountPath: /var/lib/rabbitmq/mnesia
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 750m
              memory: 512Mi
          # livenessProbe:
          #   exec:
          #     command:
          #       - 'rabbitmq-diagnostics'
          #       - 'ping'
          #       - '--quiet'
      volumes:
        - name: rabbitmq-data-volume
          persistentVolumeClaim:
            claimName: rabbitmq-pvc

And I have used amqp://service-rabbitmq:5672 to connect to it. When I deploy both the RabbitMQ and the application pods, I get the following error:

$ kubectl get pods
NAME                                       READY   STATUS    RESTARTS   AGE
deployment-service1-app-6f96656d84-pbg6r   1/1     Running   0          90s
deployment-service1-db-7bf45c9d95-jb2fq    1/1     Running   0          90s
deployment-service2-app-785b878859-lwqcj   1/1     Running   0          90s
deployment-service2-db-5546975f46-7n8kn    1/1     Running   0          90s
deployment-service3-app-b56db56d8-dzhqz    1/1     Running   0          89s
deployment-service3-db-589cbc6769-kcf5x    1/1     Running   0          89s
statefulset-rabbitmq-0                     1/1     Running   2          90s

$ kubectl logs deployment-service1-app-6f96656d84-pbg6r
[Nest] 1  - 10/14/2021, 4:56:14 PM     LOG [NestFactory] Starting Nest application...
[Nest] 1  - 10/14/2021, 4:56:14 PM     LOG [InstanceLoader] MongooseModule dependencies initialized  380ms
[Nest] 1  - 10/14/2021, 4:56:14 PM     LOG [InstanceLoader] MongooseCoreModule dependencies initialized  27ms
[Nest] 1  - 10/14/2021, 4:56:14 PM     LOG [InstanceLoader] TerminusModule dependencies initialized  2ms
[Nest] 1  - 10/14/2021, 4:56:14 PM     LOG [InstanceLoader] MongooseModule dependencies initialized  1ms
[Nest] 1  - 10/14/2021, 4:56:14 PM     LOG [InstanceLoader] HealthinessModule dependencies initialized  2ms
[Nest] 1  - 10/14/2021, 4:56:14 PM     LOG [InstanceLoader] AppModule dependencies initialized  1ms
[Nest] 1  - 10/14/2021, 4:56:14 PM   ERROR [Server] Disconnected from RMQ. Trying to reconnect.
[Nest] 1  - 10/14/2021, 4:56:14 PM   ERROR [Server] Object:
{
  "err": {
    "cause": {
      "errno": -111,
      "code": "ECONNREFUSED",
      "syscall": "connect",
      "address": "10.103.6.225",
      "port": 5672
    },
    "isOperational": true,
    "errno": -111,
    "code": "ECONNREFUSED",
    "syscall": "connect",
    "address": "10.103.6.225",
    "port": 5672
  }
}

RabbitMQ server seems to have been started successfully - I don't see any error:

$ kubectl logs statefulset-rabbitmq-0
2021-10-14 16:57:42.964882 00:00 [info] <0.222.0> Feature flags: list of feature flags found:
2021-10-14 16:57:43.032431 00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
2021-10-14 16:57:43.032495 00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
2021-10-14 16:57:43.032515 00:00 [info] <0.222.0> Feature flags:   [x] quorum_queue
2021-10-14 16:57:43.032537 00:00 [info] <0.222.0> Feature flags:   [x] stream_queue
2021-10-14 16:57:43.032649 00:00 [info] <0.222.0> Feature flags:   [x] user_limits
2021-10-14 16:57:43.032666 00:00 [info] <0.222.0> Feature flags:   [x] virtual_host_metadata
2021-10-14 16:57:43.032682 00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
2021-10-14 16:57:44.054420 00:00 [noti] <0.44.0> Application syslog exited with reason: stopped
2021-10-14 16:57:44.054519 00:00 [noti] <0.222.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
2021-10-14 16:57:44.121154 00:00 [noti] <0.222.0> Logging: configured log handlers are now ACTIVE
2021-10-14 16:57:52.058040 00:00 [info] <0.222.0> ra: starting system quorum_queues
2021-10-14 16:57:52.058172 00:00 [info] <0.222.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@statefulset-rabbitmq-0/quorum/rabbit@statefulset-rabbitmq-0
2021-10-14 16:57:52.064234 00:00 [info] <0.291.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
2021-10-14 16:57:52.064926 00:00 [noti] <0.303.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2021-10-14 16:57:52.148681 00:00 [info] <0.222.0> ra: starting system coordination
2021-10-14 16:57:52.148753 00:00 [info] <0.222.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@statefulset-rabbitmq-0/coordination/rabbit@statefulset-rabbitmq-0
2021-10-14 16:57:52.152782 00:00 [info] <0.336.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
2021-10-14 16:57:52.153150 00:00 [noti] <0.341.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
2021-10-14 16:57:52.255734 00:00 [info] <0.222.0>
2021-10-14 16:57:52.255734 00:00 [info] <0.222.0>  Starting RabbitMQ 3.9.7 on Erlang 24.1.2 [jit]
2021-10-14 16:57:52.255734 00:00 [info] <0.222.0>  Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
2021-10-14 16:57:52.255734 00:00 [info] <0.222.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  ##  ##      RabbitMQ 3.9.7
  ##  ##
  ##########  Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
  ######  ##
  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  Erlang:      24.1.2 [jit]
  TLS Library: OpenSSL - OpenSSL 1.1.1l  24 Aug 2021

  Doc guides:  https://rabbitmq.com/documentation.html
  Support:     https://rabbitmq.com/contact.html
  Tutorials:   https://rabbitmq.com/getstarted.html
  Monitoring:  https://rabbitmq.com/monitoring.html

  Logs: /var/log/rabbitmq/rabbit@statefulset-rabbitmq-0_upgrade.log
        <stdout>

  Config file(s): /etc/rabbitmq/conf.d/10-default-guest-user.conf
                  /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf

  Starting broker...2021-10-14 16:57:52.258213 00:00 [info] <0.222.0>
2021-10-14 16:57:52.258213 00:00 [info] <0.222.0>  node           : rabbit@statefulset-rabbitmq-0
2021-10-14 16:57:52.258213 00:00 [info] <0.222.0>  home dir       : /var/lib/rabbitmq
2021-10-14 16:57:52.258213 00:00 [info] <0.222.0>  config file(s) : /etc/rabbitmq/conf.d/10-default-guest-user.conf
2021-10-14 16:57:52.258213 00:00 [info] <0.222.0>                 : /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf
2021-10-14 16:57:52.258213 00:00 [info] <0.222.0>  cookie hash    : 2l58aDqPIZ5BNRjTNOxk2Q==
2021-10-14 16:57:52.258213 00:00 [info] <0.222.0>  log(s)         : /var/log/rabbitmq/rabbit@statefulset-rabbitmq-0_upgrade.log
2021-10-14 16:57:52.258213 00:00 [info] <0.222.0>                 : <stdout>
2021-10-14 16:57:52.258213 00:00 [info] <0.222.0>  database dir   : /var/lib/rabbitmq/mnesia/rabbit@statefulset-rabbitmq-0
2021-10-14 16:57:52.544775 00:00 [info] <0.222.0> Feature flags: list of feature flags found:
2021-10-14 16:57:52.544839 00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
2021-10-14 16:57:52.544930 00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
2021-10-14 16:57:52.544970 00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
2021-10-14 16:57:52.544988 00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
2021-10-14 16:57:52.545067 00:00 [info] <0.222.0> Feature flags:   [x] quorum_queue
2021-10-14 16:57:52.545092 00:00 [info] <0.222.0> Feature flags:   [x] stream_queue
2021-10-14 16:57:52.545108 00:00 [info] <0.222.0> Feature flags:   [x] user_limits
2021-10-14 16:57:52.545123 00:00 [info] <0.222.0> Feature flags:   [x] virtual_host_metadata
2021-10-14 16:57:52.545210 00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
2021-10-14 16:57:52.945793 00:00 [info] <0.222.0> Running boot step pre_boot defined by app rabbit
2021-10-14 16:57:52.945882 00:00 [info] <0.222.0> Running boot step rabbit_global_counters defined by app rabbit
2021-10-14 16:57:52.946226 00:00 [info] <0.222.0> Running boot step rabbit_osiris_metrics defined by app rabbit
2021-10-14 16:57:52.946440 00:00 [info] <0.222.0> Running boot step rabbit_core_metrics defined by app rabbit
2021-10-14 16:57:52.978102 00:00 [info] <0.222.0> Running boot step rabbit_alarm defined by app rabbit
2021-10-14 16:57:53.022595 00:00 [info] <0.351.0> Memory high watermark set to 5090 MiB (5338033356 bytes) of 12726 MiB (13345083392 bytes) total
2021-10-14 16:57:53.029390 00:00 [info] <0.353.0> Enabling free disk space monitoring
2021-10-14 16:57:53.029459 00:00 [info] <0.353.0> Disk free limit set to 50MB
2021-10-14 16:57:53.034835 00:00 [info] <0.222.0> Running boot step code_server_cache defined by app rabbit
2021-10-14 16:57:53.034965 00:00 [info] <0.222.0> Running boot step file_handle_cache defined by app rabbit
2021-10-14 16:57:53.035272 00:00 [info] <0.356.0> Limiting to approx 1048479 file handles (943629 sockets)
2021-10-14 16:57:53.035483 00:00 [info] <0.357.0> FHC read buffering: OFF
2021-10-14 16:57:53.035523 00:00 [info] <0.357.0> FHC write buffering: ON
2021-10-14 16:57:53.036032 00:00 [info] <0.222.0> Running boot step worker_pool defined by app rabbit
2021-10-14 16:57:53.036114 00:00 [info] <0.343.0> Will use 16 processes for default worker pool
2021-10-14 16:57:53.036143 00:00 [info] <0.343.0> Starting worker pool 'worker_pool' with 16 processes in it
2021-10-14 16:57:53.037184 00:00 [info] <0.222.0> Running boot step database defined by app rabbit
2021-10-14 16:57:53.039504 00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-14 16:57:53.039726 00:00 [info] <0.222.0> Successfully synced tables from a peer
2021-10-14 16:57:53.039781 00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-14 16:57:53.039985 00:00 [info] <0.222.0> Successfully synced tables from a peer
2021-10-14 16:57:53.052726 00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-14 16:57:53.052917 00:00 [info] <0.222.0> Successfully synced tables from a peer
2021-10-14 16:57:53.052951 00:00 [info] <0.222.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2021-10-14 16:57:53.053077 00:00 [info] <0.222.0> Running boot step database_sync defined by app rabbit
2021-10-14 16:57:53.053281 00:00 [info] <0.222.0> Running boot step feature_flags defined by app rabbit
2021-10-14 16:57:53.053552 00:00 [info] <0.222.0> Running boot step codec_correctness_check defined by app rabbit
2021-10-14 16:57:53.053586 00:00 [info] <0.222.0> Running boot step external_infrastructure defined by app rabbit
2021-10-14 16:57:53.053624 00:00 [info] <0.222.0> Running boot step rabbit_registry defined by app rabbit
2021-10-14 16:57:53.053799 00:00 [info] <0.222.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-10-14 16:57:53.053939 00:00 [info] <0.222.0> Running boot step rabbit_queue_location_random defined by app rabbit
2021-10-14 16:57:53.054086 00:00 [info] <0.222.0> Running boot step rabbit_event defined by app rabbit
2021-10-14 16:57:53.054373 00:00 [info] <0.222.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-10-14 16:57:53.054473 00:00 [info] <0.222.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-10-14 16:57:53.054612 00:00 [info] <0.222.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-10-14 16:57:53.054738 00:00 [info] <0.222.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-10-14 16:57:53.054837 00:00 [info] <0.222.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-10-14 16:57:53.054954 00:00 [info] <0.222.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-10-14 16:57:53.055080 00:00 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-10-14 16:57:53.055177 00:00 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-10-14 16:57:53.055297 00:00 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-10-14 16:57:53.055386 00:00 [info] <0.222.0> Running boot step rabbit_priority_queue defined by app rabbit
2021-10-14 16:57:53.055439 00:00 [info] <0.222.0> Priority queues enabled, real BQ is rabbit_variable_queue
2021-10-14 16:57:53.055560 00:00 [info] <0.222.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-10-14 16:57:53.055678 00:00 [info] <0.222.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-10-14 16:57:53.055772 00:00 [info] <0.222.0> Running boot step kernel_ready defined by app rabbit
2021-10-14 16:57:53.055800 00:00 [info] <0.222.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-10-14 16:57:53.056090 00:00 [info] <0.222.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-10-14 16:57:53.059579 00:00 [info] <0.391.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-10-14 16:57:53.059752 00:00 [info] <0.222.0> Running boot step guid_generator defined by app rabbit
2021-10-14 16:57:53.060987 00:00 [info] <0.222.0> Running boot step rabbit_node_monitor defined by app rabbit
2021-10-14 16:57:53.061357 00:00 [info] <0.395.0> Starting rabbit_node_monitor
2021-10-14 16:57:53.061560 00:00 [info] <0.222.0> Running boot step delegate_sup defined by app rabbit
2021-10-14 16:57:53.062457 00:00 [info] <0.222.0> Running boot step rabbit_memory_monitor defined by app rabbit
2021-10-14 16:57:53.062710 00:00 [info] <0.222.0> Running boot step core_initialized defined by app rabbit
2021-10-14 16:57:53.062739 00:00 [info] <0.222.0> Running boot step upgrade_queues defined by app rabbit
2021-10-14 16:57:53.111104 00:00 [info] <0.222.0> Running boot step channel_tracking defined by app rabbit
2021-10-14 16:57:53.111501 00:00 [info] <0.222.0> Setting up a table for channel tracking on this node: 'tracked_channel_on_node_rabbit@statefulset-rabbitmq-0'
2021-10-14 16:57:53.111728 00:00 [info] <0.222.0> Setting up a table for channel tracking on this node: 'tracked_channel_table_per_user_on_node_rabbit@statefulset-rabbitmq-0'
2021-10-14 16:57:53.111966 00:00 [info] <0.222.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
2021-10-14 16:57:53.112057 00:00 [info] <0.222.0> Running boot step connection_tracking defined by app rabbit
2021-10-14 16:57:53.112299 00:00 [info] <0.222.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@statefulset-rabbitmq-0'
2021-10-14 16:57:53.112540 00:00 [info] <0.222.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@statefulset-rabbitmq-0'
2021-10-14 16:57:53.112754 00:00 [info] <0.222.0> Setting up a table for per-user connection counting on this node: 'tracked_connection_table_per_user_on_node_rabbit@statefulset-rabbitmq-0'
2021-10-14 16:57:53.113041 00:00 [info] <0.222.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-10-14 16:57:53.113111 00:00 [info] <0.222.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-10-14 16:57:53.113202 00:00 [info] <0.222.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-10-14 16:57:53.113557 00:00 [info] <0.222.0> Running boot step rabbit_policies defined by app rabbit
2021-10-14 16:57:53.113911 00:00 [info] <0.222.0> Running boot step rabbit_policy defined by app rabbit
2021-10-14 16:57:53.113981 00:00 [info] <0.222.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-10-14 16:57:53.114081 00:00 [info] <0.222.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-10-14 16:57:53.114175 00:00 [info] <0.222.0> Running boot step rabbit_stream_coordinator defined by app rabbit
2021-10-14 16:57:53.114360 00:00 [info] <0.222.0> Running boot step rabbit_vhost_limit defined by app rabbit
2021-10-14 16:57:53.114486 00:00 [info] <0.222.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2021-10-14 16:57:53.114525 00:00 [info] <0.222.0> Management plugin: using rates mode 'basic'
2021-10-14 16:57:53.115086 00:00 [info] <0.222.0> Running boot step recovery defined by app rabbit
2021-10-14 16:57:53.115914 00:00 [info] <0.433.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@statefulset-rabbitmq-0/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-10-14 16:57:53.117910 00:00 [info] <0.433.0> Starting message stores for vhost '/'
2021-10-14 16:57:53.118172 00:00 [info] <0.437.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-10-14 16:57:53.121616 00:00 [info] <0.433.0> Started message store of type transient for vhost '/'
2021-10-14 16:57:53.121999 00:00 [info] <0.441.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-10-14 16:57:53.124594 00:00 [info] <0.433.0> Started message store of type persistent for vhost '/'
2021-10-14 16:57:53.126533 00:00 [info] <0.222.0> Running boot step empty_db_check defined by app rabbit
2021-10-14 16:57:53.126596 00:00 [info] <0.222.0> Will not seed default virtual host and user: have definitions to load...
2021-10-14 16:57:53.126617 00:00 [info] <0.222.0> Running boot step rabbit_looking_glass defined by app rabbit
2021-10-14 16:57:53.126641 00:00 [info] <0.222.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-10-14 16:57:53.126908 00:00 [info] <0.222.0> Running boot step background_gc defined by app rabbit
2021-10-14 16:57:53.127164 00:00 [info] <0.222.0> Running boot step routing_ready defined by app rabbit
2021-10-14 16:57:53.127193 00:00 [info] <0.222.0> Running boot step pre_flight defined by app rabbit
2021-10-14 16:57:53.127211 00:00 [info] <0.222.0> Running boot step notify_cluster defined by app rabbit
2021-10-14 16:57:53.127230 00:00 [info] <0.222.0> Running boot step networking defined by app rabbit
2021-10-14 16:57:53.127335 00:00 [info] <0.222.0> Running boot step definition_import_worker_pool defined by app rabbit
2021-10-14 16:57:53.127431 00:00 [info] <0.343.0> Starting worker pool 'definition_import_pool' with 16 processes in it
2021-10-14 16:57:53.129310 00:00 [info] <0.222.0> Running boot step cluster_name defined by app rabbit
2021-10-14 16:57:53.129352 00:00 [info] <0.222.0> Running boot step direct_client defined by app rabbit
2021-10-14 16:57:53.129475 00:00 [info] <0.481.0> Resetting node maintenance status
2021-10-14 16:57:53.154498 00:00 [info] <0.508.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2021-10-14 16:57:53.154786 00:00 [info] <0.481.0> Ready to start client connection listeners
2021-10-14 16:57:53.157103 00:00 [info] <0.552.0> started TCP listener on [::]:5672
 completed with 3 plugins.
2021-10-14 16:57:53.426031 00:00 [info] <0.481.0> Server startup complete; 3 plugins started.
2021-10-14 16:57:53.426031 00:00 [info] <0.481.0>  * rabbitmq_prometheus
2021-10-14 16:57:53.426031 00:00 [info] <0.481.0>  * rabbitmq_web_dispatch
2021-10-14 16:57:53.426031 00:00 [info] <0.481.0>  * rabbitmq_management_agent

Can anyone help me figure out where the problem is exactly?

Additional Info (1):

Out of curiosity, I attempted to retrieve status of the running RabbitMQ pod, and this is what I got:

$ kubectl exec -it statefulset-rabbitmq-0 -- rabbitmq-diagnostics status --quiet
Runtime

OS PID: 21
OS: Linux
Uptime (seconds): 101
Is under maintenance?: false
RabbitMQ version: 3.9.7
Node name: rabbit@statefulset-rabbitmq-0
Erlang configuration: Erlang/OTP 24 [erts-12.1.2] [source] [64-bit] [smp:16:1] [ds:16:1:10] [async-threads:1] [jit]
Erlang processes: 341 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_prometheus
 * rabbitmq_web_dispatch
 * prometheus
 * rabbitmq_management_agent
 * cowboy
 * cowlib
 * accept

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@statefulset-rabbitmq-0
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@statefulset-rabbitmq-0/quorum/rabbit@statefulset-rabbitmq-0   

Config files

 * /etc/rabbitmq/conf.d/10-default-guest-user.conf
 * /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf

Log file(s)

 * /var/log/rabbitmq/rabbit@statefulset-rabbitmq-0_upgrade.log
 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.1453 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 5.338 gb

reserved_unallocated: 0.0654 gb (45.03 %)
code: 0.0341 gb (23.48 %)
other_system: 0.0324 gb (22.34 %)
other_proc: 0.0193 gb (13.26 %)
other_ets: 0.003 gb (2.07 %)
atom: 0.0014 gb (0.98 %)
plugins: 4.0e-4 gb (0.3 %)
metrics: 2.0e-4 gb (0.16 %)
mnesia: 1.0e-4 gb (0.06 %)
binary: 1.0e-4 gb (0.05 %)
quorum_ets: 0.0 gb (0.02 %)
msg_index: 0.0 gb (0.02 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_other: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
mgmt_db: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 2, limit: 1048479
Sockets: 0, limit: 943629

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 4.9169 gb

Totals

Connection count: 0
Queue count: 0
Virtual host count: 1

Listeners

Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

So it seems like there's no issue with the pod, everything looks perfect so far. Then why can't I connect to it from my apps?

CodePudding user response：

So the issue was that the Service selector didn't match with Pod's label. The correct manifest would be:

apiVersion: v1
kind: Service
metadata:
  name: service-rabbitmq
spec:
  selector:
-   app: service-rabbitmq
    app: statefulset-rabbitmq