AKS LoadBalancer service creating spurious network traffic-CodePudding

I’m attempting to deploy an SFTP server using python / paramiko into AKS.

This deployed successfully into a bare metal dev server, however I am having issues deploying this into AKS.

The problem starts when creating the LoadBalancer service, this triggers a deluge of TCP traffic on the target port which renders the SFTP server useless.

Is this expected? I’m at the limit of how the AKS internals work so I don’t want to assume this is an error, but I would like to know where I might be going wrong.

The code below reproduces the issue in a newly provisioned AKS environment using the nast network sniffer. Run the first command to launch nast and then create the load balancer service using in a separate console:

kubectl run -it --rm --restart=Never --image=ubuntu --labels="app=debug-pod" debug-pod -- bash -c "apt-get update && apt-get install -y nast && nast"

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: debug-service
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app: debug-pod 
  ports:
  - name: sftp
    protocol: TCP
    port: 34567
    targetPort: 45678
EOF

Sample traffic seen ~10 seconds after the LoadBalancer is created at a rate of several every second:

Nast V. 0.2.0

Sniffing on:

- Device:       eth0
- MAC address:  66:4E:3B:41:F3:7B
- IP address:   10.244.0.61
- Netmask:      255.255.255.0
- Promisc mode: Set
- Filter:       None
- Logging:      None

---[ ARP ]-----------------------------------------------------------
2A:8E:A8:C9:D3:1E -> 66:4E:3B:41:F3:7B
Type: ARP request: Who has 10.244.0.61? Tell 10.244.0.1
Hardware size: 6 - Protocol size: 4
Packet Number: 1

---[ ARP ]-----------------------------------------------------------
66:4E:3B:41:F3:7B -> 2A:8E:A8:C9:D3:1E
Type: ARP reply: 10.244.0.61 is at 66:4E:3B:41:F3:7B
Hardware size: 6 - Protocol size: 4
Packet Number: 2

---[ TCP ]-----------------------------------------------------------
10.240.0.7:64618(unknown) -> 10.244.0.61:45678(unknown)
TTL: 126        Window: 8192    Version: 4      Length: 52
FLAGS: -S-----  SEQ: 2330618374 - ACK: 0
Packet Number: 5

---[ TCP ]-----------------------------------------------------------
10.244.0.61:45678(unknown) -> 10.240.0.7:64618(unknown)
TTL: 64         Window: 0       Version: 4      Length: 40
FLAGS: --R-A--  SEQ: 0 - ACK: 2330618375
Packet Number: 6

---[ TCP ]-----------------------------------------------------------
10.240.0.7:31026(unknown) -> 10.244.0.61:45678(unknown)
TTL: 126        Window: 8192    Version: 4      Length: 52
FLAGS: -S-----  SEQ: 2330618374 - ACK: 0
Packet Number: 7

---[ TCP ]-----------------------------------------------------------
10.244.0.61:45678(unknown) -> 10.240.0.7:31026(unknown)
TTL: 64         Window: 0       Version: 4      Length: 40
FLAGS: --R-A--  SEQ: 0 - ACK: 2330618375
Packet Number: 8

---[ TCP ]-----------------------------------------------------------
10.240.0.7:52540(unknown) -> 10.244.0.61:45678(unknown)
TTL: 126        Window: 8192    Version: 4      Length: 48
FLAGS: -S-----  SEQ: 2330618374 - ACK: 0
Packet Number: 9

---[ TCP ]-----------------------------------------------------------
10.244.0.61:45678(unknown) -> 10.240.0.7:52540(unknown)
TTL: 64         Window: 0       Version: 4      Length: 40
FLAGS: --R-A--  SEQ: 0 - ACK: 2330618375
Packet Number: 10

---[ TCP ]-----------------------------------------------------------
10.240.0.6:32242(unknown) -> 10.244.0.61:45678(unknown)
TTL: 126        Window: 8192    Version: 4      Length: 52
FLAGS: -S-----  SEQ: 2102210393 - ACK: 0
Packet Number: 11

---[ TCP ]-----------------------------------------------------------
10.244.0.61:45678(unknown) -> 10.240.0.6:32242(unknown)
TTL: 64         Window: 0       Version: 4      Length: 40
FLAGS: --R-A--  SEQ: 0 - ACK: 2102210394
Packet Number: 12

---[ TCP ]-----------------------------------------------------------
10.240.0.6:27550(unknown) -> 10.244.0.61:45678(unknown)
TTL: 126        Window: 8192    Version: 4      Length: 52
FLAGS: -S-----  SEQ: 2102210393 - ACK: 0
Packet Number: 13

---[ TCP ]-----------------------------------------------------------
10.244.0.61:45678(unknown) -> 10.240.0.6:27550(unknown)
TTL: 64         Window: 0       Version: 4      Length: 40
FLAGS: --R-A--  SEQ: 0 - ACK: 2102210394
Packet Number: 14

---[ TCP ]-----------------------------------------------------------
10.240.0.6:65391(unknown) -> 10.244.0.61:45678(unknown)
TTL: 126        Window: 8192    Version: 4      Length: 48
FLAGS: -S-----  SEQ: 2102210393 - ACK: 0
Packet Number: 15

---[ TCP ]-----------------------------------------------------------
10.244.0.61:45678(unknown) -> 10.240.0.6:65391(unknown)
TTL: 64         Window: 0       Version: 4      Length: 40
FLAGS: --R-A--  SEQ: 0 - ACK: 2102210394
Packet Number: 16

---[ TCP ]-----------------------------------------------------------
10.240.0.7:18224(unknown) -> 10.244.0.61:45678(unknown)
TTL: 126        Window: 8192    Version: 4      Length: 52
FLAGS: -S-----  SEQ: 2517447963 - ACK: 0
Packet Number: 17

---[ TCP ]-----------------------------------------------------------
10.244.0.61:45678(unknown) -> 10.240.0.7:18224(unknown)
TTL: 64         Window: 0       Version: 4      Length: 40
FLAGS: --R-A--  SEQ: 0 - ACK: 2517447964
Packet Number: 18

CodePudding user response：

The host at 10.240.0.7 is trying to connect to IP 10.240.0.7 port 45678. That host reports that the port is not open. The process repeats.

Your problem is that there is no process listening on port 45678.

CodePudding user response：

I resolved this by setting externalTrafficPolicy: Local in the LoadBalancer service, rather than the default value of Cluster.