TCP connections are stopped in ASP.NET Azure-CodePudding

I have a real head scratcher here (for me).

I have the following setup:

Kubernetes Cluster in Azure (linux VMs)
ASP.NET docker image with TCP server
Software simulating TCP clients
RabbitMQ for notifying incoming messages

Peer behaviour:

The client sends its heartbeat every 10 minutes
The server sends a keep-alive every 5 minutes (nginx-ingress kills connections after being idle for ~10 minutes)

I am testing the performance of my new TCP server. The previous one, written in Java, could easily handle the load I am about to explain. For some reason, the new TCP server, written in C#, loses the connection after about 10-15 minutes.

Here is what I do:

Use the simulator to start 500 clients with a ramp-up of 300s
All connections are there established correctly
Most of the time, the first heartbeats and keep-alives are sent and received
After 10 minutes, I receive 0 bytes from Stream.EndRead() on BOTH ends of the connection.

This is the piece of code that is triggering the error.

var numberOfBytesRead = Stream.EndRead(result);
if (numberOfBytesRead == 0)
{
    This.Close("no bytes read").Sync(); //this is where I end up
    return;
}

In my logging on the server side, I see lots of disconnected ('no bytes read') lines and a lot of exceptions indicating that RabbitMQ is too busy: None of the specified endpoints were reachable.

My guesses would be that the Azure Load Balancer just bounces the connections, but that does not happen with the Java TCP server. Or that the ASP.NET environment is missing some configuration.

Does anyone know how this is happening, and more important, how to fix this?

--UPDATE #1--

I just used 250 devices and that worked perfectly.
I halved the ramp-up and that was a problem again. So this seems to be a performance issue. A component in my chain is too busy.

--UPDATE #2--

I disabled the publishing to RabbitMQ and it kept working now. Now I have to fix the RabbitMQ performance.

CodePudding user response：

I ended up processing the incoming data in a new Task. This is my code now:

public void ReceiveAsyncLoop(IAsyncResult? result = null)
{
    try
    {
        if (result != null)
        {
            var numberOfBytesRead = Stream.EndRead(result);
            if (numberOfBytesRead == 0)
            {
                This.Close("no bytes read").Sync();
                return;
            }

            var newSegment = new ArraySegment<byte>(Buffer.Array!, Buffer.Offset, numberOfBytesRead);

            // This.OnDataReceived(newSegment)); <-- previously this
            Task.Run(() => This.OnDataReceived(newSegment));
        }

        Stream.BeginRead(Buffer.Array!, Buffer.Offset, Buffer.Count, ReadingClient.ReceiveAsyncLoop, null);
    }
    catch (ObjectDisposedException) { /*ILB*/ } 
    catch (Exception ex)
    {
        Log.Exception(ex, $"000001: {ex.Message}");
    }
}

Now, everything is super fast.