[Problem]

System.Net.Sockets.Socket's async receiving corrupts the receiving buffer, injecting a random burst of bytes into the middle of the received data.

[Background]

I have a TCP connections server. Clients establish socket connection with the server, and send encrypted packets to server.

Initially, the server operated using threads. Every incoming connection is served by a dedicated thread. All buffers to be sent and received via sockets are allocated on the heap and garbage collected. Everything worked fine.

Then, to improve performance, I did two changes:

1. Instead of allocating buffers on the heap, I allocated from using ArrayPool.Shared.Rent().

2. I changed the socket's sending and receiving to async.

I did not change how packets are serialized, deserialized, encrypted and decrypted.

After the change, everything works fine on my development PC. When deployed to production server on AWS, when packets are small (50 bytes or so), everything was fine.

When the packet size is large, 16 KB each, the server will receive a few of such packets fine, then, randomly, one of the received packets will be corrupted, with a burst of a thousand or so unknow bytes injected in the middle of the received data, causing the AES decryption to throw the "invalid padding" exception:

I have no idea where the injected extra bytes come from.

Following are the socket sending and receiving code:

    public static Task<int> ReceiveBytesAsync(this Socket socket, SocketAsyncEventArgs args) // , int offset, int count)
    {
        var tcs = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
        EventHandler<SocketAsyncEventArgs> handler = null;

        handler = (s, e) =>
        {
            args.Completed -= handler;

            if (args.SocketError != SocketError.Success)
                tcs.SetException(new InvalidOperationException(args.SocketError.ToString()));
            else
                tcs.SetResult(args.BytesTransferred);
        };

        args.Completed  = handler;

        if (!socket.ReceiveAsync(args))
        {
            args.Completed -= handler;
            if (args.SocketError != SocketError.Success)
                tcs.SetException(new InvalidOperationException(args.SocketError.ToString()));
            else
                tcs.SetResult(args.BytesTransferred);
        }

        return tcs.Task;
    }

    private static async Task<bool> ReadBufferFromSocketAsync(Socket socket, SocketAsyncEventArgs socketArgs) // , int offset, int length)
    {
        for (int bytesRead = 0; bytesRead < socketArgs.Count;)
        {
            // If bytesRead is 0, SetBuffer would have been called by the caller of this method.
            if (bytesRead > 0)
                socketArgs.SetBuffer(socketArgs.Offset   bytesRead, socketArgs.Count - bytesRead);

            int nRead = await socket.ReceiveBytesAsync(socketArgs);
            bytesRead  = nRead;

            if (nRead == 0)
            {
                socket.Close();
                return false;
            }
        }

        return true;
    }

    public static Task<int> SendBytesAsync(this Socket socket, SocketAsyncEventArgs args) // , int offset, int count)
    {
        var tcs = new TaskCompletionSource<int>(TaskCreationOptions.RunContinuationsAsynchronously);
        EventHandler<SocketAsyncEventArgs> handler = null;

        handler = (s, e) =>
        {
            args.Completed -= handler;
            if (args.SocketError != SocketError.Success)
                tcs.SetException(new InvalidOperationException(args.SocketError.ToString()));
            else
                tcs.SetResult(args.BytesTransferred);
        };

        args.Completed  = handler;

        if (!socket.SendAsync(args)) // means the operation completed synchronously & Completed handler won't fire
        {
            args.Completed -= handler;
            if (args.SocketError != SocketError.Success)
                tcs.SetException(new InvalidOperationException(args.SocketError.ToString()));
            else
                tcs.SetResult(args.BytesTransferred);
        }

        return tcs.Task;
    }

    private static async Task<bool> SendBufferToSocketAsync(Socket socket, SocketAsyncEventArgs socketArgs)
    {
        int retry = 0;

        for (int ixWrite = 0; ixWrite < socketArgs.Count;)
        {
            if (ixWrite > 0)
                socketArgs.SetBuffer(socketArgs.Offset   ixWrite, socketArgs.Count - ixWrite);

            int nSent = await socket.SendBytesAsync(socketArgs);
            ixWrite  = nSent;

            if (retry >= 10)
            {
                socket.Close();
                return false;
            }
            if (nSent == 0)
            {
                retry  ;
                await Task.Delay(10 * retry);
            }
            else
                retry = 0;
        }

        return true;
    }

CodePudding user response：

Problem disappeared after I changed the ReadBufferFromSocketAsync to the following

    private static async Task<bool> ReadBufferFromSocketAsync(Socket socket, byte[] buffer, int offset, int totalToRead) 
    {
        using (SocketAsyncEventArgs socketArgs = new SocketAsyncEventArgs())
        {
            int iAllRead = 0, iOneRead = 0;
            socketArgs.SetBuffer(buffer, offset, totalToRead);

            while (true)
            {
                iOneRead = await socket.ReceiveBytesAsync(socketArgs);

                if (iOneRead == 0)
                {
                    socket.Close();
                    return false;
                }

                iAllRead  = iOneRead;

                if (iAllRead == totalToRead)
                    return true;

                offset  = iOneRead;
                socketArgs.SetBuffer(offset, totalToRead - iAllRead);
            }
        }
    }