Unity application for HoloLens potentially overloaded using Sockets, but not receiving any exception-CodePudding

I'm currently working on a research project that involves receiving real-time data (up to 10 cycles/second) from an external scanner and displaying it on a HoloLens 2 using Unity. Currently, the system is separated into two parts:

Python server that sends a packet with data from the external scanner over a TCP socket
HoloLens client that takes the packet's processed data and displays it to the user

The HoloLens portion is built to be an asynchronous socket client, and once it receives a data packet it will update some UI and GameObject components accordingly. All of the data that I'm sending is simple JSON strings that use my own packet structure, and I've been able to successfully receive the scanner's data over the socket by the HoloLens, but it seems that once the scanner starts sending too much data the HoloLens stops accepting new packets and stops processing anything.

This is strange, as tests from running the same application in the Unity Editor on my computer do not show these problems. I understand that this is a race/infinite polling condition, but I am currently using the C# Socket.BeginReceive() API to asynchronously wait for new data, and from my understanding of the MSDN documentation this should handle any incoming data with separate threads. I don't understand where on the HoloLens the problem is coming from, as no exceptions get thrown from the debug logs and the system works with smaller data sizes (closer to 2 cycles a second). Another idea could be that the buffer for the socket could hold >1 packet and is failing to parse correctly, but wouldn't that produce a Newtonsoft.Json exception?

Here is a snippet of my HoloLens client:

    public class Manager : MonoBehaviour
    {
        public string serverSocketEndpoint = "localhost";
        public string serverCommandEndpoint = "localhost:5000/command";

        private Dictionary<string, PacketCategoryData> _tagCounts;

        private bool _connected = false;
        private static Socket _socket;
        private byte[] _socketBuffer = new byte[8192];

        private Thread _socketThread;
        
        void Start()
        {
            StartSocketConnection();
        }


        public void StartSocketConnection() {
            try
            {
                IPAddress ipAddr = Dns.GetHostAddresses(serverSocketEndpoint)[1];
            
                IPEndPoint writeEndPoint = new IPEndPoint(ipAddr, 12345);
                IPEndPoint readEndPoint = new IPEndPoint(ipAddr, 12346);
            
                _socket = new Socket(ipAddr.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
                _socket.NoDelay = true;
                _socket.ReceiveBufferSize = 8192;
                _socket.Connect(writeEndPoint);
                
                Debug.Log(String.Format("Socket connected to {0} ", _socket.RemoteEndPoint.ToString()));
                
                _socket.BeginReceive(_socketBuffer, 0, _socketBuffer.Length, SocketFlags.None, OnDataReceived, _socket);
            }
            
            catch (Exception e)
            {
                Debug.Log(e.ToString());
                debugMenu.UpdateStats();
            } 
        }
        

        private void OnDataReceived(IAsyncResult result)
        {
            Socket connection = result.AsyncState as Socket;
            int received = connection.EndReceive(result);

            Debug.Log("Got packet data");
            
            // Handle received data in buffer.
            string incomingData = Encoding.ASCII.GetString(_socketBuffer, 0, received);
            DataPacket packet = JsonConvert.DeserializeObject<DataPacket>(incomingData);
            
            // Start a new async receive on the client to receive more data.
            _socket.BeginReceive(
                _socketBuffer, 
                0,
                _socketBuffer.Length,
                SocketFlags.None,
                OnDataReceived,
                _socket);
            
            // Parse the previous packet that we picked up
            ParseIncomingPacket(packet);
        }
        
        
        public void ParseIncomingPacket(DataPacket packet)
        {
            try
            {
                if (packet != null)
                {
                    if (packet.type.Equals("count_update"))
                    {
                        // Do some data processing and UI updating on the main thread...
                        // This particular place is where things go wrong, or atleast this general area
                        // Until I surpass the threshold of about 2 or 3 cycles a second, everything is
                        // being processed correctly, but as soon as I surpass that limit everything just stops
                        // with no errors

                        UnityMainThreadDispatcher.Instance().Enqueue(() =>
                        {
                            Logger.Log("[Info] Got count_update response");
                        });
                        
                        _connected = true;
                        
                        var json = JsonConvert.SerializeObject(packet.data);
                        Dictionary<string, PacketCategoryData> categoryCounts = 
                            JsonConvert.DeserializeObject<Dictionary<string, PacketCategoryData>>(json);

                        // Since this is Unity game object specific code, it must be ran on the main thread
                        UnityMainThreadDispatcher.Instance().Enqueue(CountDataCallback());
                    }
                }
                else
                {
                    //
                    // Exception conditions...
                    //
                }
            }
            
            catch (Exception e)
            {
                UnityMainThreadDispatcher.Instance().Enqueue(() =>
                {
                    Debug.LogError(String.Format("Unexpected Exception : {0}", e.ToString()));
                    _connected = false;
                    debugMenu.UpdateStats();
                });
            }
        }


        private IEnumerator CountDataCallback() {
            UpdateTagCountUI();
            debugMenu.UpdateStats();
            yield return null;
        }
    }

Any help would be greatly appreciated.

Edit: I am using an external library for the UnityMainThreadDispatcher, as the BeginRecieve operates on different child threads, Unity only accepts game object/Unity-specific method calls to be made on the main thread. This is intended to circumvent that limitation and allow for my scene objects/UI to be updated once we receive a data packet

CodePudding user response：

As mentioned in the comments I can imagine the issue being your external library sending a wast amount of updates with a very high frame-rate

=> Your Unity app collects all those updates into the Queue of the UnityMainThreadDispatcher whih then has to iterate over all of them in

private void Update()
{
    while (_executionQueue.Count > 0) 
    {
        _executionQueue.Dequeue().Invoke();
    }
}

so imagine receiving 10 updates from your external library within one frame

=> Your Unity app has to update the UI with 10 redundant iterations since anyway only the very last one will be ever visible to the user!

At the same time it delays this frame => Within the next frame your pile up even more items into the Queue

=> Your entire app stales more and more the longer the external library keeps spamming it with update packets.

In your use case you seemingly do not really care about an ensured sequential and continuous receiving and processing of the data.

What you rather want is your UI only process and display the very latest received data state.

Thus, I would not use a Queue but rather a ConcurrentStack and only process the latest received DataPacket for this frame, otherwise basically replicate the structure of the dispatcher like e.g.

// A thread-safe Stack
private readonly ConcurrentStack<DataPacket> _lastReceivedDataPackets = new ();

private void Update()
{
    if(_lastReceivedDataPackets.TryPop(out var dataPacket))
    {
        // TODO: Put your UI update here
        UpdateUIWithReceivedData(dataPacket);

        // We don't care about any previous received data -> simply erase them
        _lastReceivedDataPackets.Clear();
    }
}

private void ParseIncomingPacket(DataPacket packet)
{
    ...

    // whatever is the last call to this before the next frame will determine what is displayed in the next frame
    _lastReceivedDataPackets.Push(packet);  
           
    ...
}