Convert a C# collection of arrays to a single array efficiently-CodePudding

I've got a ConcurrentQueue<Vector3[]> that I'm filling up and I want to read out of it in chunks for the sake of performance.

private ConcurrentQueue<Vector3[]> positionsQueue = new ConcurrentQueue<Vector3[]>();

For example I want to wait until there's 20 Vector3[]s in queue and then read them as a single, continous array (I need the data to be simplified to a single Vector3[] as I'm passing it to a buffer later on). I am looking for a better method than looping, like a memory copy or sth similar. I tried using CopyTo() that copies queue contents into 2D array, but then I still need to convert 2D array to a merged 1D array.

Vector3[][] positionsFromQueue = new Vector3[positionsQueue.Count][];
positionsQueue.CopyTo(positionsFromQueue, 0);
Vector3[] mergedResult = ???

Tried using System.Buffer.BlockCopy, but I don't think I'm able to achieve what I want with that.

What else could I try to get a merged 1D vector and make this process' performance reasonable?

(btw I'm working in Unity, but asking this questions as a generic C# problem)

CodePudding user response：

You should be able to use BlockCopy or Array.Copy, something like

var source = new Vector3[][];
var target = new Vector3[source.Sum(a => a.Length)];
var position = 0;
foreach(var chunk in source ){
    Buffer.BlockCopy(chunk, 0, target, position, target.Length);
    position  = target.Length;
}

You should however read Array.Copy vs Buffer.BlockCopy, that suggest the difference is minimal, or even negative, compared to a plain loop for a short array. So I'm doubtful if you actually will gain much.

As with everything with performance I would recommend starting with profiling. That should reveal if this is really a problem at all. It is really easy to underestimate how fast computers are at some specific kinds of tasks.

If you really have a huge number of vectors I would probably not recommend using a concurrentQueue, and instead write your own specialized container for your specific use case. For example using double or triple buffering so you can switch buffers between the reader and writer.

If you really want to optimize you code you typically want to avoid any copying at all. So it is fairly common to work with fixed size buffers and using pointer length or Span<T>/Memory<T> to refer to a region of memory.