I want to use a third party dll function which requires a stream input.

The data I need to feed it is provided by a different third party dll function, which only offers access to the source data by using a 'ReadBuffer' option to obtain chunks of data at a time, by populating a byte array of a set length.

The data I'm reading exceeds several TB, so I'm unable to just write a loop and write all the data to memory and then into a stream.

Is their a simple way to create a stream from data which is being read into a byte array buffer within a while loop as the stream is read?

I'm writing in C# & thanks for any pointers

Thanks

CodePudding user response：

You can use a MemoryStream in C# to create a stream using a byte array buffer. You can use the ReadBuffer method from the third party DLL to populate a byte array, and then use the Write method from the MemoryStream to write the byte array to the stream.

///
using (var memoryStream = new MemoryStream())
{
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = thirdPartyDll.ReadBuffer(buffer, 0, buffer.Length)) > 0)
{
    memoryStream.Write(buffer, 0, bytesRead);
}
}

You can use an extra BufferedStream to wrap around the MemoryStream and buffer more data, so you don't have to write each byte array to the stream one by one.

 ///
using (var memoryStream = new MemoryStream())
using (var bufferedStream = new BufferedStream(memoryStream))
{
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = thirdPartyDll.ReadBuffer(buffer, 0, buffer.Length)) > 0)
{
    bufferedStream.Write(buffer, 0, bytesRead);
}}

CodePudding user response：

You should inherit the Stream Class. It is possible your library offers all the data from an inner Stream you may read and set in your derived class (Current position, Data Length, if can be readed, etc).

This way you can create a new object and use in the other library

public class MyStream : Stream
{
    private LibraryClient _client;

    // Wrapper
    public MyStream(LibraryClient libraryClient)
    {
        buffer = new byte[bufferSize];
        _client = libraryClient;
    }

    // Return the client length
    public override long Length => _client.DataLength;
    

    // Specify the position in your buffer, if the client has this info, reference it
    public override long Position { get; set; }


    public override int Read(byte[] buffer, int offset, int count)
    {
        // temp array in local var for example. you may instance it or rent it as well
        var tmp = new byte[count - offset];

        _client.ReadBuffer(tmp, Position, count);

        //copy all (we skip count check because it's local variable)
        tmp.CopyTo(buffer, offset);
        return count - offset; //if the library 'ReadBuffer' returns the length readed, place it here

    }


    public override long Seek(long offset, SeekOrigin origin)
    {
        //... this method moves the Position of the Stream
        //use it to update the inner position
        if (offset > _client.DataLength) throw new ArgumentOutOfRangeException();
        long tempPosition = 0;
        switch (origin)
        {
            case SeekOrigin.Begin:
                {
                    tempPosition = offset;
                    break;
                }
            case SeekOrigin.Current:
                {
                    tempPosition = Position   offset;
                    Position = tempPosition;
                    break;
                }
            case SeekOrigin.End:
                {
                    tempPosition = _client.DataLength   offset;
                    break;
                }
        }

        if (tempPosition < 0) throw new IOException("Offset too backward");
        if (tempPosition > _client.DataLength) throw new IOException("Offset too foward");

        Position = tempPosition;

        return offset;
    }

    public override void SetLength(long value)
    {
        // ... handle if neccesary
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        // ... handle if neccesary
    }

    public override void Flush()
    {
        // handle if neccesary
    }

    // Modify if neccesary
    public override bool CanWrite => false;
    public override bool CanSeek => true;
    public override bool CanRead => Position < Length;
}