C# FlatBufferBuilder create String from Stream-CodePudding

Suppose you need to read a large string from a stream and you want to put that string into a flatbuffer.

Currently what I do is read the stream into a string and then use the FlatbufferBuilder.CreateString(string s) function.

This works fine but it does have as a drawback that the string is copied and loaded into memory twice: once by reading it from the stream into the string; and then a second time the string is copied into the flatbuffer.

I was wondering if there is a way to fill the flatbuffer string directly from a stream?

For a more concrete example: Suppose your flatbuffer schema looks like:

table Message
   {
   _Data: string;
   }

root_type Message;

We can then create a flatbuffer like this (with myData a string)

var fbb = new FlatBufferBuilder(myData.Length);
var dataOffset = fbb.CreateString(myData);
var message = Message.CreateMessage(fbb, dataOffset);
Message.FinishMessageBuffer(fbb, message);

So the question is can we somehow do the same thing, where myData is a System.IO.Stream?

Obviously the following works, but I'd like to avoid first reading the Stream into memory.

using (var reader = new StreamReader(myStream)
{
  var myData = reader.ReadToEnd();
  var fbb = new FlatBufferBuilder(myData.Length);
  var dataOffset = fbb.CreateString(myData);
  var message = Message.CreateMessage(fbb, dataOffset);
  Message.FinishMessageBuffer(fbb, message);
}

CodePudding user response：

There is currently no way to avoid that copy twice, afaik.. it should be relatively simple to implement a version of CreateString that takes a stream and reduces it to one copy. You could have a go at that and open a PR on github with the result.