Protocol buffers, C# and NetworkStream: messages are never received-CodePudding

I'm trying to use ProtocolBuffers over a NetworkStream but the messages are never fully received.

Here is my server:

var listener = new TcpListener(System.Net.IPAddress.Any, 4989);
listener.Start();

while (true)
{
    var client = listener.AcceptTcpClient();
    Task.Factory.StartNew(() =>
    {
        var message = ServerMessage.Parser.ParseFrom(client.GetStream());
        Console.WriteLine(message);
    });
}

Here is my client:

Thread.Sleep(2000);//Wait for server to start

var client = new TcpClient();
client.Connect("localhost", 4989);

while (true)
{
    var message = new ServerMessage
    {
        Time = (ulong)DateTime.UtcNow.Ticks,
        Type = MessageType.Content
    };
    message.WriteTo(client.GetStream());

    Thread.Sleep(1000);
}

A full repro solution is available here: https://github.com/IanPNewson/ProtobufNetworkStreamIssue

What's going wrong?

CodePudding user response：

protobuf is not a terminated protocol, by which I mean: there is literally nothing that says when an individual message ends. Because of this, by default: APIs like ParseFrom read until the end of the stream, and in an open Socket/NetworkStream: that would only happen if you sent a single message and then closed the outbound connection (not having any more bytes at the moment is not sufficient - it would need to see an actual EOF on the stream, which means you could only send one message per socket).

Because of this, you usually use framing with protobuf, which means: some means of denoting individual messages in an open stream. This is usually done via a length-prefix (you can't use a sentinel value to terminate, because protobuf can contain any byte value).

Some APIs include convenience methods for this (the *WithLengthPrefix APIs in protobuf-net, for example, although you can't just drop that in place here) - or you can implement it yourself manually. Alternatively, perhaps consider something like gRPC which deals with all the framing etc semantics for you, so you can concentrate on doing interesting work. I have a variant of gRPC that works on bare sockets, if you really don't want to deal with the full HTTP/2 side of gRPC.