Casting golang byte array to node json-CodePudding

Hi I have two services one in golang other in nodejs

Golang one is producing to kafka with the following code

var network bytes.Buffer        
enc := gob.NewEncoder(&network)
enc.Encode(product)
produce_to_kafka("add", network.Bytes())

And node is consuming with following

await consumer.run({
  eachMessage: async ({ topic, partition, message, heartbeat }) => {
      console.log({
          key: message.key.toString(),
          value: new Buffer.from(message.value, "binary").toString('ascii'),
          headers: message.headers,
      })
  },
})

As you can see I've tried casting with buffer.from with ascii, binary and base64 but wasn't successful in doing so.

This is the console.log I'm getting in nodejs

[categories]   key: 'add',
[categories]   value: '\x7FO\x7F\x01\x03\x01\x01\x07Product\x01\x7F\x02\x00\x01\x0F\x01\x04Name\x01\f\x00\x01\x02Id\x01\f\x00\x01\x07Storeid\x01\f\x00\x01\tStoreProd\x01\f\x00\x01\x05Price\x01\b\x00\x01\bDiscount\x01\b\x00\x01\x05Brand\x01\f\x00\x01\n'  
[categories]     'Categories\x01\x7F\x04\x00\x01\x04Tags\x01\x7F\x04\x00\x01\bLocation\x01\f\x00\x01\x07InStock\x01\x04\x00\x01\n'  
[categories]     'ExpiryDate\x01\f\x00\x01\tTotalSold\x01\x04\x00\x01\x11LastStockAddition\x01\f\x00\x01\x04Cost\x01\x04\x00\x00\x00\x16\x7F\x03\x02\x01\x01\b[]string\x01\x7F\x04\x00\x01\f\x00\x00i\x7F\x02\x01\x04Stng\x01\x05Stngp\x01\x01t\x01\x06tStngp\x01@\x01@\x01\x06String\x01\x01\x05askdl\x01\x01\x05asdas\x01\x06String\x01\x04\x01\x102021-10-03 12:13\x01\x06\x01\x102021-10-03 12:13\x01\b\x00',
[categories]   headers: {}
[categories] }

Moreover I have a consume in golang for same topic and can successfully parse that back to struct with the following code

var network bytes.Buffer;
network.WriteString(string(ev.Value))
dec := gob.NewDecoder(&network) 
var q model.Product
err = dec.Decode(&q)

Can you kindly share any resource or syntax how can I do this in nodejs

CodePudding user response：

You could be able to decode your binary encoding string with the NodeJS Buffer API.

So in your case, it should give something like this:

var binaryString = '\x7FO\x7F\x01\x03\x01\x01\x07Product\x01\x7F\x02\x00\x01\x0F\x01\x04Name\x01\f\x00\x01\x02Id\x01\f\x00\x01\x07Storeid\x01\f\x00\x01\tStoreProd\x01\f\x00\x01\x05Price\x01\b\x00\x01\bDiscount\x01\b\x00\x01\x05Brand\x01\f\x00\x01\nCategories\x01\x7F\x04\x00\x01\x04Tags\x01\x7F\x04\x00\x01\bLocation\x01\f\x00\x01\x07InStock\x01\x04\x00\x01\nExpiryDate\x01\f\x00\x01\tTotalSold\x01\x04\x00\x01\x11LastStockAddition\x01\f\x00\x01\x04Cost\x01\x04\x00\x00\x00\x16\x7F\x03\x02\x01\x01\b[]string\x01\x7F\x04\x00\x01\f\x00\x00i\x7F\x02\x01\x04Stng\x01\x05Stngp\x01\x01t\x01\x06tStngp\x01@\x01@\x01\x06String\x01\x01\x05askdl\x01\x01\x05asdas\x01\x06String\x01\x04\x01\x102021-10-03 12:13\x01\x06\x01\x102021-10-03 12:13\x01\b\x00';

const data = Buffer.from(binaryString, "binary");

console.log(data);
console.log(data.toString());

However, as mentioned in the comments, I don't know if the Go gob encoder and NodeJS binary decoder works well in all situations. Maybe it could be better to use another format.

Hope it helps anyway.

CodePudding user response：

The gob format is meant to be used primarily between Go services.

I'd recommend that you use a cross compatible binary format such as Protobuf or Avro, which are commonly used with Kafka, if not JSON.