I am working on C and trying to reduce network traffic of Flight RPC by compressing the RecordBatch object.

What's the best way to make a RecordBatch compact? I found that RecordBatch consists of Array. Is it good to directly compress ArrayData?

CodePudding user response：

With the caveat that in my own testing, I've never found compression to help substantially, Arrow already supports compression and this can be enabled (with some finagling) by setting IPC options.

For example, if you have a DoGet, pass an IpcWriteOptions with compression enabled to RecordBatchStream. Then Arrow/Flight will compress body buffers for you and the other end (assuming it supports compression) will transparently decompress them.