Best Way to Save Sensor Data, Split Every x Megabytes in Python-CodePudding

I'm saving sensor data at 64 samples per second into a csv file. The file is about 150megs at end of 24 hours. It takes a bit longer than I'd like to process it and I need to do some processing in real time.

            value = str(milivolts)
            logFile.write(str(datet)   ','   value   "\n")

So I end up with single lines with date and milivolts up to 150 megs. At end of 24 hours it makes a new file and starts saving to it.

I'd like to know if there is a better way to do this. I have searched but can't find any good information on a compression to use while saving sensor data. Is there a way to compress while streaming / saving? What format is best for this?
While saving the sensor data, is there an easy way to split it into x megabyte files without data gaps?

Thanks for any input.

CodePudding user response：

I'd like to know if there is a better way to do this.

One of the simplest ways is to use a logging framework, it will allow you to configure what compressor to use (if any), the approximate size of a file and when to rotate logs. You could start with this question. Try experimenting with several different compressors to see if speed/size is OK for your app.

While saving the sensor data, is there an easy way to split it into x megabyte files without data gaps?

A logging framework would do this for you based on the configuration. You could combine several different options: have fixed-size logs and rotate at least once a day, for example.

Generally, this is accurate up to the size of a logged line, so if the data is split into lines of reasonable size, this makes life super easy. One line ends in one file, another is being written into a new file.

Files also rotate, so you can have order of the data encoded in the file names:

raw_data_<date>.gz
raw_data_<date>.gz.1
raw_data_<date>.gz.2

In the meta code it will look like this:

# Parse where to save data, should we compress data,
# what's the log pattern, how to rotate logs etc
loadLogConfig(...)

# any compression, rotation, flushing etc happens here
# but we don't care, and just write to file
logger.trace(data)

# on shutdown, save any temporary buffer to the files
logger.flush()