For precomputation purposes I need to store a large number (tens of millions) of single-precision floating point numbers in a comma-separated file.

My performance priorities are, in this order:

1. Reading speed
2. File size
3. Writing speed

Right now I am just writing the string representations of the numbers, which can obviously be improved upon. What would be some good ways to accomplish this?

CodePudding user response：

The fastest and most efficient solution (file size) is storing binary data in binary format. In rough lines (not optimized in any way):

private static IEnumerable<float> read()
{
    using (var file = new FileStream(//whatever)
    using (var reader = new BinaryReader(file))
    {
        while (file.Position < file.Length)
            yield return reader.ReadSingle();
    }
}

private static void write(IEnumerable<float> ff)
{
    using (var file = new FileStream(//whatever)
    using (var writer = new BinaryWriter(file))
    {
        foreach (var f in ff)
            writer.Write(f);
    }
}

If you need to keep a csv format then there really isn't any option around writing and reading text representations of the floats, which makes the process considerably slower and the file size can be orders of magnitude larger.