I've made a neural network and now I need to save the results of the training process into a local file. In total, there are 7,155,264 values. I've tried with a loop like this

string weightsString = "";
string biasesString = "";

for (int l = 1; l < layers.Length; l  )
{
    for (int j = 0; j < layers[l].Length; j  )
    {
        for (int k = 0; k < layers[l - 1].Length; k  )
        {
            weightsString  = weights[l][j, k]   "\n";
        }

        biasesString  = biases[l][j]   "\n";
    }
}

File.WriteAllText(@"path", weightsString   "\n"   biasesString);

But it literally takes forever to go through all of the values. Is there no way to write the contents directly without having to write them in a string first?

(Weights is a double[][,] while biases is a double[][])

CodePudding user response：

First of writing down 7 million datasets will obviously take a lot of time. I'd suggest you split up weights and biases into two files and write them on the fly, no need to store them all in memory until you are done.

using StreamWriter weigthStream = new("weigths.txt", append: true);
using StreamWriter biasStream = new("biases.txt", append: true);

for (int l = 1; l < layers.Length; l  )
{
    for (int j = 0; j < layers[l].Length; j  )
    {
        for (int k = 0; k < layers[l - 1].Length; k  )
        {
            await weightStream.WriteLineAsync(weights[l][j, k]);
        }

        await biasStream.WriteLineAsync(biases[l][j]);
    }
}

CodePudding user response：

1. Bad variant - you can use json serialization

2. So-so variant - write in file immediately. Use File.AppendText

3. IMHO the best variant - use DB

4. IMHO good variant - use BinaryFormatter (you will not be able to read that by yourself, but application will)

5. Working variant - use StringBuilder

CodePudding user response：

StringBuilder weightsSB = new StringBuilder();
StringBuilder biasesSB = new StringBuilder();

for (int l = 1; l < layers.Length; l  )
{
    for (int j = 0; j < layers[l].Length; j  )
    {
        for (int k = 0; k < layers[l - 1].Length; k  )
        {
            weightsSB.Append(weights[l][j, k]   "\n");
        }

        biasesSB.Append(biases[l][j]   "\n");
    }
}

As suggested in the comments, I used a StringBuilder instead. Works like a charm.