When I use more data to train a tensorflow NN model, some unexpected thing happened-CodePudding

I try to this exercise of tensorflow, when I give xs and ys higher value during training, the loss starts increasing. Why this happens? Am I doing something wrong? It looks OK when max is 19 but not for 20.

import tensorflow as tf
import numpy as np
from tensorflow import keras


def main():
    print(tf.__version__)
    model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])

    model.compile(optimizer='sgd', loss="mean_squared_error")

    min = 0
    max = 20 # the loss will increse if gives more data
    xs = np.array([float(x) for x in range(min, max)], dtype=np.float32)
    ys = np.array([float(3 * x   1) for x in range(min, max)], dtype=np.float32)
    print(xs, sep=',')
    print(ys, sep=',')
    
    model.fit(xs, ys, epochs=10)
    print(model.predict([100]))
    
if __name__ == "__main__":
    main()

Results:

Epoch 1/10
1/1 [==============================] - 0s 85ms/step - loss: 1813.5785
Epoch 2/10
1/1 [==============================] - 0s 444us/step - loss: 3997.2852
Epoch 3/10
1/1 [==============================] - 0s 392us/step - loss: 8810.5635
Epoch 4/10
1/1 [==============================] - 0s 349us/step - loss: 19419.8730
Epoch 5/10
1/1 [==============================] - 0s 315us/step - loss: 42804.6797
Epoch 6/10
1/1 [==============================] - 0s 309us/step - loss: 94348.9844
Epoch 7/10
1/1 [==============================] - 0s 365us/step - loss: 207961.6875
Epoch 8/10
1/1 [==============================] - 0s 414us/step - loss: 458384.1875
Epoch 9/10
1/1 [==============================] - 0s 494us/step - loss: 1010359.8125
Epoch 10/10
1/1 [==============================] - 0s 347us/step - loss: 2227010.5000

CodePudding user response：

I tried to change some configurations, and the single biggest change was changing the learning rate from the default 0.01 to 0.001. To do that change

model.compile(optimizer="sgd",.....)

model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=0.001),.....

It uses the same optimizer, but allows you to set a learning rate.

Too big a learning rate will make you shoot over the local minima and might make your loss explode. Learning rate is a good place to start tweaking when your loss explodes.

CodePudding user response：

The problem is not about max value being 19 or 20.

Use 2 neurons instead of 1 in the Dense layer like below:

model = tf.keras.Sequential([keras.layers.Dense(units=2, input_shape=[1])])

Maybe 1 neuron can't handle that regression function.