Validation acc is very high in each fold but Test acc is very low-CodePudding

I am trying to implement a neural network. I am using CNN model for classifying. First I split the dataset into train and test.

Code Snippet:

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42, shuffle=True, stratify=Y)

then I built a CNN model and used stratified cross-validation to fit the model.

Code Snippet:

from statistics import mean, stdev
# Loop through the splits
lst_accu_stratified = []
for train_index, val_index in skf.split(X_train, y_train):
    X_train_fold, X_val_fold = X_train[train_index], X_train[val_index]
    y_train_fold, y_val_fold = y_train[train_index], y_train[val_index]
    # print('Fold :')
    ResNet50 = model.fit(X_train_fold, y_train_fold, batch_size=16, epochs=20, verbose=1)
    val_loss, val_acc = model.evaluate(X_val_fold, y_val_fold, verbose=0)
    print("Validation Loss: ", val_loss, "Validation Accuracy: ", val_acc)
    lst_accu_stratified.append(val_acc)

# Print the output.
print('List of possible accuracy:', lst_accu_stratified)
print('\nMaximum Accuracy That can be obtained from this model is:',
      max(lst_accu_stratified)*100, '%')
print('\nMinimum Accuracy:',
      min(lst_accu_stratified)*100, '%')
print('\nOverall Accuracy:',
      mean(lst_accu_stratified)*100, '%')
print('\nStandard Deviation is:', stdev(lst_accu_stratified))

Output:

Epoch 1/20
30/30 [==============================] - 9s 102ms/step - loss: 1.3490 - accuracy: 0.5756
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.4620 - accuracy: 0.8466
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1818 - accuracy: 0.9412
Epoch 4/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1106 - accuracy: 0.9727
Epoch 5/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0643 - accuracy: 0.9811
Epoch 6/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0438 - accuracy: 0.9895
Epoch 7/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0371 - accuracy: 0.9916
Epoch 8/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0212 - accuracy: 0.9958
Epoch 9/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0143 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0149 - accuracy: 0.9958
Epoch 11/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0158 - accuracy: 0.9958
Epoch 12/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0134 - accuracy: 0.9958
Epoch 13/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0072 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0031 - accuracy: 1.0000
Epoch 15/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0024 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0019 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0088 - accuracy: 0.9979
Epoch 20/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0031 - accuracy: 1.0000
Validation Loss:  0.8360670208930969 Validation Accuracy:  0.800000011920929
Epoch 1/20
30/30 [==============================] - 3s 106ms/step - loss: 0.5129 - accuracy: 0.8700
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.4789 - accuracy: 0.8784
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 0.2724 - accuracy: 0.9224
Epoch 4/20
30/30 [==============================] - 2s 72ms/step - loss: 0.2108 - accuracy: 0.9308
Epoch 5/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1081 - accuracy: 0.9706
Epoch 6/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1010 - accuracy: 0.9748
Epoch 7/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0481 - accuracy: 0.9895
Epoch 8/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0316 - accuracy: 0.9874
Epoch 9/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0483 - accuracy: 0.9811
Epoch 10/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0167 - accuracy: 0.9937
Epoch 11/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0129 - accuracy: 0.9937
Epoch 12/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0023 - accuracy: 1.0000
Epoch 13/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0024 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0093 - accuracy: 0.9979
Epoch 15/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0389 - accuracy: 0.9895
Epoch 16/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0293 - accuracy: 0.9895
Epoch 17/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7058e-04 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0011 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7595e-04 - accuracy: 1.0000
Validation Loss:  0.5674645304679871 Validation Accuracy:  0.8571428656578064
Epoch 1/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1533 - accuracy: 0.9518
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0978 - accuracy: 0.9686
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0702 - accuracy: 0.9790
Epoch 4/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0754 - accuracy: 0.9811
Epoch 5/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0362 - accuracy: 0.9874
Epoch 6/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0174 - accuracy: 0.9916
Epoch 7/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0144 - accuracy: 0.9916
Epoch 8/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0089 - accuracy: 0.9958
Epoch 9/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0017 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0044 - accuracy: 0.9979
Epoch 11/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0033 - accuracy: 1.0000
Epoch 12/20
30/30 [==============================] - 2s 73ms/step - loss: 5.9884e-04 - accuracy: 1.0000
Epoch 13/20
30/30 [==============================] - 2s 73ms/step - loss: 3.7875e-04 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 73ms/step - loss: 4.7657e-04 - accuracy: 1.0000
Epoch 15/20
30/30 [==============================] - 2s 73ms/step - loss: 2.8062e-04 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 73ms/step - loss: 4.5594e-04 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 72ms/step - loss: 2.3471e-04 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 72ms/step - loss: 2.5190e-04 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 72ms/step - loss: 1.5143e-04 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 72ms/step - loss: 2.4174e-04 - accuracy: 1.0000
Validation Loss:  0.002929181093350053 Validation Accuracy:  1.0
Epoch 1/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0035 - accuracy: 1.0000
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0048 - accuracy: 0.9979
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 7.1234e-04 - accuracy: 1.0000
Epoch 4/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0100 - accuracy: 0.9937
Epoch 5/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0041 - accuracy: 1.0000
Epoch 6/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 7/20
30/30 [==============================] - 2s 71ms/step - loss: 6.2473e-04 - accuracy: 1.0000
Epoch 8/20
30/30 [==============================] - 2s 72ms/step - loss: 4.5511e-04 - accuracy: 1.0000
Epoch 9/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0015 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0132 - accuracy: 0.9979
Epoch 11/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0106 - accuracy: 0.9958
Epoch 12/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0032 - accuracy: 0.9979
Epoch 13/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0022 - accuracy: 0.9979
Epoch 14/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0039 - accuracy: 0.9979
Epoch 15/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0023 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 73ms/step - loss: 2.7678e-04 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0022 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0034 - accuracy: 0.9979
Epoch 19/20
30/30 [==============================] - 2s 73ms/step - loss: 4.1879e-04 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 72ms/step - loss: 4.0388e-04 - accuracy: 1.0000
Validation Loss:  0.003368004923686385 Validation Accuracy:  1.0
Epoch 1/20
30/30 [==============================] - 2s 72ms/step - loss: 5.1283e-04 - accuracy: 1.0000
Epoch 2/20
30/30 [==============================] - 2s 72ms/step - loss: 8.4923e-04 - accuracy: 1.0000
Epoch 3/20
30/30 [==============================] - 2s 72ms/step - loss: 3.2774e-04 - accuracy: 1.0000
Epoch 4/20
30/30 [==============================] - 2s 72ms/step - loss: 1.3468e-04 - accuracy: 1.0000
Epoch 5/20
30/30 [==============================] - 2s 72ms/step - loss: 1.4561e-04 - accuracy: 1.0000
Epoch 6/20
30/30 [==============================] - 2s 72ms/step - loss: 1.6656e-04 - accuracy: 1.0000
Epoch 7/20
30/30 [==============================] - 2s 71ms/step - loss: 1.2794e-04 - accuracy: 1.0000
Epoch 8/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7647e-05 - accuracy: 1.0000
Epoch 9/20
30/30 [==============================] - 2s 71ms/step - loss: 1.7325e-04 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 72ms/step - loss: 6.5071e-05 - accuracy: 1.0000
Epoch 11/20
30/30 [==============================] - 2s 72ms/step - loss: 6.1966e-05 - accuracy: 1.0000
Epoch 12/20
30/30 [==============================] - 2s 71ms/step - loss: 5.9293e-05 - accuracy: 1.0000
Epoch 13/20
30/30 [==============================] - 2s 71ms/step - loss: 3.1360e-04 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 71ms/step - loss: 1.0051e-04 - accuracy: 1.0000
Epoch 15/20
30/30 [==============================] - 2s 71ms/step - loss: 1.7242e-04 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 71ms/step - loss: 5.6384e-05 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 71ms/step - loss: 8.4639e-05 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7929e-04 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 71ms/step - loss: 1.6557e-04 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 71ms/step - loss: 4.6414e-04 - accuracy: 1.0000
Validation Loss:  8.931908087106422e-05 Validation Accuracy:  1.0
List of possible accuracy: [0.800000011920929, 0.8571428656578064, 1.0, 1.0, 1.0]

Maximum Accuracy That can be obtained from this model is: 100.0 %

Minimum Accuracy: 80.0000011920929 %

Overall Accuracy: 93.1428575515747 %

Standard Deviation is: 0.09604420178372833

here the val accuracy of each fold is pretty high but when I test the model with test dataset, the accuracy is very low.

Code snippet:

model.evaluate(X_test, y_test,batch_size=32)

output:

5/5 [==============================] - 1s 222ms/step - loss: 2.3315 - accuracy: 0.6913
[2.3314528465270996, 0.6912751793861389]

My question is,

Is my method correct?
What can be the reason for low test accuracy?

CodePudding user response：

Overfitting

You are overfitting heavily, your model is too powerful for your problem. So you manage to overfit on each of your splits having awesome training and validation loss.

The good news is that is perfectly fine when following a general recipe for reaching a nicely trained deep neural network.

Regularize

You just need to follow through with the next steps and regularize your model. You have multiple options (see details here and some nice heuristics in this book here) for that:

get more training data
augment your data
use a pretrained model
reduce your input dimensionality
use a smaller model
use dropout
use early stopping (can be done with a large model)

CodePudding user response：

I totally agree with overfitting situations but wanted to state one more possible reason. Your training is really fast so imagine your inputs are kinda small. If model has BatchNorm (as far as I know ResNet50 has), I would recommend to increase batch size as BatchNorm goes a bit crazy in these kind of situations.