Keras validation accuracy is 1 but only predicts only class 0 in binary classification?-CodePudding

Using these same stratified training and test splits I'm getting >99% true predictions using Linear SVC and Decision Tree with the following configurations:

LinearSVC(C = 0.025, tol = 5e-05, random_state = 21)
DecisionTreeClassifier(max_depth=3, random_state = 21)

I also tested the above two models on a completely separate dataset with 1000 rows, they're working near perfectly. Why is the Neural Net below failing? Only predicts 0. Using Standard Scaler and Min Max scaler has no effect and only reduces the accuracy of the other two models.

print('All dtypes: float64')
print('')
print('X_train maximum value: {0}\nX_train minimum value: {1}'.format(X_train.max().max(), X_train.min().min()))
print('y_train maximum value: {0}\ny_train minimum value: {1}'.format(y_train.max(), y_train.min()))
print('')
print('X_test maximum value: {0}\nX_test minimum value: {1}'.format(X_test.max().max(), X_test.min().min()))
print('y_test maximum value: {0}\ny_test minimum value: {1}'.format(y_test.max(), y_test.min()))
print('')
print('X_train shape: {0}'.format(X_train.shape))
print('y_train shape: {0}'.format(y_train.shape))
print('')
print('X_test shape: {0}'.format(X_test.shape))
print('y_test shape: {0}'.format(y_test.shape))

model_net = Sequential()

model_net.add(Dense(12, activation = 'relu', input_dim = 24))
model_net.add(Dense(1, activation = 'sigmoid'))

print(model_net.summary())
print('')
model_net.compile(optimizer='Adam', loss='binary_crossentropy', metrics=['accuracy'])

epochs = 5
batch_size = 64

run_hist = model_net.fit(X_train, y_train,
                          batch_size = batch_size, 
                          epochs = epochs, verbose = 1, 
                          validation_split = 0.2, validation_data = (X_test, y_test))

y_pred = model_net.predict(X_test).argmax(1)

accuracy = metrics.accuracy_score(y_true=y_test, y_pred=y_pred)
print('')
print('Accuracy: {0}'.format(round(accuracy, 2)))
print('')
      
unique, counts = np.unique(y_pred, return_counts=True)
print('\nUnique prediction values:')
np.asarray((unique, counts)).T

Output:

All dtypes: float64

X_train maximum value: 21781.0
X_train minimum value: 0.0
y_train maximum value: 1.0
y_train minimum value: 0.0

X_test maximum value: 7730.25
X_test minimum value: 0.0
y_test maximum value: 1.0
y_test minimum value: 0.0

X_train shape: (66064, 24)
y_train shape: (66064,)

X_test shape: (16516, 24)
y_test shape: (16516,)
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 12)                300       
                                                                 
 dense_1 (Dense)             (None, 1)                 13        
                                                                 
=================================================================
Total params: 313
Trainable params: 313
Non-trainable params: 0
_________________________________________________________________
None

Epoch 1/5
1033/1033 [==============================] - 5s 3ms/step - loss: 0.3795 - accuracy: 0.9834 - val_loss: 0.0015 - val_accuracy: 0.9998
Epoch 2/5
1033/1033 [==============================] - 3s 3ms/step - loss: 9.6806e-04 - accuracy: 0.9999 - val_loss: 5.3518e-04 - val_accuracy: 0.9999
Epoch 3/5
1033/1033 [==============================] - 4s 3ms/step - loss: 5.0223e-04 - accuracy: 1.0000 - val_loss: 3.1895e-04 - val_accuracy: 0.9999
Epoch 4/5
1033/1033 [==============================] - 3s 3ms/step - loss: 1.8808e-04 - accuracy: 1.0000 - val_loss: 2.7151e-04 - val_accuracy: 0.9999
Epoch 5/5
1033/1033 [==============================] - 3s 3ms/step - loss: 2.3315e-04 - accuracy: 1.0000 - val_loss: 1.0257e-04 - val_accuracy: 1.0000
517/517 [==============================] - 1s 1ms/step

Accuracy: 0.23


Unique prediction values:
array([[    0, 16516]], dtype=int64)

CodePudding user response：

In second layer try relu instead of sigmoid. Sigmoid/SoftMax is generally used in last dense(fully connected layer).

Second, check if your test and train have mixed classes data as well.(Sometime it happens that the test or train have a particular class as dominant class and the model become bias).

This is a comment but I don't have 50 reputation.

CodePudding user response：

y_pred = model_net.predict(X_test)
y_pred = tf.greater(y_pred, .5)