I have created a deep neural network model with sequential() model of keras. This is a binary classification problem. I have fitted the model with train data.

I am confused about the calculation of different accuracy metrics for train and validation data. I am calculating RMSE, F1 Score, AUC of ROC and PR curve as,

# Prediction
y_pred_train = model.predict(x_train_df).ravel()
y_pred_val = model.predict(x_val_df).ravel()

# RMSE
rmse_train = mean_squared_error(y_train_df, y_pred_train)
rmse_val = mean_squared_error(y_val_df, y_pred_val)

# ROC-AUC
fpr_train, tpr_train, thresholds_roc_train = roc_curve(y_train_df, y_pred_train, pos_label=None)
fpr_val, tpr_val, thresholds_roc_val = roc_curve(y_val_df, y_pred_val, pos_label=None)

roc_auc_train = auc(fpr_train, tpr_train)
roc_auc_val = auc(fpr_val, tpr_val)

# PR-AUC
precision_train, recall_train, thresholds_pr_train = precision_recall_curve(y_train_df, y_pred_train)
precision_val, recall_val, thresholds_pr_val = precision_recall_curve(y_val_df, y_pred_val)
pr_auc_train = auc(recall_train, precision_train)
pr_auc_val = auc(recall_val, precision_val)

# F1 Score
f1_train = np.mean(2 * (precision_train * recall_train) / (precision_train   recall_train))
f1_val = np.mean(2 * (precision_val * recall_val) / (precision_val   recall_val))

The values of these accuracies are,

• RMSE Train 0.11
• RMSE Validation 0.13
• ROC-AUC Train 0.94
• ROC-AUC Validation 0.91
• PR-AUC Train 0.96
• PR-AUC Validation 0.93
• F1 Score Train 0.66
• F1 Score Validation 0.66

I am very new to machine learning. I have implemented these codes by searching various web pages. Is my code correct? I am getting this confusion as the F1 score is not very high although all other metrics have high values.

If the code is correct, then why I'm getting not so high F1 score?

Edit 1

As asked in the comment, the precision and recall values are

print(np.mean(precision_train))
print(np.mean(recall_train))
print(np.mean(precision_val))
print(np.mean(recall_val))

Output:

0.9299899169174257
0.6012312742646909
0.8988925808831595
0.6052356704530617

CodePudding user response：

Is my code correct?

Sorry, not entirely -

1. For the Precision, Recall, and f1 you should not be taking a mean of the curves such as f1_train = np.mean(.., Instead:
   Use your PR-AUC and ROC-AUC to define a threshold. Use that threshold to binarise the y_pred_ *, and then call classification_report for printing final precision, recall and f1 scores. You will, then, see the effective f1 score and how might precision and recall be impacting it.

2. RMSE: Go for a cross entropy metric may be (after you have binarised your predictions), because it is a classifier you trained.