Say I have a numpy array of confusion matrices obtained from k-fold cross-validation,

[array([[39,  4],
       [ 9,  6]], dtype=int64), array([[39,  4],
       [ 9,  5]], dtype=int64), array([[37,  6],
       [11,  3]], dtype=int64), array([[42,  1],
       [11,  3]], dtype=int64), array([[40,  3],
       [ 9,  5]], dtype=int64)]

I can find the mean average of these confusion matrices like so,

mean_conf_matrices = np.mean(conf_matrices_arr, axis=0)

// [[39.4  3.6]
    [ 9.8  4.4]]

But I am wondering how I would go about obtaining the best confusion matrix from the array of confusion matrices. Similarly to how you can get best_score_ from a GridSearchCV. My idea would be to obtain the TN, TP, FN, FP and evaluate each to find the confusion matrix with the highest TN and TP and the lowest FN and FP. Is there a more intuitive way to achieve this?

Edit: my method in more detail would be to obtain each part of the confusion matrix (TN, TP, FN, FP) and store it in an array with the same index i.e. collect all the TNs from all the confusion matrices - TN[0] would be the TN taken from confusion_matrix[0] and vice versa. There would then be a choice of which metric we want to focus on - reduced FPs or FNs. Say we wanted reduced FNs, then the lowest FN would be taken from the array of FNs. The index of this would then be obtained and the same index from the array of confusion matrices would be obtained, and that matrix would be selected as the best i.e. FN[4] is the best, therefore, confusion_matrix[4] is chosen. I am wondering if there is a more intuitive way to achieve this as my method feels cumbersome.

CodePudding user response：

You need to choose a classification metric which will enable you to compare different classifiers: for example AUC (area under the ROC curve), precision, recall, F1 (which combines precision and recall), ...
See this link from scikit-learn about the different possibilities and implementation.

CodePudding user response：

I have coded my implementation below. If you have a more intuitive way in which I may achieve the same objective please feel free to post an answer.

import math
import numpy as np

def get_tp_tn_fp_tn_from_confusion_matrix(cfmatrix):
    
    FN = cfmatrix.sum(axis=0) - np.diag(cfmatrix)
    FP = cfmatrix.sum(axis=1) - np.diag(cfmatrix)
    TN = np.diag(cfmatrix)
    TP = cfmatrix.sum() - (FP   FN   TN)
    
    return (TP[:1], TN[:1], FP[:1], FN[:1])

def get_best_confusion_matrix(cfmatrices, reduction_bias=None, debug=False):
    
    assert reduction_bias in ['FP', 'FN'], \
        f'{reduction_bias} is not a valid reduction bias. Select "FN" or "FP".'
    
    tps = list()
    tns = list()
    fps = list()
    fns = list()
    
    for cfmatrix in cfmatrices:
        
        TP, TN, FP, FN = get_tp_tn_fp_tn_from_confusion_matrix(cfmatrix)
        tps.append(int(math.ceil(TP)))
        tns.append(int(math.ceil(TN)))
        fps.append(int(math.ceil(FP)))
        fns.append(int(math.ceil(FN)))
    
    idx = 0
    best_cfmatrix = None
    
    if reduction_bias == 'FN':
        
        lowest_fn = min(fns)
        idx_fn = fns.index(lowest_fn)
        
        if debug: print('The chosen confusion matrix is:\n', cfmatrices[idx_fn])
        
        for cfmatrix in cfmatrices:
            
            _, _, chosen_FP, chosen_FN = get_tp_tn_fp_tn_from_confusion_matrix(cfmatrices[idx_fn])
            
            if (cfmatrix == cfmatrices[idx_fn]).all():
                if debug: print('Skipping the chosen confusion matrix...')
                continue
            else:
                _, _, FP, FN = get_tp_tn_fp_tn_from_confusion_matrix(cfmatrix)
                
                if FN == chosen_FN:
                    if FP < chosen_FP:
                        print('Found duplicate confusion matrix. It is better than chosen confusion matrix.')
                        print('Chosen confusion matrix replaced.')
                        idx_fp = fps.index(FP)
                        best_cfmatrix = cfmatrices[idx_fp]
                        break
                    else:
                        if debug: print('Found duplicate confusion matrix. Chosen confusion matrix is better.')
                        best_cfmatrix = cfmatrices[idx_fn]
                else:
                    if debug: print('Searching for duplicate confusion matrices. None found.')
                    best_cfmatrix = cfmatrices[idx_fn]
        
    elif reduction_bias == 'FP':

        lowest_fp = min(fps)
        idx_fp = fps.index(lowest_fp)
        
        if debug: print('The chosen confusion matrix is:\n', cfmatrices[idx_fp])
        
        for cfmatrix in cfmatrices:    
            
            _, _, chosen_FP, chosen_FN = get_tp_tn_fp_tn_from_confusion_matrix(cfmatrices[idx_fp])
            
            if (cfmatrix == cfmatrices[idx_fp]).all():
                if debug: print('Skipping the chosen confusion matrix...')
                continue
            else:
                _, _, FP, FN = get_tp_tn_fp_tn_from_confusion_matrix(cfmatrix)
                
                if FP == chosen_FP:
                    if FN < chosen_FN:
                        print('Found duplicate confusion matrix. It is better than chosen confusion matrix.')
                        print('Chosen confusion matrix replaced.')
                        idx_fn = fns.index(FN)
                        best_cfmatrix = cfmatrices[idx_fn]
                        break
                    else:
                        if debug: print('Found duplicate confusion matrix. Chosen confusion matrix is better.')
                        best_cfmatrix = cfmatrices[idx_fp]
                else:
                    if debug: print('Searching for duplicate confusion matrices. None found.')
                    best_cfmatrix = cfmatrices[idx_fp]
                        
    return best_cfmatrix