ValueError using tensorflow.metrics.Recall(class

Using Python 3.8.3 and tensorflow version 2.4.1

Wanted to use the parameter class_id in tensorflow.metrics such as Recall (see documentation )

Here is a minimal piece of code to replicate the problem. The code below crashes with class_id=1

from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.keras.layers import SimpleRNN
from sklearn.model_selection import train_test_split
from tensorflow.keras import metrics
import numpy as np
#generate data
max_length  = 200
width = 3
n_samples = 100
data = np.random.rand(n_samples, max_length, width)
label = np.random.randint(0, high =2, size = n_samples)
train_size = 0.8
x_train, x_test, y_train, y_test = train_test_split(data, label, train_size = train_size)

#create a model
rnn_size = 16
sequence_input = Input(shape=(max_length,width,), dtype='float32')
x = SimpleRNN(rnn_size)(sequence_input)
preds = Dense(1, activation='sigmoid')(x)
model = Model(sequence_input, preds)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[metrics.Recall(class_id=1)])
#fit
BATCH_SIZE = 32
history = model.fit(x_train, y_train, epochs=1, batch_size=BATCH_SIZE)

Throws ValueError

ValueError: slice index 1 of dimension 1 out of bounds. for '{{node strided_slice_1}} = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=1, end_mask=0, new_axis_mask=0, shrink_axis_mask=2](Cast_1, strided_slice_1/stack, strided_slice_1/stack_1, strided_slice_1/stack_2)' with input shapes: [?,1], [2], [2], [2] and with computed input tensors: input[1] = <0 1>, input[2] = <0 2>, input[3] = <1 1>.

But it works with metrics.Recall(class_id=0)

The same error with metrics.Precision(class_id=1) and probably all other metrics using class_id (I haven't tried them all).

I can't decipher what the error message means or find anything relevant online to answer my question.

CodePudding user response：

Documentation states that:

class_id (Optional): Integer class ID for which we want binary metrics. This must be in the half-open interval [0, num_classes), where num_classes is the last dimension of predictions.

When you are using sigmoid your outputs consist of the shape: (1, ) which is causing this error. And if you modify your network for binary classification the outputs will be the sigmoid probabilities of being class 1.

So for binary classification case you will get Precision and Recall for class 1 by default, if you want to get class 0 then you need to define your own metric. An example can be found here.

The relative error is coming from here (source code):

if class_id is not None:
  y_true = y_true[..., class_id]
  y_pred = y_pred[..., class_id]

In your example, labels should be one-hot-encoded:

from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.keras.layers import SimpleRNN
from sklearn.model_selection import train_test_split
from tensorflow.keras import metrics
from tensorflow.keras.utils import to_categorical
import numpy as np
#generate data
max_length  = 200
width = 3
n_samples = 100
data = np.random.rand(n_samples, max_length, width)
label = np.random.randint(0, high =2, size = n_samples)
label = to_categorical(label, 2)
train_size = 0.8
x_train, x_test, y_train, y_test = train_test_split(data, label, train_size = train_size)

#create a model
rnn_size = 16
sequence_input = Input(shape=(max_length,width), dtype='float32')
x = SimpleRNN(rnn_size)(sequence_input)
preds = Dense(2, activation='softmax')(x)
model = Model(sequence_input, preds)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[metrics.Precision(class_id=1),
                                                                          metrics.Recall(class_id=1)])
#fit
BATCH_SIZE = 32
history = model.fit(x_train, y_train, epochs=16, batch_size=BATCH_SIZE,
                    validation_data = (x_test, y_test))

Epoch 16/16
3/3 [==============================] - 0s 86ms/step - loss: 0.6771 - precision: 0.5676 - 
       recall: 0.5250 - val_loss: 0.6419 - val_precision: 0.2222 - val_recall: 0.6667

Validate the results by sklearn:

from sklearn.metrics import classification_report
print(classification_report(np.argmax(y_test, axis = -1), 
                            np.argmax(model.predict(x_test, batch_size = 1), 
                                      axis= -1), digits = 4))

              precision    recall  f1-score   support

           0     0.9091    0.5882    0.7143        17
           1     0.2222    0.6667    0.3333         3

    accuracy                         0.6000        20
   macro avg     0.5657    0.6275    0.5238        20
weighted avg     0.8061    0.6000    0.6571        20

If you change class_id = 0 in the last example, it will compute the metrics for the class 0.