I'm trying to create a Performance Evaluation Results for my CRF model, which depicts what part of speech does the word belong to. I've created a function to transform the data in a more 'datasetish' format. This function returns the data as two lists, one of Dicts of features and the other with the labels.
def transform_to_dataset(tagged_sentences):
X, y = [], []
for sentence, tags in tagged_sentences:
sent_word_features, sent_tags = [], []
for index in range(len(sentence)):
sent_word_features.append(extract_features(sentence, index)),
sent_tags.append(tags[index])
X.append(sent_word_features)
y.append(sent_tags)
return X, y
then I divide the set BEFORE encoding to have full sentences in training/testing sets.
penn_train_size = int(0.8*len(penn_treebank))
penn_training = penn_treebank[:penn_train_size]
penn_testing = penn_treebank[penn_train_size:]
X_penn_train, y_penn_train = transform_to_dataset(penn_training)
X_penn_test, y_penn_test = transform_to_dataset(penn_testing)
and then I load the model for it to train and test my data
penn_crf = CRF(
algorithm='lbfgs',
c1=0.01,
c2=0.1,
max_iterations=100,
all_possible_transitions=True
)
#The fit method is the default name used by Machine Learning algorithms to start training.
print("Started training on Penn Treebank corpus!")
penn_crf.fit(X_penn_train, y_penn_train)
print("Finished training on Penn Treebank corpus!")
and then I test it with
y_penn_pred=penn_crf.predict(X_penn_test)
But when I try
from sklearn.metrics import accuracy_score
print("Accuracy: ", accuracy_score(y_penn_test, y_penn_pred))
It gives off an error:
ValueError: You appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead - the MultiLabelBinarizer transformer can convert to this format.
But when I try to use MultiLabelBinarizer;
from sklearn.preprocessing import MultiLabelBinarizer
bin_y_penn_test = MultiLabelBinarizer().fit_transform(y_penn_test)
bin_y_penn_pred = MultiLabelBinarizer().fit_transform(y_penn_pred)
It gives me an error of:
ValueError: inconsistent shapes
Here's the full traceback
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) /tmp/ipykernel_5694/856179584.py in 1 from sklearn.metrics import accuracy_score 2 ----> 3 print("Accuracy: ", accuracy_score(bin_y_penn_test, bin_y_penn_pred))
~/.local/lib/python3.8/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs) 61 extra_args = len(args) - len(all_args) 62 if extra_args <= 0: ---> 63 return f(*args, **kwargs) 64 65 # extra_args > 0
~/.local/lib/python3.8/site-packages/sklearn/metrics/_classification.py in accuracy_score(y_true, y_pred, normalize, sample_weight) 203 check_consistent_length(y_true, y_pred, sample_weight) 204 if y_type.startswith('multilabel'): --> 205 differing_labels = count_nonzero(y_true - y_pred, axis=1) 206 score = differing_labels == 0 207 else:
~/.local/lib/python3.8/site-packages/scipy/sparse/base.py in sub(self, other) 431 elif isspmatrix(other): 432 if other.shape != self.shape: --> 433 raise ValueError("inconsistent shapes") 434 return self._sub_sparse(other) 435 elif isdense(other):
ValueError: inconsistent shapes
What should I do so that I could produce the confusion matrix of my model?
CodePudding user response:
Try doing
penn_train_size = int(0.7*len(penn_treebank))
Then check if the shape is still inconsistent
bin_y_penn_test.shape
bin_y_penn_pred.shape
if bin_y_penn_test.shape == bin_y_penn_pred.shape:
print('Consistent Shape')
else:
print('Inconsistent Shape')
If it gives off consistent shape, do
from sklearn.metrics import multilabel_confusion_matrix
multilabel_confusion_matrix(bin_y_penn_test, bin_y_penn_pred)
However if it still isn't consistent, try to tinker on your data.