Keep model made with TFIDF for predicting new content using Scikit for Python-CodePudding

this is a sentiment analysis model made with tf-idf for feature extraction i want to know how can i save this model and reuse it. i tried saving it this way but when i load it , do same pre-processing on the test text and fit_transform on it it gave an error that the model expected X numbers of features but got Y

this is how i saved it

filename = "model.joblib"
joblib.dump(model, filename)

and this is the code for my tf-idf model

import pandas as pd
import re
import nltk
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import BernoulliNB
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
nltk.download('stopwords')
from nltk.corpus import stopwords

processed_text = ['List of pre-processed text'] 
y = ['List of labels']
tfidfconverter = TfidfVectorizer(max_features=10000, min_df=5, max_df=0.7, stop_words=stopwords.words('english'))
X = tfidfconverter.fit_transform(processed_text).toarray()

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

text_classifier = BernoulliNB()
text_classifier.fit(X_train, y_train)

predictions = text_classifier.predict(X_test)
print(confusion_matrix(y_test, predictions))
print(classification_report(y_test, predictions))
print(accuracy_score(y_test, predictions))

edit: just to exact where to put every line so after:

tfidfconverter = TfidfVectorizer(max_features=10000, min_df=5, max_df=0.7, stop_words=stopwords.words('english'))

then

tfidf_obj = tfidfconverter.fit(processed_text)//this is what will be used again
joblib.dump(tfidf_obj, 'tf-idf.joblib')

then you do the rest of the steps you will save the classifier after training as well so after:

text_classifier.fit(X_train, y_train)

put joblib.dump(model, "classifier.joblib") now when you want to predict any text

tf_idf_converter = joblib.load("tf-idf.joblib")
classifier = joblib.load("classifier.joblib")

now u have List of sentences to predict

sent = []
classifier.predict(tf_idf_converter.transform(sent))

now print that for a list of sentiments for each sentece

CodePudding user response：

You can first fit tfidf to your training set using:

tfidfconverter = TfidfVectorizer(max_features=10000, min_df=5, max_df=0.7, stop_words=stopwords.words('english'))
tfidf_obj = tfidfconverter.fit(processed_text)

Then find a way to store the tfidf_obj for instance using pickle or joblib e.g:

joblib.dump(tfidf_obj, filename)

Then load the saved tfidf_obj and apply transform only on your test set

loaded_tfidf = joblib.load(filename)
test_new = loaded_tfidf.transform(X_test)