I have a data called:

• after_tokenize.xlsx
• positive.xlsx
• negative.xlsx after tokenize positive negative

What I want to is labelling sentiment positive and negative for data from after_tokenize.xlsx. If data on after tokenize have a lot of positive word from data positive.xlsx it will be positive and If data have a lot negative word from negative it will be negative. the result will be entered into a label named label. sample:

import pandas as pd
import nltk

df = pd.DataFrame({'data': ['i like love hate you', 'i dont hate like you']})
pos = pd.DataFrame(data=['like', 'love'], columns=['positive'])
neg = pd.DataFrame(data=['dont', 'hate'], columns=['negative'])
df['data'] = df.apply(lambda row: nltk.word_tokenize(row['data']), axis=1)

CodePudding user response：

You can use set() and operation set(...) & set(...) to get words which are in two lists.

And then you can count them using len()

len( set([i, like, love, hate, you]) & set(['like', 'love']) ) 

import pandas as pd
import nltk

df = pd.DataFrame({'data': ['i like love hate you', 'i dont hate like you']})

pos = ['like', 'love']
neg = ['dont', 'hate']

#print(df)

df['data'] = df['data'].apply(nltk.word_tokenize)

# --- get common words ---

df['pos words'] = df['data'].apply(lambda item: list(set(item) & set(pos)))
df['neg words'] = df['data'].apply(lambda item: list(set(item) & set(neg)))

# --- count common words ---

df['pos'] = df['data'].apply(lambda item: len(set(item) & set(pos)))
df['neg'] = df['data'].apply(lambda item: len(set(item) & set(neg)))

# or

df['pos'] = df['pos words'].apply(len)
df['neg'] = df['neg words'].apply(len)

# --- assing labels ---

df['label'] = '???'  # default value 

#df.['label'][ df['pos'] > df['neg'] ] = 'positive'
df.loc[ (df['pos'] > df['neg']), 'label' ] = 'positive'

#df.['label'][ df['pos'] < df['neg'] ] = 'negative'
df.loc[ (df['pos'] < df['neg']), 'label' ] = 'negative'

# ---

print(df)

Result:

                         data     pos words     neg words  pos  neg     label
0  [i, like, love, hate, you]  [love, like]        [hate]    2    1  positive
1  [i, dont, hate, like, you]        [like]  [hate, dont]    1    2  negative