Iterate function across dataframe-CodePudding

I have a dataset containing pre-processed online reviews, each row contains words from online review. I am doing a Latent Dirichlet Allocation process to extract topics from the entire dataframe. Now, I want to assign topics to each row of data based on an LDA function called get_document_topics.

I found a code from a source but it only prints the probability of a document being assign to each topic. I'm trying to iterate the code to all documents and returns to the same dataset. Here's the code I found...

text = ["user"]
bow = dictionary.doc2bow(text)
print "get_document_topics", model.get_document_topics(bow)
### get_document_topics [(0, 0.74568415806946331), (1, 0.25431584193053675)]

Here's what I'm trying to get...

                  stemming   probabOnTopic1 probOnTopic2 probaOnTopic3  topic 
0      [bank, water, bank]              0.7          0.3           0.0      0 
1  [baseball, rain, track]              0.1          0.8           0.1      1
2     [coin, money, money]              0.9          0.0           0.1      0 
3      [vote, elect, bank]              0.2          0.0           0.8      2

Here's the codes that I'm working on...

def bow (text):
    return [dictionary.doc2bow(text) in document]

df["probability"] = optimal_model.get_document_topics(bow)
df[['probOnTopic1', 'probOnTopic2', 'probOnTopic3']] = pd.DataFrame(df['probability'].tolist(), index=df.index)

CodePudding user response：

One possible option can be creating a new column in your DF and then iterate over each row in your DF. You can use the get_document_topics function to get the topic distribution for each row and then assign the most likely topic to that row.

df['topic'] = None
for index, row in df.iterrows():
    text = row['review_text']
    bow = dictionary.doc2bow(text)
    topic_distribution = model.get_document_topics(bow)
    most_likely_topic = max(topic_distribution, key=lambda x: x[1])
    df.at[index, 'topic'] = most_likely_topic

is it helpful ?