Counting occurrences in string column-CodePudding

Within a dataframe I have a variable containing different abstracts of academic literature. Below you find a example of the first 3 observations:

abstract = ['Word embeddings are an active topic in the NLP', 'We propose a new shared task for tactical data', 'We evaluate a semantic parser based on a character']

I want to split the sentences in this variable in seperate words and remove possible periods '.'

The line of code in this case should return the following list:

abstractwords = ['Word', 'embeddings', 'are', 'an', 'active', 'topic', 'in', 'the', 'NPL', 'We', 'Propose', 'a', 'new', 'shared', 'task', 'for', 'tactical', 'data', 'We', 'evaluate', 'a', 'semantic', 'parser', 'based', 'on', 'a', 'character']

CodePudding user response：

Use for..each loop to go through elements, replace "." with a space. Split the sentence, and concatenate the lists.

abstractwords = []
for sentence in abstract:
    sentence = sentence.replace(".", " ")
    abstractwords.extend(sentence.split())

CodePudding user response：

You can use nested list comprehension:

abstract = ['Word embeddings are an active topic in the NLP.', 'We propose a new shared task for tactical data.', 'We evaluate a semantic parser based on a character.']

words = [word.strip('.') for sentence in abstract for word in sentence.split()]
print(words)
# ['Word', 'embeddings', 'are', 'an', 'active', 'topic', 'in', 'the', 'NLP', 'We', 'propose', 'a', 'new', 'shared', 'task', 'for', 'tactical', 'data', 'We', 'evaluate', 'a', 'semantic', 'parser', 'based', 'on', 'a', 'character']

If you want to remove '.' in the middle of the words as well, use word.replace('.', '') instead.