Python NLTK - Tokenize sentences into words while removing numbers-CodePudding

hoping someone can assist with this! I have a list of sentences which is read from a text file. I am trying to tokenize the sentences into words, while also removing sentences while contain only numbers. There is no pattern for when the numbers will appear.

The sentences I have:

[
  ['                    1'], 
  ['This is a text file,'], 
  ['to keep the words,'],
  ['                    2'],
  ['Another line of the text:'],
  ['                    3']
]

Desired output:

[
  ['This', 'is', 'a', 'text', 'file,'], 
  ['to', 'keep', 'the', 'words,'],
  ['Another', 'line', 'of', 'the', 'text:'],
]

CodePudding user response：

After some pre processing, now you can apply tokenizing

import re

a = [
    ['                    1'],
    ['This is a text file,'],
    ['to keep the words,'],
    ['                    2'],
    ['Another line of the text:'],
    ['                    3']
]


def replace_digit(string):
    return re.sub(r'\d', '', string).strip()


data = []
process = [replace_digit(i[0]) for i in a]
filtered = filter(lambda x: x, process)
tokenize = map(lambda x: x.split(), filtered)
print(list(tokenize))