glob filenames | AttributeError: 'str' object has no attribute 'content'-CodePudding

I am running my own version of this Notebook, where section Apply DocumentClassifier is altered as below.

Objects doc in documents is str dtype. I assume it shoudln't be. What dtype should doc be?

Jupyter Labs, kernel: conda_mxnet_latest_p37.

Cells:

import glob
docs_to_classify = glob.glob('full-set-of-gri-standards-2021-english/*.pdf')

with open('filt_gri.txt', 'r') as filehandle:
    tags = [current_place.rstrip() for current_place in filehandle.readlines()]

doc_classifier = TransformersDocumentClassifier(model_name_or_path="cross-encoder/nli-distilroberta-base",
                                                task="zero-shot-classification",
                                                labels=tags,
                                                batch_size=16)

classified_docs = doc_classifier.predict(docs_to_classify)

all_docs = convert_files_to_dicts(dir_path=doc_dir)

preprocessor_sliding_window = PreProcessor(split_overlap=3,
                                           split_length=10,
                                           split_respect_sentence_boundary=False,
                                           split_by='passage')

Output:

INFO - haystack.modeling.utils -  Using devices: CUDA
INFO - haystack.modeling.utils -  Number of GPUs: 1
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-10-75f29230cd0e> in <module>
      7                                                 batch_size=16)
      8 
----> 9 classified_docs = doc_classifier.predict(docs_to_classify)
     10 
     11 all_docs = convert_files_to_dicts(dir_path=doc_dir)

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/haystack/nodes/document_classifier/transformers.py in predict(self, documents)
    134         :return: List of Document enriched with meta information
    135         """
--> 136         texts = [doc.content if self.classification_field is None else doc.meta[self.classification_field] for doc in documents]
    137         batches = self.get_batches(texts, batch_size=self.batch_size)
    138         if self.task == 'zero-shot-classification':

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/haystack/nodes/document_classifier/transformers.py in <listcomp>(.0)
    134         :return: List of Document enriched with meta information
    135         """
--> 136         texts = [doc.content if self.classification_field is None else doc.meta[self.classification_field] for doc in documents]
    137         batches = self.get_batches(texts, batch_size=self.batch_size)
    138         if self.task == 'zero-shot-classification':

AttributeError: 'str' object has no attribute 'content'

Please let me know if there is anything else I should add to post/ clarify.

CodePudding user response：

I forgot to include line, before classified_docs:

# convert to Document using a fieldmap for custom content fields the classification should run on
docs_to_classify = [Document.from_dict(d) for d in docs_sliding_window]