yield multiple item from one page-CodePudding

I need to yield multiple items from one page. Items differ by one field. I collect one item, then create new items in a loop, i.e.:

for i in new_fields:
    new_item = item
    new_item["new_field"] = i
    yield new_item

But scrapy just returns me n of same rows, where n is a len(new_fields).

What am I doing wrong?

CodePudding user response：

I recommend using Item Loaders in this case. You can read more about it on the documentation page https://docs.scrapy.org/en/latest/topics/loaders.html.
In general, it would look something like this:

from scrapy.loader import ItemLoader
from myproject.items import Product

def parse(self, response):
    for i in new_fields:
        l = ItemLoader(item=Product(), response=response)
        #  you can add data using CSS, XPath or values
        l.add_xpath('name', '//div[@]')
        l.add_css('stock', 'p#stock')
        l.add_value('last_updated', 'today')
        yield l.load_item()

CodePudding user response：

Assuming your "item" is a dictionary, try

    new_item = item.copy()

Otherwise, you are updating the same dictionary each time and yield a new reference to the one dictionary each time.