Scrapy can't access inner level data-CodePudding

So here is the problem which is pretty basic Scrapy. I am scraping the following url for the json data https://bama.ir/cad/api/search?vehicle=pride,151,se&pageIndex=1 everything works fine and I can get to the first level tags, say key value pairs in 'metadata' So far so good but I am gunning for the lower level value pairs say in data[data']['ads'] and the yield doesn't work. The question is why and how to get it.

def parse(self, response):
    data = json.loads(response.body)
    yield data['data']['ads']

CodePudding user response：

You are retrieving the information correctly. The issue is that the data['data']['ads'] is a list, which is not one of the data types scrapy allows you to yield back.

The data types scrapy does allow include dictionaries, scrapy items, or scrapy requests. So your a solution to your example could be as simple as wrapping the ads list in a dictionary, for example:

def parse(self, response):
    data = response.json()
    yield {'ads': data['data']['ads']}