Home > Net >  Getting this error when scraping JSON with scrapy: Spider must return request, item, or None, got &#
Getting this error when scraping JSON with scrapy: Spider must return request, item, or None, got &#

Time:05-08

I am trying to get a json field with key "longName" with scrapy but I am receiving the error: "Spider must return request, item, or None, got 'str'".

The JSON I'm trying to scrape looks something like this:

{
   "id":5355,
   "code":9594,

}sadsadsd

This is my code:

import scrapy
import json

class NotesSpider(scrapy.Spider):
    name = 'notes'
    allowed_domains = ['blahblahblah.com']
    start_urls = ['https://blahblahblah.com/api/123']

    def parse(self, response):
        data = json.loads(response.body)
        yield from data['longName']

I get the above error when I run "scrapy crawl notes" in prompt. Anyone can point me in the right direction?

CodePudding user response:

If you only want longName modifying your parse method like this should do the trick:

    def parse(self, response):
        data = json.loads(response.body)
        yield {"longName": data["longName"]}
  • Related