how to handle json output in scarpy-CodePudding

i'm writing some scrapy code and when i want to get the output in json with this command: «scrapy crawl mobileir -o myjayson.json -s FEED_EXPORT_ENCODING=utf-8» i get json file in a single line but i want to get in json format.what should i do?

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
                'https://www.ijpsat.com/'
    ]

    def parse(self, response):
        yield {
            'test1':response.css('#home a::text').get(),
            'test2':response.css('#about a::text').get(),
            'test3':response.css('#search a::text').get()
        }

my output is:

[
{"test1": "Home", "test2": "About", "test3": "Search"}
]

CodePudding user response：

Pretty JSON Print

Use json.dumps()

import json

d = {
    'test1': 'Home',
    'test2': 'About',
    'test3': 'Search'
}
print(f"Before: {d}")

formatted_d = json.dumps(d, indent=2)
print(f"After: {formatted_d}")

Output

Before: {'test1': 'Home', 'test2': 'About', 'test3': 'Search'}
After: {
  "test1": "Home",
  "test2": "About",
  "test3": "Search"
}

CodePudding user response：

You can add do this by adding -s FEED_EXPORT_INDENT=<number_of_spaces>. so when you execute something like this scrapy crawl quote -o items.json -s FEED_EXPORT_INDENT=2 you will get:

[
{
  "test1": "Home",
  "test2": "About",
  "test3": "Search"
}
]