I am trying to learn to make a Scrapy Desktop app using Python Tk inter. So, I'm trying to run this prebuild Scrapy desktop app using Python Tkinter. While saving the output, there is an issue with the "FEEDS". The below-mentioned function does save the output file by choosing the specific extension and saving it to a specific location. This script was developed based Scrapy version up to 1.8. So in Scrapy version 2.X the 'FEED_URI & 'FEED_FORMAT' are not working as they have been replaced with "FEEDS".
def execute_spider():
if dataset_entry.get() == '' or chosen_feed not in ['CSV', 'JSON']:
messagebox.showerror('Error', 'All entries are required')
return
try:
feed_uri = f'file:///{folder_path}/{dataset_entry.get()}.{chosen_feed}'
except:
messagebox.showerror('Error', 'All entries are required')
settings = project.get_project_settings()
settings. Set('FEED_URI', feed_uri)
settings. Set('FEED_FORMAT', chosen_feed)
configure_logging()
runner = CrawlerRunner(settings)
runner.crawl(chosen_spider)
reactor.run(installSignalHandlers=False)
To see the full script, you can also visit here
So, how can I use 'FEEDS' instead of these particular 2 lines?
settings. Set('FEED_URI', feed_uri)
settings. Set('FEED_FORMAT', chosen_feed)
CodePudding user response:
You would instead use
settings.set("FEEDS", {
feed_uri: {
'format': chosen_format,
'encoding': 'utf8'
}
})
and include any other settings that are appropriate.
Refer to the scrapy docs for available options: FEEDS