Scrapy novice consult, the parse function is not implemented-CodePudding

The import scrapy

The class getdata (scrapy. Spiders) :
Name="baidu"
Start_urls=(" www.baidu.com ")

Def parse (self, response) :
Print (1)

Print out 1 come on,

Please expert advice, program execution is not an error, is the lack of what plug-ins or configuration has a problem, baidu didn't solve this problem, a circle

CodePudding user response:

Try ROBOTSTXT_OBEY=False setting in the Settings

CodePudding user response:

reference 1st floor sinat_37982934 response:

ROBOTSTXT_OBEY=False setting set try

Thank you big brother, I changed, or is not working,,,,

CodePudding user response:

You can access without agreement==
Start_urls=(" http://www.baidu.com ")

CodePudding user response:

Problem solved, thank you, when debugging, add
The from scrapy import cmdline
Cmdline. Execute (" scrapy crawl XXX ". The split ())

So that, the command line to perform scrapy crawl XXX this can also, novice but I don't know much about,

CodePudding user response:

The from scrapy import cmdline
Cmdline. Execute (" scrapy crawl XXX ". The split ())
Where is the write?? XXX represent??
Small white problem more

CodePudding user response:

refer to 7th floor qq_36377875 response:

from scrapy import cmdline
Cmdline. Execute (" scrapy crawl XXX ". The split ())
Where is the write?? XXX represent??
Small white problem more

Crawler file filename you wrote, need not add suffix

CodePudding user response:

reference 4 floor gaozg7 response:

problem solved, thank you, when debugging, add
The from scrapy import cmdline
Cmdline. Execute (" scrapy crawl XXX ". The split ())

Is that, the command line to perform scrapy this also can crawl XXX, novice but I don't know much about,

How do you do when I was in the command is not implemented the print statement is how debugging
Novice small white

CodePudding user response:

references 9 f qq_33357094 response:

Quote: refer to 4th floor gaozg7 response:

Problem solved, thank you, when debugging, add
The from scrapy import cmdline
Cmdline. Execute (" scrapy crawl XXX ". The split ())

Is that, the command line to perform scrapy this also can crawl XXX, novice but I don't know much about,

How do you do when I was in the command is not implemented the print statement is how debugging
Novice small white

Scrapy startup mode of the crawler project directory CMD execution scrapy crawl XXX XXX - for your crawler name
The run is not directly
The above method is to define a method to perform the CMD in the start statement more convenient

CodePudding user response:

reference 1st floor sinat_37982934 response:

ROBOTSTXT_OBEY=False setting set try

This method can be

CodePudding user response:

"According to the method of the building Lord I use the tip after No module named 'GetImage. Settings', this is why

CodePudding user response:

ROBOTSTXT_OBEY=False

CodePudding user response:

ROBOTSTXT_OBEY=False setting set, the from scrapy import cmdline
Cmdline. Execute (" scrapy crawl XXX ". The split ()) also have such a boot file, but still not solve, not into the def parse (self, response) : output, data couldn't climb down, ask for help