I use "selenium" to locate info with Python3.9.6 64-bit.Sometimes it does output some correct results, but then suddenly crashes after finishing a few of the 10 tasks. But sometimes it doesn't work at all.Codes are like this:
from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
web = Chrome()
web.get("http://lagou.com")
web.find_element(By.XPATH, '//*[@id="cboxClose"]').click()
time.sleep(1)
web.find_element(By.XPATH, '//*[@id="search_input"]').send_keys("python", Keys.ENTER)
for i in range(1, 11):
web.find_element(By.XPATH, f'//*[@id="jobList"]/div[1]/div[{i}]/div[1]/div[1]/div[1]/a').click()
web.switch_to.window(web.window_handles[-1])
details = web.find_element(By.XPATH, '//*[@id="job_detail"]/dd[2]/div').text
print(details)
web.close()
web.switch_to.window(web.window_handles[0])
And the crach info are like this
File "/Users/xxxx/Library/Mobile Documents/com~apple~CloudDocs/xxxx/Code/Python/Project/selenium(2).py", line 20, in <module>
details = web.find_element(By.XPATH, '//*[@id="job_detail"]/dd[2]/div').text
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 1244, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="job_detail"]/dd[2]/div"}
(Session info: chrome=97.0.4692.71)
Stacktrace:
0 chromedriver 0x000000010fa0fe69 chromedriver 5160553
1 chromedriver 0x000000010f99a593 chromedriver 4679059
2 chromedriver 0x000000010f54e2c8 chromedriver 172744
3 chromedriver 0x000000010f583b62 chromedriver 392034
4 chromedriver 0x000000010f583d21 chromedriver 392481
5 chromedriver 0x000000010f5b6304 chromedriver 598788
6 chromedriver 0x000000010f5a13fd chromedriver 513021
7 chromedriver 0x000000010f5b40ab chromedriver 589995
8 chromedriver 0x000000010f5a1623 chromedriver 513571
9 chromedriver 0x000000010f576dce chromedriver 339406
10 chromedriver 0x000000010f578105 chromedriver 344325
11 chromedriver 0x000000010f9cb23e chromedriver 4878910
12 chromedriver 0x000000010f9e2d17 chromedriver 4975895
13 chromedriver 0x000000010f9e8a3f chromedriver 4999743
14 chromedriver 0x000000010f9e361a chromedriver 4978202
15 chromedriver 0x000000010f9bfbb1 chromedriver 4832177
16 chromedriver 0x000000010f9fffd8 chromedriver 5095384
17 chromedriver 0x000000010fa00161 chromedriver 5095777
18 chromedriver 0x000000010fa172a8 chromedriver 5190312
19 libsystem_pthread.dylib 0x00007fff204ab8fc _pthread_start 224
20 libsystem_pthread.dylib 0x00007fff204a7443 thread_start 15
How was that going?
CodePudding user response:
I've run your code many times and yes it crashes after random iterations. and that is an indication of some code some ware didn't finish its execution. also, one of my runs finished all the 10 iterations. So. that supports my theory.
So. the solution is to add this
time.sleep(3)
at the end of the loop. or just add this line before going to the link. and this is better for future updates.
web.implicitly_wait(10)
you can read more about this at selenium wait documentation
CodePudding user response:
I used the SeleniumBase Python framework to make the script reliable: Run pip install seleniumbase
and then run the following test with pytest
:
from seleniumbase import BaseCase
class MyTestClass(BaseCase):
def test_base(self):
self.open("http://lagou.com")
self.click("#cboxClose")
self.type("#search_input", "python\n")
for i in range(1, 11):
self.click('#jobList > div > div:nth-of-type(%s) a' % i)
details = self.get_text("#job_detail > dd:nth-of-type(2) > div")
print(details)
self.driver.close()
self.switch_to_window(0)
Additionally, since the website is in Chinese, you might like the script in the Chinese translation of SeleniumBase:
from seleniumbase.translate.chinese import 硒测试用例
class MyTestClass(硒测试用例):
def test_base(self):
self.开启("http://lagou.com")
self.单击("#cboxClose")
self.输入文本("#search_input", "python\n")
for i in range(1, 11):
self.单击('#jobList > div > div:nth-of-type(%s) a' % i)
details = self.获取文本("#job_detail > dd:nth-of-type(2) > div")
print(details)
self.driver.close()
self.切换到窗口(0)
Both files will run the same, reliably.