I am trying to extract data from class using selenium but it is not working-CodePudding

I need to scrape data from this website: https://www.daraz.pk/products/hy-i189662857-s1379994759.html?spm=a2a0e.searchlistcategory.list.3.70426378Fs3yJh&search=1

and html code for this is:

<div >
   <div >...</div>
   <div >...</div>
   <div >...</div>
   <div >...</div>
</div>

Each class div has 1 review and i want to extract it. I tried following code with selenium:

driver.get(site)
time.sleep(5)
data = driver.find_elements(By.CLASS_NAME,"item")

Hypothetically this should give me all item tags but for some reason this returns nothing. Am I missing something here?

CodePudding user response：

These elements only appear after you scroll down the page for a while. Try to GUI-Automate that with a javascript injection like this

for i in range(25):
    driver.execute_script("window.scrollBy(0,100)")

Sadly, that scrolling does not work until you put your mouse cursor there, so there will other gui automation be necessary as well.

But wait! you can trick it:

for i in range(10):
    driver.execute_script("window.scrollBy(0,100)")
for i in range(10):
    driver.execute_script("window.scrollBy(0,-100)")
for i in range(25):
    driver.execute_script("window.scrollBy(0,100)")