How do I click and open a collection of objects in an element in python selenium without closing and-CodePudding

Say I have an eccormece site I would want to scrape and I am interested in the top ten trending products and when dig into the html element its like this:

<div>
    <div>
        <span>
            <a href='www.mysite/products/1'>
                Product 1
            </a>
        </spa>
    </div>
    <div>
        <span>
            <a href='www.mysite/products/2'>
                Product 2
            </a>
        </spa>
    </div>
    <div>
        <span>
            <a href='www.mysite/products/3'>
                Product 3
            </a>
        </spa>
    </div>
    <div>
        <span>
            <a href='www.mysite/products/4'>
                Product 4
            </a>
        </spa>
    </div>

</div>

My first solution was to extract the href attributes and then store them in a list then I would open browser instances for each and every attribute, but then it comes at a cost as I have to close and open the browser and every time I open it I have to authenticate. I then tried solution 2. In my solution two the outer div is the parent and as per selenium way of doing things it would mean that products I stored as follows:


  product_1 = driver.find_element_by_xpath("//div/div[1]")
  product_2 = driver.find_element_by_xpath("//div/div[2]")
  product_3 = driver.find_element_by_xpath("//div/div[3]")
  product_4 = driver.find_element_by_xpath("//div/div[4]")

So my objective would is to search for a product and after getting the list target the box's a tag and then click it, go to extract more details on the product and then go back without closing the browser till my list is finished and below is my solution:

 for i in range(10):
    try:
        num = i   1
        path = f"//div/div[{num}]/span/a"
        poduct_click = driver.find_element_by_xpath(path)
        driver.execute_script("arguments[0].click();", poduct_click)
        scrape_product_detail() #function that scrapes the whole product detail
        driver.execute_script("window.history.go(-1)") # goes backwards to continue looping
    except NoSuchElementException:
        print('Element not found')

The problem is it works for the first product and it scrapes all the detail and then it goes back. Despite going back to the product page the program fails to find the second element and those coming afterwards and I am failing to understand what may be the problem. May you kindly assist. Thanks

CodePudding user response：

thanks @Debenjan you did help me a lot there. Your solution is working like a charm. For those who would want to know how I went about here is the following code:

        article_elements = self.find_elements_by_class_name("s-card-image")
        collection = []
        for news_box in article_elements:
            # Pulling the hotel name
            slug = news_box.find_element_by_tag_name(
                'a'
            ).get_attribute('href')

            collection.append(
                slug
            )
       for i in range(len(collection)):
            self.execute_script("window.open()")
            self.switch_to.window(self.window_handles[i 1])
            url = collection[i]
            self.get(url)
            print(self.title, url, self.current_url)

@A D thanks so much your solution is working too and I just will have to test and see whats the best strategy and go with it. Thanks a lot guys