List out of index error even the index is in range, selenium python-CodePudding

As title says everything i am getting this error , sometime it comments and 100 posts and sometimes 2 or 3 than i get the same error upvote[count].click() IndexError: list index out of range even the class is in range

Basically what i am trying to do is automate commenting on Quora only for educational purpose , i tried searching everywhere but didn't found solution, some told me to add time.sleep() function and all but nothing working at all. please help me to solve this it will be much appreciated

You can see the image here where it shows the class where i want to add a comment

here is my code

driver = webdriver.Chrome("chromedriver.exe")
url = "https://quora.com/"
driver.get(url)

for i in range(200):
    print(count)
    wait = WebDriverWait(driver, 20)
    upvote = wait.until(
    EC.presence_of_all_elements_located((By.NAME, "Comment")))
    upvote[count].click()
    time.sleep(3)
    click_comment_bar = wait.until(
        EC.presence_of_all_elements_located((By.XPATH,"//*[@class='doc empty']")))
    click_comment_bar[count].click()
    comments = "Hello"
    click_comment_bar[count].send_keys(comments)
    time.sleep(3)
    post_comment = wait.until(
        EC.presence_of_all_elements_located((By.XPATH, "//*[@class='q-box qu-ml--tiny']")))
    post_comment[count].click()

Error

Traceback (most recent call last):
File "c:\Users\quora.py.py", line 14
    click_comment_bar[count].click()
IndexError: list index out of range
PS C:\Users\Quora> [7244:3456:0101/120500.531:ERROR:gpu_init.cc(457)] Passthrough is not 
supported, GL is disabled, ANGLE is

CodePudding user response：

Your main issue is related to the use of presence_of_all_elements_located

I will invite you to read the py doc of the class used:

class presence_of_all_elements_located(object):
    """ An expectation for checking that there is at least one element present
    on a web page.
    locator is used to find the element
    returns the list of WebElements once they are located
    """

As mentioned as soon as one element is present, this method won't wait longer. This is an issue in your case as each iteration you click on a new "Comment" which take a certain time for the comment textfield to appear. But since there are already some loaded in the DOM the call will resolve right away.

You were definitely on the right path.. time.sleep() is to avoid as much as possible. It will just slow down your code and it adds instability.

Solution: Use nested query

This option will be more accurate and target each element, instead of relying on the previous loop query. This involves re-writing some code.. but bare with me!

With this solution, you won't have to care about indexing.. right this will fix the issue mentioned in your title.

To do so, we will target nested elements. Each post are wrapped under a specific container. We will iterate over that list using the following class

post_containers = .q-box.qu-pt--medium

Then we will use this list and iterate over it

for el in driver.find_element(By.CSS_SELECTOR, post_containers): ...

The el will become our driver. Why? Because the next call to find an element will only resolve its children. This is true for CSS_SELECTOR, but not for Xpath...

comment_button_locator = [name="Comment"]

Now we will click on that "Comment" button using the el

el.find_element(By.CSS_SELECTOR, comment_button_locator).click()

Here, you could've tested the difference using el vs driver

print(f"Nbr of elements found using el: {len(el.find_elements(By.CSS_SELECTOR, comment_button_locator))}")

print(f"Nbr of elements found using the original driver: {len(driver.find_elements(By.CSS_SELECTOR, comment_button_locator))}")

Once this is done, since the text field to enter a comment take some time, we will create our wait instance:

wait = WebDriverWait(el, 15) # Using the el.. not the driver so it must be created for each iteration

At that point we can wait for the textfield to become "present".

text_field_locator = '.doc.empty' click_comment_bar = wait.until( EC.presence_of_all_elements_located((By.CSS_SELECTOR, text_field_locator)))

The site also does not load all the post all in one, if you are looking how to load them all or more.. please have a look at this post infinite scrolling