Equivalent of time.sleep() in Selenium-CodePudding

I realize this is a relatively simple question but I haven't found the answer yet.

I'm using driver.get() in a for loop that iterates through some urls. To help avoid my IP address getting blocked, I've implemented time.sleep(5) before the driver.get statement in the for loop.

Basically, I just want a wait period to make my scraping seem more natural.

I think time.sleep may be causing page crashes. What is the equivalent of time.sleep in selenium? From what I understand, implicitly_wait just sets the amount of time before throwing an exception, but I'm not sure that's what I want here? I want a specific amount of time for the driver to wait.

CodePudding user response：

time.sleep()

The sleep() function is from the time module which suspends execution of the current thread for a given number of seconds.

Now, WebDriver being a out-of-process library which instructs the browser what to perform and at the same time the web browser being asynchronous in nature, WebDriver can't track the active, real-time state of the HTML DOM. This gives rise to some intermittent issues that arise from usage of Selenium and WebDriver those are subjected to race conditions that occur between the browser and the user’s instructions.

As of now Selenium doesn't have any identical method to time.sleep(), however there are two equavalent methods at your disposal and can be used as per the prevailing condition of your automated tests.

Implicit wait: In this case, WebDriver polls the DOM for a certain duration when trying to find any element. This can be useful when certain elements on the webpage are not available immediately and need some time to load.

def implicitly_wait(self, time_to_wait) -> None:
    """
    Sets a sticky timeout to implicitly wait for an element to be found,
       or a command to complete. This method only needs to be called one
       time per session. To set the timeout for calls to
       execute_async_script, see set_script_timeout.

    :Args:
     - time_to_wait: Amount of time to wait (in seconds)

    :Usage:
        ::

            driver.implicitly_wait(30)
  """
  self.execute(Command.SET_TIMEOUTS, {
      'implicit': int(float(time_to_wait) * 1000)})

Explicit wait: This type of wait allows your code to halt program execution, or freeze the thread, until the condition you pass it resolves. As an example:

CodePudding user response：

There is no specific method in Selenium for hardcoded pauses like a time.sleep() general Python method.
As you mentioned there is an implicitly_wait and Expected Conditions explicit WebDriverWait waits but both these are NOT a hardcoded pauses.
Both the implicitly_wait WebDriverWait are used for setting the timeout - how long time to poll for some element presence or condition, so if that condition is fulfilled or the element is presented the program flow will immediately continue to the next code line.
So, if you want to put a pause you have to use some general Python method that will suspend the program / thread run like the time.sleep().