Home > database >  How to add wait between urls in scraping python
How to add wait between urls in scraping python

Time:03-01

i want to add wait between scraping these urls. i want to scrape 2 urls every minute so 30 second wait will be enough but don't know how to add wait inbetween urls. newbie here thanks for helping!

import cloudscraper
from bs4 import BeautifulSoup
scraper = cloudscraper.create_scraper()
urls = ["https://www.brandbucket.com/names?page=1","https://www.brandbucket.com/names?page=2","https://www.brandbucket.com/names?page=3","https://www.brandbucket.com/names?page=4","https://www.brandbucket.com/names?page=5"]
for url in urls:
    r = scraper.get(url)
    html = r.content
    soup = BeautifulSoup(html, 'html.parser')
    titles = soup.find_all("div", class_="domainCardDetail")
    for title in titles:
        print(title.text)

CodePudding user response:

You can use time.sleep()

import the time module with import time

then use time.sleep("number of seconds you want to wait")

  • Related