i want to add wait between scraping these urls. i want to scrape 2 urls every minute so 30 second wait will be enough but don't know how to add wait inbetween urls. newbie here thanks for helping!
import cloudscraper
from bs4 import BeautifulSoup
scraper = cloudscraper.create_scraper()
urls = ["https://www.brandbucket.com/names?page=1","https://www.brandbucket.com/names?page=2","https://www.brandbucket.com/names?page=3","https://www.brandbucket.com/names?page=4","https://www.brandbucket.com/names?page=5"]
for url in urls:
r = scraper.get(url)
html = r.content
soup = BeautifulSoup(html, 'html.parser')
titles = soup.find_all("div", class_="domainCardDetail")
for title in titles:
print(title.text)
CodePudding user response:
You can use time.sleep()
import the time module with
import time
then use
time.sleep("number of seconds you want to wait")