How do I scrape multiple URLs using Selenium web driver?

Below is my code scraping a title from one amazon link. I want to add more links and scrape multiple titles. Thank you.

import selenium
from selenium import webdriver
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)

links = ("https://www.amazon.com/Nintendo-Switch-Neon-Blue-Joy‑/dp/B07VGRJDFY/?_encoding=UTF8&pd_rd_w=tUivq&pf_rd_p=cc3d6e86-d477-4b0e-8ce3-a46ea2292f65&pf_rd_r=C62K87403EP0W3NW9T3B&pd_rd_r=5d895c99-d866-4e36-b06c-5e9d0a5d48d4&pd_rd_wg=vuVXa&ref_=pd_gw_crs_zg_bs_468642")

driver.get(links)
driver.implicitly_wait(5)
print(driver.title)
driver.quit()

CodePudding user response：

I did something similar a while ago, I used loops for that instance. It is longer and more complicated than this, but here is the functional, shortened and customised version for you:

import selenium
from selenium import webdriver

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
website_list = ["YOUR_WEBISTE1", "YOUR_WEBISTE2"]

for website in website_list:
    driver.get(website)
    # PARSE THE WEBSITE
    driver.implicitly_wait(5)
    print(driver.title)

driver.quit()

The program parses multiple websites and uses them accordingly. But remember that depending on your parsing process, it takes time. You can always use multiprocessing, but this program was not time-critical.

CodePudding user response：

As your code successfully scrapes the title from one amazon link, so to scrape multiple titles from multiple amazon links you need to have the links in a list and then iterate through the list of urls invoking each url through get() and scrape the title using the following solution:

• Code Block:

  driver = webdriver.Chrome(service=s, options=options)
  urls = ['https://www.amazon.com/Nintendo-Switch-Neon-Blue-Joy‑/dp/B07VGRJDFY/?_encoding=UTF8&pd_rd_w=tUivq&pf_rd_p=cc3d6e86-d477-4b0e-8ce3-a46ea2292f65&pf_rd_r=C62K87403EP0W3NW9T3B&pd_rd_r=5d895c99-d866-4e36-b06c-5e9d0a5d48d4&pd_rd_wg=vuVXa&ref_=pd_gw_crs_zg_bs_468642&th=1', 'https://www.amazon.com/Minecraft-Nintendo-Switch/dp/B07D13QGXM/ref=sr_1_3?keywords=microsoft switch&pd_rd_r=ed7260ed-ee5d-4e77-8d78-f04c9f092d11&pd_rd_w=tMvvl&pd_rd_wg=wSdVv&pf_rd_p=baee3516-e45b-42ba-859e-de5632f9c487&pf_rd_r=2N2FBHCVGWEKX1810Z8P&qid=1650487774&sr=8-3']
  for url in urls:
    driver.get(url)
    print(driver.title)
  driver.quit()
  

• Console Output:

  Amazon.com: Nintendo Switch with Neon Blue and Neon Red Joy‑Con - HAC-001(-01) : Video Games
  Amazon.com: Minecraft - Nintendo Switch : Nintendo of America: Video Games