I am trying to find menu prices for certain fast food restaurants from this website by different state. There is a dropdown menu where different states are the options. After I select a state (for example, California), I want to web scrape the different prices of their Ice Cream. However, I keep getting the same error message preventing me from getting the data. I think it works fine until going to the website, but it can't seem to select it by state. What do I need to do in order to:

1. Get selenium to accurately select a certain state (California)
2. Get the menu prices for just the "Ice Cream" category?

The Code is below:

!pip3 install selenium
!pip3 install webdriver-manager

from selenium import webdriver
import time 
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
import pandas as pd

path = ("C:/path/to/chromedriver.exe")

driver = webdriver.Chrome(executable_path = path)
url = "https://www.fastfoodmenuprices.com/baskin-robbins-prices/"
driver.get(url)
select_state = driver.find_element_by_xpath("//select[@class='tp-variation']").click
## select_state = driver.find_element_by_id("variation-tablepress-34")
select_state = driver.find_element_by_id("variation-tablepress-34")
select_state.select_by_value("MS4yOA==")
time.sleep(3)

The Error I get is:

AttributeError: 'WebElement' object has no attribute 'select_by_value'

A photo of the code error is below:

CodePudding user response：

select_by_value() is a method from select class and to use it you have to use the Select(webelement) method.

Solution

To select the option California you can use the following locator strategy:

• Code Block:

  driver.get("https://www.fastfoodmenuprices.com/baskin-robbins-prices")
  Select(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//select[@class='tp-variation']")))).select_by_value("MS4yOA==")
  

• Browser Snapshot: