I am trying to make a webscraper in order to have a user search for characters, comics and other things related to their search from https://www.marvel.com/search

however, no matter what I do I can't figure out how to get the search function to work. I have two codes on my github, below is my most recent attempt that I found from stackoverflow.

from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

user_input = input("character: ")
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1080")

driver = webdriver.Chrome("C:\Program Files (x86)\chromedriver.exe")
driver.get("https://www.marvel.com/search")

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span.nav-icon.gcom-icon-search"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#searchString"))).send_keys(user_input   Keys.RETURN)

https://github.com/chrismc0723/WebScrapers/blob/MCU/MCU.py https://github.com/chrismc0723/WebScrapers/blob/MCU/MCU2.py

CodePudding user response：

Try this:

from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

user_input = input("character: ")
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1080")

driver = webdriver.Chrome("D:\Downloads\chromedriver\chromedriver.exe")
driver.get("https://www.marvel.com/search")

search_input_xpath = "//input[@placeholder='Search']"
search_input = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, search_input_xpath)))
search_input.send_keys(user_input)

first_item_in_auto_suggest_area_xpath = "//div[contains(@id,'react-autowhatever')]/ul"
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, first_item_in_auto_suggest_area_xpath)))
search_input.send_keys(Keys.ENTER)

Notes:

1. Pay attention that in the code you never use options variable.
2. Running the code you may see warning DeprecationWarning: executable_path has been deprecated, please pass in a Service object. To fix it - follow suggestions from this answer.