Parsing web-page with search bar-CodePudding

I need to parse store names (<div >) from https://www.comicshoplocator.com/StoreLocator. The thing is -- when you entering zip code (for instance 73533) in search it does not appear in the URL. Because of that python can't see elements on page. Here is my code snippet. I am receiving no output, because of that.

How to make python see input with zip code? Thanks

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup

browser = webdriver.Firefox(executable_path=r'C:\Geckodriver\Geckodriver.exe')
browser.get('https://www.comicshoplocator.com/StoreLocator')
browser.find_element(By.NAME, 'query').send_keys('73533'   Keys.RETURN)

html = browser.page_source
soup = BeautifulSoup(html, features="html.parser")
for tag in soup.find_all('div', class_="LocationName"):
    print(tag.text)

CodePudding user response：

The problem is in here: browser.find_element(By.NAME, 'query').send_keys('73533' Keys.RETURN)

The correct one would be:

search = browser.find_element(By.NAME, 'query')
search.send_keys('73533')
search.send_keys(Keys.RETURN)

Full working code:

I use chrome driver you can change that portion with no times

import time
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get('https://www.comicshoplocator.com/StoreLocator')
driver.maximize_window()
time.sleep(2)
d=driver.find_element(By.NAME, 'query')
d.send_keys('73533') 
d.send_keys(Keys.ENTER)

soup = BeautifulSoup(driver.page_source, 'lxml')
for tag in soup.find_all('div', class_="LocationName"):
    print(tag.text)

Output:

MARK DOWN COMICS
WWW.DCBSERVICE.COM

CodePudding user response：

This code worked for me:

listings = browser.find_elements(By.CLASS_NAME, 'CslsLocationItem')
for listing in listings:
    print(listing.find_element(By.CLASS_NAME,'LocationName').get_attribute('innerText'))

CodePudding user response：

Well, actually, this can be done with requests, there's no need to use Selenium. You can send a post request to:

https://www.comicshoplocator.com/StoreLocator

import re
import requests
from bs4 import BeautifulSoup

data = {
    "showAll": "false",
    "showCsls": "true",
    "query": "73533",
}

response = requests.post(
    "https://www.comicshoplocator.com/StoreLocator",
    data=data,
)

soup = BeautifulSoup(response.text, "html.parser")
string = soup.select_one("script:-soup-contains('address')").string
unformatted_data = re.search(r"\(({.*?})\)", string, re.DOTALL).group(1)
# remove all the whitespace
formatted_data = re.sub(r"\s ", "", unformatted_data)

print(formatted_data)

Prints:

{storeno:"8816",lat:"41.0671081542969",lng:"-85.1372680664063",name:"WWW.DCBSERVICE.COM",address:"6005ESHELBYDR",address2:"WWW.DCBSERVICE.COM",city:"MEMPHIS",state:"TN",zip:"38141",phone:"",hasProfile:"True",storeLogo:'/Image/CslsLogo/' "8816"}