I am looking to scrape the following web page, where I wish to scrape all the text on the page, including all the clickable elements.

I've attempted to use requests:

import requests
response = requests.get("https://cronoschimp.club/market/details/2424?isHonorary=false")
response.text

Which scrapes the meta-data but none of the actual data.

Is there a way to click through and get the elements in the floating boxes?

CodePudding user response：

As it's a Javascript enabled web page, you can't get anything as output using requests, bs4 because they can't render javascript. So, you need an automation tool something like selenium. Here I use selenium with bs4 and it's working fine. Please, see the minimal working example as follows:

Code:

from bs4 import BeautifulSoup
import time
from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
driver.maximize_window()
time.sleep(8)

url = 'https://cronoschimp.club/market/details/2424?isHonorary=false'
driver.get(url)
time.sleep(20)

soup = BeautifulSoup(driver.page_source, 'lxml')
name = soup.find('div',class_="DetailsHeader_title__1NbGC").get_text(strip=True)
p= soup.find('span',class_="DetailsHeader_value__1wPm8")
price= p.get_text(strip=True) if p else "Not for sale"
print([name,price])

Output:

['Chimp #2424', 'Not for sale']