Trouble Scraping a loading score-CodePudding

I'm trying to scrape a score from a page. But i honestly can't get myself on the path of getting anywhere close. it's sandwich between a ::before after::. Googling that has led me to probably needing selenium? I've tried Beautiful Soup and Selenium but not getting anywhere. Below is the best(it didn't return an error) that i've gotten. But didn't return anything i can understand. [<selenium.webdriver.remote.webelement.WebElement (session="d902be37b19adc23f00bcaa20ecfc885", element="4064ab3c-7da4-4223-b8bb-c2fbb6590cbe")>]

from selenium import webdriver
from selenium.webdriver.common.by import By

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
URL = "https://coolcatsnft.com/user/0x10eb84abd429fa4df8dcabbc7c2803822a5b82d9"

driver.get(URL)

search = driver.find_elements(By.CLASS_NAME,"sc-8ce97ff6-2.cgjlQk")
print(search)
driver.quit()

CodePudding user response：

You can use the site's API to grab it using just the requests library. You just need to take the unique identifier from the end of the page url and append it to the API and then use json to extract the score.

import requests
import math
api = "https://prod.journey.coolcatsnft.com/v1/score/get/"
page = "https://coolcatsnft.com/user/0x10eb84abd429fa4df8dcabbc7c2803822a5b82d9"

request_url = api   page.split("/")[-1]


resp = requests.get(request_url)

data = resp.json()
score = data["result"]["overAllScore"]

print(score)
print(math.ceil(score))

output

115.5
116

CodePudding user response：

You were close enough.

[<selenium.webdriver.remote.webelement.WebElement (session="d902be37b19adc23f00bcaa20ecfc885", element="4064ab3c-7da4-4223-b8bb-c2fbb6590cbe")>]

indicates the WebElement itself, where as you need to extract the text.

Solution

To extract the text 116 you can use:

driver.get('https://coolcatsnft.com/user/0x10eb84abd429fa4df8dcabbc7c2803822a5b82d9')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='Accept cookies']"))).click()
print(driver.find_element(By.CSS_SELECTOR,".sc-8ce97ff6-2.cgjlQk").text)