AmCharts Scraping Using Python Selenium-CodePudding

I am trying to Scrape AmCharts Graph Values in JSON Object from this

It appears like this in the web interface

How I can retrieve this dataProvider Array correctly. Thanks in advance.

CodePudding user response：

The graph is not in Selenium view port, so first we will have to deal with vertical scrolling to the desired graph, and then I see //*[name()='tspan'] xpath contains, horizontal and vertical values which is present in UI as well.

Code:

driver_path = r'D:\\chromedriver.exe'

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
wait = WebDriverWait(driver, 30)

driver.get("https://eg.pricena.com/en/product/oppo-reno-5g-price-in-egypt")

driver.execute_script("window.scrollTo(0, 1000)")

ele = driver.find_element_by_xpath("//*[name()='svg']")
driver.execute_script("arguments[0].scrollIntoView(true);", ele)

char_val = []
for elem in driver.find_elements(By.XPATH, "//*[name()='tspan']"):
    print(elem.text)

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output:

Mar
May
Jul
Sep
Nov
Mar
May
Jul
Sep
Nov
5,000
5,500
6,000
6,500
7,000
7,500
8,000

Process finished with exit code 0

CodePudding user response：

You need to add return to your execute_script call in order to access the value in your script, e.g. driver.execute_script("return AmCharts.charts[0].dataProvider")

Full code below:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait

driver_path = r"path/to/chromedriver"

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
wait = WebDriverWait(driver, 30)

driver.get("https://eg.pricena.com/en/product/oppo-reno-5g-price-in-egypt")

# scroll into the div so that the chart will render
driver.execute_script("document.getElementById('product_pricechart').scrollIntoView()")

# wait until the chart div has been rendered before accessing the data provider
wait.until(lambda x: x.find_element_by_class_name("amcharts-chart-div").is_displayed)

# display chart data
print(driver.execute_script("return AmCharts.charts[0].dataProvider"))

driver.close()