how to scrape this interactive chart data at desired datetime?-CodePudding

I am currently attempting to scrape this website to print all data in the blue rectangle from https://mempool.jhoenicke.de/#BTC,6m,weight

Desired point that I want to scrape

I would like to scrape the text in all the individual tooltips because I can see that the data is under the id="tooltip" like this

data under id="tooltip"

I tried to scrape it with selenium by click and hold on the element id = "tooltip" but it doesn't work

    from selenium import webdriver
    from selenium.webdriver import ActionChains
    from selenium.webdriver.common.action_chains import ActionChains
    import time
    from time import sleep
    from random import randint
    website = 'https://jochen-hoenicke.de/queue/#BTC,6m,weight'
     
    path = '/Users/LENOVO/Downloads/chromedriver'
    driver = webdriver.Chrome(path)
    driver.get(website)
      
    element = driver.find_element("xpath", '//*[@id="tooltip"]')
    element.click()
      
    element2 = driver.find_element("style", '//*[@id="tooltip"]')
    action = ActionChains(driver)
    action.click_and_hold(element2)
    action.perform()
       
    time.sleep(3)
    memdate = driver.find_element("xpath",'//[@id="tooltip"]/strong').text
    print(memdate)
        
    action.release(element2)
    action.perform()

but it has failed since element.click()

I just want to know that did I go to the right direction or Could you guide me the right way to get the data from the table under tooltip at the desired datetime in tag strong.

Thank you so very much in advance.

CodePudding user response：

Essentially you need to move your mouse in a horizontal line across the page, near to the bottom of the chart, and record the tooltip content each time that it changes.

# wait until key elements are loaded
canvas = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, "canvas[class='flot-overlay']")))
WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#tooltip")))
prev_date = "UNKNOWN"
action = webdriver.ActionChains(driver)
action.move_to_element(canvas).move_by_offset(0, 10).click().perform()
for i in range(20):
    action.move_by_offset(1, 0).click().perform() 
    try:
        WebDriverWait(driver, 2).until_not(EC.text_to_be_present_in_element((By.CSS_SELECTOR, "#tooltip strong"), prev_date))
        tooltip_date = driver.find_element(By.CSS_SELECTOR, "#tooltip strong")
        tooltip_table = driver.find_element(By.CSS_SELECTOR, "#tooltip table")
        print("%s\n%s" % (tooltip_date.text, tooltip_table.text))
        prev_date = tooltip_date.text
    except TimeoutException:
        # tooltip date has not changed so keep moving
        continue

Starting point, stopping point and increment will need to be refined in order to pull all values from the chart, but this shows the general idea.