Scraping javascript data from table or grid-CodePudding

I need to scrape data from a grid. I need all the data within the grid - (data containing Ticker , Score, Company, Exchange, and so on.

I am using Selenium to open the page and can't figure out what I am doing wrong. I've tried some tutorials but the code seems different. I've tried to scrape using the table "maintable-multi-1067-body" and I've also tried the grid "gridview-1070" but nothing is working. If anyone could show me how to do this, I'd appreciate it.

Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
import pandas as pd
    
options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
ser = Service("./chromedriver.exe")
browser = driver = webdriver.Chrome(service=ser)

driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
  "source": """
    Object.defineProperty(navigator, 'webdriver', {
      get: () => undefined
    })
  """
})
driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
wait = WebDriverWait(driver, 30)
driver.get("https://stockrover.com")
wait.until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[1]/div/section[2]/div/ul/li[2]"))).click()
user = driver.find_element(By.NAME, "username")
password = driver.find_element(By.NAME, "password")
user.clear()
user.send_keys("test11964")
password.clear()
password.send_keys("test21964")
driver.find_element(By.NAME, "Sign In").click()
wait = WebDriverWait(driver, 30)


wait = WebDriverWait(driver,30)

table = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'tbody#maintable-multi-1067-body tr')))

for tab in table:
  print(tab.text)

CodePudding user response：

You are very close to the working code but there are still several issues here:

Your table locator is wrong.
Instead of

'tbody#maintable-multi-1067-body tr'

Try using Instead of

'#maintable-multi-1067-body tr'

It will give you 502 results.
2) No need to define 3 times the wait object

wait = WebDriverWait(driver,30)