driver.get(f"https://www.morningstar.com/stocks/{market_value}/{ticker_value}/financials")
            WebDriverWait(driver, 50).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(., 'Income Statement')]"))).click()
            WebDriverWait(driver, 50).until(EC.element_to_be_clickable((By.XPATH, "//a[contains(., 'Expand Detail View')]"))).click()
            data = WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='sal-columns sal-small-12']"))).get_attribute("outerHTML")

I have this code to scrape the table in morningstar, i can click the buttons but i cant scrape the div that contains the data.

Here is what the i did with the selenium:

1. It will open the website and go to the tab.
2. Inside the tab, a button with 'expand' will be clicked.
3. Once click, it will show a table.

Im trying to get the data inside the table

CodePudding user response：

The data you want is contained in the div with the class equity-financials-bottom. Since it's not a table element, you'll have to do the parsing into a dataframe shape yourself.

At the same time, why not use the Export Data functionality? It gives you an .xls file that should be easier to parse.

CodePudding user response：

You can try to wait using time.sleep() after you click on the button.

I tried before to use WebDriverWait but it didn't work.

Could you give me one of the URLs and I may help you