Has anyone ever scrapped (e.g. into dataframe) financial statements available at roic.ai?

The source code of the page is very nested and obtaining the statements is not straightforward:

from gazpacho import get, Soup

ticker = 'aapl'
url = f'https://roic.ai/financials/{ticker}?fs=annual'
print(url)

html = get(url)
soup = Soup(html)

soup.find('div', {'class', "flex-col"})

CodePudding user response：

You can load the Json data from the <script> inside the page:

import re
import json
import requests
from bs4 import BeautifulSoup

ticker = "aapl"
url = f"https://roic.ai/financials/{ticker}?fs=annual"


soup = BeautifulSoup(requests.get(url).content, "html.parser")
data = json.loads(soup.select_one("#__NEXT_DATA__").text)

# umcomment this to print all data:
# print(json.dumps(data, indent=4))

# load sample data as pandas DataFrame
df = pd.DataFrame(data["props"]["pageProps"]["data"]["data"]["bsq"])
print(df.head().to_markdown(index=False))

Prints:

CodePudding user response：

from gazpacho import Soup
import json
import pandas as pd

ticker = 'aapl'
url = f'https://roic.ai/financials/{ticker}?fs=annual'
soup = Soup.get(url)
scrapped_data = soup.find('script', {'id': "__NEXT_DATA__"})
data = json.loads(scrapped_data.text)
df = pd.DataFrame(data["props"]["pageProps"]["data"]["data"]["bsq"])
print(df.head())

It can be implemented like this. Don't forget to import pandas and JSON libraries.