I'm trying to scrape this site URL: https://statusinvest.com.br/fundos-imobiliarios/urpr11 to get from a table the dividends info from this specific REIT (I'll later generalize this). This is the table that contains the info:

dividends table

I was able to get the dates and values from the table, but only for the first page. When I change the table page there's no modification in the website URL so I actually don't know how to deal with this. Any help would be appreciated.

Obs: It would be nice if the way to solve doesn't depende on the amount of pages because some REITs can have more than 2 pages of info.

This is how I'm currently taking the info from the first page

from bs4 import BeautifulSoup
import requests


URL = "https://statusinvest.com.br/fundos-imobiliarios/urpr11"
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")
test = soup.find_all("tr", class_="")

rows = []
for r in test:
  if not r.find("td", title="Rendimento"):
    continue
  row = []
  for child in r.findChildren():
    if child.text.lower()=="rendimento":
      continue
    print(child.text)
    row.append(child.text)
  rows.append(row)

CodePudding user response：

Content is provided dynamically by JavaScript, what requests per se is not rendering, so you wont get all the data that way.

How to fix?

You could use selenium to interact with the website like humans would do it in the browser - Something for later and more complicated issues.

But in this case it is much more simple and do not need selenium. Just grab the JSON data JavaScript is using to provide the table:

data = json.loads(soup.select_one('#results')['value'])

Convert it into DataFrame adjust for your needs and save it to csv,json, ....

pd.DataFrame(data).to_csv('yourFile.csv', index=False)

There are more columns as displayed on the website, take a look at the output of the example. These adjustments will give you the expected ones by only reading specific data and renaming column headers:

df = pd.DataFrame(data, columns=['et','ed', 'pd', 'v'])
df.columns = ['TIPO','DATA COM','PAGAMENTO','VALOR']
df.to_csv('yourFile.csv', index=False)

Example

from bs4 import BeautifulSoup
import requests, json
import pandas as pd


URL = "https://statusinvest.com.br/fundos-imobiliarios/urpr11"
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

data = json.loads(soup.select_one('#results')['value'])
pd.DataFrame(data)

#or with adjustment as mentioned above
#df = pd.DataFrame(data, columns=['et','ed', 'pd', 'v'])
#df.columns = ['TIPO','DATA COM','PAGAMENTO','VALOR']
#df.to_csv('yourFile.csv', index=False)

Output