I am trying toscrape some financial data but i keep getting empty lists whenever i use selenium or beautiful soup

this is the code i used with selenium

driver = webdriver.Chrome(os.getcwd()   '/chromedriver')
driver.get("https://www.amarkets.com/research-education/sentiment-widget/")
time.sleep(3)

real_soup = BeautifulSoup(driver.page_source, 'html.parser')
sentiments = real_soup.findAll("TD")
print(sentiments)

i keep getting an empty list.

CodePudding user response：

Don't scrape it - use the api (which you can find by checking network activity):

import requests
import pandas as pd

r = requests.get('https://api.prod.amarkets.dev/v1/sentiment-widget-dev/?request={"Symbols":["EURUSD","GBPUSD","USDJPY","AUDUSD","NZDUSD","USDCAD","USDCHF","EURJPY","EURGBP","XAUUSD","XAGUSD","DowJones30","S&P500","Nasdaq100","BTCUSD","BRENT"],"NodesType":"Real","CollectSuffixes":true,"RequestName":"RequestWidgetSentiments"}')
df = pd.DataFrame(r.json()['Result'])

Result: