BeautifulSoup find() returns None and findAll returns an empty list-CodePudding

Trying to scrape the change of price of the symbols on CNBC but I get an empty list, what am i doing wrong?

from bs4 import BeautifulSoup
import requests

url = "https://www.cnbc.com/nasdaq-100/"

response = requests.get(url)
content = response.text

soup = BeautifulSoup(content, "html.parser")

data = soup.find_all("td", {"class": "BasicTable-quoteDecline"})
print(data)

CodePudding user response：

As mentioned content is generated dynamically and is provided by additional calls that response with JSON data.

If you know the symbols you could request them and get all information (AMD and TSLA):

https://quote.cnbc.com/quote-html-webservice/restQuote/symbolType/symbol?symbols=AMD|TSLA&requestMethod=itv&noform=1&partnerId=2&fund=1&exthrs=1&output=json&events=1

This will give you a list of the requested symbols and additional data:

[{'symbol': 'AMD', 'symbolType': 'symbol', 'code': 0, 'name': 'Advanced Micro Devices Inc', 'shortName': 'AMD', 'onAirName': 'Advanced Micro', 'altName': 'Advanced Micro Devices', 'last': '76.95', 'last_timedate': '4:15 PM EDT', 'last_time': '2022-07-11T16:15:00.000-0400', 'changetype': 'DOWN', 'type': 'STOCK', 'subType': 'Common Stock', 'exchange': 'NASDAQ', 'source': 'Last NASDAQ LS, VOL From CTA', 'open': '77.85', 'high': '78.29', 'low': '75.84', 'change': '-2.40', 'change_pct': '-3.02%', 'currencyCode': 'USD', 'volume': '72,260,709', 'volume_alt': '72.3M', 'provider': 'CNBC QUOTE CACHE', 'previous_day_closing': '79.35', 'altSymbol': 'AMD.O', 'realTime': 'true', 'curmktstatus': 'POST_MKT', 'pe': '28.71', 'mktcapView': '124.698B', 'beta': '1.93', 'tendayavgvol': '90.44M', 'pcttendayvol': '0.8399', 'yrhiprice': '164.46', 'yrhidate': '11/30/21', 'yrloprice': '71.60', 'yrlodate': '07/05/22', 'eps': '2.68', 'sharesout': '1.621B', 'revenuettm': '18.876B', 'ROETTM': '10.98%', 'NETPROFTTM': '17.94%', 'GROSMGNTTM': '48.53%', 'TTMEBITD': '4.767B', 'DEBTEQTYQ': '3.23%', 'fpe': '17.73', 'feps': '4.34', 'streamable': '1', 'issue_id': '44307', 'issuer_id': '9238', 'countryCode': 'US', 'timeZone': 'EDT', 'ExtendedMktQuote': {'type': 'POST_MKT', 'source': 'Last NASDAQ LS, VOL From CTA', 'last': '77.18', 'last_timedate': '4:53 PM EDT', 'change': ' 0.23', 'change_pct': ' 0.30%', 'volume': '2,970,833', 'volume_alt': '3.0M', 'changetype': 'UP'}, 'EventData': {'next_earnings_date': '07/25/2022(est)', 'next_earnings_date_today': 'N', 'announce_time': 'A', 'yrhiind': 'N', 'yrloind': 'N', 'is_halted': 'N'}}, {'symbol': 'TSLA', 'symbolType': 'symbol', 'code': 0, 'name': 'Tesla Inc', 'shortName': 'TSLA', 'onAirName': 'Tesla', 'altName': 'Tesla', 'last': '703.03', 'last_timedate': '4:00 PM EDT', 'last_time': '2022-07-11T16:00:00.000-0400', 'changetype': 'DOWN', 'type': 'STOCK', 'subType': 'Common Stock', 'exchange': 'NASDAQ', 'source': 'Last NASDAQ LS, VOL From CTA', 'open': '756.31', 'high': '759.19', 'low': '700.88', 'change': '-49.26', 'change_pct': '-6.55%', 'currencyCode': 'USD', 'volume': '31,878,440', 'volume_alt': '31.9M', 'provider': 'CNBC QUOTE CACHE', 'previous_day_closing': '752.29', 'altSymbol': 'TSLA.O', 'realTime': 'true', 'curmktstatus': 'POST_MKT', 'pe': '95.02', 'mktcapView': '706.025B', 'beta': '2.13', 'tendayavgvol': '28.93M', 'pcttendayvol': '1.1417', 'yrhiprice': '1,243.49', 'yrhidate': '11/04/21', 'yrloprice': '620.57', 'yrlodate': '05/24/22', 'eps': '7.40', 'sharesout': '1.004B', 'revenuettm': '62.19B', 'ROETTM': '29.42%', 'NETPROFTTM': '13.60%', 'GROSMGNTTM': '27.10%', 'TTMEBITD': '12.776B', 'DEBTEQTYQ': '14.12%', 'fpe': '57.21', 'feps': '12.29', 'streamable': '1', 'issue_id': '24812378', 'issuer_id': '74213', 'countryCode': 'US', 'timeZone': 'EDT', 'ExtendedMktQuote': {'type': 'POST_MKT', 'source': 'Last NASDAQ LS, VOL From CTA', 'last': '702.65', 'last_timedate': '4:53 PM EDT', 'change': '-0.38', 'change_pct': '-0.05%', 'volume': '206,060', 'volume_alt': '206.1K', 'changetype': 'DOWN'}, 'EventData': {'next_earnings_date': '07/20/2022', 'next_earnings_date_today': 'N', 'announce_time': 'A', 'yrhiind': 'N', 'yrloind': 'N', 'is_halted': 'N'}}]

Example

import requests
jsonData =requests.get('https://quote.cnbc.com/quote-html-webservice/restQuote/symbolType/symbol?symbols=AMD|TSLA&requestMethod=itv&noform=1&partnerId=2&fund=1&exthrs=1&output=json&events=1').json()

for s in jsonData['FormattedQuoteResult']['FormattedQuote']:
    print(s['symbol'],s['change'] )

Output

AMD -2.40
TSLA -49.26

Else you could use selenium to render website like a browser would do.