I am trying to scrape the daily change value from the yahoo finance page, for example BTC: https://finance.yahoo.com/quote/BTC-USD and the screenshot of value I am looking for.

Below is my code:

import requests, json
from bs4 import BeautifulSoup

btc_url = 'https://finance.yahoo.com/quote/BTC-USD'
btc_page = requests.get(btc_url)
btc_soup = BeautifulSoup(btc_page.content, 'html.parser')
btc_change = btc_soup.find('div' , class_='D(ib) Mend(20px)').find('fin-streamer' ,class_ ='Fw(500) Pstart(8px) Fz(24px)').text
print (btc_change)

I do get data returned but it's always the value of -661.20 no matter what is actually on the webpage. Can someone check and let me know if I am doing something wrong? Thanks

CodePudding user response：

You're getting nonsense data because the request does not look legitimate. You need to include a user agent.

I'm trying to scrape stock prices off of Yahoo Finance, and I'm targeting the price. But when I run my code I get "None" in my output

Try this:

import requests, json
from bs4 import BeautifulSoup

btc_url = 'https://finance.yahoo.com/quote/BTC-USD'
headers = {'User-agent': 'Mozilla/5.0'}
btc_page = requests.get(btc_url, headers=headers)
btc_soup = BeautifulSoup(btc_page.content, 'html.parser')
btc_change = btc_soup.find('fin-streamer', {'data-symbol':'BTC-USD', 'data-field':'regularMarketChange'}).text

print(btc_change)