find() Method Can't Find What I Can-CodePudding

I have been working on a web scraper in python to scrape Google Finance, but I can't find the specific tag I'm looking for using the find() method. Finally, I got so annoyed that I decided to write the returned data to a file and look for it myself. So I wrote it to testing.html in the same directory, and opened it with Google Chromium so I could use the inspect tool. Within minutes, I found the element I was looking for. What am i doing wrong? My code is attached below:

import dryscrape

session = dryscrape.Session()


def get(url):
    global session
    try:
        session.visit(url)
        data = session.body()
    except:
        print('Connection Failed')
    return str(data)

def save(price, stockname):
    pass

def extract(data):
    return data.find('<div >')

class following():
    apple = "https://www.google.com/finance/quote/AAPL:NASDAQ"
    tesla = "https://www.google.com/finance/quote/TSLA:NASDAQ"
    google = "https://www.google.com/finance/quote/GOOGL:NASDAQ"
    amazon = "https://www.google.com/finance/quote/AMZN:NASDAQ"
    microsoft = "https://www.google.com/finance/quote/MSFT:NASDAQ"
    netflix = "https://www.google.com/finance/quote/NFLX:NASDAQ"
    def __init__():
        global apple
        global tesla
        global google
        global amazon
        global microsoft
        global netflix
        save(extract(get(following.apple)), following.apple)
        save(extract(get(following.tesla)), following.tesla)
        save(extract(get(following.google)), following.google)
        save(extract(get(following.amazon)), following.amazon)
        save(extract(get(following.microsoft)), following.microsoft)
        save(extract(get(following.netflix)), following.netflix)

f = open("testing.html")
print(extract(f.read()))
f.close()

CodePudding user response：

Why don't you try using requests and BeautifulSoup library instead. The following is what I meant.

import requests
from bs4 import BeautifulSoup


class following():

    def __init__(self):
        self.session = requests.Session()
        self.session.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'

    def get_last_price(self,link):
        r = self.session.get(link)
        return BeautifulSoup(r.text,"lxml")

    def extract(self,soup):
        return soup.select_one("[data-exchange='NASDAQ']")['data-last-price']


if __name__ == '__main__':
    base = "https://www.google.com/finance/quote/{}:NASDAQ"
    scraper = following()

    for ticker in ['AAPL','TSLA','GOOGL','AMZN','MSFT','NFLX']:
        soup_object = scraper.get_last_price(base.format(ticker))
        print(scraper.extract(soup_object))

CodePudding user response：

Found the issue: It's not YMLKec but YMlKec. Not a capital L.

data = open("testing.html", "r").read()
class_ = "YMlKec fxKbKc"
print(data.find(class_))
>>> 992880