AttributeError: 'NoneType' object has no attribute 'get' helpp-CodePudding

I need to take link to the latest user tiktok video. it worked, but then stopped working

import bs4
import requests
from fake_useragent import UserAgent
from bs4 import BeautifulSoup

link = "https://www.tiktok.com/@dava_m?"
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36 OPR/81.0.4196.61'}
full_usd_page = requests.get(link, headers=headers).text
soup = BeautifulSoup(full_usd_page, 'html.parser')
product_link = soup.find("a", class_="").get("href")
print(link)

product_link = soup.find("a", class_="").get("href")
AttributeError: 'NoneType' object has no attribute 'get'

it returns none, but an hour ago it worked. pls help to get latest video link https://www.tiktok.com/@dava_m?

CodePudding user response：

This means soup.find("a", class_="") is returning a nonetype object. Try running it by itself without the get bit and check the output. Make sure the output of the following is not None.

soup.find("a", class_="")

It's possible that the function can't find anything.

CodePudding user response：

Probably the page is dynamically generated, and it's not sufficient to use simply requests. But other methods are quite more sophisticated. They might be selenium, scrapy-splash, etc. Search the web for python scrape dynamic content/page.