I want to scrap my YouTube subscriptions list into one csv file. I typed this code (but I didn't finish coding yet):
import requests
from bs4 import BeautifulSoup
import csv
url = 'https://www.youtube.com/feed/channels'
source = requests.get(url)
soup = BeautifulSoup(source, 'lxml')
I found this error:
File "/Users/hendy/YouTube subscriptions scraping.py", line 7, in soup = BeautifulSoup(source, 'lxml') File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/bs4/init.py", line 312, in init elif len(markup) <= 256 and ( TypeError: object of type 'Response' has no len()
I don't know what's the problem.
CodePudding user response:
What happens?
You using the whole response
object and push it to BeautifulSoup
what would not work.
How to fix?
To generate a BeautifulSoup
object use the content
or text
of your response:
BeautifulSoup(source.content, 'lxml')
Example
from bs4 import BeautifulSoup
import requests
headers ={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'}
url = 'https://www.youtube.com/feed/channels'
source = requests.get(url, headers=headers)
soup = BeautifulSoup(source.content, 'lxml')