How do I scrape?-CodePudding

import requests
from bs4 import BeautifulSoup

url = "https://www.jab.de/tr/en/productadvancedsearch?searchTerm=&page=1"
website = requests.get(url)
html = website.content
soup = BeautifulSoup(html,"html.parser")

urunListesi = soup.find("section",{"class":"results"}).find("div",{"class":"col-item details"})
# print(urunListesi)

for urun in urunListesi:
    link = urun.div.a.get("href")
    print(link)
    print("----------------------------\n")

when I operate to that code it returns None, can you help me ?

CodePudding user response：

Actually, data is generating by javascript from backdoor api calls json response that's why you can't grab data using BeautifulSoup and requests only. You can collect data from api response.

CodePudding user response：

try this

import requests
from bs4 import BeautifulSoup

url = "https://www.jab.de/tr/en/productadvancedsearch?searchTerm=&page=1"
website = requests.get(url)
html = website.content
print(html)
soup = BeautifulSoup(html, "html.parser")

c = soup.findAll("section",class_="results")

# print(c)


for x in c:
    d = x.find(class_="col-item details")
    print(d)