I am trying to scrape the href attribute from links from a page, but I end up with [] as the output
My desired output is: https://www.pigiame.co.ke/listings/nissan-latio-2016-36000-kms-5300124
CodePudding user response：
You can try:
import re
import requests
import urllib.parse
from bs4 import BeautifulSoup
url = "https://www.pigiame.co.ke/cars"
headers = {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0",
}
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")
links = [
item.a["href"] for item in soup.select(".listings-cards__list-item:has(a)")
]
for link in links:
soup = BeautifulSoup(requests.get(link).content, "html.parser")
data = soup.select_one(".btn-whatsapp")["href"]
data = urllib.parse.unquote(data)
phone = re.search(r"phone=(.*?)&", data).group(1)
print("{:<20} {}".format(phone, link))
Prints:
254723099904 https://www.pigiame.co.ke/listings/nissan-note-5253578
254722935411 https://www.pigiame.co.ke/listings/honda-freed-7-seater-2015-5291221
254722763845 https://www.pigiame.co.ke/listings/2006-bmw-x3-2500cc-petrol-155000kms-5241375
254722710833 https://www.pigiame.co.ke/listings/mazda-mpv-2006-5273382
254713193417 https://www.pigiame.co.ke/listings/landrover-109-very-clean-accident-free-5282118
254708467397 https://www.pigiame.co.ke/listings/landcrusser-prado-tx-fully-loaded-with-sunroof-2016-model-5304294
254708467397 https://www.pigiame.co.ke/listings/landcrusser-prado-tx-fully-loaded-with-sunroof-5304293
254708467397 https://www.pigiame.co.ke/listings/mistubishi-canter-2016-model-3tones-5304291
254708467397 https://www.pigiame.co.ke/listings/hillux-revolution-2016-model-fully-loaded-5304288
254708467397 https://www.pigiame.co.ke/listings/toyota-prado-2016-model-fully-loaded-with-sunroof-5304285
254708467397 https://www.pigiame.co.ke/listings/toyota-prado-txl-diesel-fuel-2017-model-5304283
254769333436 https://www.pigiame.co.ke/listings/toyota-fielder2015kdkbelow-50000km-5304279
254708467397 https://www.pigiame.co.ke/listings/subaru-forester-sti-turbo-2016-model-5304280
254769333436 https://www.pigiame.co.ke/listings/honda-fit2015kdkbelow-60000km-5304276
254769333436 https://www.pigiame.co.ke/listings/honda-vezel2015kdklow-milleage-5304273