I am writing a web scraper and am struggling to get the href link from a web page. The URL is https://fortune.com/100-fastest-growing-companies/2021/search/. I am trying to get the href pull all of the article links. Here is an example of an href I am trying to pull"
<a trackerdata="[object Object] href="https://fortune.com/company/atlanticus-holdings/100-fastest-growing-companies/"><div ><span><div>Atlanticus Holdings</div></span></div></a>
Here is my code:
import pandas as pd
import os
import requests
from bs4 import BeautifulSoup
import re
URL = "https://fortune.com/100-fastest-growing-companies/2021/search/"
headers={'User-Agent':'mozilla/5.0'}
page = requests.get(URL,headers=headers)
soup = BeautifulSoup(page.text, "html.parser")
links = []
for link in soup.findAll(class_="searchResults__cellWrapper--39MAj"):
links.append((link.get('href')))
print(links)
When I run my code, it gives me nothing. Can someone point me in the right direction?
CodePudding user response：
This page uses JavaScript to add elements
But it reads data as JSON from
and it can be used to get links
import requests
url = 'https://fortune.com/wp-json/irving/v1/data/franchise-search-results?list_id=3159847&token=Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM='
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(url, headers=headers)
data = response.json()
for item in data[1]['items']:
print(item['permalink'])
Result:
https://fortune.com/company/progressive/100-fastest-growing-companies/
https://fortune.com/company/adobe-systems/100-fastest-growing-companies/
https://fortune.com/company/salesforce-com/100-fastest-growing-companies/
https://fortune.com/company/newmont-mining/100-fastest-growing-companies/
https://fortune.com/company/lennar/100-fastest-growing-companies/
https://fortune.com/company/kla-tencor/100-fastest-growing-companies/
https://fortune.com/company/jd-com/100-fastest-growing-companies/
https://fortune.com/company/jacobs-engineering-group/100-fastest-growing-companies/
https://fortune.com/company/cigna/100-fastest-growing-companies/
https://fortune.com/company/charles-river-laboratories-intl/100-fastest-growing-companies/
https://fortune.com/company/advanced-micro-devices/100-fastest-growing-companies/
https://fortune.com/company/paypal-holdings/100-fastest-growing-companies/
https://fortune.com/company/netflix/100-fastest-growing-companies/
https://fortune.com/company/alphabet/100-fastest-growing-companies/
https://fortune.com/company/amazon-com/100-fastest-growing-companies/
https://fortune.com/company/veeva-systems/100-fastest-growing-companies/
https://fortune.com/company/align-technology/100-fastest-growing-companies/
https://fortune.com/company/etsy/100-fastest-growing-companies/
https://fortune.com/company/lululemon-athletica/100-fastest-growing-companies/
https://fortune.com/company/trade-desk/100-fastest-growing-companies/
https://fortune.com/company/emergent-biosolutions/100-fastest-growing-companies/
https://fortune.com/company/hibbett/100-fastest-growing-companies/
https://fortune.com/company/business-first-bancshares/100-fastest-growing-companies/
https://fortune.com/company/marinemax/100-fastest-growing-companies/
https://fortune.com/company/spirit-of-texas-bancshares/100-fastest-growing-companies/
https://fortune.com/company/ies-holdings/100-fastest-growing-companies/
https://fortune.com/company/brooks-automation/100-fastest-growing-companies/
https://fortune.com/company/mid-penn-bancorp/100-fastest-growing-companies/
https://fortune.com/company/strategic-education/100-fastest-growing-companies/
https://fortune.com/company/waterstone-financial/100-fastest-growing-companies/
https://fortune.com/company/g-willi-food-international/100-fastest-growing-companies/
https://fortune.com/company/marketaxess-holdings/100-fastest-growing-companies/
https://fortune.com/company/ameris-bancorp/100-fastest-growing-companies/
https://fortune.com/company/shyft-group/100-fastest-growing-companies/
https://fortune.com/company/smartfinancial/100-fastest-growing-companies/
https://fortune.com/company/superior-group-of-companies/100-fastest-growing-companies/
https://fortune.com/company/meta-financial-group/100-fastest-growing-companies/
https://fortune.com/company/addus-homecare/100-fastest-growing-companies/
https://fortune.com/company/metropolitan-bank-holding/100-fastest-growing-companies/
https://fortune.com/company/1-800-flowers-com/100-fastest-growing-companies/
https://fortune.com/company/fs-bancorp/100-fastest-growing-companies/
https://fortune.com/company/arch-capital-group/100-fastest-growing-companies/
https://fortune.com/company/atlanticus-holdings/100-fastest-growing-companies/
https://fortune.com/company/b-riley-financial/100-fastest-growing-companies/
https://fortune.com/company/joint-corp/100-fastest-growing-companies/
https://fortune.com/company/quidel/100-fastest-growing-companies/
https://fortune.com/company/acm-research/100-fastest-growing-companies/
https://fortune.com/company/victory-capital-holdings/100-fastest-growing-companies/
https://fortune.com/company/freedom-holding/100-fastest-growing-companies/
https://fortune.com/company/staar-surgical/100-fastest-growing-companies/
https://fortune.com/company/neurocrine-biosciences/100-fastest-growing-companies/
https://fortune.com/company/kornit-digital/100-fastest-growing-companies/
https://fortune.com/company/enstar-group/100-fastest-growing-companies/
https://fortune.com/company/logitech-international/100-fastest-growing-companies/
https://fortune.com/company/pjt-partners/100-fastest-growing-companies/
https://fortune.com/company/mvb-financial/100-fastest-growing-companies/
https://fortune.com/company/repligen/100-fastest-growing-companies/
https://fortune.com/company/harborone-bancorp/100-fastest-growing-companies/
https://fortune.com/company/capstar-financial-holdings/100-fastest-growing-companies/
https://fortune.com/company/ssr-mining/100-fastest-growing-companies/
https://fortune.com/company/merchants-bancorp/100-fastest-growing-companies/
https://fortune.com/company/boston-beer/100-fastest-growing-companies/
https://fortune.com/company/first-horizon-national-corp/100-fastest-growing-companies/
https://fortune.com/company/lhc-group/100-fastest-growing-companies/
https://fortune.com/company/mercury-systems/100-fastest-growing-companies/
https://fortune.com/company/national-bank-holdings/100-fastest-growing-companies/
https://fortune.com/company/facebook/100-fastest-growing-companies/
https://fortune.com/company/monolithic-power-systems/100-fastest-growing-companies/
https://fortune.com/company/perkinelmer/100-fastest-growing-companies/
https://fortune.com/company/malibu-boats/100-fastest-growing-companies/
https://fortune.com/company/svb-financial-group/100-fastest-growing-companies/
https://fortune.com/company/first-bancshares/100-fastest-growing-companies/
https://fortune.com/company/veritex-holdings/100-fastest-growing-companies/
https://fortune.com/company/myr-group/100-fastest-growing-companies/
https://fortune.com/company/mi-homes/100-fastest-growing-companies/
https://fortune.com/company/slm/100-fastest-growing-companies/
https://fortune.com/company/dow-chemical/100-fastest-growing-companies/
https://fortune.com/company/fox-factory-holding/100-fastest-growing-companies/
https://fortune.com/company/lgi-homes/100-fastest-growing-companies/
https://fortune.com/company/take-two-interactive-software/100-fastest-growing-companies/
https://fortune.com/company/paylocity-holding/100-fastest-growing-companies/
https://fortune.com/company/alarm-com-holdings/100-fastest-growing-companies/
https://fortune.com/company/ssc-technologies-holdings/100-fastest-growing-companies/
https://fortune.com/company/stonex-group/100-fastest-growing-companies/
https://fortune.com/company/century-communities/100-fastest-growing-companies/
https://fortune.com/company/teradyne/100-fastest-growing-companies/
https://fortune.com/company/nmi-holdings/100-fastest-growing-companies/
https://fortune.com/company/virtu-financial/100-fastest-growing-companies/
https://fortune.com/company/ubiquiti/100-fastest-growing-companies/
https://fortune.com/company/scotts-miracle-gro/100-fastest-growing-companies/
https://fortune.com/company/vertex-pharmaceuticals/100-fastest-growing-companies/
https://fortune.com/company/medifast/100-fastest-growing-companies/
https://fortune.com/company/epam-systems/100-fastest-growing-companies/
https://fortune.com/company/medpace-holdings/100-fastest-growing-companies/
https://fortune.com/company/green-brick-partners/100-fastest-growing-companies/
https://fortune.com/company/leucadia-national/100-fastest-growing-companies/
https://fortune.com/company/fortinet/100-fastest-growing-companies/
https://fortune.com/company/appfolio/100-fastest-growing-companies/
https://fortune.com/company/kinsale-capital-group/100-fastest-growing-companies/
https://fortune.com/company/pennymac-financial-services/100-fastest-growing-companies/
To get other information you may have to digg in JSON.
for item in data[1]['items']:
print(item['permalink'])
for f in item['fields']:
if f['key'] == 'fgc_industry':
print(f['value'])
if f['key'] == 'rank':
print(f['value'])
if f['key'] == 'title':
print(f['value'])
Url has field token but I'm not sure if it may change value. You would have to observer it in browser for few requests. If it changes then you may have to search it in original HTML.
After decoding base64
import base64
base64.b64decode('Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM=')
it gives 'fortune:BpsrfkMd#yJwcZI!4qj2wD93' and this values you may have to search in HTML.