Home > Enterprise >  BeautifulSoup web scrape href issue
BeautifulSoup web scrape href issue

Time:04-21

I am writing a web scraper and am struggling to get the href link from a web page. The URL is https://fortune.com/100-fastest-growing-companies/2021/search/. I am trying to get the href pull all of the article links. Here is an example of an href I am trying to pull"

<a  trackerdata="[object Object] href="https://fortune.com/company/atlanticus-holdings/100-fastest-growing-companies/"><div ><span><div>Atlanticus Holdings</div></span></div></a>

Here is my code:

import pandas as pd
import os
import requests
from bs4 import BeautifulSoup
import re



URL = "https://fortune.com/100-fastest-growing-companies/2021/search/"
headers={'User-Agent':'mozilla/5.0'}
page = requests.get(URL,headers=headers)
soup = BeautifulSoup(page.text, "html.parser")

links = []
for link in soup.findAll(class_="searchResults__cellWrapper--39MAj"):
    links.append((link.get('href')))
print(links)

When I run my code, it gives me nothing. Can someone point me in the right direction?

CodePudding user response:

This page uses JavaScript to add elements

But it reads data as JSON from

https://fortune.com/wp-json/irving/v1/data/franchise-search-results?list_id=3159847&token=Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM=

and it can be used to get links

import requests

url = 'https://fortune.com/wp-json/irving/v1/data/franchise-search-results?list_id=3159847&token=Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM='
headers = {'User-Agent': 'Mozilla/5.0'}

response = requests.get(url, headers=headers)
data = response.json()

for item in data[1]['items']:
    print(item['permalink'])

Result:

https://fortune.com/company/progressive/100-fastest-growing-companies/
https://fortune.com/company/adobe-systems/100-fastest-growing-companies/
https://fortune.com/company/salesforce-com/100-fastest-growing-companies/
https://fortune.com/company/newmont-mining/100-fastest-growing-companies/
https://fortune.com/company/lennar/100-fastest-growing-companies/
https://fortune.com/company/kla-tencor/100-fastest-growing-companies/
https://fortune.com/company/jd-com/100-fastest-growing-companies/
https://fortune.com/company/jacobs-engineering-group/100-fastest-growing-companies/
https://fortune.com/company/cigna/100-fastest-growing-companies/
https://fortune.com/company/charles-river-laboratories-intl/100-fastest-growing-companies/
https://fortune.com/company/advanced-micro-devices/100-fastest-growing-companies/
https://fortune.com/company/paypal-holdings/100-fastest-growing-companies/
https://fortune.com/company/netflix/100-fastest-growing-companies/
https://fortune.com/company/alphabet/100-fastest-growing-companies/
https://fortune.com/company/amazon-com/100-fastest-growing-companies/
https://fortune.com/company/veeva-systems/100-fastest-growing-companies/
https://fortune.com/company/align-technology/100-fastest-growing-companies/
https://fortune.com/company/etsy/100-fastest-growing-companies/
https://fortune.com/company/lululemon-athletica/100-fastest-growing-companies/
https://fortune.com/company/trade-desk/100-fastest-growing-companies/
https://fortune.com/company/emergent-biosolutions/100-fastest-growing-companies/
https://fortune.com/company/hibbett/100-fastest-growing-companies/
https://fortune.com/company/business-first-bancshares/100-fastest-growing-companies/
https://fortune.com/company/marinemax/100-fastest-growing-companies/
https://fortune.com/company/spirit-of-texas-bancshares/100-fastest-growing-companies/
https://fortune.com/company/ies-holdings/100-fastest-growing-companies/
https://fortune.com/company/brooks-automation/100-fastest-growing-companies/
https://fortune.com/company/mid-penn-bancorp/100-fastest-growing-companies/
https://fortune.com/company/strategic-education/100-fastest-growing-companies/
https://fortune.com/company/waterstone-financial/100-fastest-growing-companies/
https://fortune.com/company/g-willi-food-international/100-fastest-growing-companies/
https://fortune.com/company/marketaxess-holdings/100-fastest-growing-companies/
https://fortune.com/company/ameris-bancorp/100-fastest-growing-companies/
https://fortune.com/company/shyft-group/100-fastest-growing-companies/
https://fortune.com/company/smartfinancial/100-fastest-growing-companies/
https://fortune.com/company/superior-group-of-companies/100-fastest-growing-companies/
https://fortune.com/company/meta-financial-group/100-fastest-growing-companies/
https://fortune.com/company/addus-homecare/100-fastest-growing-companies/
https://fortune.com/company/metropolitan-bank-holding/100-fastest-growing-companies/
https://fortune.com/company/1-800-flowers-com/100-fastest-growing-companies/
https://fortune.com/company/fs-bancorp/100-fastest-growing-companies/
https://fortune.com/company/arch-capital-group/100-fastest-growing-companies/
https://fortune.com/company/atlanticus-holdings/100-fastest-growing-companies/
https://fortune.com/company/b-riley-financial/100-fastest-growing-companies/
https://fortune.com/company/joint-corp/100-fastest-growing-companies/
https://fortune.com/company/quidel/100-fastest-growing-companies/
https://fortune.com/company/acm-research/100-fastest-growing-companies/
https://fortune.com/company/victory-capital-holdings/100-fastest-growing-companies/
https://fortune.com/company/freedom-holding/100-fastest-growing-companies/
https://fortune.com/company/staar-surgical/100-fastest-growing-companies/
https://fortune.com/company/neurocrine-biosciences/100-fastest-growing-companies/
https://fortune.com/company/kornit-digital/100-fastest-growing-companies/
https://fortune.com/company/enstar-group/100-fastest-growing-companies/
https://fortune.com/company/logitech-international/100-fastest-growing-companies/
https://fortune.com/company/pjt-partners/100-fastest-growing-companies/
https://fortune.com/company/mvb-financial/100-fastest-growing-companies/
https://fortune.com/company/repligen/100-fastest-growing-companies/
https://fortune.com/company/harborone-bancorp/100-fastest-growing-companies/
https://fortune.com/company/capstar-financial-holdings/100-fastest-growing-companies/
https://fortune.com/company/ssr-mining/100-fastest-growing-companies/
https://fortune.com/company/merchants-bancorp/100-fastest-growing-companies/
https://fortune.com/company/boston-beer/100-fastest-growing-companies/
https://fortune.com/company/first-horizon-national-corp/100-fastest-growing-companies/
https://fortune.com/company/lhc-group/100-fastest-growing-companies/
https://fortune.com/company/mercury-systems/100-fastest-growing-companies/
https://fortune.com/company/national-bank-holdings/100-fastest-growing-companies/
https://fortune.com/company/facebook/100-fastest-growing-companies/
https://fortune.com/company/monolithic-power-systems/100-fastest-growing-companies/
https://fortune.com/company/perkinelmer/100-fastest-growing-companies/
https://fortune.com/company/malibu-boats/100-fastest-growing-companies/
https://fortune.com/company/svb-financial-group/100-fastest-growing-companies/
https://fortune.com/company/first-bancshares/100-fastest-growing-companies/
https://fortune.com/company/veritex-holdings/100-fastest-growing-companies/
https://fortune.com/company/myr-group/100-fastest-growing-companies/
https://fortune.com/company/mi-homes/100-fastest-growing-companies/
https://fortune.com/company/slm/100-fastest-growing-companies/
https://fortune.com/company/dow-chemical/100-fastest-growing-companies/
https://fortune.com/company/fox-factory-holding/100-fastest-growing-companies/
https://fortune.com/company/lgi-homes/100-fastest-growing-companies/
https://fortune.com/company/take-two-interactive-software/100-fastest-growing-companies/
https://fortune.com/company/paylocity-holding/100-fastest-growing-companies/
https://fortune.com/company/alarm-com-holdings/100-fastest-growing-companies/
https://fortune.com/company/ssc-technologies-holdings/100-fastest-growing-companies/
https://fortune.com/company/stonex-group/100-fastest-growing-companies/
https://fortune.com/company/century-communities/100-fastest-growing-companies/
https://fortune.com/company/teradyne/100-fastest-growing-companies/
https://fortune.com/company/nmi-holdings/100-fastest-growing-companies/
https://fortune.com/company/virtu-financial/100-fastest-growing-companies/
https://fortune.com/company/ubiquiti/100-fastest-growing-companies/
https://fortune.com/company/scotts-miracle-gro/100-fastest-growing-companies/
https://fortune.com/company/vertex-pharmaceuticals/100-fastest-growing-companies/
https://fortune.com/company/medifast/100-fastest-growing-companies/
https://fortune.com/company/epam-systems/100-fastest-growing-companies/
https://fortune.com/company/medpace-holdings/100-fastest-growing-companies/
https://fortune.com/company/green-brick-partners/100-fastest-growing-companies/
https://fortune.com/company/leucadia-national/100-fastest-growing-companies/
https://fortune.com/company/fortinet/100-fastest-growing-companies/
https://fortune.com/company/appfolio/100-fastest-growing-companies/
https://fortune.com/company/kinsale-capital-group/100-fastest-growing-companies/
https://fortune.com/company/pennymac-financial-services/100-fastest-growing-companies/

To get other information you may have to digg in JSON.

for item in data[1]['items']:
    print(item['permalink'])

    for f in item['fields']:
        if f['key'] == 'fgc_industry':
            print(f['value'])
        if f['key'] == 'rank':
            print(f['value'])
        if f['key'] == 'title':
            print(f['value'])

Url has field token but I'm not sure if it may change value. You would have to observer it in browser for few requests. If it changes then you may have to search it in original HTML.

After decoding base64

import base64

base64.b64decode('Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM=')

it gives 'fortune:BpsrfkMd#yJwcZI!4qj2wD93' and this values you may have to search in HTML.

  • Related