Two part question -- First, when running the code without final if statement, I'm not getting all of the HREF tags... I see many more links in Inspector that don't seem to come through.

Looking for a fix, but also trying to understand general knowledge on this - is there a reason why some links would work and others would not?

Similarly, I wanted to pull the HREF tags that contain "Surf-Report". I've used this code with p.startswith, and it works... but I couldn't find what the function call would be to say "contains".

I'm new to all of this, looking but don't fully understand either of these.

import requests
from bs4 import BeautifulSoup

profiles = []
urls = [
    'https://magicseaweed.com/New-Jersey-Monmouth-County-Surfing/277/',
    'https://magicseaweed.com/New-Jersey-Ocean-County-Surfing/278/'
]
for url in urls:
    req = requests.get(url)
    soup = BeautifulSoup(req.text, 'html.parser')
    for profile in soup.find_all('a'):

        profile = profile.get('href')

        profiles.append(profile)

# print(profiles)

for p in profiles:
    if p.contains('Surf-Report'):
        print(p)

For context, my overall goal is to go to these different county pages, and get all of the HREF tags there. Once I have those, I want to visit each individual link and pull the wave sizes from each of the links stored there.

I'm looking to build a way to monitor all waves in New Jersey daily... no purpose, just a fun practice project with something I find interesting.

CodePudding user response：

Those urls in page appears to be fed into dynamically, via an (or more?) XHR call. Upon a brief inspection of that page' Dev tools - network tab, I noticed a call to an api (from which I stripped the variables). Scraping that api returns over 8k results:

import requests
import pandas as pd
import json

r = requests.get('https://magicseaweed.com/api/mdkey/spot?&limit=-1')
df = pd.DataFrame(r.json())
print(df)

Result:

Is this what you're after?

CodePudding user response：

Since the website using JavaScript to load content you have to render the page before trying to search for a tags, so you should use requests_html instead of requests.
first you need to install requests-html library by typing pip install requests-html

from requests_html import HTMLSession
from bs4 import BeautifulSoup
profiles = []
session = HTMLSession()
urls = [
    'https://magicseaweed.com/New-Jersey-Monmouth-County-Surfing/277/',
    'https://magicseaweed.com/New-Jersey-Ocean-County-Surfing/278/'
]
for url in urls:
    r = session.get(url)
    r.html.render(sleep=3, timeout=20) # wait for 3s until the page fully loaded
    soup = BeautifulSoup(r.html.raw_html, "html.parser")
    for profile in soup.find_all('a'):
        profile = profile.get('href')
        profiles.append(profile)
for p in profiles:
    if p and 'Surf-Report' in p:
        print(p)

The result obtained :

/Belmar-Surf-Report/3683/     
/Manasquan-Surf-Report/386/   
/Ocean-Grove-Surf-Report/7945/
/Asbury-Park-Surf-Report/857/ 
/Avon-Surf-Report/4050/       
/Bay-Head-Surf-Report/4951/   
/Belmar-Surf-Report/3683/
/Boardwalk-Surf-Report/9183/
/Bradley-Beach-Surf-Report/7944/
/Casino-Surf-Report/9175/
/Deal-Surf-Report/822/
/Dog-Park-Surf-Report/9174/
/Jenkinsons-Surf-Report/4053/
/Long-Branch-Surf-Report/7946/
/Long-Branch-Surf-Report/7947/
/Manasquan-Surf-Report/386/
/Monmouth-Beach-Surf-Report/4055/
/Ocean-Grove-Surf-Report/7945/
/Point-Pleasant-Surf-Report/7942/
/Sea-Girt-Surf-Report/7943/
/Spring-Lake-Surf-Report/7941/
/The-Cove-Surf-Report/385/
/Belmar-Surf-Report/3683/
/Avon-Surf-Report/4050/
/Deal-Surf-Report/822/
/34th-Street-Surf-Report/9058/
/Lavenia-Ave-Surf-Report/9060/
/Casino-Pier-Surf-Report/387/
/Harvey-Cedars-Surf-Report/7938/
/Hudson-Ave-Surf-Report/9062/
/Mantoloking-Surf-Report/9063/
/Ortley-Beach-Surf-Report/9064/
/Seaside-Park-Surf-Report/9280/
/Ship-Bottom-Surf-Report/9065/
/Spray-Beach-Surf-Report/9066/
/Surf-City-Surf-Report/388/
/34th-Street-Surf-Report/9058/
/Barnegat-Light-Surf-Report/7940/
/Beach-Haven-Surf-Report/7939/
/Casino-Pier-Surf-Report/387/
/Harvey-Cedars-Surf-Report/7938/
/Holyoke-Surf-Report/389/
/Hudson-Ave-Surf-Report/9062/
/Island-Beach-State-Park-Surf-Report/4052/
/Lavenia-Ave-Surf-Report/9060/
/Mantoloking-Surf-Report/9063/
/New-Jersey-Hurricane-Surf-Report/1094/
/Ortley-Beach-Surf-Report/9064/
/Seaside-Park-Surf-Report/9280/
/Ship-Bottom-Surf-Report/9065/
/Spray-Beach-Surf-Report/9066/
/Surf-City-Surf-Report/388/
/New-Jersey-Hurricane-Surf-Report/1094/
/New-Jersey-Hurricane-Surf-Report/1094/
/Casino-Pier-Surf-Report/387/